Preface
The system programmers are often criticizing threads. The usage scenarios they consider for ordinary application programmers may not encounter in their lifetime... In 99% of the cases encountered by application programmers, you only need to know how to derive a bunch of independent threads and then use a queue to collect the results.
This article records my key knowledge and personal experience in the Basic Control Process of Python. Friends who plan to get started with Python can learn and communicate together.
Focus of this article:
1. Master the relevant concepts of asynchronous programming;
2. Understand the concept, significance and usage methods of periodical futures;
3. Understand the characteristics of blocking I/O functions in Python to release GIL.
1. Related concepts of asynchronous programming
Blocking: The state where the program is suspended when the required computing resources are not obtained. In other words, if the program is unable to continue doing other things while waiting for an operation to complete, it is said that the program is blocked in the operation.
Concurrency: describes the organizational structure of the program. It means that the program is designed to be multiple subtasks that can be independently executed. Concurrency is intended to utilize limited computer resources to enable multiple tasks to be executed in real time or near real time.
Parallel: refers to the program state where multitasking is executed simultaneously, with the purpose of using multi-core CPU to accelerate the completion of multitasking.
Asynchronous: A way to complete a task without communication and coordination between different program units.
Unrelated program units can be asynchronous. In short, asynchronous means disorder.
Asynchronous programming: a programming method that uses processes, threads, coroutines, functions/methods as the basic units for executing tasks, combined with callbacks, event loops, semaphores and other mechanisms to improve overall execution efficiency and concurrency capabilities.
2. Futures
Among the three clients that aim to download the national flag, the two HTTP concurrent clients have much higher performance than the scripts downloaded in sequence.
This shows that using concurrency can efficiently handle network I/O.
Future refers to an object that represents an operation performed asynchronously.
Period object: or instance of class.
Three major methods:
- (): Create period item.
- .as_completed(): The period item at the end of the iteration run, returning an iterator.
- (): Handle the same callable object with different parameters.
Summary: The combination of () plus futures.as_completed() is more flexible than () because submit() can handle different callable objects and parameters.
The main features of the module are the ThreadPoolExecutor and ProcessPoolExecutor classes. The interfaces implemented by these two classes can execute callable objects in different threads or processes respectively.
Notice:Normally, one should not create periods by itself, but can only be instantiated by a concurrent framework (or asyncio).
Example: Module Application
from concurrent import futures
from flags import save_flag, get_flag, show, main
MAX_WORKERS = 20
def download_one(cc):
image = get_flag(cc)
show(cc)
save_flag(image, () + '.gif')
return cc
def download_many(cc_list):
workers = min(MAX_WORKERS, len(cc_list))
with (workers) as executor:
res = (download_one, sorted(cc_list))
return len(list(res))
if __name__ == '__main__':
main(download_many)
3. Obstructive I/O and GIL
All blocking I/O functions in the Python standard library release global interpreter locks (GILs) to allow other threads to run.
Therefore, despite GIL, Python threads are still suitable for use in I/O-intensive systems.
IV. Alternatives for threads and multi-processes
For CPU-intensive work, multiple processes need to be started to circumvent GIL.
The easiest way to create multi-process is to use classes.
threading and multiprocessing modules: are low-level implementations of multi-threading and multi-process concurrency in Python.
Summarize
The above is the entire content of this article. I hope that the content of this article has certain reference value for everyone's study or work. If you have any questions, you can leave a message to communicate. Thank you for your support.