python aiohttp creates many threads
Recently, I used python aiohttp to send asynchronously. I used py-spy to monitor it on the command line. I found that running python programs created 125 threads, which scared me.
solve
Google has been finding a solution for a long time.
reason
Query dns every time a request is sent. This query dns is blocking, so it opens a thread every time it querys dns.
So I specified it in the code that the dns query object worked.
from import AsyncResolver resolver = AsyncResolver() tcp_conn = (resolver=resolver) async with (connector=tcp_conn) as session: await process_spider(spider, session)
When running the program, it may be prompted that aiodns is required. Just install it: pip install aiodns.
python aiohttp module use
asyncioSingle-threaded concurrent IO operations can be implemented. If used only on the client, it will not be very powerful.
IfasyncioUsed on the server side, such as a web server, since HTTP connection is an IO operation, you can use a single thread +coroutineImplement high concurrency support for multiple users.
asyncioImplement TCP, UDP, SSL and other protocols.aiohttpIt is based onasyncioImplemented HTTP framework.
Let's install it firstaiohttp：
pip install aiohttp
Then write an HTTP server and process the following URLs separately:
-
/- Home page backb'<h1>Index</h1>'； -
/hello/{name}- Return text according to URL parametershello, %s!。
The code is as follows:
import asyncio
from aiohttp import web
async def index(request):
await (0.5)
return (body=b'<h1>Index</h1>')
async def hello(request):
await (0.5)
text = '<h1>hello, %s!</h1>' % request.match_info['name']
return (body=('utf-8'))
async def init(loop):
app = (loop=loop)
.add_route('GET', '/', index)
.add_route('GET', '/hello/{name}', hello)
srv = await loop.create_server(app.make_handler(), '127.0.0.1', 8000)
print('Server started at http://127.0.0.1:8000...')
return srv
loop = asyncio.get_event_loop()
loop.run_until_complete(init(loop))
loop.run_forever()NoticeaiohttpInitialization functioninit()Also onecoroutine，loop.create_server()UseasyncioCreate a TCP service.
Summarize
The above is personal experience. I hope you can give you a reference and I hope you can support me more.