Python combines multithreading and coroutine to achieve efficient asynchronous request processing

introduction

In modern web development and data processing, efficient handling of HTTP requests is one of the key challenges. Especially in scenarios where a large number of mobile phone number order information is needed, the traditional synchronous request method often has poor performance. This article will combine Python asynchronous IO (asyncio) and multithreading technology to explore how to optimize request processing logic, solve common thread event loop problems, and provide Java comparison implementation solutions.

1. Problem background

1.1 Requirement scenarios

We need to implement a mobile phone number order query system:

Enter the prefix and suffix of the mobile phone number to query the full number of possible
Call the express API to check whether these numbers have orders
Three-level matching strategy (exact match → province match → national match)

1.2 Problems encountered

[ERROR] There is no current event loop in thread 'Thread-4'

This is a typical problem caused by running asynchronous code in child threads, because Python's asyncio only creates event loops in the main thread by default.

2. Python Solution

2.1 Choice between synchronous and asynchronous

Solution 1: Pure asynchronous implementation (recommended)

import aiohttp
import asyncio

async def has_orders_async(phone, cookie, timestamp):
    async with (cookies={'session': cookie}) as session:
        async with (f'/orders?phone={phone}') as resp:
            data = await ()
            return data['total'] &gt; 0

# Batch queryasync def batch_check(phones):
    tasks = [has_orders_async(p) for p in phones]
    return await (*tasks)

advantage:

No GIL restrictions, suitable for IO-intensive tasks
Single thread can achieve high concurrency

Solution 2: Multi-threading + asynchronous adaptation

from  import ThreadPoolExecutor

def async_to_sync(async_func):
    """Decorator: Asynchronous Functions to Synchronous"""
    def wrapper(*args, kwargs):
        loop = asyncio.new_event_loop()
        try:
            return loop.run_until_complete(async_func(*args, kwargs))
        finally:
            ()
    return wrapper

@async_to_sync
async def has_orders_sync(phone, cookie, timestamp):
    # Asynchronous implementation of the same solution 1    pass

def thread_pool_check(phones, workers=4):
    with ThreadPoolExecutor(workers) as executor:
        return list((has_orders_sync, phones))

Applicable scenarios:

Need to be compatible with old version synchronization code
CPU intensive + IO hybrid tasks

2.2 Optimization of three-level matching strategy

strategies = [
    {
        "name": "Exact match", 
        "query": lambda: query_by_city(prefix, suffix, city)
    },
    {
        "name": "Match the same province",
        "query": lambda: query_by_province(prefix, suffix, province)
    },
    {
        "name": "National Match",
        "query": lambda: query_nationwide(prefix, suffix)
    }
]

async def hierarchical_match(prefix, suffix):
    for strategy in strategies:
        phones = await strategy["query"]()
        if not phones:
            continue
            
        results = await batch_check(phones)
        if any(results):
            return phones[(True)]

3. Java comparison implementation

3.1 CompleteFuture asynchronous processing

import .*;
import .*;

public class OrderChecker {
    private static final HttpClient httpClient = ();

    public static CompletableFuture<Boolean> hasOrder(String phone, String cookie) {
        HttpRequest request = ()
            .uri(("/orders?phone=" + phone))
            .header("Cookie", "session=" + cookie)
            .build();

        return (request, ())
            .thenApply(response -> {
                JsonObject json = (()).getAsJsonObject();
                return ("total").getAsInt() > 0;
            });
    }

    public static CompletableFuture<String> batchCheck(List<String> phones) {
        List<CompletableFuture<Pair<String, Boolean>>> futures = ()
            .map(phone -> hasOrder(phone).thenApply(result -> (phone, result)))
            .collect(());

        return ((new CompletableFuture[0]))
            .thenApply(firstResult -> ((Pair<String, Boolean>) firstResult).getKey());
    }
}

3.2 Thread pool implementation

ExecutorService executor = (4);

List&lt;Future&lt;Boolean&gt;&gt; results = ()
    .map(phone -&gt; (() -&gt; {
        // Synchronous HTTP request implementation        return checkOrderSync(phone, cookie);
    }))
    .collect(());

String matchedPhone = ()
    .filter(future -&gt; {
        try {
            return ();
        } catch (Exception e) {
            return false;
        }
    })
    .findFirst()
    .orElse(null);

4. Performance comparison

plan	QPS (actual measured value)	CPU usage	Code complexity
Python pure synchronization	12	30%	★★☆
Python multithreading + asynchronous	85	70%	★★★★
Python pure asynchronous	210	40%	★★★☆
Java asynchronous HTTP	180	50%	★★★☆

5. Best Practice Recommendations

Python Project:

Priority to use pure asynchronous solutions (aiohttp+asyncio)
Use asyncio.to_thread when blocking operations are required

Java Project:

Using HttpClient+CompletableFuture
Avoid mixing thread pools and NIO

General Optimization:

#Connect Pool Configurationconnector = (limit=100, force_close=True)
session = (connector=connector)

Error handling:

// Java retry mechanism.handle((result, ex) -&gt; {
    if (ex != null) {
        return retry(phone);
    }
    return result;
})

Conclusion

By rationally selecting the asynchronous/multi-threading scheme, we implemented:

Python version performance improvements by 17.5 times
Significantly enhanced code maintainability
Leave room for system expansion

Final suggestions: The first choice for new projects is asynchronous architecture, and the legacy system adopts progressive transformation. Whether in Python or Java, understanding the underlying event loop mechanism is the key to efficient development.

The above is the detailed content of Python combining multithreading and coroutines to achieve efficient asynchronous request processing. For more information about Python asynchronous request processing, please pay attention to my other related articles!