Detailed usage methods and case explanations of httpx library in python

1. Install httpx

First, make sure httpx is already installed. It can be installed through the following command:pip install httpx

If you need to support HTTP/2, you can install additional dependencies:pip install httpx[http2]

2. Synchronize requests

Send a GET request

import httpx

# Send a GET requestresponse = ('/get')
print(response.status_code)  # Status Codeprint()         # Response content

Send a POST request

# Send a POST requestdata = {'key': 'value'}
response = ('/post', json=data)
print(())  # parse JSON response

Set request header

headers = {'User-Agent': 'my-app/1.0.0'}
response = ('/headers', headers=headers)
print(())

Set query parameters

params = {'key1': 'value1', 'key2': 'value2'}
response = ('/get', params=params)
print(())

Processing timeout

try:
    response = ('/delay/5', timeout=2.0)
except :
    print("Request timeout")

3. Asynchronous request

httpx supports asynchronous operation and is suitable for high-performance scenarios.

Send an asynchronous GET request

import httpx
import asyncio

async def fetch(url):
    async with () as client:
        response = await (url)
        print()

(fetch('/get'))

Send an asynchronous POST request

async def post_data(url, data):
    async with () as client:
        response = await (url, json=data)
        print(())

(post_data('/post', {'key': 'value'}))

Concurrent requests

async def fetch_multiple(urls):
    async with () as client:
        tasks = [(url) for url in urls]
        responses = await (*tasks)
        for response in responses:
            print()

urls = ['/get', '/ip']
(fetch_multiple(urls))

4. Advanced features

Using HTTP/2

# Enable HTTP/2client = (http2=True)
response = ('/get')
print(response.http_version)  # Output protocol version

File upload

files = {'file': open('', 'rb')}
response = ('/post', files=files)
print(())

Streaming request

# Streaming Uploaddef generate_data():
    yield b"part1"
    yield b"part2"

response = ('/post', data=generate_data())
print(())

Streaming response

# Streaming downloadwith ('GET', '/stream/10') as response:
    for chunk in response.iter_bytes():
        print(chunk)

5. Error handling

httpx provides a variety of exception classes to facilitate handling of errors.

Handle network errors

try:
    response = ('')
except :
    print("Network Error")

Handle HTTP error status code

response = ('/status/404')
if response.status_code == 404:
    print("Page not found")

6. Configure the client

Global settings can be configured via or .

Set timeout

client = (timeout=10.0)
response = ('/get')
print()

Set up a proxy

proxies = {
    "http://": ":8080",
    "https://": ":8080",
}
client = (proxies=proxies)
response = ('/get')
print()

Set the basic URL

client = (base_url='')
response = ('/get')
print()

7. Use with Beautiful Soup

httpx can be used in conjunction with Beautiful Soup to crawl and parse web pages.

import httpx
from bs4 import BeautifulSoup

# Crawl the web pageresponse = ('')
html = 

# parse web pagessoup = BeautifulSoup(html, 'lxml')
title = ('title').text
print("Web title:", title)

8. Example: Crawl and parse web pages

Here is a complete example showing how to use httpx to crawl and parse web data:

import httpx
from bs4 import BeautifulSoup

# Crawl the web pageurl = ''
response = (url)
html = 

# parse web pagessoup = BeautifulSoup(html, 'lxml')

# Extract titletitle = ('title').text
print("Web title:", title)

# Extract all linkslinks = soup.find_all('a', href=True)
for link in links:
    href = link['href']
    text = 
    print(f"Link text: {text}, Link address: {href}")

9. Things to note

Performance: The asynchronous mode of httpx is suitable for high concurrency scenarios.

Compatibility: httpx's API is highly compatible with requests and has low migration costs.

HTTP/2: If you need to use HTTP/2, make sure httpx[http2] is installed.

Through the above methods, you can use httpx to efficiently send HTTP requests, and combine them with other tools (such as Beautiful Soup) to realize data crawling and parsing.

This is the article about the detailed usage methods and case explanations of httpx library in python. For more related python httpx library usage and case content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!