SoFunction
Updated on 2025-03-03

How to use Selenium to crawl dynamic web pages in Python

1. What is dynamic web crawling?

The main difference between dynamic web crawling and traditional static web crawling is:Dynamic web content is generated on the client via JavaScript. This means that directly requesting the HTML file of the web page cannot obtain complete data, and you need to wait for JavaScript to execute to load the data. To solve this problem, there are usually several methods for dynamic web crawling:

  1. Using an automated browser(such as Selenium): Let the browser execute JavaScript directly, and then crawl the loaded web page content.
  2. Analyze network requests: Some websites will send additional requests (usually data in JSON format) to the server in the background, which can directly simulate these requests to get the data.
  3. With the help of a tool library (such as Pyppeteer): Some libraries can simulate browser behavior and directly render the page to obtain complete content.

2. Preparation

Before we start dynamic web crawling, we need to install some necessary libraries.

2.1 Install Selenium

Selenium is an automated testing tool that can execute JavaScript by controlling the browser to load dynamic content. After installing Selenium, you also need to download a matching browser driver (such as ChromeDriver).

pip install selenium

Download ChromeDriver (assuming a Chrome browser) and add its path to the environment variable.

2.2 Install Pyppeteer

Pyppeteer is another powerful dynamic crawler library that can perform browser tasks in interfaceless mode. Pyppeteer is the Python version of Puppeteer, a tool used for Puppeteer.

pip install pyppeteer

2.3 Install Requests and BeautifulSoup

While Requests and BeautifulSoup are mainly used for static web crawling, they are also useful when fetching some backend interface data in dynamic web pages.

pip install requests beautifulsoup4

3. Use Selenium to crawl dynamic web pages

We will crawl dynamic web pages through Selenium. First, let’s look at how to start the browser, access the web page and wait for the page to load. Suppose we want to grab a dynamically loaded product list.

from selenium import webdriver
from  import By
from  import Service
from  import WebDriverWait
from  import expected_conditions as EC
import time

# Set the path to ChromeDriverservice = Service(executable_path='path/to/chromedriver')

# Initialize Chrome browserdriver = (service=service)

# Open the landing pageurl = '/dynamic-page'
(url)

# Wait for the page to loadtry:
    # Wait for a specific element to load, suppose we wait for a product list element to load    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "product-list"))
    )

    # Crawl the HTML content of the page    page_content = driver.page_source
    print(page_content)

finally:
    # Close the browser    ()

3.1 Find elements and extract data

When the page is loaded, we can use the search method provided by Selenium to locate and extract the content of a specific element.

# Find the element of the product name (assuming class is product-name)products = driver.find_elements(By.CLASS_NAME, 'product-name')

for product in products:
    print()

3.2 Scrolling page to load more content

Some dynamic web pages will load more content when the user scrolls, and Selenium can crawl more data by simulated scrolling:

# Simulate scrolling and load more contentSCROLL_PAUSE_TIME = 2

# Get the height of the current pagelast_height = driver.execute_script("return ")

while True:
    # Scroll to the bottom    driver.execute_script("(0, );")

    # Wait for new content to load    (SCROLL_PAUSE_TIME)

    # Get the height of the new page    new_height = driver.execute_script("return ")
    if new_height == last_height:
        break  # If the height remains unchanged, it means it has been loaded
    last_height = new_height

4. Use Pyppeteer to crawl dynamic web pages

Pyppeteer is a Python version of Puppeteer. It can also be used to crawl dynamic web pages and supports headless browsers.

4.1 Simple Example

Here is a simple example of crawling dynamic content using Pyppeteer:

import asyncio
from pyppeteer import launch

async def fetch_dynamic_content(url):
    # Start the browser    browser = await launch(headless=True)
    page = await ()
    
    # Open the web page    await (url)
    
    # Wait for the specific element to load    await ('.product-list')

    # Get page content    content = await ()
    print(content)

    # Close the browser    await ()

# Start asynchronous tasksurl = '/dynamic-page'
asyncio.get_event_loop().run_until_complete(fetch_dynamic_content(url))

4.2 Screenshots and debugging

Pyppeteer also supports screenshot function, which can help us debug.

await ({'path': ''})

5. Use Requests to crawl API data from dynamic web pages

Many dynamic websites actually send requests to the backend to get data when loading content. We can find the URLs for these requests in the "Network" panel of the browser and use the Requests library to simulate these requests to crawl data.

import requests

# Target URL (API request URL found in the browser network panel)url = '/api/products'

# Send a request and get dataresponse = (url)
data = ()

# Print dataprint(data)

6. Frequently Asked Questions and Tips for Dynamic Crawling

6.1 JavaScript rendered content is not loaded

If the content in the page is rendered through JavaScript, then use it directlyrequestsGetting HTML does not contain these contents. This situation can be considered:

  • Using Selenium or Pyppeteer, let the browser actually execute JavaScript and load the content.
  • Find the API request behind JavaScript and get the data directly.

6.2 Encounter verification code or anti-crawler measures

Many websites have anti-crawler measures. In this case, you can try:

  • Adjust request frequency: Increase the interval between requests to avoid high frequency access.
  • Using a proxy: Avoid using fixed IP addresses.
  • Setting up the browser header: Add browser header information to the request to simulate normal browser access.
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
response = (url, headers=headers)

7. Practical application scenarios of dynamic crawling

  • E-commerce data collection: Crawl product list and details.
  • Social Media Analytics: Get dynamic content on social platforms, such as tweets or comments.
  • News data collection: Crawl news entries dynamically loaded by news websites.

8. Summary

Dynamic web crawling is much more complex than static web crawling because it requires simulated browser behavior to execute JavaScript. However, using tools like Selenium and Pyppeteer, we can easily crawl dynamic web content. In addition, analyzing network requests and directly crawling API data can also be a more efficient way. Hopefully, through this article, you can understand the basics of Python dynamic web crawling and use these tools to obtain the data you need.

The above is the detailed content of the method and steps for Python to crawl dynamic web pages using Selenium. For more information about Python Selenium crawling web pages, please follow my other related articles!