SoFunction
Updated on 2025-04-14

How to use BeautifulSoup to crawl and parse web page data in Python

introduction

Data is an indispensable resource in the fields of data analytics and machine learning. As a rich source of information, web page data often needs to be crawled through crawlers. Python's BeautifulSoup is a powerful tool for processing HTML and XML. It can parse complex web documents into actionable data structures, allowing us to easily extract and process information.

This article will introduce the basic usage of BeautifulSoup in detail and demonstrate how to use BeautifulSoup to crawl and parse web data in a practical case to help beginners understand and master this skill.

1. Introduction to BeautifulSoup

BeautifulSoup is a Python library for parsing HTML and XML. It supports multiple parsers, and the default is, in addition it can also be usedlxmlandhtml5lib. BeautifulSoup can flexibly extract web content through various methods such as tags, attributes, text, etc.

1. Features of BeautifulSoup

  • Simple and easy to use: Intuitive code, suitable for parsing HTML pages with complex structures.
  • Flexible parser selection: Supports multiple parsers to deal with different HTML structures.
  • Strong compatibility: Able to handle web pages with irregular formats.

2. Install BeautifulSoup

The BeautifulSoup and lxml parsers can be installed using the following commands:

pip install beautifulsoup4 lxml

After the installation is complete, we can start to learn the basic usage and actual cases of BeautifulSoup.

2. Basic usage of BeautifulSoup

Before using BeautifulSoup to grab web page data, we first understand some common basic operations, such as creating BeautifulSoup objects, selecting elements, and extracting data.

1. Create a BeautifulSoup object

We first need to obtain HTML content from the web page, generally throughrequestsThe library is done. Here is a simple example:

import requests
from bs4 import BeautifulSoup

# Get web contenturl = ''
response = (url)
html_content = 

# Create BeautifulSoup objectsoup = BeautifulSoup(html_content, '')

2. Find elements

BeautifulSoup provides a variety of ways to find elements, such asfind、find_all、selectwait. Here are some commonly used search methods:

  • find: Find the first element that meets the criteria
  • find_all: Find all elements that meet the criteria
  • select: Use the CSS selector to find elements
# Find the first h1 elementh1_tag = ('h1')
print(h1_tag.text)

# Find all linkslinks = soup.find_all('a')
for link in links:
    print(('href'))

# Find elements using CSS selectoritems = ('.item .title')
for item in items:
    print()

3. Extract element content

We can usetext、get_text()orattrsand other methods to extract the text content and attribute values ​​of elements:

# Extract tag texttitle = ('h1').text

# Extract propertieslink = ('a')
href = ('href')   # or link['href']

3. BeautifulSoup practical case: grab and extract news titles

To better understand the application of BeautifulSoup, let’s do a simple practical case: grab news titles and links from news websites and save them to local files. WeBBC NewsWebsite as an example.

1. Requirements Analysis

In this case, our goal is to crawlBBC NewsNews titles and links to the homepage of the website and save them to a CSV file. We need to do the following:

  1. Get the HTML content of the web page.
  2. Use BeautifulSoup to parse HTML and extract news titles and links.
  3. Save the data to a CSV file.

2. Case implementation steps

Step 1: Get the HTML content of the web page

We userequestsThe library sends a request to get HTML content.

import requests

# Destination URLurl = '/news'

# Send a requestresponse = (url)

# Check the request statusif response.status_code == 200:
    html_content = 
else:
    print("Failed to retrieve the webpage")

Step 2: parse and extract news titles and links

After getting the HTML content, we parse the web page using BeautifulSoup and select the news title and link through a specific CSS class. We can check web page elements in the browser to find the element class name that contains the news title.

from bs4 import BeautifulSoup

# parse HTML contentsoup = BeautifulSoup(html_content, '')

# Find news titles and linksnews_list = []
for item in ('.gs-c-promo-heading'):
    title = item.get_text()
    link = ('href')
    if link and not ('http'):
        link = '' + link  # Complete the relative link    news_list.append([title, link])

Here we usedselectMethod, positioning.gs-c-promo-headingClass to find the title and link for each news.

Step 3: Save data to a CSV file

We can use Python'scsvThe module saves the extracted data to a CSV file:

import csv

# Write data to CSV filewith open('bbc_news.csv', 'w', newline='', encoding='utf-8') as csvfile:
    writer = (csvfile)
    (['Title', 'Link'])
    (news_list)

print("Data saved to bbc_news.csv")

By this point, we have completed the entire process of crawling news titles and links from BBC News. After running the program, you will find a name calledbbc_news.csvfile containing the crawled news data.

4. Further optimization

Our practical cases have been basically completed, but further optimization can be made in actual applications. For example:

1. Handle errors

During the crawling process, you may encounter network request errors or changes in the web structure. We can improve the stability of our code by adding exception handling.

try:
    response = (url)
    response.raise_for_status()
except  as e:
    print(f"Error: {e}")

2. Avoid frequent requests

To avoid being banned by the website, we can add delays between each request. use()Can make the crawler look more like the behavior of normal users:

import time
(1)  # Delay 1 second

3. Use multi-threaded or asynchronous requests

When crawling large amounts of data, you can use multi-threaded or asynchronous requests to speed up crawling. Python'soraiohttpIt's a good choice.

5. Complete code example

Here is a complete code example that combines previous steps together:

import requests
from bs4 import BeautifulSoup
import csv
import time

def fetch_news(url):
    try:
        response = (url)
        response.raise_for_status()
        return 
    except  as e:
        print(f"Error: {e}")
        return None

def parse_news(html_content):
    soup = BeautifulSoup(html_content, '')
    news_list = []
    for item in ('.gs-c-promo-heading'):
        title = item.get_text()
        link = ('href')
        if link and not ('http'):
            link = '' + link
        news_list.append([title, link])
    return news_list

def save_to_csv(news_list, filename='bbc_news.csv'):
    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
        writer = (csvfile)
        (['Title', 'Link'])
        (news_list)
    print(f"Data saved to {filename}")

def main():
    url = '/news'
    html_content = fetch_news(url)
    if html_content:
        news_list = parse_news(html_content)
        save_to_csv(news_list)
        (1)

if __name__ == "__main__":
    main()

6. Summary

Through this article's case, we have a deeper understanding of how to use BeautifulSoup to crawl and parse web content. The steps cover the entire process of web page request, data parsing, and CSV file storage. What makes BeautifulSoup powerful is its flexibility to handle different web structures. With the requests library, BeautifulSoup can help us easily implement data crawling tasks. In practical applications, by adding optimization measures such as error handling and delay, crawlers can be made more stable and reliable.

The above is the detailed content of Python's operation method of using BeautifulSoup to crawl and parse web page data. For more information about Python BeautifulSoup web page data, please pay attention to my other related articles!