In this article, the example of Python3 implementation of the crawler to crawl the Catch.com list function. Shared for your reference, as follows:
python3 crawler of crawling catch.com listings. These days have been learning to use python3 crawl data, today record, the code is very simple and easy to get started.
First of all, you need to install python3. If you haven't installed it yet, you can refer to the previous article about python3 installation and configuration.
First you need to install the request and BeautifulSoup modules.
Requests is Python's HTTP web request module. Using Requests, you can easily perform any action available to the browser.
pip install requests
BeautifulSoup is an HTML/XML parser written in Python that handles irregular markup and generates a parse tree.
pip install beautifulsoup4
Code:
from urllib import request
from bs4 import BeautifulSoup
# Construct headers to simulate browser access
url="/meirdjm/o2/"
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}
page = (url,headers=headers)
# Send a request to get content
page_info = (page).read().decode('utf-8')
# Convert the fetched content into BeautifulSoup format and will be used as a parser
soup = BeautifulSoup(page_info, '')
# Find all a tags with class='list-info-title' in them.
titles = soup.find_all('a',class_="list-info-title")
# print the captured title
for title in titles:
print()
Results:
More about Python related content can be viewed on this site's topic: thePython Socket Programming Tips Summary》、《Python Regular Expression Usage Summary》、《Python Data Structures and Algorithms Tutorial》、《Summary of Python function usage tips》、《Summary of Python string manipulation techniques》、《Python introductory and advanced classic tutorialsand theSummary of Python file and directory manipulation techniques》
I hope that what I have said in this article will help you in Python programming.