Recently prepared to do a battle on scrapy framework, crawl Tencent social recruiting information and storage, this blog to record the steps to create the project
It is not possible to create a scrapy project with pycharm.
Therefore, we need to create a new scrapy project using the command line method
Make sure you have scrapy, twisted, pypiwin32 installed.
I: Go to the path you need, which stores the project you created
Mine will be in the Scrapy directory on the E drive.
Second: create the project: scrapy startproject *** (this is the project name)
This creates a project called tencent.
Third: enter the project to create a new crawler: scrapy genspider tencent_spider
Here we should note that for the command above, the bolded is the name of the crawler and the italicized is the domain name
In this way, we have created a new crawler project, open the folder to view the
Open spiders
Then we open it with pycharm
Click File ->open, find the folder where the project is located, and open it!
In this way, we have created a new scrapy project, if you install the required libraries, scrapy red, remember to go to switch the interpreter
In File->settings marked in red
It is also recommended that you create a new file in the root directory and write the
from scrapy import cmdline
("scrapy crawl tencent_spider".split())
This way, we can just run, run, run each time, without having to go to the command line to execute the run command
It can be created by right mouse clicking on tencent and creating a new python file.
The above is a small introduction to the python3 Scrapy crawler to create a project detailed integration, I hope to help you, if you have any questions please leave me a message, I will reply to you in a timely manner. I would also like to thank you very much for your support of my website!