SoFunction
Updated on 2025-03-02

Python pass parameters to Scrapy via command line

When using Scrapy for web crawling, you may encounter the need to perform crawler tasks according to different parameters in different operating environments. For example, crawl different pages, adjust the time range of crawls, or dynamically change certain configuration items. To solve this problem, Scrapy provides a way to pass parameters to the crawler via the command line.

This article will explain in detail how to pass parameters from the command line in Scrapy, and how to get these parameters in the crawler code to enhance the flexibility and configurability of the crawler.

1. Why do you need to pass parameters through the command line?

In many practical application scenarios, the behavior of crawlers may change with different operating environments. for example:

  • You may need to specify the target URL or keyword to crawl from the command line.
  • You may want to pass the start and end times through the command line to define the time range of crawl.
  • The configuration of the scheduling needs to be controlled through parameters, such as delay, concurrency quantity, etc.
  • Passing parameters through the command line allows your crawler to adapt to different needs more flexibly without having to modify the code or configuration file every time.

2. Pass the parameter using the -a parameter

Scrapy provides the -a option to pass parameters. The use of -a is very simple. The passed parameters will be used as attributes of the crawler class, or passed to start_requests(),init() method.

2.1 Basic usage

Suppose you have a crawler that needs to receive a URL from the command line as the start address of the crawl, which you can pass through with the -a parameter.

First, write a simple Scrapy crawler and define a crawler class that receives url parameters:

import scrapy

class MySpider():
    name = 'myspider'

    def __init__(self, url=None, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.start_urls = [url] if url else []

    def parse(self, response):
        (f"Crawling URL：{}")

In this example, url is a parameter passed in from the command line. If the url is specified, the crawler will use it as the starting URL in start_urls.

Next, start the crawler via the command line and pass the url parameters:

scrapy crawl myspider -a url=

When you run this command, the crawler will crawl.

2.2 Passing multiple parameters

You can also pass multiple parameters via -a. For example, suppose you need to pass two parameters url and category to crawl data from different categories:

import scrapy

class MySpider():
    name = 'myspider'

    def __init__(self, url=None, category=None, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.start_urls = [url] if url else []
         = category

    def parse(self, response):
        (f"Crawling URL：{}")
        (f"Classification parameters：{}")

When running, you can pass parameters through the following command:

scrapy crawl myspider -a url= -a category=books

The crawler will record the crawled URL and the classification parameter category.

2.3 Passing parameters from the command line to the start_requests() method

In Scrapy, the reptile classinitThe () method and the start_requests() method are the most common places to receive parameters. If your parameters need to be processed in start_requests(), you can use them like this:

import scrapy

class MySpider():
    name = 'myspider'

    def __init__(self, category=None, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
         = category

    def start_requests(self):
        urls = [
            '/category1',
            '/category2'
        ]
        for url in urls:
            if :
                url += f'?category={}'
            yield (url=url, callback=)

    def parse(self, response):
        (f"Crawling：{}")

Start the crawler and pass the category parameter:

scrapy crawl myspider -a category=books

The crawler will dynamically build the URL based on the passed category parameters and start crawling.

3. Pass parameters through Scrapy settings (settings)

In addition to passing parameters through -a, Scrapy also allows direct modification of some configuration items through the command line, which will be passed into the crawler's settings to override the default configuration.

3.1 Use -s to modify Scrapy settings

The -s option allows you to modify Scrapy's settings on the command line. For example, you can change the crawler's USER_AGENT or DOWNLOAD_DELAY via the command line:

scrapy crawl myspider -s USER_AGENT="Mozilla/5.0" -s DOWNLOAD_DELAY=2

In crawlers, you can get these settings by:

import scrapy

class MySpider():
    name = 'myspider'

    def parse(self, response):
        user_agent = ('USER_AGENT')
        delay = ('DOWNLOAD_DELAY')
        (f"User Agent: {user_agent}, Download delay: {delay}")

3.2 Using dynamic configuration in

Sometimes you may want to dynamically modify the configuration based on the parameters passed by the command line, such as adjusting the number of concurrency or enabling/disabling a middleware. This can be achieved by passing the configuration in the command line:

scrapy crawl myspider -s CONCURRENT_REQUESTS=10 -s LOG_LEVEL=INFO

In this way, CONCURRENT_REQUESTS will be set to 10 and the log level will be set to INFO, overwriting the default value in .

4. Pass parameters through environment variables

In addition to -a and -s, you can pass parameters through environment variables, especially when deploying crawlers using containerized deployments (such as Docker), which is useful. Scrapy allows you to dynamically modify the behavior of crawlers by obtaining environment variables.

import scrapy
import os

class MySpider():
    name = 'myspider'

    def start_requests(self):
        url = ('TARGET_URL', '')
        yield (url=url, callback=)

    def parse(self, response):
        (f"Crawl URL：{}")

Set environment variables at runtime:

export TARGET_URL=""
scrapy crawl myspider

The crawler will dynamically decide the URL to crawl based on the environment variable TARGET_URL.

5. Summary

Scrapy provides multiple ways to pass parameters from the command line, making crawlers more flexible and configurable. Common ways include:

  • Use the -a parameter to pass data directly to the crawler class or the start_requests() method to dynamically specify crawling content.
  • Use the -s parameter to directly modify the settings of Scrapy, such as concurrency number, download delay and other configurations.
  • Passing parameters through environment variables is especially suitable for containerized deployment scenarios.

In these ways, Scrapy crawlers can easily adapt to a variety of different operating environments and needs without requiring code modifications every time. This is extremely important for projects that require frequent configuration adjustments or flexible scheduling of crawlers in production environments.

By using the command line to pass parameters reasonably, Scrapy crawlers not only become more flexible, but can also be easily integrated into various automated processes, such as timing tasks, CI/CD pipelines, etc.

This is the article about Python passing parameters to Scrapy through the command line. For more related content to pass parameters to Python Scrapy, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!