How can I add a new spider arg to my own template in Scrapy/Zyte

Question

I am working on a paid proxy spider template and would like the ability to pass in a new argument on the command line for a Scrapy crawler. How can I do that?

Answer 1

This is achievable by using kwargs in your spider's __init__-Method:

import scrapy


class YourSpider(scrapy.Spider):
    name = your_spider

    def __init__(self, *args, **kwargs):
        super(YourSpider, self).__init__(*args, **kwargs)
        self.your_arg = kwargs.get("your_cmd_arg", 42)

Now it would be possible to call the spider as follows:
scrapy crawl your_spider -a your_cmd_arg=foo

For more information on the topic, feel free to check this page in the Scrapy documentation.