Python Scrapy: how to code the parameter instead of using cmd: use Custom code in Scrapy

Question

I am using Scrapy 0.20 with Python 2.7.

I used to do this in cmd,

 -s JOBDIR=crawls/somespider-1

to handle the dublicated items. Note please, I already did the changes in setting

I dont' want to use that in cmd.

Is there a way I can type it in code inside my spider?

Edit the topic such that the new one is more relevent..

Nabin
– Nabin

2014-03-03 10:18:34 +00:00
Commented Mar 3, 2014 at 10:18 — Nabin
– Nabin, Commented Mar 3, 2014 at 10:18

Nabin · Accepted Answer · 2014-03-02 20:16:28Z

2

It's so easy. Use dropitem in pipelines.py to drop the item. And you can use custom command to code the parameter inside of program.

Here is example of custom code in scrapy

Using the custom command (say : scrapy crawl mycommand)

you can run -s JOBDIR=crawls/somespider-1

Example:

Create a directory commands where you have scrapy.cfg file Inside the directory create a file mycommand.py

from scrapy.command import ScrapyCommand
from scrapy.cmdline import execute



class Command(ScrapyCommand):
    requires_project = True

    def short_desc(self):
        return "This is your custom command"


    def run(self, args, opts):
        args.append('scrapy')
        args.append('crawl')
        args.append('spider')##add what ever your syntax needs.In my case i want to get "scrapy crawl spider" in cmd
        execute(args)#send a list as parameter with command as a single element of it

Now go to cmd line and type scrapy mycommand. Then your magic is ready :-)

edited Mar 2, 2014 at 20:16

answered Mar 2, 2014 at 19:08

Nabin

11.8k8 gold badges70 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Marco Dinatsoli Over a year ago

could you clarify more

Nabin Over a year ago

i will give you a sample then

Nabin Over a year ago

and i will explain you it. just try to figure out how it works. and if you have problem, you can ask again

Marco Dinatsoli Over a year ago

I understood that i can do this self.jobdir = 'something, right?

Nabin Over a year ago

let us continue this discussion in chat

Collectives™ on Stack Overflow

Python Scrapy: how to code the parameter instead of using cmd: use Custom code in Scrapy

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related