Running Scrapy from a script with file output

Question

I'm currently using Scrapy with the following command line arguments:

scrapy crawl my_spider -o data.json

However, I'd prefer to 'save' this command in a Python script. Following https://doc.scrapy.org/en/latest/topics/practices.html, I have the following script:

import scrapy
from scrapy.crawler import CrawlerProcess

from apkmirror_scraper.spiders.sitemap_spider import ApkmirrorSitemapSpider

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(ApkmirrorSitemapSpider)
process.start() # the script will block here until the crawling is finished

However, it is unclear to me from the documentation what the equivalent of the -o data.json command line argument should be within the script. How can I make the script generate a JSON file?

Possible duplicate of scrapy from script output in json

Casper
– Casper

2017-04-18 09:49:01 +00:00
Commented Apr 18, 2017 at 9:49 — Casper
– Casper, Commented Apr 18, 2017 at 9:49
Do refer this answer

Jaysheel Utekar
– Jaysheel Utekar

2017-04-18 09:50:52 +00:00
Commented Apr 18, 2017 at 9:50 — Jaysheel Utekar
– Jaysheel Utekar, Commented Apr 18, 2017 at 9:50
Possible duplicate of scrapy from script output in json

Gallaecio
– Gallaecio

2019-07-16 16:27:18 +00:00
Commented Jul 16, 2019 at 16:27 — Gallaecio
– Gallaecio, Commented Jul 16, 2019 at 16:27

vold · Accepted Answer · 2017-04-18 09:48:36Z

14

You need to add the FEED_FORMAT and FEED_URI to your CrawlerProcess:

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'data.json'
})

answered Apr 18, 2017 at 9:48

vold

1,5491 gold badge15 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

softmarshmallow Over a year ago

how to make it overridable?

Collectives™ on Stack Overflow

Running Scrapy from a script with file output

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related