How to make scrapy output to stdout to be read from Python

Question

I have a spider that I want to output its results to standard output so that it can be read by subprocess.check_output. I don't want to output to a file as an intermediary.

I've tried adding the flag '-o', 'stdout' but it doesn't work.

test = subprocess.check_output([
        'scrapy', 'runspider', 'spider.py',
        '-a', f"keywords={keywords}", '-a', f'domain={domain}', '-a', f'page={1}',
        '-s', 'USER_AGENT=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
    ])

What does your test variable contain? Please explain why it doesn't work. — Valentino
– Valentino, Commented Feb 8, 2019 at 1:41
Oh well it's just an empty string because it's not working but I want it to contain the output of my spider. It works if I do -o test.json at the end of that command, but that puts it into a file, which I don't want. — FLOWMEEN
– FLOWMEEN, Commented Feb 8, 2019 at 3:27
yea! as a string of course, but i want to consume it through the subprocess check_output method — FLOWMEEN
– FLOWMEEN, Commented Feb 8, 2019 at 6:35

Иван Васильев · Accepted Answer · 2019-02-08 14:12:41Z

1

Try this: Main .py

from subprocess import Popen, PIPE

command = ["scrapy runspider yourspider.py -a some additional commands"]
proc = Popen(command, shell=True, stdout=PIPE, stderr=PIPE)
proc.wait()
res = proc.communicate()
if proc.returncode:
    print(res[1])
print('result:', res[0])

Sub yourspider.py

import sys

# your code

print(something what you need to transfer)

edited Feb 8, 2019 at 14:12

answered Feb 8, 2019 at 13:41

Иван Васильев

1424 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to make scrapy output to stdout to be read from Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related