How to store scrapy shell output/response to variable instead of html file

Question

I am trying store the html code into variable called response using cmdline.execute as shown in below code ,but it is unavailable to store and program code breaks at scrapy shell, can anyone tell me how to store raw html to variable

import scrapy

from scrapy import cmdline

linkedinnurl = "https://stackoverflow.com/users/5597065/adnan-stab=profile"

response = cmdline.execute("scrapy shell https://stackoverflow.com/users/5597065/adnan-s?tab=profile".split()))

print(response)

@vezunchik Clearly not a duplicate. The linked question seeks to store the value of requests.post, whereas this question seeks to store the result of an operation initiated by cmdline.execute. Completely different scenario. — Brian Warshaw
– Brian Warshaw, Commented May 16, 2019 at 11:58

Amrit · Accepted Answer · 2019-05-17 05:42:52Z

2

You can do like this to store raw html to variable:

 class MySpider(scrapy.Spider):
        def parse(self, res):
            with open(dynamic_file_name_function(res.url), 'w') as f:
                f.write(res.body)

if you don't need dynamic file name then just do :

 class MySpider(scrapy.Spider):
        def parse(self, res):
            with open(your_file_path, 'w') as f:
                f.write(res.body)

edited May 17, 2019 at 5:42

answered May 16, 2019 at 12:54

Amrit

2,1831 gold badge24 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Amrit Over a year ago

that was a function to create dynamic filename.You can remove that if you don't need dynamic file name. I have updated the answer

Collectives™ on Stack Overflow

How to store scrapy shell output/response to variable instead of html file

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related