19

I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:

I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?

Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.

Improve this question

2 Answers 2

Reset to default
36

First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

  • open browser developer tools, network tab
  • go to the target site
  • click submit button and see what XHR request is going to the server
  • simulate this XHR request in your spider

Also see:

Hope that helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

i was referring to this url blog.scrapy.org/scraping-ajax-sites-with-scrapy‎ which is no longer available, thanks for reminding me of readthedocs.com
Got it. If you have problems with the spider implementation, consider posting another question with what url are you trying to crawl, what button to click etc. Happy scraping!
@Lynob here's the URL you're talking about: web.archive.org/web/20130525095330/http://blog.scrapy.org/…
4

I found the answer very useful but I would like to make it more simple.

response = requests.post(request_url, data=payload, headers=request_headers)

request.post takes three parameters url, data and headers. Values for these three attributes can be found in the XHR request.

Copy the whole request header and form data to load into the above variables and you are good to go

Improve this answer

1 Comment

What if I can't find the XHR request? It's not visible in Chrome or FoxFox webtools. However, there is an AJAXSetup.js file. Is there any other way to find that information?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.