0

Let's say i want to scrape this page: https://twitter.com/nfl

from bs4 import BeautifulSoup
import requests

page = 'https://twitter.com/nfl'
r = requests.get(page)
soup = BeautifulSoup(r.text)
print soup

The more i scroll down on the page, the more results show up. But this above request only gives me the initial load. How do i get all the information of the page as if I were to manually scroll down?

Improve this question
1
  • Hi, I am in similar situation as yours, my recommendation is to learn a little bit of js ( that is what I am doing right now). You can actually call the js file with appropriate parameters to make it directly output the data to a file (json most likely). But since I am learning it now, I can't provide a better solution. Correct me if I am wrong. The case I am working on is stocktwits.com/symbol/aapl . I hope it will you a bit. Commented Mar 28, 2016 at 8:06

4 Answers 4

Reset to default
4

First parse the data-max-id="451819302057164799" value from the html source.

Then using the id 451819302057164799 construct an url like below:

https://twitter.com/i/profiles/show/nfl/timeline?include_available_features=1&include_entities=1&max_id=451819302057164799

Now get the html source of the link and parse using simplejson or any other json library.

Remember, the next page load(when you scroll down) is available from the value "max_id":"451369755908530175" in that json.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

https://twitter.com/i/profiles/show/nfl/timeline?include_available_features=1&include_entities=1&max_id=451819302057164799 is this a generic solution for all twitter pages? how do you know how to construct that specific url?
@jason_cant_code can be. I didn't check. May be the nfl is the key for different pages.
i don't think this works. I'm getting a much shorter page than expected.
Use requests session to ensure that you keep alive your session in every GET
1

Better solution is to use Twitter API.

There are several python twitter API clients, for example:

Improve this answer

3 Comments

good suggestion. I'll probably give that a shot if there is no good generic solution.
@jason_cant_code well, this is definitely the way to go if you need the data from twitter.
Definitely if you can use a API, forget scraping. This is your solution.
1

If the content is dynamically added with javascript, your best chance is to use selenium to control a headless browser like phantomjs, use the selenium webdriver to simulate the scrolldown, add a wait for the new content to load, and only then extract the html and feed it to your BS parser.

Improve this answer

3 Comments

See stackoverflow.com/questions/14583560/… for scrolling down
I'll try this if there is no better solution
Yes, Selenium is always an option.... but IMO is not the best one. I preffer to figure out the http traffic between browser and server and simulate it using requests, urllib or whatever.... much faster than Selenium.
0

For dynamically generated content, the data is usually in json format. So we have to inspect the page, go to network option and find the link which will give us the data/response on the fly. For example : The page - https://techolution.app.param.ai/jobs/ the data is generated dynamically. For that I got this link - https://techolution.app.param.ai/api/career/get_job/?query=&locations=&category=&job_types=

After that the web scraping becomes a bit easy and I have done that in python using Anaconda Navigator. Here is the github link for that - https://github.com/piperaprince01/Webscraping_python/blob/master/WebScraping.ipynb

If you can make any changes to make it better then feel free to do so. Thank You.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.