Authentication while scraping via BeautifulSoup in python

Question

I have created a piece of code to scrape an article off the ft.com website.

url = ""
r = requests.get(url)
soup = bs4.BeautifulSoup(r.content, "html.parser")
for a in soup.find_all('div', {"id":"storyContent"}):
    print a

1) On the website, there is a div tag with id:storyContent but I get no output as a result of this code which means that it didn' enter the loop at all! What might the reason be?
Now ft.com does not give access to articles without entering username and password.
I have logged into ft.com using chrome.
Suppose my username, password details are the following:
Username : [email protected]
Pass: 12345
I need to know either of the following:
2) How can I provide this authentication in my code?
3) How can I use the session on chrome (on which I'm already logged in) to acces the webpage/article details.
4) If authentication is the resson behind no output!
5) I am trying to get the article's body out of the webpage.
Thanks!

use python mechanize for authentication and fetch the source and then use beautiful soup — Harish
– Harish, Commented Aug 30, 2016 at 11:45
Try and get the form, then you can submit with r = requests.post('http://httpbin.org/post', data = {'key':'value'}) — Daniel Lee
– Daniel Lee, Commented Aug 30, 2016 at 11:50
@DanielLee, if you could help me with one more little thing please. I am trying to extract the p tags within the div tags for div in soup.find_all('div', {"id":"storyContent"}): and then for p in div.find_all('p'): print p.string. There are certain p tags that also contain anchors. So the p.string returns a None. How do I get the entire article out? Thanks! — Blabber
– Blabber, Commented Aug 30, 2016 at 12:38

Daniel Lee · Accepted Answer · 2016-08-30 11:59:04Z

1

Rather start with this.

url = "http://www.ft.com"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")
for a in soup:
    print a

Then add a requests when you find the key:value pair required

r = requests.post('http://www.ft.com/xxx', data = {'key':'value'})

answered Aug 30, 2016 at 11:59

Daniel Lee

8,1994 gold badges31 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Authentication while scraping via BeautifulSoup in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related