Python beautiful soup form input parsing

Question

My goal is to grab a list of all input names and values. To pair them up and submit the form. The names and values are randomised.

from bs4 import BeautifulSoup # parsing

html = """
<html>
<head id="Head1"><title>Title Page</title></head>
<body>
    <form id="formS" action="login.asp?dx=" method="post">

    <input type=hidden name=qw1NWJOJi/E8IyqHSHA== value='gDcZHY+nV' >
    <input type=hidden name=sfqwWJOJi/E8DFDHSHB== value='kgDcZHY+n' >
    <input type=hidden name=Jsfqw1NdddfDDSDKKSL== value='rNg4pUhnV' >
    </form>

</body>

</html>
"""

html_proc = BeautifulSoup(html)

This bit works fine:

print html_proc.find("input", value=True)["value"]
> gDcZHY+nV

However the following statements don't work or don't work as hoped:

print html_proc.find("input", name=True)["name"]
> TypeError: find() got multiple values for keyword argument 'name'

print html_proc.findAll("input", value=True, attrs={'value'})
> []  

print html_proc.findAll('input', value=True)
> <input name="qw1NWJOJi/E8IyqHSHA==" type="hidden" value="gDcZHY+nV">
> <input name="sfqwWJOJi/E8DFDHSHB==" type="hidden" value="kgDcZHY+n">
> <input name="Jsfqw1NdddfDDSDKKSL==" type="hidden" value="rNg4pUhnV">
> </input></input></input>, <input name="sfqwWJOJi/E8DFDHSHB==" type="hidden" 
> value="kgDcZHY+n">
> <input name="Jsfqw1NdddfDDSDKKSL==" type="hidden" value="rNg4pUhnV">
> </input></input>, <input name="Jsfqw1NdddfDDSDKKSL==" type="hidden" value="rNg4p
> UhnV"></input>

alecxe · Accepted Answer · 2014-04-11 01:09:22Z

23

You cannot submit a form with BeautifulSoup, but here's how you can get the list of name,value pairs:

print [(element['name'], element['value']) for element in html_proc.find_all('input')]

prints:

[('qw1NWJOJi/E8IyqHSHA==', 'gDcZHY+nV'), 
 ('sfqwWJOJi/E8DFDHSHB==', 'kgDcZHY+n'), 
 ('Jsfqw1NdddfDDSDKKSL==', 'rNg4pUhnV')]

edited Apr 11, 2014 at 1:09

answered Apr 11, 2014 at 0:59

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sarasimple Over a year ago

Thank you. Brilliant. Parsimonious solutions to code are intoxicating.

sarasimple Over a year ago

When my reputation is higher I'll come back and rate your answer up.

alecxe Over a year ago

@sarasimple thanks, but don't worry about it, just glad it helped. Happy web-scraping!

An old man in the sea. Over a year ago

alcxe, what if I need to submit a form? What would you recommend?

Community · Accepted Answer · 2017-05-23 12:32:10Z

7

d = {e['name']: e.get('value', '') for e in html_proc.find_all('input', {'name': True})}
print(d)

prints:

{'sfqwWJOJi/E8DFDHSHB==': 'kgDcZHY+n', 
 'qw1NWJOJi/E8IyqHSHA==': 'gDcZHY+nV', 
 'Jsfqw1NdddfDDSDKKSL==': 'rNg4pUhnV'}

Building on @alecxe, this avoids KeyErrors, and parses the form into a dictionary, more ready for requests.

url = 'http://example.com/' + html_proc.form['action']
requests.post(url , data=d)

Though if this gets any more complicated (cookies, scripts) you might want to Mechanize.

The reason for the TypeError is confusion over the first parameter to find() being 'name'. Instead html_proc.find("input", attrs={'name': True}). Also for the attrs parameter, instead of the set {'value'} use the dictionary {'value': True}.

edited May 23, 2017 at 12:32

CommunityBot

11 silver badge

answered Aug 18, 2015 at 14:04

Bob Stein

17.5k11 gold badges92 silver badges108 bronze badges

Collectives™ on Stack Overflow

Python beautiful soup form input parsing

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related