Posting form data with requests and python

Question

Posting form data isn't working and since my other post about this wasn't working, I figured I would try to ask the question again so maybe I can get another perspective. I am currently trying to get the requests.get(url, data=q) to work. When I print, I am getting a page not found. I have resorted just to set variables and join them to the entire URL to make it work but I really want to learn this aspect about requests. Where am I making the mistake? I am using the HTML tag attributes name=search_terms and name=geo_location_terms for the form.

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com"
q = {'search_terms': search_terms, 'geo_locations_terms': location}
page = requests.get(url, data=q)
print(page.url)

Lev Zakharov · Accepted Answer · 2018-08-21 01:50:19Z

2

You have few little mistakes in your code:

Check form's action parameter. Then url = "https://www.yellowpages.com/search"
Second parameter is geo_location_terms not geo_locations_terms.
You should pass query parameters in requests.get as params not as request data (data).

So, the final version of code:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
print(page.url)

Result:

https://www.yellowpages.com/search?search_terms=Bars&geo_location_terms=New+Orleans%2C+LA

answered Aug 21, 2018 at 1:50

Lev Zakharov

2,4371 gold badge13 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kamikaze_goldfish Over a year ago

So when using requests.get() use ‘params’ and when requests.post() use data to pass the values?

Lev Zakharov Over a year ago

@Kamikaze_goldfish not really, but usually it's so. Check this answer.

Dani Mesejo · Accepted Answer · 2018-08-21 02:55:37Z

Besides the issues pointed by @Lev Zakharov, you need to set the cookies in your request, like this:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"

with requests.Session() as session:
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36',
        'Cookie': 'cookies'
    })

    q = {'search_terms': search_terms, 'geo_locations_terms': location}
    response = session.get(url, params=q)

    print(response.url)
    print(response.status_code)

Output

https://www.yellowpages.com/search?search_terms=Bars&geo_locations_terms=New+Orleans%2C+LA
200

To get the cookies you can see the requests using some Network listener for instance using Chrome Developer Tools Network tab, then replace the value 'cookies'

Collectives™ on Stack Overflow

Posting form data with requests and python

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related