Sending an jquery AJAX GET request with Python request library

Question

I have a website I need to scrape and it is using jquery AJAX function to get information from the server. I have been investigating the code for a while and I successfully get the response from server using:

data = {'part_number':'1234'}
r = $.ajax({
        type : 'GET',
        url : 'ajaxurl',
        data : data
    })

Note that this is done via js console so I am alreadt logged in. When I try to do it in python I need to login first of course:

import requests
headers = {'User-Agent': 'Mozilla/5.0','Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
payload = {'username':'me','password':'1234'}
link = 'login url'
session = requests.Session()
resp = session.get(link,headers=headers)
cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
resp = session.post(link,headers=headers,data=payload,cookies =cookies)

#until here sucesss!"############

url = "ajaxurl"
my_params={'part_number':'1234'}
r = session.get( url = url, data = my_params, cookies =cookies,headers =headers )

The post request for login goes well but for the GET response I receive BAD REQUEST 400. I cannot figure out how to format my request. I don't know what ajax does to my request. Anyone has any ideas?

Thanks in Advance!

Ok I got something else. When I add 'X-Requested-With': 'XMLHttpRequest' the status code of the response changes to 500 Internal Server Error with the content : ''SOAP request failed with fault code "1543" and fault string "An error occurred while processing diagram node "Output 2". Attempting to convert data with a picture clause into a string failed. Source field, invalid source value and picture clause: Root.PW00118O.DOCDATA.PART-FOLDER-ISSUE-PUB, , 9(2)."' — bop
– bop, Commented Feb 20, 2015 at 12:45

bop · Accepted Answer · 2015-02-23 08:36:55Z

7

Solved it! I added 'X-Requested-With': 'XMLHttpRequest' to headers and did:

pn = '1234'
r = requests.get(ajaxurl + '?part_number=' + pn, headers=headers, cookies=cookies)

Did not understand why though :(

answered Feb 23, 2015 at 8:36

bop

4451 gold badge3 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alex W Over a year ago

It tells the server you're requesting the content via a XMLHttpRequest, e.g. AJAX.

Shivam Sharma Over a year ago

A bit late, but, servers can very easily filter requests that are not AJAX generated (request type: xhr). I guess that's why you got a 400 response.

Artur Filipiak · Accepted Answer · 2015-02-19 16:02:06Z

0

BAD REQUEST 400 means that the server was unable to understand or process your request (the data you sent via AJAX).

Since you're using GET method in AJAX, I would give a try to traditional url with query string:

var request = 'part_number=1234';
$.get('http://your.website/your_server_file?'+request, function(data){
    // handle the server response (data) here ...
});

edited Feb 19, 2015 at 16:02

answered Feb 19, 2015 at 15:56

Artur Filipiak

9,1874 gold badges33 silver badges56 bronze badges

1 Comment

bop Over a year ago

Well the ajax request with jquery is successful. The question is how to do it using python namely emulating it.

Collectives™ on Stack Overflow

Sending an jquery AJAX GET request with Python request library

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related