Get Ajax Json Output with Python Requests Library

Question

I am requesting an Ajax Web site with a Python script and fetching cities and branch offices of http://www.yurticikargo.com/bilgi-servisleri/Sayfalar/en-yakin-sube.aspx

I completed the first step with posting {cityID: 34} to this url and fetc the JSON output.

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetTownByCity

But I can not retrive the JSON output with Python although i get succesfully with Chrome Advanced Rest Client Extension, posting {cityID:54,townID:5416,unitOnDutyFlag:null,closestFlag:2}

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-unitservices.aspx/GetUnit

All of the source code is here

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests
import json


class Yurtici(object):

    baseUrl = 'http://www.yurticikargo.com/'
    ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/'
    getTown = 'GetTownByCity'
    getUnit = 'GetUnit'
    urlGetTown = baseUrl + ajaxRoot + getTown
    urlGetUnit = baseUrl + ajaxRoot + getUnit 
    headers = {'content-type': 'application/json','encoding':'utf-8'}

    def __init__(self):
        pass

    def ilceler(self, plaka=34): # Default testing value
        payload = {'cityId':plaka}
        url = self.urlGetTown
        r = requests.post(url, data=json.dumps(payload), headers=self.headers)
        return r.json() # OK

    def subeler(self, ilceNo=5902): # Default testing value
        # 5902 Çerkezköy 
        payload=  {'cityID':59,'townID':5902,'unitOnDutyFlag':'null','closestFlag':0}
        url = self.urlGetUnit
        headers = {'content-type': 'application/json','encoding':'utf-8'}
        r = requests.post(url, data=json.dumps(payload), headers=headers)
        print  r.status_code, r.raw.read()

if __name__ == '__main__':        
    a = Yurtici()
    print a.ilceler(37) # OK
    print a.subeler()   # NOT OK !!!

This is a trivial point, but the documentation clearly states that you should not use Response.raw unless you've set stream=True on your original request. — Lukasa
– Lukasa, Commented Nov 24, 2013 at 7:52

Lukasa · Accepted Answer · 2013-11-24 08:08:30Z

2

Your code isn't posting to the same url you're using in your text example.

Let's walk through this backwards. First, let's look at the failing POST.

url = self.urlGetUnit
headers = {'content-type': 'application/json','encoding':'utf-8'}
r = requests.post(url, data=json.dumps(payload), headers=headers)

So we're posting to a URL that is equal to self.urlGetUnit. Ok, let's look at how that's defined:

baseUrl = 'http://www.yurticikargo.com/'
ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/'
getUnit = 'GetUnit'
urlGetUnit = baseUrl + ajaxRoot + getUnit

If you do the work in urlGetUnit, you get that the URL will be http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetUnit. Let's put this alongside the URL you used in Chrome to compare the differences:

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetUnit
http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-unitservices.aspx/GetUnit

See the difference? ajaxRoot is not the same for both URLs. Sort that out and you'll get back a JSON response.

answered Nov 24, 2013 at 8:08

Lukasa

15.7k4 gold badges35 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

guneysus Over a year ago

Thanks a lot, i was sleepy :) Now i solved it. By fixing the ajax url. And also,the response was encoded gzip. I decompressed it with zlib as in here : stackoverflow.com/questions/2695152/…

Lukasa Over a year ago

Requests can do that for you. Use Response.content, Response.text or Response.json() and Requests will automatically decompress it.

Collectives™ on Stack Overflow

Get Ajax Json Output with Python Requests Library

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related