Python Requests to CSV

Question

I've been trying to get the extract a JSON requests API result into a csv but I'm struggeling,

so far the below code return the value in the terminal

import requests
import csv
import json
from pprint import pprint
import urllib
import pandas as pd

url = "https://esi.evetech.net/latest/markets/10000002/orders/?datasource=tranquility&order_type=all&page=1"

payload={}
headers = {}
r = {}

response = urllib.request.urlopen(url)

text = response.read()
json_data = json.loads(text)

pprint(json_data)
df = pd.read_json(r)
df.to_csv("output.csv")

however Pandas return me with an error in regards of the Dictionary class? ( apologize I'm not really familiar when it come to coding )

Last question would be, what would be the logic to continue the url request beyond page=1 in the url variable? (order_type=all&page=1 ) I don't know how many page the system have

thanks

Can you post the error trace you are getting to the question — Mithun Manohar
– Mithun Manohar, Commented Jan 6, 2021 at 12:04

Mithun Manohar · Accepted Answer · 2021-01-06 12:19:24Z

1

The error is reproducible with the code above and looks like some encoding error with theurlib library. But I was able to load the data successfully using the python requests module

import requests
url = "https://esi.evetech.net/latest/markets/10000002/orders/datasource=tranquility&order_type=all&page=1"
response = requests.get(url)
data = response.text
df = pd.read_json(data)
df.to_csv("output.csv")

answered Jan 6, 2021 at 12:19

Mithun Manohar

5961 gold badge7 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Quentin D Over a year ago

That worked as well thanks a lot Justa! I'm trying to figure out a way for the page=1 now

aneroid · Accepted Answer · 2021-01-06 12:45:11Z

1

You've done r = {} and then df = pd.read_json(r), so why would that work?

You can do either one of the following:

# from your code, above
text = response.read()

# create DF from serialized JSON data (a string)
df = pd.read_json(text)

Or:

# from your code, above
text = response.read()
json_data = json.loads(text)

# create DF from loaded JSON data (in your case a list of dicts)
df = pd.DataFrame.from_records(json_data)

And the line import urllib should be import urllib.request. Or just use the 'requests' package, which you've also imported already.

Part 2:

To get all the pages, keep incrementing the value of page until you get an error or an empty response. (It's bad form to ask multiple unrelated questions on SO.) See this answer for guidance on how.

edited Jan 6, 2021 at 12:45

answered Jan 6, 2021 at 12:27

aneroid

16.7k3 gold badges42 silver badges77 bronze badges

3 Comments

Quentin D Over a year ago

Perfect, that worked out I've got confuse as I never used pandas before - any suggestion for the page=1 ?

aneroid Over a year ago

Btw, if you're happy with the answer, "accept" it.

aneroid Over a year ago

Edited. Either try writing the code for that with the link provided, or post a new question.

Collectives™ on Stack Overflow

Python Requests to CSV

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related