Saving json object to csv file python

Question

I am trying to save a json object that I scraped from a webpage into a csv file for geoprocessing. This is the code:

from bs4 import BeautifulSoup
import json
import urllib2
import csv
import os
import requests
import re

page1 = urllib2.urlopen("http://runkeeper.com/user/212579518/route/513771")
soup = BeautifulSoup(page1)

point_re = re.compile('.*routePoints =(.*);')
point_json = point_re.search(str(soup)).group(1)
point_data = json.loads(point_json)

##thislineworks
with open('test2.csv','wb') as f:
    w = csv.writer(f)
    w.writerows(point_data())

When I execute the code I got this message:

Traceback (most recent call last):
  File "C:\Users\Jesus\Desktop\scrapping1advance1.py", line 19, in <module>
    w.writerows(point_data())
TypeError: 'list' object is not callable

Any ideas on what i am doing wrong?

Thanks

amccormack · Accepted Answer · 2014-02-26 03:38:06Z

1

Your error is with the code:

point_data = json.loads(point_json)

##thislineworks
with open('test2.csv','wb') as f:
    w = csv.writer(f)
    w.writerows(point_data())

point_data is not a function. Use:

point_data = json.loads(point_json)

##thislineworks
with open('test2.csv','wb') as f:
    w = csv.writer(f)
    w.writerows(point_data) #notice there are no parens here

answered Feb 26, 2014 at 3:38

amccormack

14k11 gold badges41 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

asado23 Over a year ago

I tried the code you suggested but it returned this error: Traceback (most recent call last): File "C:\Users\Jesus\Desktop\scrapping1advance1.py", line 18, in <module> w.writerows(point_data) Error: sequence expected

amccormack Over a year ago

The point_data = json.loads(point_json) is not assigning point_data to a sequence. This is a good time to use the python shell (such as IDLE) and run through your script in the shell, by typing in each line yourself. After you assign point_data, print it to the screen to see what kind of data it is. It may work if you change w.writerows to w.writerow. But I'm guessing that isn't what you intended.

asado23 Over a year ago

When I change the code to write.row(point_json) it saves all the observations in a single row with the following format: {u'altitude': 40, u'longitude': -77.036478, u'deltaDistance': 0, u'latitude': 38.918704, u'type': u'StartPoint', u'deltaPause': 0} However I would like this ouput to look more like a dataframe. Any ideas on how to do this?

amccormack Over a year ago

@jesusleal Yup. Thats what I meant by "guessing that isn't what you intended." Looking at the site you are pulling down, I think you may want to grab the entire json line, and not just one line? If so you will want to use the re.MULTILINE flag when creating your regex. Then you can use the writerows method

asado23 Over a year ago

Thanks for your help. So I inspecte the object created and it is a list of dictionaries. With this code I got what I needed:

keys = ['altitude','longitude','deltaDistance','latitude','type','deltaPause']  with open('test3.csv','wb') as f:         dict_writer = csv.DictWriter(f, keys)         dict_writer.writer.writerow(keys)         dict_writer.writerows(point_data)

Collectives™ on Stack Overflow

Saving json object to csv file python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related