python: Inserting JSON data into dataframe

Question

{
"total_count": 2316913,
"incomplete_results": false,
"items": [
    {
        "id": 13737149,
        "name": "gremlins.js",
        "full_name": "marmelab/gremlins.js",
        "owner": {
            "login": "marmelab",
            "id": 3116319,
            "avatar_url": "https://avatars1.githubusercontent.com/u/3116319?v=4",
            "gravatar_id": "",
            "url": "https://api.github.com/users/marmelab",
            "html_url": "https://github.com/marmelab",
            "followers_url": "https://api.github.com/users/marmelab/followers",
            "following_url": "https://api.github.com/users/marmelab/following{/other_user}",
            "gists_url": "https://api.github.com/users/marmelab/gists{/gist_id}",
            "starred_url": "https://api.github.com/users/marmelab/starred{/owner}{/repo}",
            "subscriptions_url": "https://api.github.com/users/marmelab/subscriptions",
            "organizations_url": "https://api.github.com/users/marmelab/orgs",
            "repos_url": "https://api.github.com/users/marmelab/repos",
            "events_url": "https://api.github.com/users/marmelab/events{/privacy}",
            "received_events_url": "https://api.github.com/users/marmelab/received_events",
            "type": "Organization",
            "site_admin": false
        },

The above is data from the Github search API. What I am attempting to do is pull certain fields out and insert it into a dataframe.

Using a standard for loop like:

for i in jdata['items']:
  print(i['name']  + "\t" + i['html_url'])

prints the data I need, but, its ugly, hard to read, and since I have more things to extract, I am at a stopping point.

url = "https://api.github.com/search/repositories?q={}".format(sTerm)
jdata = requests.get(url).json()

I am using this to get input from the user, and storing the results in jdata.

Dataframes are new to me, and it seems that no matter what I try, it either displays everything in the search results, or nothing. Not sure what I'm doing wrong. (I know I didn't paste any dataframe code, nothing is working, so I figured I'd save everyone the eye cancer.)

fuglede · Accepted Answer · 2017-12-06 18:53:43Z

1

It sounds like you want to do something like

import pandas as pd
df = pd.DataFrame(jdata['items'], columns=['name', 'html_url'])

For example, if sTerm is 'hest',

In [15]: pd.DataFrame(jdata['items'], columns=['name', 'html_url'])
Out[15]:
                 name                                        html_url
0                Hest           https://github.com/MegaCakeEater/Hest
1              heston           https://github.com/daleroberts/heston
2   NotoSansKR-Hestia  https://github.com/theeluwin/NotoSansKR-Hestia
3                hest                https://github.com/mastensg/hest
4              Heston                https://github.com/jcfrei/Heston
5                Hest           https://github.com/abirk2thebone/Hest
6                Hest                   https://github.com/h6899/Hest
7                Hest                  https://github.com/hsurce/Hest
8                hest                   https://github.com/kmajo/hest

answered Dec 6, 2017 at 18:53

fuglede

18.3k3 gold badges62 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Josh Over a year ago

Yep, that worked once I added print(df.to_string()). Thanks for that, I was no where near that.

fuglede Over a year ago

You're welcome. If you found the answer helpful, you can accept it.

Collectives™ on Stack Overflow

python: Inserting JSON data into dataframe

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related