Getting a Nested List into a Dataframe in Python

Question

I saw a few other similar questions with converting nested lists into a Pandas DataFrame, but my seems a bit more complicated. I currently have a list with many nests inside of it (did I say that right? lol).

For example:

    [{'date': 'yyyy-mm-dd',
    'total_comments':1,
    'id': 123456,
    'engagements_by_type': {'url clicks': 111, 'other clicks':222},
    'url': 'https://hi.com/stackoverflow/is/the/best',
    'posts_by_paid_unpaid': {'paid': 1, 'total': 100, 'unpaid': 99}
    'organic_impressions': ,
    'social_media_impressions': {'facebook': 2, 'twitter': 4, 'instagram': 4, 'twitch': 6,
    'total_social_media-impressions' : 10}
    {'date':....
    ......}]

*Notice the 'total_social_media_impressions' is a total of the nested list preceeding it, 'social_media_impressions'. This makes it very tricky.

...and so on. There are way more columns than I mentioned, but I am just trying to show a shortened example.

Does anyone know how to turn this type of long nested list into a pandas dataframe?

Update: I used a for loop to identify which columns inside the list are nested:

df = pandas.DataFrame(data)
columns = df.columns

for i in columns:
    if str(df[i][0]).startswith('{'):
        print('True')
    else:
        print('False')

Next, step is to figure out how to properly manipulate them to be in the DataFrame as a normal column and not be nested.

why don't you update your question with sample output as well that would help people to understand your problem — suraj deshmukh
– suraj deshmukh, Commented Jul 20, 2019 at 12:00

suraj deshmukh · Accepted Answer · 2019-07-20 12:38:29Z

I am posting my solution assuming you wish to convert nested dict keys into columns as well.

import pandas as pd

data = [
            {'date': 'yyyy-mm-dd',
            'total_comments':1,
            'id': 123456,
            'engagements_by_type': {'url clicks': 111, 'other clicks':222},
            'url': 'https://hi.com/stackoverflow/is/the/best',
            'posts_by_paid_unpaid': {'paid': 1, 'total': 100, 'unpaid': 99},
            'organic_impressions': 1,
            'social_media_impressions': {'facebook': 2, 'twitter': 4, 'instagram': 4, 'twitch': 6}},
            {'date': 'yyyy-mm-dd',
            'total_comments':1,
            'id': 123456,
            'engagements_by_type': {'url clicks': 111, 'other clicks':222},
            'url': 'https://hi.com/stackoverflow/is/the/best',
            'posts_by_paid_unpaid': {'paid': 1, 'total': 100, 'unpaid': 99},
            'organic_impressions': 1,
            'social_media_impressions': {'facebook': 2, 'twitter': 4, 'instagram': 4, 'twitch': 6}},
            {'date': 'yyyy-mm-dd',
            'total_comments':1,
            'id': 123456,
            'engagements_by_type': {'url clicks': 111, 'other clicks':222},
            'url': 'https://hi.com/stackoverflow/is/the/best',
            'posts_by_paid_unpaid': {'paid': 1, 'total': 100, 'unpaid': 99},
            'organic_impressions': 1,
            'social_media_impressions': {'facebook': 2, 'twitter': 4, 'instagram': 4, 'twitch': 6}}
    ] 

def create_plain_dict(ip):
    for i in list(ip):
        if type(ip[i]) == dict: #check whether value associated with that key is dict and if yes then update it with original dict and pop that key
            temp = ip.pop(i) #in this way we are basically converting nested dict into plain dict 
            ip.update(temp)
    return ip

mod_data = list(map(create_plain_dict, data))

df = pd.DataFrame(data)

dataframe looks like this

Collectives™ on Stack Overflow

Getting a Nested List into a Dataframe in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related