Adding multiple dictionaries into a single Dataframe pandas

Question

I have a set of python dictionaries that I have obtained by means of a for loop. I am trying to have these added to Pandas Dataframe.

Output for a variable called output

{'name':'Kevin','age':21}
{'name':'Steve','age':31}
{'name':'Mark','age':11}

I am trying to append each of these dictionary into a single Dataframe. I tried to perform the below but it just added the first row.

df = pd.DataFrame(output)

Could anyone advice as to where am going wrong and have all the dictionaries added to the Dataframe.

Update on the loop statement

The below code helps to read xml and convert it to a dataframe. Right now I see I am able to loop in through multiple xml files and created dictionaries for each xml file. I am trying to see how could I add each of these dictionaries to a single Dataframe:

def f(elem, result):
    result[elem.tag] = elem.text
    cs = elem.getchildren()
    for c in cs:
        result = f(c, result)
    return result

    result = {}
    for file in allFiles:
        tree = ET.parse(file)
        root = tree.getroot()
        result = f(root, result)
        print(result)

@anky_91, Dataframe that stores these values from the dictionaries — coding learner
– coding learner, Commented Mar 6, 2019 at 11:03
Possible duplicate of Append dictionary to pandas dataframe in a loop — entropy
– entropy, Commented Mar 6, 2019 at 11:04

Guru · Accepted Answer · 2021-10-08 03:38:17Z

3

You can append each dictionary to list and last call DataFrame constructor:

out = []
for file in allFiles:
    tree = ET.parse(file)
    root = tree.getroot()
    result = f(root, result)
    out.append(result)
        
df = pd.DataFrame(out)

edited Oct 8, 2021 at 3:38

Guru

2,3716 gold badges32 silver badges50 bronze badges

answered Mar 6, 2019 at 11:04

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

coding learner Over a year ago

thanks for the help. I tried the first approach you mentioned above and see that I am able to only get the header column returned back to the Dataframe df

coding learner Over a year ago

I have problem such that in future I would be running this across different files that would have different header columns so I am not able to define what the header columns would be each time.

jezrael Over a year ago

@codinglearner - So in lopp is not possible create list of dictionaries?

coding learner Over a year ago

have added the loop statement, I am trying to overcome. However I am unable to add it to a Dataframe. Hope this helps.

jezrael Over a year ago

@codinglearner - it is better, also is possible add some data sample working with your code?

|

perl · Accepted Answer · 2019-03-06 14:22:48Z

1

We can add these dicts to a list:

ds = []
for ...:      # your loop
    ds += [d] # where d is one of the dicts

When we have the list of dicts, we can simply use pd.DataFrame on that list:

ds = [
    {'name':'Kevin','age':21},
    {'name':'Steve','age':31},
    {'name':'Mark','age':11}
]
pd.DataFrame(ds)

Output:

    name  age
0  Kevin   21
1  Steve   31
2   Mark   11

Update: And it's not a problem if different dicts have different keys, e.g.:

ds = [
    {'name':'Kevin','age':21},
    {'name':'Steve','age':31,'location': 'NY'},
    {'name':'Mark','age':11,'favorite_food': 'pizza'}
]
pd.DataFrame(ds)

Output:

   age favorite_food location   name
0   21           NaN      NaN  Kevin
1   31           NaN       NY  Steve
2   11         pizza      NaN   Mark

Update 2: Building up on our previous discussion in Python - Converting xml to csv using Python pandas we can do:

results = []
for file in glob.glob('*.xml'):
    tree = ET.parse(file)
    root = tree.getroot()
    result = f(root, {})
    result['filename'] = file # added filename to our results
    results += [result]

pd.DataFrame(results)

edited Mar 6, 2019 at 14:22

answered Mar 6, 2019 at 11:42

perl

9,9811 gold badge14 silver badges23 bronze badges

5 Comments

coding learner Over a year ago

thanks for the help. I am continuing from the other ticket where you helped me build the xml to csv convertor. Since it was a new issue I raised a new ticket on this. I have multiple dictionaries from each of the xml file but now trying to add them to Dataframe

coding learner Over a year ago

I tried to add all dicts to a list by doing the below ds = [] for i in result: ds += [i] print(ds) The above converts the dicts to a list but not dicts inside a list as shown below. Sample output: ['name':'Kevin','age':21] ['name':'Steve','age':31] ['name':'Mark','age':11]

perl Over a year ago

OK, so let me clarify it a bit, on each step we get a dict like {'name': 'Kevin', ...}, right?

perl Over a year ago

OK, great, can you check the code from Update 2 please?

perl Over a year ago

Added filename to the results with result['filename'] = file # added filename to our results

Collectives™ on Stack Overflow

Adding multiple dictionaries into a single Dataframe pandas

2 Answers 2

7 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related