How to dynamically refer to dataframes in a for loop in Python

Question

I am very new to python and it's probably a simple question, but I cannot seem to find a solution.

I have several pandas data frames with names going like: output_1, output_2, ..., output_n

I want to sum their lengths (as in the number of their rows) and I came up with something like this:

sum =0
for num in range(1,n):
    nameframe="output_"+str(num)
    sum+=nameframe.shape[0]

The problem is that Python sees nameframe as a string, not as the name of a dataframe.

Looking around I found a potential solution:

sum =0
for num in range(1,n):
    x = globals()["output_urls_%s" % num] 
    sum+=x.shape[0]

This seems to work, however the usage of globals() seem to be very discouraged. Therefore, what is the most pythonic way to achieve my purpose?

You're in a messed up situation that requires a lot of work because you didn't stick with standards to begin with. Instead of manually creating all your data frames and giving them df_id like names, create them in a loop and stick them into a list. Then you could iterate over that list. — FooBar
– FooBar, Commented Aug 25, 2014 at 13:05
you can use nameframe = eval("output_"+str(num)) in the loop, but I agree with @FooBar you should be storing these as a list upon creation, or if you want to keep names use a dictionary. — ZJS
– ZJS, Commented Aug 25, 2014 at 14:08
OK, thank you. I indeed inserted the data frames in a list. It's much more tidy and easy to access. — thepule
– thepule, Commented Aug 26, 2014 at 9:22

chrisb · Accepted Answer · 2014-08-25 13:05:29Z

3

The most pythonic way would probably be to store your dataframes in a list. E.g.,

dfs = [output_1, output_2, ...]
df_length = sum(x.shape[0] for x in dfs)

Alternatively, you could look at storing your data in a combined pandas data structure, assuming they are all related in some way. E.g., if each dataframe is a different group, you could set a MultiIndex on the combined frame, like

df = pd.concat([output_1, output_2, ...], keys=['group_a', 'group_b', ..])

Then you could just take the length of the combined frame.

answered Aug 25, 2014 at 13:05

chrisb

52.7k8 gold badges73 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

EdChum Over a year ago

Your second line can be simplified to df_length = sum(len(x) for x in dfs)

thepule Over a year ago

Thanks, as suggested throughout the comments, storing the data frames in a list is the best solution.

Collectives™ on Stack Overflow

How to dynamically refer to dataframes in a for loop in Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related