creating one data frame using Pandas for multiple CSV files from many URL's

Question

I'd like to create 1 data frame/structure using Pandas from multiple CSV files from URL's, keeping the initial header line.

With a single URL everything works as expected:

df = pd.read_csv('http://www.URL1.csv')

I have attempted the following with multiple URL's:

df = pd.read_csv('http://www.URL1.csv', 'http://www.URL2.csv', ...)

However, when attempting to print for testing, the result is spaced out over thousands of lines and is far from the standard layout. Since I am new to Pandas, it is clear I am doing something wrong.

I'd expect the layout to be as followed:

Header1 Header2 Header3 ...
DATA    DATA    DATA    ...

jezrael · Accepted Answer · 2016-12-07 22:34:21Z

4

I think you need list comprehension with list of urls where output is list of DataFrames. Then use concat for join together:

urls = ['http://www.URL1.csv', 'http://www.URL2.csv']
dfs = [pd.read_csv(url) for url in urls]

df = pd.concat(dfs, ignore_index=True)

edited Dec 7, 2016 at 22:34

answered Dec 7, 2016 at 22:28

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Enigmatic Over a year ago

That's amazing! Works like a charm! Is there any chance you could explain the need for list comprehension?

jezrael Over a year ago

You need list comprehension if need apply some function like read_csv for each element of another list of elemens.

jezrael Over a year ago

... output of list comprehension is list of outputs (dataframes) which return function (like read_csv). And fastest way for create dataframe from list of dataframes is concat.

Enigmatic Over a year ago

Thanks, is there a purpose behind ignore_index=True? I can't seem to spot the difference

jezrael Over a year ago

You can avoid duplicates in index. Please check docs

Collectives™ on Stack Overflow

creating one data frame using Pandas for multiple CSV files from many URL's

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related