Reading in header information from csv file using Pandas

Question

I have a data file that has 14 lines of header. In the header, there is the metadata for the latitude-longitude coordinates and time. I am currently using

pandas.read_csv(filename, delimiter",", header=14)

to read in the file but this just gets the data and I can't seem to get the metadata. Would anyone know how to read in the information in the header? The header looks like:

CSD,20160315SSIO
NUMBER_HEADERS = 11
EXPOCODE = 33RR20160208
SECT_ID = I08
STNBBR = 1
CASTNO = 1
DATE = 20160219
TIME = 0558
LATITUDE = -66.6027
LONGITUDE = 78.3815
DEPTH = 462
INSTRUMENT_ID = 0401
CTDPRS,CTDPRS_FLAG,CTDTMP,CTDTMP_FLAG
DBAR,,ITS-90,,PSS-78

read_csv() has a header option so that you can populate the resulting dataframe with the headers as column names. Is this what you are looking for? — Akshat Mahajan
– Akshat Mahajan, Commented Apr 6, 2016 at 4:18
Probably you should use header=range(14)? Because with header=14 you will only use row 14 for header and skip 1-13 rows. — Anton Protopopov
– Anton Protopopov, Commented Apr 6, 2016 at 4:19
What format is the header in? You should post an example of what it looks like. You probably won't be able to read it with pandas.read_csv because there is no standard metadata format for csv's, you may have to write some custom code. — Marius
– Marius, Commented Apr 6, 2016 at 4:40

Community · Accepted Answer · 2016-12-21 16:47:51Z

You have to parse your metadata header by yourself, yet you can do it in an elegant manner in one pass and even by using it on the fly so that you can extract data out it / control the correctness of the file etc.

First, open the file yourself:

f = open(filename)

Then, do the work to parse each metadata line to extract data out it. For the sake of the explanation, I'm just skipping these rows:

for i in range(13):  # skip the first 13 lines that are useless for the columns definition
    f.readline()  # use the resulting string for metadata extraction

Now you have the file pointer ready on the unique header line you want to use to load the DataFrame. The cool thing is that read_csv accepts file objects! Thus you start loading your DataFrame right away now:

pandas.read_csv(f, sep=",")

Note that I don't use the header argument as I consider by your description you have only that one last line of header that is useful for your dataframe. You can build and adjust hearder parsing values / rows to skip from that example.

takachanbo · Accepted Answer · 2016-04-17 03:55:10Z

1

Although the following method does not use Pandas, I was able to extract the header information.

with open(fname) as csvfile:
    forheader_IO2016 = csv.reader(csvfile, delimiter=',')
    header_IO2016 = []
    for row in forheader_IO2016:
        header_IO2016.append(row[0])

date = header_IO2016[7].split(" ")[2]
time = header_IO2016[8].split(" ")[2]
lat = float(header_IO2016[9].split(" ")[2])
lon = float(header_IO2016[10].split(" ")[4])

answered Apr 17, 2016 at 3:55

takachanbo

6832 gold badges6 silver badges17 bronze badges

Collectives™ on Stack Overflow

Reading in header information from csv file using Pandas

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related