Python numpy: Data file with multiple datasets

Question

Suppose you have a data file which includes several data sets separated by the string "--" in the following format:

--
<x0_val> <y0_val>
<x1_val> <y1_val>
<x2_val> <y2_val>
--
<x0_val> <y0_val>
<x1_val> <y1_val>
<x2_val> <y2_val>
...

How can you read the whole file into an array of arrays so that you can plot all data sets afterwards to the same picture with a for loop looping over the outer array ?

genfromtxt('data.dat', delimiter=("--"))

gives lots of

Line #1550 (got 1 columns instead of 2)

What have you tried? You can use split via two delimiters (first --, then ' '. — Andy Hayden
– Andy Hayden, Commented Sep 11, 2012 at 10:31

oz123 · Accepted Answer · 2012-09-11 10:31:39Z

1

I will update ...

I would first split the file into multiple files, which can reside in memory as objects or on the filesystems as new files.

You can locate the string -- with the module re.

Then you can use the link I posted above.

answered Sep 11, 2012 at 10:31

oz123

29k30 gold badges133 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ritter Over a year ago

There are 130 data sets. Really split into multiple files?

oz123 Over a year ago

@Frank, why not? If it is only in the memory, no harm is done. If it is persistent, it will be easier to work later.

Community · Accepted Answer · 2017-05-23 11:55:55Z

1

If you're 100% certain that you have no negative values in your file, you can try a quick:

np.genfromtxt(your_file, comments="-")

The comments="-" will force genfromtxt to ignore all the characters after -, which of course will give weird results if you have negative variables. Moreover, the result will be just a lump of your dataset in a single array

Otherwise, the safest route is to iterate on your file and store the lines that do not match -- in one list per block, something along the lines:

blocks = []
current = []
for line in your_file:
    if line.startswith("-"):
        blocks.append(np.array(current))
        current = []
    else:
        current += line.split()

You may have to get rid of the first block if empty.

You could also check a mmap based solution already posted.

edited May 23, 2017 at 11:55

CommunityBot

11 silver badge

answered Sep 11, 2012 at 10:31

Pierre GM

20.5k3 gold badges58 silver badges67 bronze badges

2 Comments

ritter Over a year ago

No, with commments it just combines all data sets. It doesn't create an array of arrays.

Pierre GM Over a year ago

Yes, it will jst make ahuge array of all your dataset concatenated. The comments="-" forces np.genfromtxt to skip the lines that have a - in them. Once again, bad idea if you have negative values. If you want an array of array, construct individual lists per block, then transform each list into an array.

Collectives™ on Stack Overflow

Python numpy: Data file with multiple datasets

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related