Memory Issues : Shell Scripting

Question

I have 5 files in a folder a.csv b.csv ....

I need to combine these files into one file called X.csv and in future keep merging the contents of X.csv with a,b,c etc.

Even with only two lines in each file I have error message saying that there is not enough memory - its essentially only copying 10 lines across. Im using the following command :

 paste -d, *.csv >> X.csv.

However when i use

 paste -d, *.csv > X.csv

There are no memory issues. This however I canont use since i need the information on the X.CSV file previously as well so I should only append not edit contents of the whole file.

Would anyone know how I achieve this? These are ',' separated CSV files and I would like to not copy the header (Row 1) which names the columns more than once.

I use MAC OSX Mavericks 8 GB Ram.

Thank You :)

Thank You but I am having Similar memory issue with cat as well. cat *.csv > X.csv. I cannot use a..e since some files have slightly different names but its 5 files in total. — user2233834
– user2233834, Commented Jul 7, 2014 at 12:01
I think you may have a weird file in there. Try ls -al and see if there is anything big and odd! Else try rebooting. — Mark Setchell
– Mark Setchell, Commented Jul 7, 2014 at 12:04
Seriously, save your output to a temporary file whose extension is NOT ".csv" and rename when you're done. — loreb
– loreb, Commented Jul 7, 2014 at 13:12

jimm-cl · Accepted Answer · 2014-07-07 13:51:12Z

2

As @loreb said, *.csv is matching X.csv as well. You can avoid that using a proper globbing variable, but that depends if your shell is capable of handling that... I know bash can, and you posted your question with the bash tag, but you said you are using Mac OSX - well, I guess you can try it anyway.

paste -d, [a-z]*.csv >> X.csv

That will run the paste command for the files named from a.csv to z.csv, lowercase, so it will not take X.csv, which I think is the current problem.

answered Jul 7, 2014 at 13:51

jimm-cl

5,4823 gold badges27 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kojiro Over a year ago

I don't understand what you intend to imply about OS X and bash. OS X comes with an older version of bash, but it still has extglob. shopt -s extglob; paste -d, !(X).csv >> X.csv

jimm-cl Over a year ago

@kojiro, I'm not a OS X user, so I'm not sure if that's ok for its shell or not... so, better to try than not, but I wasn't implying something bad, by the way :) - Thanks for clarifying it for me ;)

l0b0 · Accepted Answer · 2014-07-07 13:34:23Z

0

If you have a lot of data, and in the absence of a tool which keeps track of the line number at which processing last stopped in [a-z].csv (which I don't think exists) you can use the following process:

Ensure that the writing program is not going to write to the CSV files anymore. Some possible ways to do this:
- Move the files to a read-only filesystem.
- Stop the file writing program.
- Somehow force the program to start writing to a new file descriptor.
Rename or move the files if necessary to avoid the writer opening them again.
Restart the writer if necessary.
paste -d, /temporary_directory/*.csv >> /final_destination/X.csv

answered Jul 7, 2014 at 13:34

l0b0

59.6k32 gold badges155 silver badges247 bronze badges

Collectives™ on Stack Overflow

Memory Issues : Shell Scripting

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related