What is the best way to sort large csv files in php, up to 5GB ?
Should I use some CSVReader or use unix commands?
-
easy with unix coreutilsRomanPerekhrest– RomanPerekhrest2017-12-27 12:28:46 +00:00Commented Dec 27, 2017 at 12:28
-
How often do you need to sort and is it dependent on user input?Sam Kool– Sam Kool2017-12-27 12:29:36 +00:00Commented Dec 27, 2017 at 12:29
-
@Sam Kool 5 times / minuteAndrius– Andrius2017-12-27 12:46:01 +00:00Commented Dec 27, 2017 at 12:46
-
The problem is that I have large CSV file and i need to use it for CRUD pageAndrius– Andrius2017-12-27 12:46:37 +00:00Commented Dec 27, 2017 at 12:46
Add a comment
|
1 Answer
PHP pages are usually made for processing quick web pages for browsers, for a file in GB and to not keep crashing into memory and time limits, call a unix command independently.
Here is a good reference to a similar question: https://stackoverflow.com/a/222445/6288442
" > /dev/null 2>/dev/null &"
That will redirect both stdio (first >) and stderr (2>) to /dev/null and run in the background.
There are other ways to do the same thing, but this is the simplest to read.
An alternative to the above double-redirect:
" &> /dev/null &"
and https://stackoverflow.com/a/223745/6288442
`echo "the command"|at now`;
As for the sort command: https://stackoverflow.com/a/9471139/6288442
sort --field-separator=';' --key=2,1,3
1 Comment
C. Geek
performance using a unix command is usually always better since it runs directly on the shell as opposed to running over the php interpreter which is an extension on the web server you are using