3

I have a PHP script which produces a error log. Now I have my script set to append a line on each successful SQL insert.

But after 1,5 million records it gets kinda full. I'd like to have my log a max amount of lines but I don't really know how to read out/remove lines without removing the file.

Let's say my file can have 20 files max, after that it should remove the first line and append another.

Can someone guide me in the correct direction? Or if a script will be posted, please elaborate.

Improve this question
3
  • 1
    for such opterations i would recommend you to use OS dependig scripts, e.g. wc -l log.txt gives you the lines tail head cut sed can be your friends on php side you have to read the whole file and then handle with it, witch can be slow down your task .... Commented Aug 24, 2015 at 10:56
  • @donald123: tail, sed and any other Unix tool will have to do the exact same thing, read the whole file and write it back (although it should do it faster, as it is optimised in native code to do it). Commented Aug 24, 2015 at 11:00
  • @Amadan i know that but it is still faster as using php Commented Aug 24, 2015 at 11:01

2 Answers 2

Reset to default
3

it should remove the first line and append another.

This is an expensive operation, as it involves reading the whole file, ignoring the first line, then writing the whole file back. It is not a big deal if you only want 20 lines; if you want 20K lines, and your update frequency is not rare, it will be a big deal.

A better way to manage this is to rotate whole files; when one file fills up, start writing in another file, and only keep N last files. If on Linux, you can use logrotate in crontab to solve this. There are a lot of good tutorials on how to set it up (here is one). This is the approach a lot of tools already take, for example Apache web server.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

It will be merely 50 lines max, So (I think) it won't really affect performance of my server.
Still, I wouldn't recommend to read the entire file, remove a line, and then write it back. It would be less memory-inefficient to create a temporary file, write to it while reading from the source file, all except the line to remove, and - when done - close both files, and then either delete the source file and move (rename) the temporary to the original name, or rename the original to a "backup" name and rename the temporary to the original.
@DmitriSologoubenko: Actually, without a benchmark, I won't say I'm certain, but I am reasonably certain that using file_get_contents, explode, implode and file_write_contents would be faster than reading line-by-line, as there is no buffering needing to be done and all the iteration is being done in C rather than in PHP. It is still a good idea to write to a temp file first (as a safety precaution against failure). But I was not advocating any specific way of reading/writing; both line-by-line and whole-file-at-a-time are expensive, and would be best avoided.
@Amadan, I didn't say "it would be faster", I said it would be less memory-inefficient. When you have to deal with large files, loading the entire file in memory will consume at least as much memory as the file size. If you keep using this approach on a production server, this will take as much virtual memory as the file size times number of concurrent processes. In any case, I would also suggest to implement some kind of locking, to avoid 2 concurrent processes do this at the same time, which can cause data corruption and loss.
@DmitriSologoubenko: You might have noted that in my answer, with large files, I strongly discourage removing the head line in the first place. I also reiterate that I never said you need to read in the whole file at once; just that you would have to read it in and write it out in its entirety. ("then" might actually imply it, but that was not what I meant to write.)
0

Maybe there is something PHP specific (doubt it, and I couldn't find it). Two do it manually with PHP it will require reading/writing whole log file and that would be really resource consuming (with millions of lines). You can check for some OS tools that can do that like @donald123 said.

My approach would be to using one log file per day (or other period that is suitable). And cleaning up old log files once a while.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.