Processing log files using bash tools

Question

I have a .csv file with approx 10 columns that is logging data. I want to use something like this:

How to get the first line of a file in a bash script?

Where it is grabbing the first line of each file and then processing the logs. However, once the line is processed, I want to mark it as processed (can be as simple as adding a new column on the end with a *** or something). So basically, I want to grab the first line not processed, process it, and move to the next unprocessed line, etc.

I need to do this using sed, awk, grep, and/or other standards. The bash script will sit and run in the background on an infinite while loop. Essentially, I am trying to read and process this log file in real-time, but need the log for history.

Edit: Also, I need this to mark the lines that have been read in the file. That way if the server stops, I can know right where to pick up processing. So tail will work if I can figure out a way to do that.

Thanks!

So you don't want the first line in the file, but rather the first unprocessed line? — bountiful
– bountiful, Commented Aug 20, 2012 at 13:51
Correct - I do want the first unprocessed line. Logrotate is a possibility ... but like the original post said this will be running and I need to grab this data as close to real-time as possible. — slthomason
– slthomason, Commented Aug 20, 2012 at 14:46

bountiful · Accepted Answer · 2012-08-20 13:59:51Z

2

Rather than sitting in a infinite loop you could do this:

tail -n +1 -f your_log_file | some_processing_pipeline

This will start reading your logfile at line 1, then continuously wait for new lines to appear and pass them to some_processing_pipeline.

answered Aug 20, 2012 at 13:59

bountiful

8041 gold badge8 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

slthomason Over a year ago

Is there a way to have that line get marked once it gets passed? In case of system catastrophe (or simple error somewhere) I need to have the line mark itself as processed. Otherwise, that solution sounds great.

bountiful Over a year ago

Maybe in that case you want to create a new log file, like Lev Levitsky suggested. I'm not sure it's wise to edit a log file after it has been created.

slthomason Over a year ago

Big question to creating a new file is if something happens and the log processing stops, and I need to start it again, how do I know where to start?

bountiful Over a year ago

You could tell your processor that if something happens to move the log else where and start a new one. But that might not be as easy or sensible as it sounds.

slthomason Over a year ago

Right ... and in the case of server failure, patching reboot, etc ... that might not always be practical. Thus, the reason I was looking for some way to just mark the line as processed. There isn't a way to piggy-back a sed command onto that?

|

Collectives™ on Stack Overflow

Processing log files using bash tools

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related