UNIX | Use awk to pull out specific values from a log file

Question

I am trying to extract specific values from a logfile like below :

Table "xxx"."xxxx":

  3785568 Rows successfully loaded.
  0 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.

Bind array size not used in direct path.
Column array  rows :    5000
Stream buffer bytes:  256000
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:       3785568
Total logical records rejected:         0
Total logical records discarded:        0
Total stream buffers loaded by SQL*Loader main thread:      878
Total stream buffers loaded by SQL*Loader load thread:      796

Run began on Fri Sep 01 04:00:26 2017
Run ended on Fri Sep 01 04:04:45 2017

Elapsed time was:     00:04:19.24
CPU time was:         00:00:08.56

What i would like to retrieve are :

3785568 as number_rows
Sep 01 04:00:26 2017 as start_time
Sep 01 04:04:45 2017 as end_time

How is this possible this extraction with awk?

Any help would be really much appreciated :)

Thank you very much for your time.

Akshay Hegde · Accepted Answer · 2017-09-05 11:07:04Z

awk '/Rows successfully loaded/{
        print $1 " as number_rows"
        next
    }
    /Run began on/{ 
        sub(/Run began on /,""); 
        print $0 " as start_time"
        next 
   }
   /Run ended on/{
        sub(/Run ended on /,"");    
        print $0 " as end_time"
   }' infile

Input

$ cat infile
Table "xxx"."xxxx":

  3785568 Rows successfully loaded.
  0 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.

Bind array size not used in direct path.
Column array  rows :    5000
Stream buffer bytes:  256000
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:       3785568
Total logical records rejected:         0
Total logical records discarded:        0
Total stream buffers loaded by SQL*Loader main thread:      878
Total stream buffers loaded by SQL*Loader load thread:      796

Run began on Fri Sep 01 04:00:26 2017
Run ended on Fri Sep 01 04:04:45 2017

Elapsed time was:     00:04:19.24
CPU time was:         00:00:08.56

Output

$ awk '/Rows successfully loaded/{
      print $1 " as number_rows"
      next
  }
  /Run began on/{ 
      sub(/Run began on /,""); 
      print $0 " as start_time"
      next 
  }
  /Run ended on/{
      sub(/Run ended on /,""); 
      print $0 " as end_time"
  }' infile

3785568 as number_rows
Fri Sep 01 04:00:26 2017 as start_time
Fri Sep 01 04:04:45 2017 as end_time

JFS31 · Accepted Answer · 2017-09-05 14:11:23Z

0

So for your given file this works:

awk '/Rows/{ if (++n==1){ print $1 } }/began/ || /ended/{ print $5,$6,$7,$8 }' log.file

output:

3785568
Sep 01 04:00:26 2017
Sep 01 04:04:45 2017

edited Sep 5, 2017 at 14:11

answered Sep 5, 2017 at 11:03

JFS31

5185 silver badges14 bronze badges

5 Comments

Ed Morton Over a year ago

BEGIN{ OFS = " " } is doing nothing useful, just setting OFS to the default value it already has.

JFS31 Over a year ago

Yep, you're right. I was in a hurry in getting to lunch there :)

tln_jupiter Over a year ago

@JFS31, another hot question please! If I want to add to this awk you provided one more argument extractrion, the table name without quotes how could this be achieved?????

JFS31 Over a year ago

This should do: awk '/^Table/{ gsub(/"||:/,"",$2); print $2 }/Rows/{ if (++n==1){ print $1 } }/began/ || /ended/{ print $5,$6,$7,$8 }' it gives you the table name without quotes and the : at the end. And next time it would be nice if you would ask it in a separate question.

tln_jupiter Over a year ago

@JFS31 Thanks, but this awk does not do the desired output.For more details, I logged stackoverflow.com/questions/46073193/… could you please have a look? :)

bukkojot · Accepted Answer · 2017-09-05 11:05:30Z

0

For this purpose better solution is grep

ROWS=`grep "Total logical records read" logfile.txt | sed 's/[^0-9]*//g'` 
START=`grep "Run began on " | cut -d" " -f4-`

answered Sep 5, 2017 at 11:05

bukkojot

1,5481 gold badge12 silver badges16 bronze badges

Comments

Raman Sailopal · Accepted Answer · 2017-09-05 11:12:32Z

0

awk '/[[:digit:]]+[[:blank:]]Rows successfully/ { print $1" as number_rows" } /^Run began on .*$/ { print $4" "$5" "$6" "$7" "$8" as start_time" } /^Run ended on .*$/ { print $4" "$5" "$6" "$7" "$8" as end_time"}' filename

answered Sep 5, 2017 at 11:12

Raman Sailopal

13k2 gold badges15 silver badges21 bronze badges

1 Comment

kayess Over a year ago

While this code snippet may solve the question, including an explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion. Please also try not to crowd your code with explanatory comments, this reduces the readability of both the code and the explanations!

RomanPerekhrest · Accepted Answer · 2017-09-05 11:14:38Z

0

Short awk approach:

awk '/Rows success/{ print $1 }/^Run (began|ended)/{ print $5,$6,$7,$8 }' file

The output:

3785568
Sep 01 04:00:26 2017
Sep 01 04:04:45 2017

answered Sep 5, 2017 at 11:14

RomanPerekhrest

93.1k4 gold badges75 silver badges112 bronze badges

1 Comment

tln_jupiter Over a year ago

Dear Roman, very good one as well, thank you, already answered by JFS31 above!

Shakiba Moshiri · Accepted Answer · 2017-09-05 11:26:03Z

0

if you do not mind Perl or grep with -P

perl -lne 'print $& if /\d+ (?=Rows successfully)|^Run (began|ended) on Fri \K[^\n\r]+/g' file

it outputs:

3785568 
Sep 01 04:00:26 2017
Sep 01 04:04:45 2017

or:

grep -Po '\d+ (?=Rows successfully)|^Run (began|ended) on Fri \K[^\n\r]+' file

edited Sep 5, 2017 at 11:26

answered Sep 5, 2017 at 11:08

Shakiba Moshiri

24.6k3 gold badges41 silver badges49 bronze badges

1 Comment

tln_jupiter Over a year ago

Very good idea, although I dont prefer perl on my prod machine!

Collectives™ on Stack Overflow

UNIX | Use awk to pull out specific values from a log file

6 Answers 6

Comments

5 Comments

Comments

1 Comment

1 Comment

or:

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

5 Comments

Comments

1 Comment

1 Comment

or:

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related