Better way to parse files and log output

Question

The purpose of this bit of my script is to extract certain pieces from logs files, previously collected in this script. However, there is an entry added in %parse% per line per for condition.

What I am wanting is IF something is found in the findstr section, then input found data in desired format, into the %parse% file.

Current output is (Its not very pretty here, than it is in my log file)

Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - -------------- Start c:\programdata\dctool\Data\<Removed_HostName>\MA\McScript.log -------------
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> -
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start Virus Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End Virus Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start Blocked Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End Blocked Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start Detected Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End Detected Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start Deleted Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End Deleted Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start File-cksum-mismatch Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End File-cksum-mismatch Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start encrypted Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End encrypted Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - Start Failed Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - End Failed Search 
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> -
Tue 04/03/2018 - 13:51:23.19 - <Removed_UN> - <Removed_HostName> - -------------- End c:\programdata\dctool\Data\<Removed_HostName>\MA\McScript.log ------------

-

I like the first and last line of this code. That lets me know that the file was parsed. However, I dont want all the middle junk in there, unless something is found. For example, "Start encrypted search" and "End encrypted search" would only be present, had it found something.

::::::::::::::::::
:: Analyze Data ::
::::::::::::::::::
:analyze
echo %date% - %time% - %un% - %host% --- Log Parser --- >>%logfile%
echo --- Log Parser ---
echo.
for /f "tokens=*" %%a in (%filelocation%) do (
    echo %date% - %time% - %un% - %host% - -------------- Start %%a ------------->>%parse%
    echo %date% - %time% - %un% - %host% ->>%parse%
    echo %date% - %time% - %un% - %host% - Start Virus Search >>%parse%
    echo Start Virus Search
    for /f "tokens=*" %%b in ('findstr /r /n "virus" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End Virus Search >>%parse%
    echo End Virus Search
    echo %date% - %time% - %un% - %host% - Start Blocked Search >>%parse%
    echo Start Blocked Search
    for /f "tokens=*" %%b in ('findstr /r /n "Blocked" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End Blocked Search >>%parse%
    echo End Blocked Search
    echo %date% - %time% - %un% - %host% - Start Detected Search >>%parse%
    echo Start Detected Search
    for /f "tokens=*" %%b in ('findstr /r /n "detected" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End Detected Search >>%parse%
    echo End Detected Search
    echo %date% - %time% - %un% - %host% - Start Deleted Search >>%parse%
    echo Start Deleted Search
    for /f "tokens=*" %%b in ('findstr /r /n "deleted" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End Deleted Search >>%parse%
    echo End Deleted Search
    echo %date% - %time% - %un% - %host% - Start File-cksum-mismatch Search >>%parse%
    echo Start File-cksum-mismatch Search
    for /f "tokens=*" %%b in ('findstr /r /n "File-cksum-mismatch" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End File-cksum-mismatch Search >>%parse%
    echo %date% - %time% - %un% - %host% - Start encrypted Search >>%parse%
    echo Start encrypted Search
    for /f "tokens=*" %%b in ('findstr /r /n "encrypted" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End encrypted Search >>%parse%
    echo End Failed Search
    echo %date% - %time% - %un% - %host% - Start Failed Search >>%parse%
    echo Start Failed Search
    for /f "tokens=*" %%b in ('findstr /r /n "Failed" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End Failed Search >>%parse%
    echo End Failed Search
    echo %date% - %time% - %un% - %host% - Start inv_partial_sync Search >>%parse%
    echo Start inv_partial_sync Search
    for /f "tokens=*" %%b in ('findstr /r /n "inv_partial_sync" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    echo %date% - %time% - %un% - %host% - End inv_partial_sync Search >>%parse%
    echo End inv_partial_sync Search
    echo %date% - %time% - %un% - %host% ->>%parse%
    echo %date% - %time% - %un% - %host% - -------------- End %%a ------------->>%parse%
)
echo %date% - %time% - %un% - %host% - Parser log Location: %parse%>>%logfile%
echo %date% - %time% - %un% - %host% - Done Parsing>>%logfile%
echo Done Parsing
echo.
exit /b

Is there a specific string you can search for to determine whether something was found or not? If you see that string, set or clear an environment variable, then suppress or echo the output based on that flag. You haven't provided enough information for a definitive answer here. — jwdonahue
– jwdonahue, Commented Apr 3, 2018 at 19:45
Please provide an example of what the output would look like based on your description. — Squashman
– Squashman, Commented Apr 3, 2018 at 20:22
Why are you encapsulating all of these checks inside a FOR /F command. You could just get rid of that and use the %filelocation% variable directly with all of your nested FOR /F commands. This will also give you an accurate time and date. As it stands the date and time will be the same for every check. — Squashman
– Squashman, Commented Apr 3, 2018 at 20:36

dbenham · Accepted Answer · 2018-04-03 21:21:07Z

Squashman answered your direct question the way I would handle it.

But you have another problem.

Using %date% and %time% within your FOR /F loop is pointless because it will be a constant value. Percent expansion occurs when the loop is parsed, and it only gets parsed once, before the loop is executed.

Possible solutions:

use call echo %%date%% - %%time%% ..., but this is relatively slow. It also doubles any quoted ^ characters.
or enable delayed expansion and use echo !date! - !time!, but this causes problems with ! literals that may be in your content when %%b is expanded. This can be solved by toggling delayed expansion on and off within the loop.
CALL a :timestamp subroutine that prints the timestamp and any message you provide as an argument. An additional argument can be used to flag printing to the screen in addition to the file. This isn't as fast as delayed expansion, but it avoids the toggle issue, and it is faster than CALL ECHO. Percent expansion works here because it is out of the scope of the parenthesized block, so it gets parsed with each call.

Also, repeatedly appending to the same file with redirection wastes time because the file must be opened and the file pointer moved to the end of file each time. It is much faster to redirect only once.

You have lots of repeating code that could be encapsulated in a :search routine, thus greatly simplifying your code.

>>"%parse%" (
  for /f "delims=" %%a in (%filelocation%) do (
    call :timestamp "-------------- Start %%a -------------"
    call :timestamp ""
    for %%s in (
      virus
      Blocked
      detected
      File-cksum-mismatch
      encrypted
      Failed
      inv_partial_sync
    ) do call :search "%%a" %%s
    call :timestamp ""
    call :timestamp "-------------- End %%a -------------"
  )
)
.... other stuff
exit /b

:search  File  String
  setlocal
  set "found="
  for /f "tokens=*" %%b in ('findstr /r /n %2 %1') do (
    if not defined found (
      call :timestamp "Start %~2 Search" CON
      set found=1
    )
    call :timestamp "%%b"
  )
  if defined found call :timestamp "End %~2 Search" CON
exit /b

:timestamp "message" conFlag
  echo %date% - %time% - %un% - %host% - %~1
  if "%~2" neq "" >con echo %date% - %time% - %un% - %host% - %~1
exit /b

Yes. You make some valid points. I had commented above as to why there is a need to use the outer FOR /F at all. All it is doing it putting a single file name into the FOR metavariable. The file location variable could just be used directly.

Squashman · Accepted Answer · 2018-04-03 20:28:39Z

2

If I understand you correctly, this type of logic should work. You essentially can use a variable as a flag to determine whether to output any information.

    echo Start encrypted Search
    for /f "tokens=*" %%b in ('findstr /r /n "encrypted" %%a') do (
        IF NOT DEFINED encryptflag (
            echo %date% - %time% - %un% - %host% - Start encrypted Search >>%parse%
            set "encryptflag=0"
        )
        echo %date% - %time% - %un% - %host% - %%b>>%parse%
    )
    IF DEFINED encryptflag echo %date% - %time% - %un% - %host% - End encrypted Search >>%parse%

answered Apr 3, 2018 at 20:28

Squashman

14.4k5 gold badges30 silver badges38 bronze badges

1 Comment

dbenham Over a year ago

You beat me to it.

Stephan · Accepted Answer · 2018-04-03 20:12:18Z

0

replace each block with something like the following:

for %%a in (a.txt) do (
    ...
    echo Start Blocked Search
    findstr "Blocked" %%a >nul && (
      echo %date% - %time% - %un% - %host% - Start Blocked Search >>%parse%
      for /f "tokens=*" %%b in ('findstr /r /n "Blocked" %%a') do (
        echo %date% - %time% - %un% - %host% - %%b >>%parse%
      )
      echo %date% - %time% - %un% - %host% - End Blocked Search >>%parse%
    )
    echo End Blocked Search
    ...
)

findstr "Blocked" %%a >nul && ( ... is quite exactly, what you wrote in your question: IF something is found in the findstr section, then ...

answered Apr 3, 2018 at 20:12

Stephan

56.8k11 gold badges66 silver badges97 bronze badges

Collectives™ on Stack Overflow

Better way to parse files and log output

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related