2

I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying.

$ sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 | 
  sudo tar x -C /home/pi/test

...this works perfectly. Example output:

191MiB 0:00:03 [58.1MiB/s] [<=>       ]

...in the terminal. But I want to do something more with this output and I have simple python script /home/pi/screentest.py:

import sys
print "start"
for line in sys.stdin:
    print "line: " + line
print "end"

Test run:

$ (   sudo tar c --files-from /tmp/camfilenames | 
      pv --width 40 | 
      sudo tar x -C /home/pi/test   ) | 
   python /home/pi/screentest.py

Output:

start
191MiB 0:00:03 [55.1MiB/s] [<=>       ]
end

I understand that pv returns everything in one buffer, so how to force it to return it line by line and flush the buffer every time to output something like:

start
line: 0MiB 0:00:00 ...
line: 60MiB 0:00:01 ...
line: 120MiB 0:00:02 ...
line: 191MiB 0:00:03 ...
end
Improve this question
1
  • 1
    If you like a fast copy, I recommend to use star -copy. Commented Nov 11, 2019 at 15:58

2 Answers 2

Reset to default
1

pv outputs the progress to stderr, so pipe to the python script didn't get anything from pv. It printed end after pv's output stopped probably because by then the last tar had exited, and so the script's stdin had been closed.

You might need something like:

( sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 -f | 
  sudo tar x -C /home/pi/test ) 2>&1 >/dev/null | python /home/pi/screentest.py

Where we discard the stdout of the tar pipeline, and instead send stderr to the pythons script. Note that we need to force progress output from pv with -f, since pv won't progress if stderr is not a terminal.

But you might want to look in to pv's --format option. You can use something like:

sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 --format $'%b %t\n' | 
  sudo tar x -C /home/pi/test

to get similar output.

Improve this answer
1

Thank you @muru, I tried you'r solution, but in the meantime I found that main issue is with forcing PV to output new lines instead of just updating constantly one line.

So I wrote this:

$ tail -f /home/pi/log | pv -f |& stdbuf -oL tr '\r' '\n'

And It worked, so now question was - how to call my python script to do something with every line - and I found that I can use do-while loop for that:

$ tail -f /home/pi/log | pv -f |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do echo $LINE; done;)

So having this I just use my python script (little LCD handler) to show PV status on LCD:

tail -f /home/pi/log | pv -f --width 20 -i 2 |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do python /home/pi/screen.py $LINE; done;)

And I re-introduced my TAR script to copy many small files into it:

$ (sudo tar c --files-from /tmp/camfilenames | pv -f --width 20 -i 1 | sudo tar x -C /home/pi/test) |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do python /home/pi/screen.py $LINE; done;)

So at the end when I add more parameters to PV, and I made few string replaces before sending PV output to LCD, and I add few break lines I have this:

cd /home/pi/cam \ && ( \ sudo tar c --files-from /tmp/camfilenames \ | pv -f -s $( \ du -sb /home/pi/cam \ | awk '{print $1}' \ ) --width 64 -i 5 \ | sudo tar x -C /mnt/usb \ ) \ |& stdbuf -oL tr '\r' '\n' \ | ( \ while read -r LINE; \ do LINE="${LINE// /_}"; \ LINE="${LINE//_\[/ [}"; \ LINE="${LINE//\]_/] }"; \ lcd $LINE; \ done; \ ) \ && cd /home/pi

This is how PV output looks on LCD: This is how PV output looks on LCD

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.