Have you ever been gzipping a log file and wondered what the progress was? Or have you ever wanted to slow down a pipeline, for example throttling an SSH connection? I was reading at [Tollef Fog Heen's](http://err.no/personal/blog/tech/2008-05-19-22-32_new_backup_system) blog (a friend and former co-worker) about his backup solution. There, nestled inside sshconfig options, I learned about a beautiful gem called ‘pv.’

Tollef uses it in his ssh config like this (see the last line):

Host backup-$hostname
Hostname $backupserver.err.no
User backup-$hostname
IdentityFile /root/.ssh/id_rsa_rdup
ProxyCommand pv -L 40k -q | nc %h %p

This limits the speed of this ssh connection to 40k (according to the man page, this would be 40 kilobytes per second, not kbits)

I did a quick test to see it in action. To watch the progress of zipping a log file:

cat bandwidth_log | pv | gzip > ~/b.gz
188MB 0:00:10 [18.5MB/s] [ <=> ]

There are some command line options you can use to give it hints as to the amount of data to expect which gives it the ability to show an ETA and makes the progress bar a ratio of how much is done. Here’s an example of a large file:

$ size=`ls -sl full_log | cut -f 1 –delimiter=’ ‘`k
$ echo $size
3210360k
$ cat full_log | pv -s $size | gzip > ~/full_log.gz
712MB 0:00:47 [15.2MB/s] [============> ] 22% ETA 0:02:39

(note that $size has a k appended to it so that pv knows this value is in KB)
Therefore, something like this might be handy:

for logfile in *.log
do
size=`ls -sl $logfile | cut -f 1 –delimiter=’ ‘`k
echo “Processing $logfile ($size)…”
cat $logfile | pv -s $size | gzip > ~/$logfile.gz
done

Thanks for sharing your wisdom Tollef!

By the way, if you want to redirect the contents of a pipeline to a file and see what’s going on at the same time the command you want is _tee_ (which I learned from [Tom Pohl](http://www.tcpconsulting.com)).

[some command] | tee output.txt

instead of

[some command] > output.txt

I don’t think this would work good combined with pv but I haven’t tried it.