Comments on: Pigz – parallel gzip OMG

By: Raul Martinez

Raul Martinez — Thu, 05 Aug 2010 10:53:18 +0000

I like pigz but could not find a statically compiled binary of it.
Problem is one our server still has outdated version of zlib so I can not build the pigz source.

Appreciate if you know where I can find one..

By: Robert Stewart

Robert Stewart — Sun, 04 Apr 2010 19:08:41 +0000

Database backups are a sweet spot for me, too. When I need to set up a new MySQL replication slave of an existing master, the faster I can get the snapshot loaded on the slave, the faster the slave can catch up to the ongoing changes on the master and become available for reads. Of course, the additional load on the master during the parallel gzip has to be taken into account.

It might be interesting to combine pigz with mk-parallel-dump for quick test exports on quiescent systems, depending on how much mk-parallel-dump is already saturating the cores. mk-parallel-dump currently defaults to gzip.

By: allspaw

allspaw — Sat, 03 Apr 2010 13:48:49 +0000

Alex: in our case, we have some database backups that are massive, but yeah assuming there’s some recursive compression going on many many files, that would be interesting to see.

David: Looks good:
[jallspaw@server01 ~]$ ./pigz-2.1.6/pigz daemon.log.2
[jallspaw@server01 ~]$ gunzip ./daemon.log.2.gz
[jallspaw@server01 ~]$ gzip ./daemon.log.2
[jallspaw@server01 ~]$ ./pigz-2.1.6/unpigz ./daemon.log.2.gz
[jallspaw@server01 ~]$

By: David Gerard

David Gerard — Sat, 03 Apr 2010 10:55:20 +0000

So how’s the file format compatibility with gzip, each way?

By: Alex

Alex — Sat, 03 Apr 2010 09:26:43 +0000

It’d be interesting to see how long each method takes to gzip 16 files. I suspect for most of us the problem is not compressing one log file – but compressing the 1000′s generated every hour.

Alex

By: Alex madden

Alex madden — Sat, 03 Apr 2010 07:21:06 +0000

Also parallel bzip2

http://compression.ca/pbzip2/

Alex

By: Aaron Bush

Aaron Bush — Fri, 02 Apr 2010 22:08:51 +0000

Excellent boost. Makes me wonder if several of the GNU utils need a refresh for parallel opportunities?

By: allspaw

allspaw — Fri, 02 Apr 2010 22:03:53 +0000

For the 1.55 seconds, yeah the load average I think went above 1. The load average will definitely go up because of the number of simultaneous jobs running.

By: Denish Patel

Denish Patel — Fri, 02 Apr 2010 22:01:10 +0000

Seems nice improvement in total run time.

isn’t load average pretty high with pigz?

Denish