Questions tagged [zstd]
Zstandard is a lossless data compression algorithm developed by Yann Collet. Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest) to 22 (slowest in compression speed, but best compression ratio).
20 questions
0
votes
2
answers
135
views
Why did my backup folder with large amounts of repeated data compress so poorly?
I have a folder with around seventy subfolders, each containing a few tarballs which are nightly backups of a few directories (the largest being /home) from an old Raspberry Pi. Each is a full backup; ...
- 261
1
vote
1
answer
592
views
`pzstd` = parallelized Zstandard, how to watch progress in 4TB large file/disk?
I am brand new to zstd/pzstd, trying out its features, compression, benchmarking it, and so on. (I run Linux Mint 22 Cinnamon.) This computer has 32 GB RAM.
The basic command appears to be working, ...
- 31.3k
1
vote
0
answers
219
views
Recoverably recompress gzip files into zstd, preserving original checksums?
I need to archive a lot of gzip-compressed data. The problem is that compared to zstd, gzip is wasteful, both in terms of ratio and CPU time required to decompress the data. Because of that, I want to ...
- 1,298
0
votes
1
answer
183
views
btrfs mount with compress fails with udisksctl but succeeds with mount?
$ sudo mkfs.btrfs -fL borgbackups /dev/vgxubuntu/borgbackups
$ udisksctl mount -o compress=ztsd:15 -b /dev/mapper/vgxubuntu-borgbackups
Error mounting /dev/dm-3: GDBus.Error:org.freedesktop.UDisks2....
- 156
4
votes
1
answer
2k
views
How do you ssh using zstd dynamic compression?
I want to set up my ssh tunnel to do dynamic compression so that I get maximum throughput for the bandwidth and CPU resources I have available on each end. Since gzip compression is built-in and not ...
- 41
2
votes
2
answers
6k
views
Decompress folder.tar.gz with files inside
I'm trying to decompress folder.tar.gz which was previously compressed with zstd, so I did:
unzstd folder.tar.gz.zst
Now if I try to decompress by using:
tar -xvf folder.tar.gz
It will decompress ...
- 273
1
vote
1
answer
1k
views
create initrd image compressed with zstd
I have initrd image compressed with xz. This is how I created it from image file initrd:
e2image -ar initrd - | xz -9 --check=crc32 > initrd.xz
now I need same image compressed using zstd ...
- 848
2
votes
1
answer
3k
views
How do I uncompress or unarchive the contents of initramfs img file in Arch Linux?
I have been using Arch Linux for a while and studying about the initramfs. I thought of looking into the contents of the file to get a clear idea of it. I googled for various ways to peek through the ...
- 408
0
votes
1
answer
1k
views
Recompress 7z archives to tar.zst on the fly
I have a bunch of 7z archives (containing directories and files) that I would like to recompress as tar.zst (which offers much better decompression speeds if / when I need to unarchive them).
I could ...
- 101
0
votes
1
answer
546
views
Using Debian Stretch to build 5.15.x kernel with zstd compression fails with incorrect parameters
I had to move from Jessie to Stretch as kernel 5.15.49 requires gcc 5.x version (Jessie had 4.9 version Stretch 6.x). I decided to try the ZSTD module compression option in 5.15.x. I ensured to apt-...
- 293
28
votes
3
answers
59k
views
zst compression not supported by apt/dpkg
I'm running Debian, namely:
# uname -A
Linux martlins2 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux
and for some time I see some errors telling that some parts of some ...
- 403
4
votes
1
answer
6k
views
How to tell if btrfs subvolumes are actually compressed or not
I have mounted Btrfs subvolumes (including /home) with the compress=no option in /etc/fstab. However, when I run btrfs inspect-internal dump-super -a <device_name> (both, on the running system, ...
- 331
0
votes
1
answer
625
views
tar creates an absolute sub directory within the compressed file
I would like to tar a folder using zstd. My folder contains subfolders a and b.
I am tar-ing using this:
tar --zstd -cvf /dest/path/to/output/archive.tar.zstd /full/path/to/src/folder
However, this ...
- 1,075
2
votes
0
answers
253
views
Is there an equivalent of zdiff for zstandard
Is there an equivalent of the gzip utility zdiff(1), but for zstandard?
I have not been able to find anything in the zstandard repository other than zstdgrep and zstdless.
- 659
9
votes
1
answer
2k
views
Is there a compression tool with an arbitrarily large dictionary?
I am looking for a compression tool with an arbitrarily large dictionary (and "block size"). Let me explain by way of examples.
First let us create 32MB random data and then concatenate it to itself ...
- 381
2
votes
2
answers
4k
views
How can I separate multiple files from single file caused by zstd -r folder -o output.zst?
I didn't read enough the manual and run the following command
$ zstd -r folder -o output.zst
The following command gave me back a single file called output
$ unzstd output.zst
The output file has ...
- 171
2
votes
0
answers
81
views
Avoid pipeline stalls from write caching to usb drive
I have the following process I use for backups:
btrfs send snapshots/home-2016-06-04 |
zstd --verbose -T4 - |
gpg --batch --passphrase-file /tmp/secret --compress-algo none --symmetric - >
/mnt/...
- 141
18
votes
2
answers
40k
views
compressing and decompressing dd image - zstd instead of gzip
Earlier I was using fsarchiver to create compressed partition image. Due to some weird behavior I am choosing to replace it with dd.
However, I like how fsarchiver compressed with zstd.
So, I ...
- 3,134
3
votes
1
answer
2k
views
Is zstd for zram actually available in Linux 4.15?
I found zstd in /drivers/block/zram/zcomp.c, but I can't find anything zstd-related in /crypto.
So is zstd for zram actually available in Linux 4.15 or not?
- 1,023
17
votes
3
answers
16k
views
How to set a non default zstd compression level at btrfs filesystem defragment?
# btrfs filesystem defragment -r -v -czstd:15 /
ERROR: unknown compression type zstd:15
# btrfs filesystem defragment -r -v -czstd_15 /
ERROR: unknown compression type zstd_15
# btrfs filesystem ...
- 5,124