[vim/vim] xxd: incredible bad performance (#8090)
Bignaux Ronan
I'm using xxd to write big image and test regression on a few software (pfsshell & hdl-dump).
The image of 8GB is created with this script https://github.com/bignaux/pfsshell/blob/mkpart/tests/pfsshelltest.tcl .
Basicly, it's just few APA partition and zeros.
$ time hexdump -C test.img > test.hex
real 0m20,809s
user 0m16,284s
sys 0m4,502s
$ time xxd -a test.img test.xxd
real 20m7,453s
user 19m53,414s
sys 0m9,422s
So here, using hexdump take 20 seconds, using xxd for same things (but the two are incompatibles), it took 20minutes. I think i can qualify that of issue.
To Reproduce
wget https://gist.githubusercontent.com/bignaux/895e658101e7f26bc3ef1238077661ec/raw/5140a62e292df5d07cd0567ed8179033c2cebb42/test.hex
perform a revert xxd (fast):
time xxd -r test.hex test.img
then you can xxd -a test.img test.xxd
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, copy/paste the text or add screenshots to help explain your problem.
versions:
xxd V1.10 27oct98 by Juergen Weigert
vim-8.2.1522
nixos 20.09
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
paul-ollis
I have done some playing around and concluded that to 'fix' this would require changing the code to avoid formatting each line until it is determined that the line should be written to the output. That change would make xxd about 2 time slower than hexdump for this particular case.
My personal view this use-case is too special to warrant changing xxd, but I am willing to make the changes if others feel that we should, in order to be able to close this issue.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Jared Johnstone
Running into slowness problems with reverse mode (-r) using xxd for a niche case.
I'm trying to send a bitlockered Windows 10 NVMe block device to another machine over ssh with a 1GBPS link between them. Naturally the bitlocker partition is incompressible and as such using gzip pipes anywhere in the chain or ssh -C (which is also gzip-powered) causes a major slowdown (>30-40MB/s) due to the overhead of trying to compress the incompressible data in flight and getting no returns.
But there's a lot of zeroed/trimmed unused space in various sections on the disk especially towards the end of the source NVMe drive where the space has never been used. So I went looking for a Zero-length encoding (Run-length encoding) solution which doesn't seem to readily exist as a Linux command for binary data and xxd -a provides this solution by skipping repeating chunks in its output. This output can also be achieved with hexdump, but xxd -a is faster when dealing with both binary data and repeating data in the same source.
This solution fits perfectly for Run-length encoding allowing me to send the binary dat can send the incompressible binary stream to a remote while still truncating repeating blocks of zeros throughout the disk. xxd -a is fast enough to saturate a 1GBPS link getting about 250MB/s in hex-dumping speed.
But xxd -r lets me down with a significantly slower conversion of the hexdumped data back to binary on the receiving side at a speed closer to ~30MB/s. This can be quickly demonstrated with pv:
xxd -a /dev/urandom | xxd -r | pv >/dev/null
37.5MiB 0:00:01 [37.6MiB/s]
I had a suspicion this may be the result of line-buffered processing for xxd -r and that may be true to an extent as changing -c to the maximum value of 256 seems to increase xxd -r's speed to about ~60MB/s. But xxd cannot exceed 256 characters per line by default.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.