Torrentzipped
Giuseppina Worster
Be careful, when you see a ZIP archive with torrentzip's typical comment, it means that that ZIP was torrentzipped at least once, but you can't say if the ZIP is torrentzipped now. The only way to know if that ZIP is torrentzipped is torrentzipping it: if the torrentzipping process is skipped, the file was already torrentzipped.
Another interesting fact is that all files when recompressed have their "last modified" timestamp set to 24 December 1996 at 11:32 PM. Some old games in torrents like eXoDOS and Win3xO rely on file timestamps as copy protection measures and because of that these torrents are not torrentzipped.
As of version 2.8.0, a new compression library was added, so it reads many more compression types.Uses code page 437 (IBM) for the default ZIP filenames, and US Unicode if characters are not found in code page 437.Fixed a bug that was unnecessarily adding a ZIP64 central directory header to some ZIP64 files.
As of version 2.9, the cancel button was fixed. Added a GIF to the drop box to know it's working. Changed the tmp filename to __.tztmp. If you drop in any files named __.tztmp, these will be deleted, as they are temporary files that should not exist outside of torrentzip running (they can be left behind if Torrentzip.NET crashes out, so it cleans up after itself).
Notes:
See above 'General format of a torrentzipped .zip file with n files' for SOCD & EOCD
The TorrentZipped Files Comments
The .ZIP file comments in the End of Central directory is used to check the validity of the torrentzipped file.
The comment must be formatted as the 22 bytes of TORRENTZIPPED-XXXXXXXX
The XXXXXXXX is the CRC32 of the central directory records stored as hexadecimal upper case text.
(the CRC32 of the bytes in the file between SOCD & EOCD)
This comment ensures that if any change is made to the files within the zip this checksum will no longer match the byte data in the central directory, and in this way we can check the validity of a torrentzip file.
The set1\ entry should be removed, as it is implied by the set1\test1.rom file. The set2\ entry should be kept to create the empty directory, as removing it would completely remove the set2 directory.
Another test that could be performed is checking for repeat file entries inside the zip, most zip programs have a hard time handling this and will just ignore this repeat giving the user no way of knowing there is a repeat filename problem. So it would fix another possible inconsistency if torrentzip scanning at least warned about repeat filename being found inside a zip.
I've used ClrMamePro for about 2 decades, and just started using RomVault, on and off and REALLY like the simplicity of RomVault. I've used RomVault for only about 2 years or so, and there is alot of quirks. I'm trying to build the Retro/Old DOS Collections and find it impossible to preserve empty directories or timestamps based on the DATs. RomVault can disable torrentzip, but no way of restoring timestamps to previously torrentzipped files from the DAT-timestamps. In theory a time-stamp-respecting torrentzipped program would still be Torrentzippped friendly if the dats contained timestamps. Preservationist would frown upon losing timestamps, I guess. I also tried carefully crafting ClrMamePro to treat archives/chds as files, and RomVault doesn't seems to work at all. Some DOS games have strange "protection" in them that checks file dates, or empty directories. Are there any solutions to such issues?
I'm also wondering what method do you use to store ROMs, I'm thinking of getting a used server with ALOT of memory and run TruNAS OpenZFS w/ Dedupe enabled (I believe it is block-based Dedupe). Has anyone tried Dedupe on the FileSystem-level, rather than the DAT-level. How efficient is the Dedupe? Has anyone tried the new preview "Fast Dedupe" of TrueNAS?
Yes, but I don't personally. Using FS level compression like zstd is more common. Some people tack on dedupe on top of that if you have the resources to do so. Not sure how efficient it is. I use RomVault in a Windows 11 VM sitting on top of my unraid array with an XFS filesystem. I'm in the process of compressing a lot of archives to zstd zips.
Yeah zstd had threading & large dictionaries. I think zstd zips do NOT support solid archives. I wish they'd update the zip specs to support solid zips. WinRAR 7.0 now support 64GB solid dictionary sizes. RAR also has the blake2sp crypto secure hash as an option since 5.0. I also wish the Zip specs gets updated with an option to add a cryptographically secure HASH to its extra fields, in addition to the collision-prone CRC-32. I remember a decade or so ago, a group had created alot of broken roms with same size & crc-32.
I'm referring to deduplicating already compressed archives. As ALOT of emulator sources from identical sources, mainly MAME/No-Intro/Redump. In theory, eg, storing a torrentzipped merged, non-merged, AND split set on a single ZFS Deduped Pool would only occupy about the same amount of space as a single merged set, as duplicate blocks are stored as pointers.
What this does is make sure the CRCs of good ROMs align with those in the torrent. So even if you have a set of ROMs from a few months/years ago, you'll only get the updated ROMs. That way instead of downloading 11+GB of ROMs every time, you only need a few hundred MB, get up to date quickly, and often seed way more than you leech (my ratio sits about about 5:1 seed:leech, which means I always get full speed from the torrents - updates between stable releases take no more than an hour or so usually).
My friend, it is in the documentation (that would be the file labelled "README" at the top of the zip file containing trrntzip.exe). As the wise swami once said: "By all means pray to the gods, but first RTFM my child". :)
Then watch it fly! It will take each ROM zipfile, decompress it, and then add the ROMs back in in alphabetical order with best compression (you might find you save a few MB in the process) to a new zipfile, replacing the old. It uses the same compression library (zlib) under all OSes, so it creates identical files no matter if your on Windows/Linux/Mac/whatever.
When complete, you will have a fresh folder full of ROMs that match the CRCs of the set held on the torrent site. Best of all, it works on anything. If you've got ROMs for other systems in .zip format, always TorrentZip them first before joining torrent trackers, and you'll save yourself heaps of bandwidth.
"Braindead"? A bit harsh, don't you think? Are you suggesting it is the job of P2P systems to peek inside each and every binary file and try to decode the contents, matching the unziped CRC/SHA with that of the other end's, in realtime?
I think the scope you're proposing there is a bit much (not to mention the resource requirements it would add to the peers). Additionally, P2P systems like torrent don't care about things like "files". All they care about are bytestreams. Whether you have 10x 1MB files or 1x 10MB files, it's all the same. Just a stream of bytes. If that stream of bytes are in two different orders, then they are deemed different. Again, I think that's far from "braindead".
Some common-sense preparation work by the end user in the form of torrentzip is more than reasonable, if you ask me. Particularly considering the whole system from P2P software to torrentzip is all free of cost.
Once I run this ZipTorrent program, what next? Just point my torrent program to my roms folder and away I go or? I honestly can't see what this thing does and/or how it associates with your roms and your torrent program.
Why do this? Well, it's a good way to check that data isn't corrupt. I send you a file, and it has a CRC of 12345678. You get the file, do a CRC and see that your's is 1234569, and we see that the CRCs are different, we know that either the file corrupted in transit (bad link), or someone's been naughty and tried to hack it midstream (maybe add a trojan or virus).
It will have a different CRC. If you join a torrent site, they are considered different files (despite being named the same, and being the same size). Why? Well again, it's trivial to make something called "game.zip", whack a virus in it, and "pad" it to be the same size as a normal "game.zip". So it's a safety measure.
So if two people on opposite sides of the world have MAME ROMs, but the contents of zips where added by different people at different times, they both run torrentzip on their roms, and "game.zip" now looks like:
It will have the same CRC for both users. Now when they connect to the torrent tracker, the torrent system will see these files with the same CRCs and say "hey, these guys have got the right data, don't overwrite it with a new copy". It means they don't have to download all games (only the ones that changed within the MAME release from the version they had to the version they are currently downloading), and it means they can start seeding their good/up-to-date ROMs straight away.
Technically speaking, you should only have to run TorrentZip once ever - the first time you join a torrent tracker (assuming the torrent uploader also uses it, which most do - in fact it's mandatory on PleasureDome). From there on, all ROM updates you download would have been torrentzipped by the torrent uploader (and thus all the seeders/leechers get identical versions). From there you should happily be in sync. You can if you like run it once every now and then as a sanity check, but again if the torrent uploaders are sensible people (which most are), then once at your first connect is fine.
Not running TorrentZip is not a sin. All it means is you're torrent program will misinterpret quite a number of ROMs as being "incorrect" due to their mismatching CRC, and you'll re-download a lot of data that you simply didn't need to.