A Plague Tale Innocence Update V1 05-CODEX
Canumil Flowers
For CODEX releases I was able to use 2016-08-15_rar540.exe no problem. But now when they're gone some major releases show up from other groups. For FairLight I haven't found a RAR version that works. Anybody know of a version that is compatible with their releases? Are they using something as ancient as themselves? ;)
As for Razor 1911, having the same type of problems rebuilding the release as with FairLight. But maybe I just haven't selected enough RAR versions. Here is a more recent release I'm having problems with:
I'm not entirely sure, but I feel like the problem has something to do with the volume size. I.e. a similar case to the packing method depending on the volume size. Although in this case it's only a single file, so it's not quite that.
There is some logic in rar that decides on when to use the "Extended time field", and when that field is present the file header is a bit larger, leaving less room for compressed data with the fixed volume sizes. So some of the compressed bytes that should go at the end of volume A end up at the start of volume B, and sometimes this make a difference for the final compressed output it seems.
However pyrescene uses a seemingly random value for the volume size, 189417906b in this case, so even if the correct timestamp is set on the file, it still won't be a match even if it uses the right rar version and -mt argument. And then there seems to be yet another difference that I can't explain, because after forcing pyrescene to use the correct volume size it still doesn't detect the correct rar version from the sample file, so perhaps the actual size of the input file also plays a role? Probably because LHD_LARGE doesn't get set, which again also affects the size of the headers by omitting some fields (leaving more room for compressed data).
Regardless, I haven't had to mess around with modified dates for e.g. CODEX releases when recreating them with pyReScene in the past. Is that really affecting the resulting checksums? Or maybe it was something else in the header that was a bit too large and using touch you just reset it sort of? The date doesn't really matter? And the PowerShell command does not do "enough" here?
Anyway, after having changed the modify date, I ran your command above and it happily created the volumes, but for me all the checksums were wrong still. Maybe I misunderstood you, weren't you able to reacreate the proper files manually above? I still failed at least. :)
It's strange that it fails for you, perhaps something else is still not quite right. For me it recreated all .rar files exactly (correct checksums) with the above command, after setting the correct date.
I don't know why it's the date that seems to trigger this, I have observed this with other release too. You can also try to manually control which timestamps get included and with what precision using the -ts[N] switch(es).
I noticed that the checksum of the first volume (rzr-aplaguetalerequiem1300.rar) turned out the same both times, so the date(s?) don't seem to be affecting the end result. I'm getting 7aae9836, but it's supposed to be 3fcff4ef.
However, I did generate an .srr file from the latest attempt now and compared to the one from srrDB. There are a lot of differences, like mine doesn't have the stored .nfo or .sfv files and the application name in the beginning is "pyReScene 0.7", not "pyReScene Auto 0.7" like the real one has.
Exactly right :-). It looks like you're really close, the only remaining difference is indeed that ATTR, everything else is the same now and you can even see that the date-time is picked up correctly, you have both the FTIME and the modification time entries matching (so the date and time do end up in the archive, and so they do matter). The checksums will match once all the header data is the same.
Good question :p. I had to trace this a bit, but with a bit of guessing and reasoning I think these are the raw "file attribute" values that windows uses for the file. I.e. looking at the extraction code, near the end of CmdExtract::ExtractCurrentFile(), you can see how Arc.FileHead.FileAttr is used in a call to SetFileAttr(), which on windows is implemented using SetFileAttributes(). So the ATTR should correspond to the windows file attributes.
Sparse file. A file that is a sparse file. A sparse file has an attribute that causes the I/O subsystem to allocate only meaningful (nonzero) data. Nonzero data is allocated on disk, and non-meaningful data (large strings of data composed of zeros) is not.
The documentation for the function that controls this bit mentions that at least in the past (Windows Server 2003 and Windows XP) The only way to clear this attribute is to overwrite the file (for example, by calling the CreateFile function with the CREATE_ALWAYS flag). This suggest that you may try copying the file as the easiest way to clear this bit, and a quick test seems to confirm this:
That did it! Copying the 50GB file cleared that attribute as theorized. That of course meant a new create date, so I ran the PowerShell command for that again and double checked that the sparse attribute had not returned and it hadn't, so both dates were in the past and the sparse file attribute was non-present.
The reason that sparse file is set on the file I think comes from how I got the file, which is from qBittorrent, which in turn is based on libtorrent. Apparently sparse file is by default on to (I guess) save on actual writes to the drive, since large strings of zeroes are actually not written? However "Size" and "Size on disk" are both still the full number of bytes (51 229 491 200) for both the sparse, downloaded file and the non-sparse, copied file, so maybe there was actually nothing to be "sparse with" on this particular file, so it had to write the whole thing. Only the attribute is set, without any actual effect on the file itself.
There doesn't seem to be an easy way to disable this behavior in qBittorrent. And the concept (inherent to NTFS) is maybe not bad to just have anyways. Why write something you don't have to, until you actually need to? Just unnecessary wear on the drive. Although practically you're not going to wear out your SSD before it becomes obsolete anyway, even if you do download some large .iso:s now and then.
You CAN enable preallocation though where it starts by writing zeroes to the whole area where the file is going to be. This disables sparse files also. This could be somewhat useful for hard drives to minimize fragmentation, but is completely irrelevant for SSDs, which purposely fragment files for both parallelized performance gains and wear leveling reasons anyway. So if you preallocate it's not going write the data to the SSD any differently than how it would have otherwise. You're just stuck waiting for it to write the empty file first. In fact, it's gonna mean more wear, since it first writes zeroes and then replaces some of them with ones.
But modification time is there, so that needs to match. The CODEX release .srr had none of that stuff, so I guess they did not use to store the .iso file modification time in their .rar files, whereas Razor 1911 do? And that's why those lines appear in the .srr file maybe? Maybe if even only one of them is there in the archive, then all four lines are included in the .srr or something?
It seems the default behavior in WinRAR is "-tsm4 -tsc0 -tsa0", which means full NTFS precision on the modification time and the other ones are omitted. I.e. if you don't specify anything this is what you get. Maybe CODEX went out of their way to not include anything.
when it creates its sample, it inadvertently also loses the sparse file attribute on its sample copy. So the resulting checksums check out and it starts the process. I then also think that WinRAR - at least when compressing data - does not (or can't) consider sparse files, so the sparsity is then again lost in the actual compression process. I even tried to extract the second release from the files I had generated before and the resulting .iso was not sparse, even though the .iso I had used to create the rars was.
The reason for this is that pyReScene works differently, it only checks for the actual compressed data to match, and doesn't need the rest of the RAR file (headers) to still be able to recreate the release. How this works is that it goes through all the settings, compressing the sample and checking the FILE_CRC header to match. Once it finds a match it will compress the (whole) file and then perform a 'merge' of the .srr and .rar files by copying the header information from the .srr and the compressed data from the .rar. You can see it create these intermediate rar files in the temporary folder.
Because it doesn't use the headers from the .rar it doesn't matter that the ATTR is wrong or the timestamp is wrong. It only matters in edge cases like this, where the result of compression (apparently) depends on the exact size of the data that is getting compressed. In such cases pyReScene will never find a "good" rar version. In theory it should be so that if you set the correct timestamps and ATTR and force the correct volume size in main.py, pyReScene should be able to find a "good" rar and reconstruct the release. A specific volume size can be forced by replacing the self.split by a hard coded volume size like "-v250000000b" in function arglist().
7fc3f7cf58