Propsed EDSK enhancements for unsupported disks
Simon Owen
I've spent a while looking at +3/CPC copy-protected disks, and in particular
the ones that are beyond what EDSK can currently represent. Rather than
create an entirely new disk format, it seems possible to extend the existing
format in a (reasonably) backwards compatible way. From the disks I've
examined so far, two things would need changing:
1) Increase Maximum EDSK Data Size
A perfect 250Kbps track written at 300rpm is 6250 bytes long. With a single
MFM sector on the track, there is 116 bytes for gap4a, 62 bytes overhead for
sector headers/gaps, and a minimum gap3 of 1 byte. This leaves
6250-179=6071 bytes for sector data, which is less than the 6144 bytes EDSK
stores (and probably why that size was chosen). However, once you shorten
gap4a (I've seen down to 50 bytes on original disks), and/or write past the
end of the track into the start of gap4a, you quite easily exceed the size
stored by EDSK. The +3 Coin Ops disk loader reads 6304 bytes from at least
one 8K track, appearing to need something that is beyond the stored 6144
bytes (thanks Woody and Luca!).
I propose extending the maximum size up to 8192 bytes, so an 8K sector can
be stored in its entirety. This ensures that everything returned by the
controller from an original disk is retained in the image, without making
any assumptions about how much is used by a given game. In a related
change, 16K sectors appear to be completely ignored by the current images,
with no data saved at all. Perhaps these should also have the same 8K data
cap? Or if John Elliot's proposed rate/encoding extensions are used for a
500Kbps track, perhaps the full 16K should be stored?
I realise that increasing the maximum data size could cause problems for
EDSK programs that blindly read the contents into a fixed-sized buffer. The
few emulators I tried had no problems, though though DskMan2.exe considered
them to be bad images (which strictly they are at present).
2) Store Multiple Versions Of Weak/Random Sectors
Speedlock uses partially-random sectors whose contents change in successive
reads. Only parts of the sector are variable however, with at least 2
flavours of randomness. The first appears to be caused by small variations
in sync position around write-splice points, where blocks of bytes have been
written in a separate pass from the original format. The data after these
points isn't always aligned to the same position within a byte, giving up to
8 variations. The second is less predictable, and could be explained by
writing patches using a weaker magnetic strength when writing the track.
In one Speedlock case (there are a number of variations), the first 256
bytes of a 512-byte sector must be a fixed 0xE5 value. It then checks for a
16-byte block a little later in the sector, which aren't a fixed value but
must all be the same (this is after a splice point, so the actual value can
change). In other games, it simply checks for any difference in the first
or last half of the sector.
Rather than attempt to interpret the changing region during disk image
creation (which could later be found to be wrong), it seems safer to store a
few copies of the data, as returned using multiple reads from an original
disk. This would only be needed for sectors with a data CRC error, where
the data is seen to change in successive reads. Error sectors with
unchanging content would require no special handling, since the single
sector data and status flags is enough.
I propose allowing multiple copies of the sector data to be stored in the
existing sector data block. The extra data would be part of the data length
in the EDSK sector header, so older programs will (or should!) safely
advance past it. To get the number of sector copies stored you simply
divide the stored size by the expected size going from the ID header size
code. When the sector is read by an emulator, a random copy from the stored
selection would be returned. I understand that existing emulators use a
hard-coded knowledge of specific games, changing specific bytes to pass the
copy-protection checks - this information belongs in the disk image instead.
The utility writing the images back to disk will be responsible for
analysing the random sectors, to determine how the differences need to be
represented. It's not possible to write a perfect pattern matching the
original (of course), but I've had good results in writing partial
pseudo-random areas to different points in a single sector, which seem
enough to keep Speedlock happy.
One minor issue is that 8K sectors will span the end of the writing point,
and the sync noise will likely trigger multiple copies of the sector to be
stored. If multiple copies are stored for the maximum 39 sectors, it will
add around 700K to the size of the image! This would still be needed to
ensure that random data wrapped round into the start of the track is
properly stored, but is a very unlikely case.
In the vast majority of cases, existing EDSK dumps need no changes.
Speedlock-protected images would ideally be redumped for random sector
copies, but if the specific checks are understood, it would also be possible
to add pseudo-random versions manually. Any game reading more than 6K from
a sector will definitely need redumping, to recover the missing data.
I realise that EDSK isn't an advanced format (like IPF/UDI/etc.), but it
closely matches what can be read and written on the PC. It would be good to
squeeze a little more out of it, and support a few more protected games.
Any comments anyone?
Si
(I won't have Net access next week, but will reply to anything ASAP)
John Elliott
: I realise that EDSK isn't an advanced format (like IPF/UDI/etc.), but it
: closely matches what can be read and written on the PC. It would be good to
: squeeze a little more out of it, and support a few more protected games.
: Any comments anyone?
There's another limitation of DSK and EDSK, though it probably hasn't shown
up in real-world dumps: You can have at most 29 sectors per track.
--
John Elliott
Luca
Great stuff Simon, I'm definitely positive on extending the EDSK format;
it only takes some little, non-traumatic changes to support those
problematic
loaders which cannot be dumped yet.
"Simon Owen" <nos...@simonowen.com> ha scritto nel messaggio
news:cbldk1t90i2fnpt6j...@4ax.com...
>
> 1) Increase Maximum EDSK Data Size
I always wondered why limit the stored data to 0x1800.
> 2) Store Multiple Versions Of Weak/Random Sectors
In RS we detected weak sectors and altered a couple of bytes at the
beginning
or the end randomly; I think this is what other emulators do too.
Either we store multiple copies as you suggest, or provide a way to mark
those
sectors and specify how to alter them when parsed by the emulator.
Concerning John's observation about the maximum number of 29 sector
descriptors
available in the track header, I don't think it represents an actual problem
(never
encountered a disk with more than 18 sectors) but we might extend the size
of the track
header to allocate more descriptors using one of the reserved bytes to
signal that.
Regards,
Luca
Simon Owen
wrote:
> There's another limitation of DSK and EDSK, though it probably hasn't shown
>up in real-world dumps: You can have at most 29 sectors per track.
Even if it's not currently needed, it may still be worth extending if any of
the other EDSK enhancements are accepted. Perhaps the existing 256-byte
track header would be considered a minimum size, with additional sector
headers simple added to the end of it? That way only disks with >29 sectors
wouldn't be backwards compatible.
Si
Simon Owen
> I'm definitely positive on extending the EDSK format;
> it only takes some little, non-traumatic changes to support those
> problematic loaders which cannot be dumped yet.
That's the hope/plan :-)
>In RS we detected weak sectors and altered a couple of bytes at the
>beginning or the end randomly; I think this is what other emulators do too.
The games Woody and I looked at would work if either the start or end were
changed, even if they made additional checks in the middle of the data. The
difficulty is that some areas must be constant too, so (as you'll already
know) you can't change both for every game.
A simpler solution would be to use a couple of bits to store whether the
start of end needed changing, but that's a very problem-specific. It seems
worth having a more general solution, just in case.
>Either we store multiple copies as you suggest, or provide a way to mark
>those sectors and specify how to alter them when parsed by the emulator.
Marking those sectors would be difficult though, as you'd have to understand
what about them is changing at the time the image is created (which could
later turn out to be wrong). In the Speedlock case, the common block after
a sync point can change, so it appears random but isn't.
Storing multiple copies ensures it's only recording what's seen, which is
always going to be 'correct'. Emulators would not have to do anything but
choose a copy to return. The more difficult analysis part would then fall
on the utility writing them back to real disks. If a problem is found with
the analysis, you'd just fix the utility - the original disk images are
still valid.
Ideally, you'd only want/need access to a physical disk once to create an
image of it. You should record as much raw information about it as you can,
with as few assumptions as possible.
>(never encountered a disk with more than 18 sectors)
I've never seen one in the wild either, though I suppose it's easy to create
a one containing 32x128-byte sectors without special tricks. The only other
real-world case I can think of is a PC disk protection that has > 100 id
headers on a single track (none with data fields). Not easy to create on
the PC, and not really much of a worry for EDSK either.
> we might extend the size of the track header to allocate more descriptors
> using one of the reserved bytes to signal that.
Would that also keep the sector count <= 29, so older utilties would see
only the first 29 sectors? If not, I suppose the sector count being > 29
would be enough to identify the extended track?
Si
John Elliott
: On Sat, 8 Oct 2005 16:32:38 +0200, "Luca" <ram...@nospam.bbk.org> wrote:
:>(never encountered a disk with more than 18 sectors)
: I've never seen one in the wild either, though I suppose it's easy to create
: a one containing 32x128-byte sectors without special tricks. The only other
: real-world case I can think of is a PC disk protection that has > 100 id
: headers on a single track (none with data fields). Not easy to create on
: the PC, and not really much of a worry for EDSK either.
I came across this while developing the LibDsk serial driver. I also wrote a
CP/M server which allows drives on CP/M boxes to be imaged by the LibDsk
tools, and when I was testing the CP/M 2 version it presented everything in
128-byte sectors. Consequently the disc I was testing with claimed to have
36 sectors; next thing I knew, I had a buffer overflow :-)
:> we might extend the size of the track header to allocate more descriptors
:> using one of the reserved bytes to signal that.
: Would that also keep the sector count <= 29, so older utilties would see
: only the first 29 sectors? If not, I suppose the sector count being > 29
: would be enough to identify the extended track?
The problem with setting the sector count to > 29 is that old utilities
might try to read the 30th sector header when they only loaded 29 sectors.
From that point of view, the best solution would be to leave the count as
29, and set a flag in the header.
The sector info for sectors 30, 31 etc. couldn't immediately follow the
Track-Info header, because the old utilities would expect to find sector 1
there. It would have to go after the first 29 sectors; my suggestion would
be to have a second Track-Info header with a slightly different magic number
such as "More-Info", and then the remaining sectors.
If there were more than 58 sectors (I suppose this might be possible at the
HD or ED data rates) then you'd have to have three headers in the linked
list, and so on.
Actually, I wonder if this would help with your other problem, of sectors
that have to be imaged multiple times? Read the track twice, as suggested,
and store the second pass in a linked More-Info block so it doesn't confuse
older utilities.
--
------------- http://www.seasip.demon.co.uk/index.html --------------------
John Elliott |BLOODNOK: "But why have you got such a long face?"
|SEAGOON: "Heavy dentures, Sir!" - The Goon Show
:-------------------------------------------------------------------------)
Woody
wrote:
>Either we store multiple copies as you suggest, or provide a way to mark
>those
>sectors and specify how to alter them when parsed by the emulator.
I'd prefer to use real data (there's a reasonably small limit of
possible shifted bit arrangements after the splice point, apparently)
than return fictitious sector data.
If the majority go for marking random sectors with 'alter points' then
I'd certainly help with modifying images too (+3 side only, sorry CPC
fans!), but this is just faking data and is probably unwise as well as
inaccurate[1].
I'm all for storing the full 8K of data for 8K sectors too. Not sure
who's going be brave enough to redump and distribute the
permission-denied CoinOp Hits image as a test case though ;-)
[1] Says the guy who spent many hours disassembling speedlock loaders
to come up with the best faked sector data possible!
Woody
wrote:
>The games Woody and I looked at would work if either the start or end were
>changed, even if they made additional checks in the middle of the data. The
>difficulty is that some areas must be constant too, so (as you'll already
>know) you can't change both for every game.
Remember changing the two central bytes of a disk sector was working
great for the vast majority of disks? That was having good results for
those disks that wanted valid data at the sector start, and for those
with valid data at the end, yet allowing both sector halves to return
random data if required.
I would've been happy sticking with that as a cool random data method.
At least until fooking Dragon Ninja came along and screwed that up
too! :(
Luca
> possible shifted bit arrangements after the splice point, apparently)
> than return fictitious sector data.
I'm for real data too, more accurate and even easier.
Luca
Kevin Thacker
> Hi,
Hi,
> I've spent a while looking at +3/CPC copy-protected disks, and in particular
> the ones that are beyond what EDSK can currently represent. Rather than
> create an entirely new disk format, it seems possible to extend the existing
> format in a (reasonably) backwards compatible way. From the disks I've
> examined so far, two things would need changing:
>
> 1) Increase Maximum EDSK Data Size
No problem here, as the specification allows the size in bytes to be
defined within the sector ID field.
It was only decided to store about 6k as this was, at the time,
considered to be the only used data.
I have no problem with updating the spec to support the full 8k.
> 2) Store Multiple Versions Of Weak/Random Sectors
Again, I have no problem here. I would like to see some data flag to
indicate the sector should be treated in this way as it would make it
easier for programs which write edsks back to real discs.
> I realise that EDSK isn't an advanced format (like IPF/UDI/etc.), but it
> closely matches what can be read and written on the PC. It would be good to
> squeeze a little more out of it, and support a few more protected games.
Again no problem here.
If others have no issues with these extensions then I will happily
update the spec at my website.
Kev
Woody
<ams...@aiind.upv.es> wrote:
>If others have no issues with these extensions then I will happily
>update the spec at my website.
I'm in favour of these changes :)
Luca
"Woody" <woody.6031769...@ntlworld.com> ha scritto nel messaggio
news:nd12l11fpfki1eknf...@4ax.com...
So do I :)
Luca
Bill H
Duncan Snowden
> Wait a minute.. the Spectrum hade a disk drive?
Yep, some of 'em did.
--
Duncan Snowden.
O Invalid stream, 18:1
Paul Dunn
> Wait a minute.. the Spectrum hade a disk drive?
Yes - mine also has a hard drive.
D.
nur...@gmx.de
> > 2) Store Multiple Versions Of Weak/Random Sectors
> Again, I have no problem here. I would like to see some data flag to
> indicate the sector should be treated in this way as it would make it
> easier for programs which write edsks back to real discs.
How is a program that writes edsks back to real discs supposed to handle
this case?
A normal PC floppy controller probably can not do that anyway, but maybe
special hardware (catweasel controller?) can write weak data? In this
case the program that writes back the image to real discs would need
instructions what to write rather than multiple examples of what the CPC
is expected to read afterwards, or else it would need to analyze the
given examples and guess where to write weak bits.
The idea of storing multiple versions of weak/random sectors sounds very
emulator friendly to me but not the best way to produce a physical 1:1
copy. However I can't come up with a better idea. So feel free to ignore
me. ;-)
Simon Owen
wrote:
>I have no problem with updating the spec to support the full 8k.
And the full 16K for size=7 too?
This also brings up the issue of how much to store for sectors that use
invalid values >= 8, as found in a handful of protections. I've run some
tests on 2 different PCs, with differing results:
The first PC (Intel D875PBZ mobo) did what I'd guessed, using only the
bottom 3 bits of the size value. So a size of 0xDD was treated as size=5,
with the controller returning only 4K of data.
The second PC (cheap SiS mobo) sent at least 16K of data before I stopped
it, with the track contents wrapping (with sync splice issues on wrapping).
I'll have to see if it sends even more, though I have vague memories of a
~28K DMA transfer limit _somewhere_. It seems unlikely it'd try to send all
128*2^221 bytes anyway!
I don't currently have a working +3/CPC to try it out for real, so I'm not
sure which one matches the behaviour of an original machine. It would
perhaps be safest to store the maximum sector size of 16K for any, though
the first machine above would either store just 4K or pad it to 16K (not
ideal). More tests needed, unless anyone knows the real behaviour already?
In the case of an FDC emulator, it should always offer up the required
length for specific sector sizes. If only 6K of an 8K sector is stored, it
will need to pad the remainder with the fill-byte (or zeros).
>> 2) Store Multiple Versions Of Weak/Random Sectors
>
>Again, I have no problem here. I would like to see some data flag to
>indicate the sector should be treated in this way as it would make it
>easier for programs which write edsks back to real discs.
Any thoughts on where this would go? The only place I could see (that
didn't rely on <= 16 sectors/track) would be one of the unused bits in a
status byte. Not really ideal, especially as I've run into a couple of
existing images with 'unused' bit 7 set in status 2 for no apparent reason!
Si
Simon Owen
wrote:
> my suggestion would be to have a second Track-Info header with a slightly
> different magic number such as "More-Info", and then the remaining sectors.
I guess this wouldn't be backwards compatible for images that needed the
extra header, but would at least cause a graceful failure when the unknown
signature is encountered?
> Actually, I wonder if this would help with your other problem, of sectors
> that have to be imaged multiple times? Read the track twice, as suggested,
> and store the second pass in a linked More-Info block so it doesn't confuse
> older utilities.
I imagined hiding the repeated data in the existing sector data would be
relatively safe. It would be an issue for programs that blindly return the
store length for sector reads, but they'd already be handling short 8K
sectors incorrectly anyway.
Do you have any additional details on where/how a More-Info block would be
linked in to the existing EDSK structure for the >29 sector and repeated
sector cases?
Si
Simon Owen
>How is a program that writes edsks back to real discs supposed to handle
>this case?
>
>A normal PC floppy controller probably can not do that anyway,
You can't write _real_ weak data with the PC controller, but you can do
interrupted writes to generate enough splice noise to give sync variations
between reads. Sector data from after the point will vary enough to keep
protections expecting data differences happy.
I won't be able to try it with a real Speedlock loader until I can get a new
PSU for my +3, but I can certain generate the fake random areas on demand in
my test program.
> but maybe special hardware (catweasel controller?) can write weak data?
I'm not sure the CatWeasel can do it either, but I might be wrong. Having
it as part of the image means it's ready for when a writing solution is
available - the SPS (formerly CAPS) is in the same position with its very
low-level IPF images too.
> or else it would need to analyze the given examples and guess where to
> write weak bits.
This is what I'm aiming for, with a comparison done between the different
copies to identify areas that are static, changing but predictable (from
bit-slipping) and random.
Reproducing the known (but different) Speedlock patterns disks shouldn't be
too much of a problem. I guess other non-working games could have slightly
different requirements.
>The idea of storing multiple versions of weak/random sectors sounds very
>emulator friendly to me
Having multiple copies also means you can tweak/improve the pattern analysis
in the writer program if something isn't being recognised correctly. The
disk image doesn't need redumping as you're simply intepreting the same data
in a different way.
> but not the best way to produce a physical 1:1 copy.
Starting from a high-level EDSK image will never give a true 1:1 copy, but
it's hopefully good enough to "work" :-)
Si
Woody
wrote:
>In the case of an FDC emulator, it should always offer up the required
>length for specific sector sizes. If only 6K of an 8K sector is stored, it
>will need to pad the remainder with the fill-byte (or zeros).
Been doing this from day 1. Some +3 titles require this behaviour.
John Elliott
Loading Simon Owen <nos...@simonowen.com>.SYS.....
:> my suggestion would be to have a second Track-Info header with a slightly
:> different magic number such as "More-Info", and then the remaining sectors.
: I guess this wouldn't be backwards compatible for images that needed the
: extra header, but would at least cause a graceful failure when the unknown
: signature is encountered?
: I imagined hiding the repeated data in the existing sector data would be
: relatively safe. It would be an issue for programs that blindly return the
: store length for sector reads, but they'd already be handling short 8K
: sectors incorrectly anyway.
: Do you have any additional details on where/how a More-Info block would be
: linked in to the existing EDSK structure for the >29 sector and repeated
: sector cases?
My idea was: Assume we want to image a track with 32 128-byte sectors.
Then the track would go:
Track-Info @ 256 bytes
Sectors 1...29 @ 128 bytes each
More-Info @ 256 bytes
Sectors 30...32 @ 128 bytes each
for a total size of 4608 (1200h) bytes, so set track length in the EDSK
header to 12h. The Track-Info header would say that there were 29 sectors
and have a flag to say "More-Info header present". Then the More-Info would
say there were 3 sectors. It is to be hoped that a downlevel utility would
use the EDSK header to work out the offset of each track in the file, rather
than by adding up the sector sizes in Track-Info; if it did, it would hit
the different magic number in the More-Info block and (one hopes) stop.
Then for a DSK with a multiply-imaged sector (say, sector 4), the scheme
would look like this:
Track-Info @ 256 bytes
Sectors 1...9 @ 512 bytes each
More-Info @ 256 bytes
Sector 4 again @ 512 bytes
Total track size: 5632 bytes (1600h), so set track length in EDSK header
to 16h.
That way, a downlevel utility would only see the first copy while a
More-Info aware utility would be able to choose between the two copies of
sector 4. The question of how to write back such a disc is a good point, and
one to which I have no answer; if it can be done on a standard 765, then
emulators would also have to detect the sequence of instructions that does
it, and save both copies in the DSK file they wrote.
My apologies if you find this suggestion over-engineered and/or baroque;
it would probably be simpler to break backwards compatibility, change the
EDSK magic string, and allow variable-size Track-Info headers. If so, I can
come up with a similar list of ambitious suggestions for that; last time, I
seem to recall proposing adding metadata blocks as an extra 'track' at the
end of the file.
PS: I don't know if you plan to support multiply-imaged 32k sectors, but bear
in mind that the maximum size of a track in an EDSK is FF00h :-)
--
--------------------------- ,@@. ,@@. TRYMON BEAR 1
John Elliott | o @@@@ @@@@ \\.....\_O_
CHAOS in a sig... | \/|> '||` '||` /o/ \I `
--------------------------- |\ JL JL \\ /\\
rpalmer
Ray Palmer
Simon Owen
wrote:
<snip>
> for a total size of 4608 (1200h) bytes, so set track length in the EDSK
>header to 12h.
This was the key bit I missed when I read your first post - using the space
at the end of the track data block (enlarging it if necessary).
> Then for a DSK with a multiply-imaged sector (say, sector 4), the scheme
>would look like this:
>
> Track-Info @ 256 bytes
> Sectors 1...9 @ 512 bytes each
> More-Info @ 256 bytes
> Sector 4 again @ 512 bytes
This does look good, though what about when there are duplicate sector IDs,
and one has an error? You'd need to know which one of them the duplicate
was for, so you couldn't just rely on the sector number.
Even more extreme, if you had >29 sectors and the extended block contains a
repeated sector ID (with both having CRC errors), you'd need repeated copies
for both. How would you know whether the 2nd repeated sector is a real
extra sector or a repeated copy for the CRC?
> The question of how to write back such a disc is a good point, and
> one to which I have no answer; if it can be done on a standard 765, then
> emulators would also have to detect the sequence of instructions that does
> it, and save both copies in the DSK file they wrote.
I guess FDC emulators would be safe in picking a random copy to return,
perhaps ideally making sure it doesn't pick the same version on subsequent
reads. When a computer emulators are using the FDC emulator, it will behave
as seen on a real machine.
Image writing utilities would need to know about the repeated copies, so
they can be examined for difference patterns, and to decide if it's possible
to write it. Additional logic may also be needed if the track doesn't
naturally fit, since EDSK lacks additional position/size information (and
the stored gap/size information isn't always reliable). It seems safest to
have as much raw information in the image as possible, and leave other
programs up to the interpretation of it.
DOS and W2K/XP should be able to write almost everything I've seen so far.
I'm not sure about the Linux support for interrupted writes, which would
exclude 8K sectors, CRC errors and weak data areas.
>it would probably be simpler to break backwards compatibility, change the
>EDSK magic string, and allow variable-size Track-Info headers.
I'm still torn about the whole idea of a new format, as EDSK seems so close
to doing what's needed already. With the proposed extensions (however
they're implemented) it should cover all existing disks, as well as all
"normal" real-world cases like your >29 128-byte sectors.
It would be possible to hand-craft real disks that can't be represented in
in the updated format, and it depends whether that's really a problem. I'd
be tempted to ignore those, even if the image dumping utility has to admit
defeat or resort to using Teledisk format for them!
>PS: I don't know if you plan to support multiply-imaged 32k sectors, but bear
>in mind that the maximum size of a track in an EDSK is FF00h :-)
Aren't 16K (size=7) sectors the largest legal ones? Even so, FF00h is a bit
limiting when even a few illegal size codes are used!
A common protection trick is to have 16x128=byte sectors on a track, using
CHRN values of 1,1,1,1 up to 15,15,15,15. If (as I suggested previously) a
full 16K is stored for each of the size >= 7 cases, it would overflow the
track space available.
One solution would be to determine the format size code needed to write the
track, and only store that amount for the oversized sectors. That way a
single 16K sector would be stored as 16K, but what appeared to be an
impossible 4x16K sectors would perhaps only store 1K (the actual value would
depend on which sectors have errors). I suppose there's still the issue of
multiple copies since the short sectors will all have CRC errors.
Nothing's ever simple eh? :-)
Si
Simon Owen
wrote:
>What is a catweasel controller ?
It's an add-on card for the PC/Amiga that includes a flexible floppy
controller: http://www.jschoenfeld.de/products/cwmk3_e.htm
Si
Simon Owen
>The second PC (cheap SiS mobo) sent at least 16K of data before I stopped
>it, with the track contents wrapping (with sync splice issues on wrapping).
>I'll have to see if it sends even more
I've retested this and it stops sending at 32K, continuing the wrapped track
data until that point. We still need to confirm what the real machine does
in this case. I'm still kinda hoping it only uses the bottom 3 bits of the
size!
Si
John Elliott
: repeated sector ID (with both having CRC errors), you'd need repeated copies
: for both. How would you know whether the 2nd repeated sector is a real
: extra sector or a repeated copy for the CRC?
If they've got the same sector ID, they're a repeated copy.
:> The question of how to write back such a disc is a good point, and
:> one to which I have no answer; if it can be done on a standard 765, then
:> emulators would also have to detect the sequence of instructions that does
:> it, and save both copies in the DSK file they wrote.
: I guess FDC emulators would be safe in picking a random copy to return,
: perhaps ideally making sure it doesn't pick the same version on subsequent
: reads. When a computer emulators are using the FDC emulator, it will behave
: as seen on a real machine.
You're talking about what an FDC emulator should do when it's emulating
reads. I'm talking about when it's emulating writes. If it's possible (for
example) to write a program that when run on a real CPC produces these
effects; then when that program's run on a CPC emulator, the FDC emulation
has to realise what's going on and insert duplicate sectors in its output
file accordingly. Which leads to another problem with EDSK: you can't
dynamically grow tracks beyond their original size without rewriting the
whole file.
Simon Owen
wrote:
> If they've got the same sector ID, they're a repeated copy.
Tomahawk's track 3 contains 9 sectors, all of which use a sector ID of zero.
None is weak/random, but if they were, the repeated sectors would need to be
tied to the correct original sectors.
It is an extreme example and is too far beyond what the suggested EDSK
extensions were for: to represent existing copy protections.
>If it's possible (for example) to write a program that when run on a real
>CPC produces these effects; then when that program's run on a CPC emulator,
>the FDC emulation has to realise what's going on and insert duplicate
> sectors in its output file accordingly.
The emulator would need to work at the track level, with cycle-accurate
timings for the head tracking the rotating disk. That part is isn't a
problem, but the simple/high-level EDSK is. You could have a fair stab at
guessing the physical positioning/spacing of sectors when loading an image,
but you'd lose much of the detail when saving it back.
The main TRS-80 emulator does most of what you're suggesting, using a DMK
disk image to preserve the raw track layout (including clock bits). You can
do interrupted writes and overwrite some of the sectors, with the rest
remaining as a real disk would. UDI is another format used by Spectrum
emulators, to hold a similar amount of detail.
All that said, none of those formats would really support what's needed for
the faked weak sectors. That would require another bitmap storing where
writes have begun/ended, for some artificial noise.
> Which leads to another problem with EDSK: you can't dynamically grow
> tracks beyond their original size without rewriting the whole file.
It does seem like EDSK works best as a read-only archive format. If you
want low-level writing, you'll definitely need a new format.
Si
Simon Owen
>1) Increase Maximum EDSK Data Size
It seems this part is pretty much agreed upon, with 8K and 16K sectors
allowing their true sizes to be stored. For existing images that hold 6K
for 8K sectors, the implemetation will already be padding the remainder with
the track filler byte when read.
Size codes above 7 are not strictly valid, but do exist on some disks.
CPDRead seemed to ignore their contents, with all the images I've examined
having no stored sector data. It would perhaps be useful to treat them as
size=7, storing up to 16K for each (if the available track space allows it).
As with the 6K issue above, any shortfall would be padded out anyway.
>2) Store Multiple Versions Of Weak/Random Sectors
It seems pretty clear that a completely new disk image format is needed to
work with disks at a low enough level to reproduce the various
copy-protection effects inside an emulated environment.
For now, I still think it's worth allowing multiple sector copies within the
data for a single EDSK sector. It's not a perfect solution covering every
possible case, but it would mean we can correctly image all the Speedlock
(and other) titles. It's also a trivial extension to to any existing FDC
emulator.
Only a small number of titles would need new images with the extra sector
data, with the rest completely unaffected by the change. The fact that some
older tools might not like the additional data doesn't seem like a major
problem, especially since they're already missing out on an important aspect
of the protection.
Identifying a multiple sector is easy: if the stored data size is larger
than the size code indicates, you've got more than 1 copy. To get the
number of copies, simply divide the stored size by the indicated sector
size. It's not too important which one gets returned for each read, as long
as it's not always the same.
Any other format needs (including >29 sectors) can probably wait for a new
format to be drawn up, as none of the commercial disks I've seen need it.
Time is running out for existing titles, so we need a solution now.
Kev? John? How about it?
Si
John Elliott
No objections here; though a sample EDSK in the appropriate extended
format might be a nice toy for utility authors to play with.
Simon Owen
wrote:
> a sample EDSK in the appropriate extended format might be a
> nice toy for utility authors to play with.
Here's an archive containing two +3 disks, each in old and new formats:
http://simonowen.com/misc/edsk_samples.zip
Robocop is protected with Speedlock, and sector 2 on track 0 is semi-random.
Without any emulator fiddles, the legacy version will fail to load. The new
version contains 3 copies read from a real disk.
World Cup Soccer Italia 90 has a number of 8K sectors, which were previously
truncated at 6144 bytes. The new version contains 3 copies of the full 8K
sector. A comparison between the sector copies shows the leading 0x18A1
bytes to be identical in some cases, which is only 1 byte more than the
length Hexagon reads from them. The legacy version did work in some
emulators, but that may be down to a existing protection workarounds.
I hoped to include a suitable CPC sample, but the only real disks I own with
random sectors are marked as distribution-denied :-/
Si
Simon Owen
>Here's an archive containing two +3 disks, each in old and new formats:
> http://simonowen.com/misc/edsk_samples.zip
<snip>
>I hoped to include a suitable CPC sample, but the only real disks I own with
>random sectors are marked as distribution-denied :-/
I've since found a CPC disk for World Cup Soccer Italia 90. I've updated
the archive so Robocop is the +3 version and World Cup Soccer is for CPC.
Hopefully that gives both camps a better chance to try it.
The new imaging tool itself is still under test, but is looking promising.
I've concentrated on the reading side at the moment, and will get back to
the more complicated writing back after that's done.
Si
Chris
<nos...@simonowen.com> writes
Have u guys seen http://www.softpres.org/ ?
Paul Dunn
Yes - they're generally considered a "very bad thing indeed", as they intend
to control the entire process. Only people who are chosen by them can make
images, and they have refused to release the file specification to the
general public.
They've tried badgering us before about our preservation methods (ie, "why
arent' you doing it our way?") and we've pretty much told them to sod off
and get their act together.
D.
Simon Owen
wrote:
>Have u guys seen http://www.softpres.org/ ?
Indeed - I've been in touch with Richard there quite recently.
On the positive side, a single SPS dump from a good disk should be all
that's needed to preserve it forever. Dumping at almost the flux level
means that anything stored on the disk will be preserved, regardless of copy
protection used. Their raw dump processing makes it possible to confirm
that the dump is good, and that the mastered disk hasn't been modified since
it was originally created.
On the negative side, you need more specialised hardware to create the dumps
(Amiga with 68020 or higher, HDD/CF/serial-link for up to 20MB of dumped
data, and a suitable drive connected to it for disk you're dumping). The
dump made is of no immediate use, and must be submitted to SPS for
processing before a final IPF disk image is created. Only original
commercial titles are supported, so you need another solution for homebrew
stuff (generally simple formats anyway). The IPF images are in a closed
format, and a binary access library is needed to use them - your emulator
needs to include this support, and some authors have reservations about it.
There is currently no support for writing the images back to real disks, so
they're currently for emulator use only. Writing is likely to be possible
in the future, with additional custom hardware.
It does do a very thorough job, but it isn't an option open to many people.
+3/CPC dumping on the PC doesn't pretend to be anywhere near 100% accurate,
but does a good enough job most of the time. Given the chance (I don't have
a suitable setup yet) I'd dump in both formats, with EDSK for my day to day
use, and IPF for peace of mind that it's safely preserved.
Si
Kevin Thacker
> Kev? John? How about it?
what you wanted.
Kev
John Elliott
I've updated LibDsk to 1.1.7; this release should be able to read DSK
files that use this extension.
<http://www.seasip.demon.co.uk/Unix/LibDsk/>
Si
wrote:
>The spec has been updated. Please check it to ensure I have described
>what you wanted.
Thanks for the update, which does cover what I need.
I've not confirmed the 3-bit vs N=7 situation when N>=8 on real +3/CPC
hardware, but I don't think it makes any real difference. I've yet to find
any software that actually reads data from them.
Si