NIB and DSK questions

[Egan Ford]

1. Is there a CLI tool (with source, preferably POSIX) that will
convert unprotected NIB to DSK? I've searched and have not found much
(with links that still exist). BTW, is there a NIB format description
floating around anywhere?

2. From time to time I encounter a DSK file that is not 143360 in
length. Is the assumption that the rest of the file is zeros?

Thanks.

[BLuRry]

1: I don't have one handy but if you don't mind java, I can make one for you. JACE has all the necessary routines to do the conversion for you so it would really not be too much of a burden.

2: I believe that most interpretations will just ignore the trailing part -- but I have seen some HDV/2mg images with the wrong extension. When in doubt, open it up and see if there is any sort of header (which would indicate 2mg). Secondly, check that it doesn't just have a lot of trailing zeros. If the file size is a multiple of 512 bytes then it might be a HDV image in disguise.

[Egan Ford]

On Feb 5, 10:44 am, BLuRry <brendan.rob...@gmail.com> wrote:
> 1: I don't have one handy but if you don't mind java, I can make one for you.  JACE has all the necessary routines to do the conversion for you so it would really not be too much of a burden.
> 2: I believe that most interpretations will just ignore the trailing part -- but I have seen some HDV/2mg images with the wrong extension.  When in doubt, open it up and see if there is any sort of header (which would indicate 2mg).  Secondly, check that it doesn't just have a lot of trailing zeros.  If the file size is a multiple of 512 bytes then it might be a HDV image in disguise.

1. Thanks! I do not mind java as long as Ubuntu does not. :-)
2. Got it. Thanks.

[BLuRry]

Ubuntu doesn't care for Oracle's shenanigans. But otherwise, Java works fine and OpenJDK seems to agree with JACE just fine. I'll update you when I have something to play with.

-Brendan

[BLuRry]

Ok, grab the new build of JACE from sourceforge. Since this is not the default startup class you'll have to use an alternate way of starting Java to get to it, namely:

java -cp jace.jar jace.ConvertDiskImage

I haven't tested it yet (out the door for a gathering with friends) but please let me know if it works for you!

Note: THIS VERSION HAS MOUSE SUPPORT!! w00t!!!

-Brendan

[Egan Ford]

Thanks. I think you need to change:

if (in.exists()) {

to:

if (!in.exists()) {

[BLuRry]

Man, you are right. Bad validation. Sorry for not testing it -- was literally out the door after writing the core of the tool. I've done some testing and kicked out some rather silly bugs. Seems to work fine now AFAIK and I've committed code and updated the sourceforge download to this fixed copy.

As I said, it basically takes advantage of the fact that JACE already performs conversion to and from NIB formats as part of emulating the disk drive, since the raw NIB format is required for proper disk drive emulation but to support writes, it has to convert the raw data back per track on the fly. Converting a DSK or PO to NIB file is simply a matter of creating the FloppyDisk object and grabbing the generated NIB byte array. Converting NIB to DSK is a little more interesting: I read in the disk as a NIB and then I change the disk file to the output file and monkey with the variables that tell JACE what kind of disk image it is. Once I loop through all tracks to flag them to be written, JACE converts the data and saves it as appropriate, respecting the desired sector ordering.

------------

ConvertDiskImage
----------------
Usage: java -cp jace.jar jace.ConvertDiskImage DISK_INPUT_NAME DISK_OUTPUT_NAME
where DISK_INPUT_NAME is the path of a valid disk image,
and DISK_OUTPUT_NAME is the path where you want to
save the converted disk image.
Supported input formats:
DSK (assumes DO), DO, PO, 2MG (140kb), NIB
Supported output formats:
DO/DSK, PO, NIB
-------

-B

[Egan Ford]

Command runs, but the output is wrong. NIB->DSK, the DSK file is the
same size as the NIB file (232960). It should be 143360.

Thanks again.

[BLuRry]

Odd... I'll take another look at it. Sorry for the mixup. DSK->NIB seems to work at least. :->

[schmidtd]

On Feb 5, 11:30 am, Egan Ford <dataj...@gmail.com> wrote:
> 1.  Is there a CLI tool (with source, preferably POSIX) that will
> convert unprotected NIB to DSK?  I've searched and have not found much
> (with links that still exist).  BTW, is there a NIB format description
> floating around anywhere?

Looks like BLuRry took care of you there. But what do you have an
unprotected .NIB for in the first place? I can imagine their use in
preserving a volume number. But if the .NIB had anything significant
in it (different headers, etc.) the .DSK you convert it to won't be of
much use.

A .NIB captures all of the nibbles on a track - including all those
headers, and data in whatever encoding scheme is in force (see Beyond
DOS or ProDOS for encoding descriptions). It's a de-facto standard
that started with the SST tool. Most emulators take a .DSK and expand
it into a .NIB anyway - which is why Brendan had such an easy time
making the conversion external.

> 2.  From time to time I encounter a DSK file that is not 143360 in
> length.  Is the assumption that the rest of the file is zeros?

No such assumption of any kind is made. If they're not 143360,
they're bad - or they're not an undecorated .dsk/.do/.po. Like
Brendan said, inspection usually will show you what the problem is.
I've seen random garbage at the end of too-long images, and if they're
short... zeroes are your only option.

[Egan Ford]

On Feb 6, 10:19 am, schmidtd <schmi...@my-deja.com> wrote:
> On Feb 5, 11:30 am, Egan Ford <dataj...@gmail.com> wrote:
>
> > 1.  Is there a CLI tool (with source, preferably POSIX) that will
> > convert unprotected NIB to DSK?  I've searched and have not found much
> > (with links that still exist).  BTW, is there a NIB format description
> > floating around anywhere?
>
> Looks like BLuRry took care of you there.  But what do you have an
> unprotected .NIB for in the first place?

Is it safe to assume that some .nibs floating around out there were
created just because that is the imaging format they used? Or should
the safe assumption be that .nib was used because nothing else would
work?

> No such assumption of any kind is made.  If they're not 143360,
> they're bad - or they're not an undecorated .dsk/.do/.po.  Like
> Brendan said, inspection usually will show you what the problem is.
> I've seen random garbage at the end of too-long images, and if they're
> short... zeroes are your only option.

Thanks.

[BLuRry]

You could look at the file size and see if it is a multiple of 35 -- anything not a multiple of 35 is a disk image of uneven size and should be thrown out. Take the file size and divide by 35, and that is the size of each track. Then make a ring buffer for each track, such that reading the end of the track jumps back to the start of the track. There's your disk emulation. :-D

The way that the computer reads a given track is to move the head until it finds sector headers for that track, and then it keeps reading until it finds the header for the given track/sector. After that it grabs the next chunk (the 6/2 encoded data) and then decodes the data. it is possible that your image has more junk inbetween the sectors -- or there is extra and/or repeated data at the end of each track.

If you scan for sector headers and inventory what is actually in the image, it might make more sense -- assuming that the disk uses the same sort of sector header format that Dos and Prodos use. Otherwise all bets are off and you have to roll your own deprotection scheme to translate it and gut the custom RWTS in favor of your own. Or buy something nice for Antoine and he'll probably crack it. :-D

-b

[schmidtd]

On Feb 6, 12:27 pm, Egan Ford <dataj...@gmail.com> wrote:
> On Feb 6, 10:19 am, schmidtd <schmi...@my-deja.com> wrote:
>
> > On Feb 5, 11:30 am, Egan Ford <dataj...@gmail.com> wrote:
>
> > > 1.  Is there a CLI tool (with source, preferably POSIX) that will
> > > convert unprotected NIB to DSK?  I've searched and have not found much
> > > (with links that still exist).  BTW, is there a NIB format description
> > > floating around anywhere?
>
> > Looks like BLuRry took care of you there.  But what do you have an
> > unprotected .NIB for in the first place?
>
> Is it safe to assume that some .nibs floating around out there were
> created just because that is the imaging format they used?  Or should
> the safe assumption be that .nib was used because nothing else would
> work?

It is safe to assume that .NIBs were used in order to capture some
light forms of copy protection, or a volume identifier in DOS.
Otherwise, the much simpler .dsk/.do/.po would have sufficed. No one
would have gone through the trouble if it weren't required for some
reason or another. Oh, just occurring to me now: unprotected 13-
sector disks can be imaged using a .NIB. That might have happened if
they didn't have Linards' 13-sector ADT yet.

[Antoine Vignau]

On 6 fév, 18:47, BLuRry <brendan.rob...@gmail.com> wrote:
> If you scan for sector headers and inventory what is actually in the image, it might make more sense -- assuming that the disk uses the same sort of sector header format that Dos and Prodos use.  Otherwise all bets are off and you have to roll your own deprotection scheme to translate it and gut the custom RWTS in favor of your own.  Or buy something nice for Antoine and he'll probably crack it.  :-D
>
> -b

Miam miam, I am hungry but... I still have dozens of disks to crack
(Micro League Baseball, old Apple titles, Captain Crunch stuff,
Mandragore, Hold-Up, Faial, Calendar crafter...) Pfewwww, too many
things, too few time (ahem)

Antoine

[Vladimir Ivanov]

On Sun, 5 Feb 2012, Egan Ford wrote:

> BTW, is there a NIB format description floating around anywhere?

Perhaps, but it's dead simple - 35 tracks of 6656 bytes (6.5 KB) each.
Each byte represents "nibble" as read from the disk controller. Since this
size is bigger than real track, it also contains leftovers that at least
in the case of unprotected software should be harmless (they can be even
partial sector). Each track should be regarded as circular and there is no
guarantee how it will start (like, in the middle of a sector).

HTH.

[BLuRry]

Absolutely correct! Each track is just a circular buffer, but there's no specific size it really has to be -- it's just a certain size by convention because that's how it was originally implemented. As for what goes in each track, it is really up to the format. The RWTS on the firmware is responsible for locating the header for a desired sector and translating the data, and that's where protection schemes start off (assuming they don't do anything funky like half or quarter tracks)

-B

[Linards Ticmanis]

On 02/06/2012 06:27 PM, Egan Ford wrote:

> Is it safe to assume that some .nibs floating around out there were
> created just because that is the imaging format they used?

Some yes, but not many. I think I've read somewhere that some very early
emulators (maybe up to 1993 or so?) used only NIB, both internally and
externally, so it is conceivable that some of the oldest disk images
were made in that format simply because it was the only one around at
that time.

After that time, almost all NIB images were made to preserve some (of
the simpler) copy protection schemes. Since NIB uses fixed track lengths
and doesn't distinguish between sync and non-sync bytes, it's very hard
to write back to real disks correctly, essentially requires manual
intervention.

As for volume numbers: DOS disks store them *inside* some block too, so
emulators that use DSK could use that byte to build the block headers
for the emulated drive to read, instead of defaulting to $FE. Doesn't
help with non-DOS disks though (that includes ProDOS disks), if the
volume number is important on those, NIB is the way to go. No idea why
nobody thought of putting in a one-byte volume number at the end of DSK
but since they didn't, things are as they are: DSK can't preserve volume
numbers on non-DOS disks, even when everything else is totally standard.

--
Linards Ticmanis

[D Finnigan]

Linards Ticmanis wrote:
>
> Doesn't
> help with non-DOS disks though (that includes ProDOS disks), if the
> volume number is important on those, NIB is the way to go.

Isn't it that ProDOS disks don't have a volume number?

[Mark Stock]

Here's a program for item 2. I used this make some very old partial DSK
files somewhat to mostly usable again.
To build and use it:

make zerofill
/zerofill < undersized.dsk > therightsize.dsk

http://hoop-la.ca/apple2/src/zerofill.c

#include <stdio.h>

#define DISK_LENGTH 143360

int main(int argc, char * argv[])
{
int c;
int i;
int not_end_of_file;

not_end_of_file = 1;
for (i = 0; i < DISK_LENGTH; i++) {
if (not_end_of_file) {
c = getchar();
if (c == EOF) {
c = not_end_of_file = 0;
}
}
putchar(c);
}

return 0;
}

[Egan Ford]

Thanks.

Here is my perl version if any are interested.

my $dsk = shift;

if(-s $dsk < 143360) {
open(DSK,">>$dsk") || die("Cannot append to '$dsk'");
print DSK chr(0) x (143360 - (-s $dsk));
close DSK;
}

While I am at it, here is my .po to .do, quick and dirty, needs error
checking:

#include <stdio.h>

typedef struct s {
unsigned char bytes[256];
} sector;

typedef struct t {
sector sectors[16];
} track;

typedef struct d {
track tracks[35];
} disk;


int main(int argc, char **argv)
{
disk po;
FILE *ifp, *ofp;
int i,j,k;
unsigned int xref[]={0x0,0xE,0xD,0xC,0xB,0xA,
0x9,0x8,0x7,0x6,0x5,0x4,0x3,0x2,0x1,0xF};

if ((ifp = fopen(argv[1], "rb")) == NULL) {
fprintf(stderr,"Cannot read: %s\n\n",argv[1]);
return 1;
}

if ((ofp = fopen(argv[2], "w")) == NULL) {
fprintf(stderr,"Cannot write: %s\n\n",argv[2]);
return 1;
}

fread(&po, 143360, 1, ifp);

for(i=0;i<35;i++)
for(j=0;j<16;j++)
for(k=0;k<256;k++)

fprintf(ofp,"%c",po.tracks[i].sectors[xref[j]].bytes[k]);

return 0;
}

[Antoine Vignau]

"long time, no C"
but is your interleaving table OK?
antoine

[Deckard]

On 5 fév, 17:30, Egan Ford <dataj...@gmail.com> wrote:
> 1.  Is there a CLI tool (with source, preferably POSIX) that will
> convert unprotected NIB to DSK?

If you have only few .NIB to convert and if you're sure they are not
protected, use Applewin:
.NIB in drive 1,
.dsk (created by CiderPress or a copy of an existing one) in drive 2,
and use a copy program such as Locksmith Fast Copy.

JM

[Egan Ford]

It should be. IOW, it appears to work.

[Egan Ford]

I have ~100 I would like to convert. This was for my
http://asciiexpress.net/diskserver project. When I harvested
http://virtualapple.org's Apple II collection a number of the images
were not .dsk. .do, .po, .bin where easy enough to work around as were
the occasional few bytes long and short. .nib and .2mg are the last
holdouts.

I would like to keep this a fully automated process for future
virtualapple.org DB updates, so for now, .nib remains elusive. And
given the feedback in this thread, may remain elusive.

Thanks.

[Vladimir Ivanov]

On Wed, 8 Feb 2012, Egan Ford wrote:

> I have ~100 I would like to convert.

I can stitch you small CLI tool for NIB -> DSK from my code.

> I would like to keep this a fully automated process for future
> virtualapple.org DB updates, so for now, .nib remains elusive. And given the
> feedback in this thread, may remain elusive.

The tool is easy. The result, though - probably elusive. If it's NIB it's
most likely kind of protected and/or not 16-sector. Both won't convert to
DSK. The remaining .NIBs that are unprotected 16-sector must be very few,
if existing at all.

[Egan Ford]

I wanted to avoid this, but perhaps I need to create a nibble writer.
Is there any Apple ASM code floating around that I can look at?

[David Schmidt]

Also elusive. :-) SST is out there. I was working with Gerard Putter a
couple of years ago to get .nib writing going inside the DOS ADT/A2V2
client (before grafting it onto ADTPro). The results worked after a
fashion, but it just wasn't deterministic enough to be reliable.

[Antoine Vignau]

Some nice code related to nibbles can be found in the source code of
ciderpress on sourceforge.
Some nice (and in French) SST information on Deckard's site at
boutillon.free.fr

D5AA96

[Linards Ticmanis]

On 02/08/2012 06:40 PM, Egan Ford wrote:

> I wanted to avoid this, but perhaps I need to create a nibble writer.
> Is there any Apple ASM code floating around that I can look at?

NIB depends on an emulated Disk II controller that always reads whole
bytes at a time. It has too little information to work on a real Disk II
(or other Apple 5.25") controller that reads data bit by bit, assembling
them to bytes with the help of those special, longer "sync" bytes.

Also, the tracks of (most) NIBs are longer than real floppy tracks so
you have to know what to truncate, a problem for which there is probably
no fully automatic solution.

These two problems prevent a general solution.

Once you manage to throw out the garbage/duplicate data and to guess
correctly which bytes are "sync", writing it back to a real disk is
eminently possible with a fairly short assembler loop construct. I did
it once and used a mixture of assembler and C (cc65 compiled) to write
the disk. I can send you the code if you're interested, I think it's
originally from "Beyond Apple DOS" or some similar book.

--
Linards Ticmanis

[Egan Ford]

Please do. Thanks.

[BLuRry]

If you want to see how a DSK gets turned into a NIB, here's the source I wrote/adopted from information largely obtained by reading Beneath Apple Dos

http://java-ace.svn.sourceforge.net/viewvc/java-ace/jace/src/jace/hardware/FloppyDisk.java?revision=169&view=markup

The nibbilize routine starts at line 106 and it writes out a whole NIB image of a DSK.

Converting back is something I do one track at a time using scheduled writes (so that way when there are many updates to the same track, I just write out the track at once). The process converts a whole track back to raw binary for the DSK image, in the desired sector order. This process starts at line 241. Notice that I had to change my scan logic at line 268 to work around a potential dos 3.3 bug that munges header byte signatures!

-Brendan

[Linards Ticmanis]

ProDOS doesn't *do* anything with volume numbers, and it never displays
them, that is true. But the volume number is part of every floppy disk
block header, on 5.25" disks at least. Since the decision was made by
Apple that ProDOS should use the same low level format for 5.25" disks
as DOS 3.3, the now-useless volume number field in the block headers had
to stay. I think the standard formatting programs for ProDOS, such as
FILER and the later Apple Disk Utilities, always use #1 as the number
they put in there, but don't quote me on it. In DOS the standard value
is #254 but can be changed of course.

--
Linards Ticmanis

[David Schmidt]

My first reaction - "it's been tried, you can't do it."

After sleeping on it, my reaction is now - "that's probably why you
should try."

The code in ADTPro that does the nibble track analysis, finds the gap,
etc. is here:
http://adtpro.cvs.sourceforge.net/viewvc/adtpro/adtpro/org/adtpro/NibbleBufferMap.java?revision=1.9&view=markup

You'll find a number of files starting with Nibble in that directory
that do the various bits of processing. Gerard Putter's A2V2 (an ADT
derivative) can do the nibble reading; that code is also in the most
recent CVS snapshot here:
http://adtpro.cvs.sourceforge.net/viewvc/adtpro/adtpro/client/src/dos/adt.s?revision=1.18&view=markup

The only reliable nibble reading is in A2V2, though, not my code. There
isn't any nibble write in either code base. See:
http://www.virtualii.com/A2V2Help/A2V2Help.html

[Michael J. Mahon]

The very earliest versions of popular bit copiers were pure nibble
copiers--like Locksmith 1.0, Nibbles Away, etc.

These programs read each track into a large buffer (more than one
revolution), then searched for the large gap (usually the end of
the track) and wrote the data between the two large gaps to the
target disk, assuming that the nibbles between epilogs and prologs
were self-sync nibbles.

Since a .nib image represents track data without sync identification,
just as early nibble copiers, it has the same information as those
copiers had. This data can be written to a disk with essentially the
same result as the early nibble copiers.

So, assuming that the .nib contained enough information to successfully
run the program, the nibble-written disk should also be usable.

Of course, if the protection scheme used more information than was
captured in the .nib, the disk would be detected as a copy, but
that's just the way it is...

There is one complication that must be dealt with in writing the
nibbles to a disk: the speed of the target disk must be slow
enough to fit all the relevant nibbles, and, if a rotation is
somewhat longer than the track data, provision must be made for
padding the nibbles with something that will not cause problems.

Typically, writing a lot of additional sync nibbles before writing
the track data will do the job.

-michael

NadaNet 3.1 for Apple II parallel computing!
Home page: http://home.comcast.net/~mjmahon/

"The wastebasket is our most important design
tool--and it's seriously underused."