DVDs created with too large files
Gary Houston
to write backups to DVD. This generally works. However I encountered
a problem when one of the files was 2351679431 bytes in size: the disk
was written with no errors reported, but on testing proved to be unreadable
to linux 2.4.21. On mounting the disk a warning was reported:
Warning: defective CD-ROM. Enabling "cruft" mount option.
The large file now appeared to be only 2869191 bytes and is truncated
on reading.
Examining the Linux sources shows the following comment in fs/isofs/inode.c:
/*
* The ISO-9660 filesystem only stores 32 bits for file size.
* mkisofs handles files up to 2GB-2 = 2147483646 = 0x7FFFFFFE bytes
* in size. This is according to the large file summit paper from 1996.
* WARNING: ISO-9660 filesystems > 1 GB and even > 2 GB are fully
* legal. Do not prevent to use DVD's schi...@fokus.gmd.de
*/
if ((inode->i_size < 0 || inode->i_size > 0x7FFFFFFE) &&
inode->i_sb->u.isofs_sb.s_cruft == 'n') {
printk(KERN_WARNING "Warning: defective CD-ROM. "
"Enabling \"cruft\" mount option.\n");
inode->i_sb->u.isofs_sb.s_cruft = 'y';
}
I think this is implying that files > 2GB are legal, but can not be
handled by the Linux kernel. The comment also suggests that mkisofs
won't handle them. However it doesn't (at least for this particular
version of mkisofs) produce any warning or error when the disk is
created.
Something to beware of, since it may cause confusion or lost data.
Gary
--
To UNSUBSCRIBE, email to cdwrite...@other.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@other.debian.org
Lourens Veen
> At 5:27 PM +0000 11/2/03, Gary Houston wrote:
> >I have been using dvd+rw-tools (5.13.4.7.4) and mkisofs
> > (cdrtools 1.11a29) to write backups to DVD. This generally
> > works. However I encountered a problem when one of the files
> > was 2351679431 bytes in size: the disk was written with no
> > errors reported, but on testing proved to be unreadable to
> > linux 2.4.21. On mounting the disk a warning was reported:
> >
> >Warning: defective CD-ROM. Enabling "cruft" mount option.
>
> "Interchange Level 3 ISO-9660 volumes are not handled by this OS"
> would be more accurate.
>
> >=09/*
> >=09 * The ISO-9660 filesystem only stores 32 bits for file size.
>
> But the ISO-9660 _standard_ stores 32 bits for the size of each
> _extent_ and there can be a virtually unlimited number of
> _extents_ in a single file at ISO-9660 Interchange Level 3.
But can a file span multiple extents? The way I read the comment=20
Gary quoted, it's legal to have an image that is over 2GB in size,=20
as long as each file inside that image is no larger than 2GB. I=20
haven't actually read the spec though.
But then the whole comment seems odd. It looks to me like the=20
WARNING was added later, and written by J=F6rg. That seems to me like=20
the only logical way to explain why the comment says it's legal,=20
but the code claims it's illegal to have files that are more than=20
2GB in size.
Lastly, at the top of the file, there is
* 1998 Eric Lammerts - ISO 9660 Level 3
so apparently at least someone looked at ISO Level 3 support. I'd=20
say send a message to linux-kernel and see what they say about=20
it...
Lourens
--=20
GPG public key: http://home.student.utwente.nl/l.e.veen/lourens.key
ljknews
>On Sun 2 November 2003 19:03, ljknews wrote:
>> At 5:27 PM +0000 11/2/03, Gary Houston wrote:
>> >I have been using dvd+rw-tools (5.13.4.7.4) and mkisofs
>> > (cdrtools 1.11a29) to write backups to DVD. This generally
>> > works. However I encountered a problem when one of the files
>> > was 2351679431 bytes in size: the disk was written with no
>> > errors reported, but on testing proved to be unreadable to
>> > linux 2.4.21. On mounting the disk a warning was reported:
>> >
>> >Warning: defective CD-ROM. Enabling "cruft" mount option.
>>
>> "Defective" CD-ROM is a misleading statement.
>> "Interchange Level 3 ISO-9660 volumes are not handled by this OS"
>> would be more accurate.
>>
>> > * The ISO-9660 filesystem only stores 32 bits for file size.
>>
>> But the ISO-9660 _standard_ stores 32 bits for the size of each
>> _extent_ and there can be a virtually unlimited number of
>> _extents_ in a single file at ISO-9660 Interchange Level 3.
>
>But can a file span multiple extents?
Absolutely yes, at ISO-9660 Interchange Level 3. That is the only
difference between Interchange Level 3 and lower levels. Note that
one therefore must be at Interchange Level 3 to have a file span
multiple volumes (and thus to have a file larger than the size of
a single volume.
> The way I read the comment
>Gary quoted, it's legal to have an image that is over 2GB in size,
>as long as each file inside that image is no larger than 2GB.
That is the case for ISO-9660 Interchange Level 1 and Level 2.
> I
>haven't actually read the spec though.
I have :-)
Volker Kuhlmann
> as long as each file inside that image is no larger than 2GB.
Careful - the comment was about mkisofs, although it was in the kernel
source. It definitely says file*systems* >2GB are legal, otherwise it
says mkisofs can't handle single *files* >2GB - that doesn't
necessarily mean they're illegal. The comment may also be old and no
longer true for current versions of standards.
> But then the whole comment seems odd. It looks to me like the
> WARNING was added later, and written by Jörg.
Yes
> That seems to me like
> the only logical way to explain why the comment says it's legal,
> but the code claims it's illegal to have files that are more than
> 2GB in size.
You're mixing up file with filesystem here?
> so apparently at least someone looked at ISO Level 3 support. I'd
> say send a message to linux-kernel and see what they say about
> it...
Yes, together with a raft of other iso9660 issues :(
Perhaps mkisofs is now able to handle files >2GB, the lack of a
suitable error when creating the filesystem does suggest so. However,
for Linux that's a moot point as Linux doesn't handle >2GB, but mkisofs
isn't only used on Linux. Thanks Gary for the warning about that.
It should be possible to retrieve the file with isoinfo -x, which
doesn't use any filesystem code.
Volker
--
Volker Kuhlmann is possibly list0570 with the domain in header
http://volker.dnsalias.net/ Please do not CC list postings to me.
Lourens Veen
> > But can a file span multiple extents? The way I read the
> > comment Gary quoted, it's legal to have an image that is over
> > 2GB in size, as long as each file inside that image is no
> > larger than 2GB.
>
> Careful - the comment was about mkisofs, although it was in the
> kernel source. It definitely says file*systems* >2GB are legal,
> otherwise it says mkisofs can't handle single *files* >2GB - that
> doesn't necessarily mean they're illegal. The comment may also be
> old and no longer true for current versions of standards.
Good point.
[...]
> > That seems to me like
> > the only logical way to explain why the comment says it's
> > legal, but the code claims it's illegal to have files that are
> > more than 2GB in size.
>
> You're mixing up file with filesystem here?
Erm, actually, what I said doesn't make sense at all. Never mind...
> > so apparently at least someone looked at ISO Level 3 support.
> > I'd say send a message to linux-kernel and see what they say
> > about it...
>
> Yes, together with a raft of other iso9660 issues :(
>
> Perhaps mkisofs is now able to handle files >2GB, the lack of a
> suitable error when creating the filesystem does suggest so.
> However, for Linux that's a moot point as Linux doesn't handle
> >2GB, but mkisofs isn't only used on Linux. Thanks Gary for the
> warning about that.
Linux in general does have large file support now doesn't it?=20
Incidentally, I had a quick look at the same file in 2.6.0-test9,=20
but apart from adding support for compressed iso9660 filesystems=20
(zisofs?) and some stuff apparently to do with multisession=20
handling nothing much seems to have changed here.
Lourens
--=20
GPG public key: http://home.student.utwente.nl/l.e.veen/lourens.key
Volker Kuhlmann
Yes, for a little while. This is filesystem-dependent though!! Looks
like iso9660 has missed out so far.
Volker
--
Volker Kuhlmann is possibly list0570 with the domain in header
http://volker.dnsalias.net/ Please do not CC list postings to me.
Joerg Schilling
>I have been using dvd+rw-tools (5.13.4.7.4) and mkisofs (cdrtools 1.11a29)
>to write backups to DVD. This generally works. However I encountered
>a problem when one of the files was 2351679431 bytes in size: the disk
>was written with no errors reported, but on testing proved to be unreadable
>to linux 2.4.21. On mounting the disk a warning was reported:
>Warning: defective CD-ROM. Enabling "cruft" mount option.
>The large file now appeared to be only 2869191 bytes and is truncated
>on reading.
>Examining the Linux sources shows the following comment in fs/isofs/inode.c:
> /*
> * The ISO-9660 filesystem only stores 32 bits for file size.
> * mkisofs handles files up to 2GB-2 = 2147483646 = 0x7FFFFFFE bytes
> * in size. This is according to the large file summit paper from 1996.
> * WARNING: ISO-9660 filesystems > 1 GB and even > 2 GB are fully
> * legal. Do not prevent to use DVD's schi...@fokus.gmd.de
> */
> if ((inode->i_size < 0 || inode->i_size > 0x7FFFFFFE) &&
> inode->i_sb->u.isofs_sb.s_cruft == 'n') {
> printk(KERN_WARNING "Warning: defective CD-ROM. "
> "Enabling \"cruft\" mount option.\n");
> inode->i_sb->u.isofs_sb.s_cruft = 'y';
> }
The hacky change is from me, but note:
- As the source before did not support files > 512 MB, nobody could
have tested multi extent files!
- If I would have removed the "limitation" completely, it would most
likely not have been accepted as a patch :-(
Jörg
--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni) If you don't have iso-8859-1
schi...@fokus.fraunhofer.de (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily
Joerg Schilling
>> > * The ISO-9660 filesystem only stores 32 bits for file size.
>>
>> But the ISO-9660 _standard_ stores 32 bits for the size of each
>> _extent_ and there can be a virtually unlimited number of
>> _extents_ in a single file at ISO-9660 Interchange Level 3.
>But can a file span multiple extents? The way I read the comment
>Gary quoted, it's legal to have an image that is over 2GB in size,
>as long as each file inside that image is no larger than 2GB. I
>haven't actually read the spec though.
>But then the whole comment seems odd. It looks to me like the
>WARNING was added later, and written by Jörg. That seems to me like
>the only logical way to explain why the comment says it's legal,
>but the code claims it's illegal to have files that are more than
>2GB in size.
>Lastly, at the top of the file, there is
> * 1998 Eric Lammerts - ISO 9660 Level 3
>so apparently at least someone looked at ISO Level 3 support. I'd
>say send a message to linux-kernel and see what they say about
>it...
You should ask him why he did not test his code....
If he did test it, then the mount induced limitation would have gone
long ago.
Joerg Schilling
>> But can a file span multiple extents? The way I read the comment
>> Gary quoted, it's legal to have an image that is over 2GB in size,
>> as long as each file inside that image is no larger than 2GB.
>Careful - the comment was about mkisofs, although it was in the kernel
>source. It definitely says file*systems* >2GB are legal, otherwise it
>says mkisofs can't handle single *files* >2GB - that doesn't
>necessarily mean they're illegal. The comment may also be old and no
>longer true for current versions of standards.
It is hard to implement things in mkisofs that cannot be tested.
Gary Houston
>> so apparently at least someone looked at ISO Level 3 support. I'd
>> say send a message to linux-kernel and see what they say about
>> it...
>
>Yes, together with a raft of other iso9660 issues :(
>
>Perhaps mkisofs is now able to handle files >2GB, the lack of a
>suitable error when creating the filesystem does suggest so. However,
>for Linux that's a moot point as Linux doesn't handle >2GB, but mkisofs
>isn't only used on Linux. Thanks Gary for the warning about that.
>
>It should be possible to retrieve the file with isoinfo -x, which
>doesn't use any filesystem code.
This was interesting advice, or at least kept me busy for a while.
The man page for isoinfo says "The user interface really sucks." No
worries, I figured it out with a debugger:
isoinfo -x '/SRC.TAR;1' -i /dev/dvd > src.tar.gz
Unfortunately extract_file in isoinfo.c uses signed ints for file
sizes, so has a 2GB size limit. Files larger than this are truncated
to 0 bytes. For this exercise I changed int to size_t, but it would
be better to use large file I/O.
I made a new DVD with a large file (the old one has been used as a coaster
already and I wasn't going to put it back in the drive) and read it back
successfully with the modified isoinfo.
This indicates that mkisofs is working OK, or at least that mkisofs and
isoinfo are consistently with each other for large files.
I found that isoinfo from cdrtools 1.11a29 could give a listing of the
correct file sizes on the DVD, but then I upgraded to 2.00.3 and the
large file came out negative sized with this version.
To test another theory, I removed the "automatic cruft" check from the
kernel and mounted the disc: the file is still truncated, so it seems that
the ISO 9660 Level 3 support is not complete. Unfortunately I don't have
access to a DVD drive on a non-Linux machine for comparison.
Gary
ljknews
> >From: ljknews <ljk...@mac.com>
>
>>>It is hard to implement things in mkisofs that cannot be tested.
>
>>that *IX developers may not have non-*IX systems readily available.
>
>Large files may be used on Win32 since a few weeks when Cygwin started to
>support them.....
Large files may be used on OpenVMS since V6.1 in about 1994.
Joerg Schilling
>>It is hard to implement things in mkisofs that cannot be tested.
>Well such a resulting volume could certainly be tested, but I understand
>that *IX developers may not have non-*IX systems readily available.
Large files may be used on Win32 since a few weeks when Cygwin started to
support them.....
Jörg
--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni) If you don't have iso-8859-1
schi...@fokus.fraunhofer.de (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily
Rob Bogus
--------------090905010205020005000506
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Joerg Schilling wrote:
>
>
>
>>>But can a file span multiple extents? The way I read the comment
>>>Gary quoted, it's legal to have an image that is over 2GB in size,
>>>as long as each file inside that image is no larger than 2GB.
>>>
>>>
>>Careful - the comment was about mkisofs, although it was in the kernel
>>source. It definitely says file*systems* >2GB are legal, otherwise it
>>says mkisofs can't handle single *files* >2GB - that doesn't
>>necessarily mean they're illegal. The comment may also be old and no
>>longer true for current versions of standards.
>>
>>
>
>It is hard to implement things in mkisofs that cannot be tested.
>
>
>
But surely you could test it with Solaris?
--
E. Robert Bogusta
It seemed like a good idea at the time
--------------090905010205020005000506
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title></title>
</head>
<body>
Joerg Schilling wrote:<br>
<blockquote type="cite"
cite="mid200311031250...@burner.fokus.fraunhofer.de">
<pre wrap=""><!---->
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">But can a file span multiple extents? The way I read the comment
Gary quoted, it's legal to have an image that is over 2GB in size,
as long as each file inside that image is no larger than 2GB.
</pre>
</blockquote>
</blockquote>
<blockquote type="cite">
<pre wrap="">Careful - the comment was about mkisofs, although it was in the kernel
source. It definitely says file*systems* >2GB are legal, otherwise it
says mkisofs can't handle single *files* >2GB - that doesn't
necessarily mean they're illegal. The comment may also be old and no
longer true for current versions of standards.
</pre>
</blockquote>
<pre wrap=""><!---->
It is hard to implement things in mkisofs that cannot be tested.
</pre>
</blockquote>
But surely you could test it with Solaris?<br>
<br>
<br>
<pre class="moz-signature" cols="$mailwrapcol">--
E. Robert Bogusta
It seemed like a good idea at the time
</pre>
<br>
</body>
</html>
--------------090905010205020005000506--