Re: Archiving content of a directory on a DVD-R.

[pe...@easthope.ca]

* From: "Thomas Schmitt" scdb...@gmx.net
* Date: Sun, 04 Nov 2018 18:48:45 +0100
> If you want to make use of xorriso's capability to create a mountable ISO
> filesystem with a copy of your directory, then you do not need tar.
>
> xorriso -for_backup -outdev /dev/sr0 -map . / \
> -close on -commit \
> -toc -check_md5 failure -- \
> -eject all
>
> This run ...

Thanks Thomas. That's what I ended up using. Two shell scripts are
in http://easthope.ca/DebianPage.html under heading Backups. Backup()
is used interactively to protect against failure of the SD card.
FilesToDVD() makes archival records allowing convenient access to old
copies of individual files. Typically, Backup() executes in less than
a minute. Good for interactive use when working.

An example run of FilesToDVD() follows.

Thanks again, ... Peter E.

peter@dalton:~$ time FilesToDVD
Insert blank DVD-R.
xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev '/dev/sr0'
Media current: DVD-R sequential recording
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 4489m free
xorriso : UPDATE : 2668 files added in 1 seconds
Added to ISO image: directory '/'='/home/peter/MY0.Bak'
xorriso : UPDATE : Thank you for being patient. Working since 0 seconds.
xorriso : UPDATE : Thank you for being patient. Working since 1 seconds.
xorriso : UPDATE : Thank you for being patient. Working since 2 seconds.
xorriso : UPDATE : Writing: 16s 0.0% fifo 100% buf 0% 0.0xD
...
xorriso : UPDATE : Writing: 380133s 100.0% fifo 0% buf 100% 0.0xD
ISO image produced: 379983 sectors
Written to medium : 380133 sectors at LBA 0
Writing to '/dev/sr0' completed successfully.

xorriso : NOTE : Re-assessing -outdev '/dev/sr0'
...
xorriso : UPDATE : 742.1m content bytes read in 112 seconds = 5.0xD
Ok, session data match recorded md5.

real 5m12.980s
user 0m10.536s
sys 0m6.148s




--
Message composed and transmitted by software designed to avoid the
complication and vulnerability of antivirus software.

123456789 123456789 123456789 123456789 123456789 123456789 123456789
Pender Is., BC: +1 604 670 0140 Washington State: +1 360 639 0202
http://easthope.ca/Peter.html Bcc: peter at easthope. ca

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> Media current: DVD-R sequential recording

> [...]

> xorriso : UPDATE : 742.1m content bytes read in 112 seconds = 5.0xD

It seems that next the joy of incremental backups could be explored.


> http://easthope.ca/DebianPage.html

First precondition would be to give up xorriso command
-close on
to keep the DVD-R writable on its yet unused area.

Next replace -outdev by -dev and -map by -update_r.
I.e.:

xorriso -for_backup -dev /dev/sr0 \
-update_r . / \

-commit \
-toc -check_md5 failure -- \
-eject all

Now you can write to the DVD-R until it is full. Each run will produce a
complete directory tree of the current state. Data file content will
only be written if the file is new or its content changed since the
previous backup session (if there is such a session on DVD already).
If not many files changed, the add-on session

When you mount the DVD on GNU/Linux, the youngest session will be used by
default. Older sessions can be mounted by mount(8) option "-o sbsector="
using the numbers told by column "sbsector" of xorriso command -toc:

$ sudo mount -o sbsector=567890 /dev/sr0 /mnt/iso

Each session shows the complete directory state as it was at the time
when it was written to DVD.

When the DVD-R has not enough room for the upcomming session, xorriso will
fail without altering the DVD-R content. In this case, just insert a new
blank DVD-R to write a new first session.

------------------------------------------------------------------------

The update decision is based on the timestamps and recorded MD5s on DVD.
For the comparison, the MD5s of the files in /home/peter/MY0.Bak have
to be computed by reading the full file content. If the storage device
is slow or your CPU runs hot, then consider to bet on device numbers and
inode numbers:

If the device number of the storage device is persistent over unmounting
or reboot, then insert
-disk_dev_ino on
between -for_backup and -outdev.

If the device number changes, but the inode numbers are persistent, then
use
-disk_dev_ino ino_only

Whether those numbers are persistent can be inquired by printing them
with shell command "stat" before and after reboot or re-mounting:

$ stat --format="device=%d inode=%i" /.../some_file
device=2051 inode=3540166

(Device number changes are a matter of the operating system's bus
management, inode number changes are mostly a matter of the filesystem's
type. extN should be safe at least with inode numbers.)


Have a nice day :)

Thomas

[pe...@easthope.ca]

From: "Thomas Schmitt" <scdb...@gmx.net>
Date: Wed, 06 Mar 2019 17:28:39 +0100

> xorriso -for_backup -dev /dev/sr0 \
> -update_r . / \
> -commit \
> -toc -check_md5 failure -- \
> -eject all

With a blank disk,
...
xorriso : FAILURE : Image size 1072576s exceeds free space on media 359844s
libisofs: MISHAP : Image write cancelled
...

root@joule:/home/root# df | grep sd
/dev/sda1 6227904 4218756 1673072 72% /
/dev/sda4 25764060 4672576 19759672 20% /home
/dev/sdb1 3658244 2214144 1258252 64% /home/root/MY

Building temporary data structures in / space is limited?

How can xorriso to told to work in /home?

Thx, ... P.

--
mobile: +1 778 951 5147
VoIP: +1 604 670 0140
48.7693 N 123.3053 W

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> With a blank disk,

> xorriso : FAILURE : Image size 1072576s exceeds free space on media 359844s

359844 * 2048 = 736,960,512 bytes = 702.82 MiB.
This looks more like the size of a "700 MB" CD-R medium.

> Building temporary data structures in / space is limited?

Not in particular. Processing of directories with very many files can
become a bit sloww. There is memory needed for each file name and its
attribute when the emerging ISO 9660 filesystemi s planned. But unless
you have millions of files this should not be a problem even with an
old computer.

The FAILURE complaint rather says that libburn told xorriso that the
medium offers ~700 MiB for writing. But the emerging ISO 9660 filesystem
would need 2094.875 MiB of medium capacity.
A blank DVD-R would take that filesystem. A blank CD-R would not.


Question:
What messages do you get printed when the CD-or-DVD medium is inserted
and you do:

xorriso -outdev /dev/sr0 -toc

With a blank "700 MB" CD-R i get:

Drive current: -outdev '/dev/sr0'
Media current: CD-R
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 703m free
Drive current: -outdev '/dev/sr0'
Drive access : exclusive:unrestricted
Drive type : vendor 'ASUS' product 'BW-16D1HT' revision '1.01'
Drive id : 'K9CF92A5012 '
Media current: CD-R
Media product: 97m34s23f/79m59s73f , Mitsubishi Chemical Corporation
Media status : is blank
Media blocks : 0 readable , 359846 writable , 359846 overall
Media summary: 0 sessions, 0 data blocks, 0 data, 703m free

(The capacity of CD-R varies by nominal size: "650 MB", ... "900 MB"
and in smaller range between manufacturers which use the exact start
and end address to distinguish their products from other companies.)

With a blank DVD-R i get:

Drive current: -outdev '/dev/sr0'

Media current: DVD-R sequential recording

Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 4489m free

Drive current: -outdev '/dev/sr0'
Drive access : exclusive:unrestricted
Drive type : vendor 'ASUS' product 'BW-16D1HT' revision '1.01'
Drive id : 'K9CF92A5012 '

Media current: DVD-R sequential recording

Media product: RITEKF1 , Ritek Corp
Media status : is blank
Media blocks : 0 readable , 2298496 writable , 2298496 overall

Media summary: 0 sessions, 0 data blocks, 0 data, 4489m free

[pe...@easthope.ca]

From: "Thomas Schmitt" <scdb...@gmx.net>
Date: Sat, 26 Feb 2022 18:29:45 +0100

> What messages do you get printed when the CD-or-DVD medium is inserted
> and you do:
>
> xorriso -outdev /dev/sr0 -toc

root@joule:/home/root# xorriso -outdev /dev/sr0 -toc
xorriso 1.5.2 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev '/dev/sr0'
Media current: CD-R
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 703m free
Drive current: -outdev '/dev/sr0'
Drive access : exclusive:unrestricted

Drive type : vendor 'SONY' product 'CD-RW CRX320E' revision 'NYK2'
Media current: CD-R
Media product: 97m26s66f/79m59s71f , CMC Magnetics Corporation
Media status : is blank
Media blocks : 0 readable , 359844 writable , 359844 overall

Media summary: 0 sessions, 0 data blocks, 0 data, 703m free

root@joule:/home/root#

Thanks. I should have read the fine print at the center of the disk.

To squeeze the data, add -zisofs.

FilesToDVD () { \
printf "Insert open or new DVD-R.";
read t;
cd ~/MY0.Bak/;

xorriso -for_backup -dev /dev/sr0 \
-update_r . / \

-zisofs \

-commit \
-toc -check_md5 failure -- \

-eject all ;
}

I have a supply of CD-Rs and DVDs aren't accessible at present.
Likely xorriso provides a way to spread an archive over multiple CDs.
Documentation or example?

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> Drive type : vendor 'SONY' product 'CD-RW CRX320E' revision 'NYK2'

Ouch. That's a "Combo" drive which can write only CD-R and CD-RW.
With DVD media it can only do reading.

So there is no use buying writable DVDs for it.

> To squeeze the data, add -zisofs.

This can help if the data are compressible by a factor 3.
But that would be quite boring data.

> xorriso -for_backup -dev /dev/sr0 \
> -update_r . / \
> -zisofs \
> -commit \

Command -zisofs sets parameters for the compression. It needs an argument
or else it would eat the command -commit and complain that it is not
a suitable parameter for -zisofs.

To cause the files to be compressed you have to equip them by zisofs
filters. For best and slowest compression choose zlib level 9:

xorriso -for_backup -dev /dev/sr0 \
-update_r . / \

-zisofs level=9 \
-find / -type f -pending_data -exec set_filter --zisofs -- \
-commit \
...

----------------------------------------------------------------------

> Likely xorriso provides a way to spread an archive over multiple CDs.
> Documentation or example?

That would be the job of an older project of mine:
http://scdbackup.sourceforge.net/main_eng.html
http://scdbackup.sourceforge.net/scdbackup-0.9.2.tar.gz

Installation and configuration are some work:
http://scdbackup.sourceforge.net/examples.html#configure

(That's a reason why it is not in Debian. Another reason is that scdbackup
gets installed by particular users, which is good for isolating their
backup data but not suitable for the sysadmin-driven model of a Linux
distro.)

For building the software you need the C compiler and its companions
from Debian package "build-essential". Compilation happens automatically
during the run of ./CONFIGURE_CD or ./CONFIGURE_DVD.

After the installation you may simply run

scdbackup_home

See
http://scdbackup.sourceforge.net/examples.html#scdbackup_home

The more general command

scdbackup

takes pathspecs as known from mkisofs and exclusions of paths:

scdbackup /home /home2=/usr/home -not /home/guest

This example backups /home and /usr/home . It gives on CD the name /home2
to the directory tree that is /usr/home on hard disk. Subdir /home/guest
will not get into the backup.

(Back in 1999 a 650 MB CD-R was a respectable backup medium, competing
with 500 MB QIC tapes and 1.2 GB DDS tapes.
Nowadays i use them only for small backups and for software delivery.
My bread-and-butter backup media are now "25 GB" BD-RE media. They are
not much more expensive than DVD+RW. Only the prices of burner drives
are significantly higher for BD than for DVD.)

[pe...@easthope.ca]

From: "Thomas Schmitt" <scdb...@gmx.net>
Date: Sun, 27 Feb 2022 09:15:04 +0100

> Ouch. That's a "Combo" drive which can write only CD-R and CD-RW.
> With DVD media it can only do reading.

...

> That would be the job of an older project of mine:
> http://scdbackup.sourceforge.net/main_eng.html
> http://scdbackup.sourceforge.net/scdbackup-0.9.2.tar.gz
>
> Installation and configuration are some work:
> http://scdbackup.sourceforge.net/examples.html#configure

...

> (Back in 1999 a 650 MB CD-R was a respectable backup medium, competing
> with 500 MB QIC tapes and 1.2 GB DDS tapes.
> Nowadays i use them only for small backups and for software delivery.
> My bread-and-butter backup media are now "25 GB" BD-RE media. They are
> not much more expensive than DVD+RW. Only the prices of burner drives
> are significantly higher for BD than for DVD.)

My conclusion: invest in more recent hardware.
https://en.wikipedia.org/wiki/Blu-ray_Disc_recordable#Pricing

Backing data in a 4 or 16 GB SD card is a modest requirement.
PC Galore might have a drive in stock. http://www.pcgalore.com/

Tips to select a drive welcome.

Thanks, ... P.

[Thomas Schmitt]

Hi,

> My conclusion: invest in more recent hardware.
> https://en.wikipedia.org/wiki/Blu-ray_Disc_recordable#Pricing

Those prices are slightly optimistic. But not too much.

> Tips to select a drive welcome.

Beware of "Combo" drives. Make sure the product description mentions
_write_ speed for BD-RE and BD-R, not only read speed for BD-ROM.
("Combo" drives are usually a bit cheaper than real BD burners.)

Buy from a seller who takes defective or wrongly chosen hardware back
without offering much resistance.
The quality of new drives varies much more individually than by the
brand or manufacturer. (Even 80 EUR are not enough to pay for a
convincing quality management in the factory.)

If my ASUS BW-16D1HT would die, i would by a new one of the same model.
It is the only one which writes more than 128 sessions to a single BD-R.

Another good drive which i have and is still at sale is LG BH16NS40.

I would not buy a new Pioneer BDR-S09 in the next years because mine
creates cracks on contemporary Verbatim BD-RE media after reading
them a dozen times. The cracks start at the inner hole and eventually
reach the recording area which then causes read failure.
I did not observe this damage with RITEK BD-RE, which are sold by about
any brand except Verbatim, or with older Verbatim BD-RE media.
(I'd blame the problem on the 10x read speed of the drive, its hard
grip on the medium, and the little engraved letters at the inner rim
of the modern Verbatim BD-RE. Forcing the drive to read more slowly
protects the victim media. It does not react on speed setting commands
but only slows down when i let the reader program wait between read
operations.)

I have no experience with slim BD drives which are sold in USB enclosures.
My USB attached drives are full-height "internal" drives which i put
into USB boxes which cost nearly as much as the drives.
("Man goennt sich ja sonst nichts." = I hardly grant myself anything else.)

[rhkr...@gmail.com]

On Monday, February 28, 2022 12:37:49 AM pe...@easthope.ca wrote:
> Backing data in a 4 or 16 GB SD card is a modest requirement.
> PC Galore might have a drive in stock. http://www.pcgalore.com/

What has been your experience with reliability of SD cards for backup?

[Chris Ramsden]

Mine has been bad. I have accumulated several dead SD cards, yet over many years, I can think of only one USB memory device failing. 
 
USB and SD memory devices are based on similar (if not identical) flash technology, are they not? 
 
Under this assumption, I'd guess that the USB interface is rather more robust than that found in SD cards. SD cards seem to be fine if installed and left there (IP cameras, dashcams, phones) but fail when handled. And yes, I am aware of ESD (electrostatic discharge) issues and take steps to minimize risk. 
 
I wouldn't trust my data to SD cards.   
-- 
Regards,
Chris

[pe...@easthope.ca]

From: "Thomas Schmitt" <scdb...@gmx.net>
Date: Mon, 28 Feb 2022 08:55:40 +0100

> Beware of "Combo" drives. Make sure the product description mentions
> _write_ speed for BD-RE and BD-R, not only read speed for BD-ROM.
> ("Combo" drives are usually a bit cheaper than real BD burners.)

Critical point, thanks.

> Buy from a seller who takes defective or wrongly chosen hardware back
> without offering much resistance.

First stop will be PC Galore which mostly sells used equipment.
They're knowledgeable and honest. If nothing there is suitable will go
to the big retailers where sales clerks can be unhelpful.

> The quality of new drives varies much more individually than by the
> brand or manufacturer. (Even 80 EUR are not enough to pay for a
> convincing quality management in the factory.)

Your instructions in 2019 were used with a DVD burner at work.
Completely effective. Thanks. The current interest is to establish
backups at home.

Considering the earlier experience and modest quantity of data (< 2
GB) another DVD drive might be adequate. Will see what is at PC
Galore.

Thx, ... P.

[pe...@easthope.ca]

From: rhkr...@gmail.com
Date: Mon, 28 Feb 2022 09:52:35 -0500

> What has been your experience with reliability of SD cards for backup?

My explanation was ambiguous. =8~| The primary medium here is SD.
The backup is optical.

The oldest SD card I have was purchased about 2012. The label was
NextTech. Purchased via eBay from a seller in Ontario or Quebec. It's
been reformatted 2 or 3 times over the years and still works. =8~)
Recently parts of some files disappeared or showed garbled
information. The card has served better than I expected. Probably due
for retirement. =8~)

Since 2012 I've purchased three Kingston 8 GB SDs. Two are used in
mobile phones; one is system store in a OLPC XO 1.5. Those SDs
continue to work with no apparent difficulties.

General policy. Buy only name brands: Kingston, SanDisk, Lexar & etc.
Buy only from local businesses maintaining a reputation in the
marketplace. Noname cards only waste time and money.

Regardless of reliability of a specific medium, data preservation
comes from a good backup system. =8~)

Regards, ... P.

[John Conover]

pe...@easthope.ca writes:
> From: rhkr...@gmail.com
> Date: Mon, 28 Feb 2022 09:52:35 -0500
> > What has been your experience with reliability of SD cards for backup?
>
> My explanation was ambiguous. =8~| The primary medium here is SD.
> The backup is optical.
>
> The oldest SD card I have was purchased about 2012. The label was
> NextTech. Purchased via eBay from a seller in Ontario or Quebec. It's
> been reformatted 2 or 3 times over the years and still works. =8~)
> Recently parts of some files disappeared or showed garbled
> information. The card has served better than I expected. Probably due
> for retirement. =8~)
>
> Since 2012 I've purchased three Kingston 8 GB SDs. Two are used in
> mobile phones; one is system store in a OLPC XO 1.5. Those SDs
> continue to work with no apparent difficulties.
>
> General policy. Buy only name brands: Kingston, SanDisk, Lexar & etc.
> Buy only from local businesses maintaining a reputation in the
> marketplace. Noname cards only waste time and money.
>

Also, make sure the SD manufacturing technology has wear
leveling. Many cheap SDs do NOT. (They are manufactured in out dated
facilities.)

If used for backups, write the file, then write the check sum of the
file, (perhaps using something linke md5sum(1),) to the disk, with the
same file name plus an extension, so that md5sum can be used to check
the integrity of the data before installing the backup.

For long term archival data, use a new SD, and write once, (perhaps on
multiple SDs, for file reconstruction; any corrupt file on one SD can
probably be retrieved from another SD.) Store at STP for the plastics.

>
> Regardless of reliability of a specific medium, data preservation
> comes from a good backup system. =8~)
>

John

--

John Conover, con...@panix.com, http://www.johncon.com/