Help Creating A Bootable USB Disk

[Christian Peeples]

Friends More Competent Than I:

I am finally getting around to upgrading to 20.4, but I seem to be stuck on step one. I went to Ubuntu for instructions and read the instructions and watched the video. The first thing that one is to do is create a bootable USB stick. I got a couple of brand new 8 GB USB sticks from Frys. I opened Startup Disk Creator from the applications. It detected the USB stick I had inserted and asked if I wanted to erase it. I clicked on that. The program did its thing and opened the USB stick in "Files" as a blank disk. When I went back to "Make a Startup Disk," the selection of how much space to allocate to documents and settings was greyed out. More importantly, the box for "Make Startup Disk" was greyed out. I tried it several times with the same result. What am I doing wrong?

-- Chris Peeples --

[Michael Paoli]

> From: "'Christian Peeples' via BerkeleyLUG" <berke...@googlegroups.com>
> Subject: Help Creating A Bootable USB Disk
> Date: Sun, 2 Aug 2020 08:24:07 +0000 (UTC)

Ewey GUI bits and/or other overly "high level" tools may be making it
more complicated/messy/problematic, by hiding what they're actually
doing.

So, first bit, Ubuntu (or the like - *buntu), will typically also
have some Ewey GUI file management thingy installed by default,
and also by default, anytime it sees newly inserted/attached
media, it is sees stuff on there that it thinks it can mount, it will
generally do so. So, first of all - that can get in the way.
If I recall correctly, the typically guilty program/utility on
*buntu is nautilus. To get around that first problem, minimally
will probably want to go into nautilus's preferences/settings,
and check/uncheck/change whatever setting(s) are necessary, to
prevent it from automatically mounting anything.
And ... why might automatically mounting be problematic?
Say one inserts USB flash, nautilus sees some filesystem(s) on
there and mounts them (even a "factory fresh" "blank"(-ish)
such USB flash, typically comes with an empty (or not so
empty - sometimes they throw some gunk utility software in there)
FAT/VFAT filesystem on there. Anyway with that (or old
ISO image, or whatever) mounted from the USB, many operations
attempted on the USB may fail with "device busy" or the like - as
it has filesystem(s) mounted from that device. We want to
avoid that. So, unplug the USB, disable - at least temporarily -
anything that's automatically mounting stuff from it when it's
inserted.
One can also check/confirm what's mounted, by looking at the output
of the command mount:
$ mount
... but that can be rather long list of stuff to look over.
However, before inserting USB, and after, that output from
mount should look the same - if it doesn't - most notably
additional line(s) of stuff - there's likely a problem, as
something probably mounted filesystem(s) from the USB device.

Next, you'll need the ISO file you want to write to the USB,
if you've already got that, you're that much closer.
Should also verify that image before bothering to write it to
USB.

Writing it to USB - with the USB device inserted, you'll need to
identify what device the USB came up as.
Typically it'll be some /dev/sd* device.
So, probably do:
$ ls -d /dev/sd[a-z] /dev/sd[a-z][a-z] 2>>/dev/null
With the USB inserted, typically you'll see 2 USB devices,
/dev/sda and /dev/sdb
And typically sda will be your boot/main drive and sdb the USB
device. But you may have more, depending upon your hardware
configuration.
In any case, will want to check which device is the USB.
One hint is to do that ls command before inserting the USB,
and then again after. Something new show up? That's probably
your USB device.
In any case, also good to check further. E.g.:
# blkid /dev/sd[a-z]*
That will generally look at the sd devices (and their partitions),
and try and identify what's on them.
So, you should be able to see which sd device (e.g. sda) and it's
partitions correspond to your installed operating system.
And which (e.g. sdb) corresponds to the USB flash.
Once you've identified and confirmed the flash device (e.g. /dev/sdb)
we're almost there.
Now you want to write - in "raw" format, to the USB, your ISO image
file. Also, for efficiency of flash writing, we'll want to
specify a large write block size - we want no less than the flash's
erase block size, and some integer multiple of that is fine.
But we don't know that size, but we can get it's some power of 2.
So, we just do something more than big enough, and that will do us
just fine.
$ expr 4 \* 1024 \* 1024
4194304
$
That would be 4 MiB - that will quite more than suffice.
So, let's say our USB device is /dev/sdb (be sure to use whatever your
actual device is - wrong device and one can do great damage - such as
destroying one's installed operating system ... with great power comes
great responsibility).
So, now just dd to write the USB device:
# dd if=ubuntu-20.04-desktop-amd64.iso obs=4194304 of=/dev/sdb
That may take a moderate while to complete. If the USB has
an LED activity indicator, it will generally light of flash to
indicate write activity.
Once that's completed, you can check it matches your .iso file, e.g.:
# cmp ubuntu-20.04-desktop-amd64.iso /dev/sdb
Expect that it will complain about EOF, e.g.:
cmp: EOF on buntu-20.04-desktop-amd64.iso
... but it shouldn't otherwise complain. And the reason it's
"complaining" about EOF - it read both files, all the bytes read up
to the length of the shorter matched ... but then it hit the
end of the shorter - hence the EOF "complaint", as it will
almost always be the case that the size of the USB flash
device is larger than that of the ISO image (were it
smaller, it wouldn't fit - and that would be a different problem).

That's basically it. Made it through to there, and you've got your
ISO written to your USB flash device.

[tom r lopes]

If you want stop by at the SF-Lug Jitsi meet.  

meet.jit.si/sf-lug.org

I just ran the startup disk creator in Ubuntu 20.04 and I'm not seeing 

the stuff you describe.  Maybe it was different in 14.04  I only saw the one 

option to write the disk.  No option for making space for files and all that.  

Is it making a live usb with persistence?  

Thomas

--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/351992075.7480532.1596356647846%40mail.yahoo.com.

[tom r lopes]

On Sun, Aug 2, 2020 at 3:21 AM Michael Paoli <Michae...@cal.berkeley.edu> wrote:

> From: "'Christian Peeples' via BerkeleyLUG" <berke...@googlegroups.com>
> Subject: Help Creating A Bootable USB Disk
> Date: Sun, 2 Aug 2020 08:24:07 +0000 (UTC)

So, now just dd to write the USB device:
# dd if=ubuntu-20.04-desktop-amd64.iso obs=4194304 of=/dev/sdb

I didn't know about the "obs" option. I've always used "bs" 

So reading the manpage I see obs is the blocksize for writing (output) 

ibs for reading (input) and bs to specify both.  I would have thought 

making the read and write block size the same is best.  

Also you can abbreviate 4194304 by 4M (so obs=4M) 

And I like to add "status=progress" because if your USB doesn't have 

any lights it is hard to know if anything is happening.  

I have been also doing "conv=sync"  Because I am just copying something 

I read somewhere.  I had thought it was like doing "sync" command after dd 

is done.  But NO!! I should check the manpages more often.  "conv=sync" 

pads the output block with zeros.  

Why would you want to do that?  

https://askubuntu.com/questions/706670/why-is-the-sync-option-used-with-the-dd-command

Second answer by Mark Plotnick

I think what I want is "oflag=sync"  

Thomas

[Michael Paoli]

SF-LUG meeting today (11a-1p) - yes, excellent point!
I thought to mention it ... then forgot by the time I
got the email sent. Thanks for covering that.

https://meet.jit.si/sf-lug.org
http://linuxmafia.com/pipermail/sf-lug/2020q3/014944.html
https://www.wiki.balug.org/wiki/doku.php?id=balug:covid-19

> https://groups.google.com/d/msgid/berkeleylug/CAGpvfspevQtF%2BCQ9YS97-QJ6baUfwY-s%2By9wbZD76ihLV3EJWw%40mail.gmail.com.
>

[Michael Paoli]

> From: "tom r lopes" <tomr...@gmail.com>

> Subject: Re: Help Creating A Bootable USB Disk
> Date: Sun, 2 Aug 2020 04:14:38 -0700

> On Sun, Aug 2, 2020 at 3:21 AM Michael Paoli <Michae...@cal.berkeley.edu>
> wrote:
>
>> > From: "'Christian Peeples' via BerkeleyLUG" <
>> berke...@googlegroups.com>
>> > Subject: Help Creating A Bootable USB Disk
>> > Date: Sun, 2 Aug 2020 08:24:07 +0000 (UTC)
>>
>> So, now just dd to write the USB device:
>> # dd if=ubuntu-20.04-desktop-amd64.iso obs=4194304 of=/dev/sdb
>>
>> I didn't know about the "obs" option. I've always used "bs"
> So reading the manpage I see obs is the blocksize for writing (output)
> ibs for reading (input) and bs to specify both. I would have thought
> making the read and write block size the same is best.

Yes, it's the output write block size that's often critical -
especially for flash, but for any device that's
block-oriented write at the physical level (and
also when applicable, erase block size), good to have the
output blocks writes set at (or some integer multiple of) that.
Also, if there's some higher level blocking also involved, e.g. on
LVM, zfs, ext[234], etc., might gain a wee bit more on performance by ensuring
obs is integer multiple of LCM of that block size and the physical
write (and erase, if applicable) block size. So, typically
that might be, e.g. 4MiB or some multiple thereof.

ibs doesn't much matter - may save teensy bit of overhead in
the program, but not much else. OS will read input
device/stream at whatever it's physical block size is (if it's
a block device) - and it'll buffer that for the application to
read from - so doesn't much matter. On the other hand, obs matters
up to hugely. E.g. if we have an obs that's a fraction of the
write, or erase, block size, on the output devices, it may require
multiple writes, or erase/write cycles, to complete the writing of
the physical write/erase block - whereas if we picked obs
at the size (or integer multiple of) the physical, then the
(erase/)write is done in single pass/cycle. So, too small,
and the output can go orders of magnitude slower, and also
likewise increase the write wear - most notably on flash/SSD,
even on rotating rust it'll generally be more head passes, rotations,
and bit more seeks, so can also slow it down quite a bit.

> Also you can abbreviate 4194304 by 4M (so obs=4M)

Depends on your vintage/flavor of dd.
Integer always works - and is bytes.
Let's see ... standards ...
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html
At current, b for 512 byte blocks, k for KiB.
None other are standard.
Hmm, "ancient" history ...
https://plan9.io/7thEdMan/v7vol1.pdf
ibs, obs, b, and k all work there too.
But none 'o that newfangled M - nor is that in current standards.

> And I like to add "status=progress" because if your USB doesn't have
> any lights it is hard to know if anything is happening.

I usually don't bother with that. But if/when I want to know ...
GNU's dd will give current status to stderr upon receipt of
SIGURS1. So, I'll sometimes use that. The -o option to lsof can
also sometimes come in handy (and even more so when what's
reading/writing isn't dd). Can also use the proc filesystem
to determine the file(s) ... but I don't know off-hand if
offset of read/write is available or easily determined from there.

> I have been also doing "conv=sync" Because I am just copying something
> I read somewhere. I had thought it was like doing "sync" command after dd
> is done. But NO!! I should check the manpages more often. "conv=sync"
> pads the output block with zeros.

Yeah, most of the time one doesn't want to pad out the last output block
written.

> Why would you want to do that?
> https://askubuntu.com/questions/706670/why-is-the-sync-option-used-with-the-dd-command
> Second answer by Mark Plotnick
> I think what I want is "oflag=sync"

I don't really see a reason for that.
After the dd, one can always do:
$ sync && sync
And ... why the double sync?
Because sync can return before it's completed.
But another sync, can't return
until earlier sync has completed.
That's why, for, e.g. fast emergency shutdown/halt, one may see
something like:
# sync && syncn && halt
... or nowadays on GNU/Linux, with all the goop added to halt
to keep folks from shooting themselves in the foot:
# sync && sync && halt -f -f
or:
# sync && sync && halt -d -f -f -n
... or something like that.
Future versions will probably require:
# sync && sync && halt -d -f -f -f -f --force='really really please I
mean it!' -n --reason='power supply on fire! Please halt now now now
now now!' --comment='I though one of the reasons we went from Windows
to Linux was so we would't have to ask Mother May I Please?'

[Rick Moen]

Quoting 'Christian Peeples' via BerkeleyLUG (berke...@googlegroups.com):

> I am finally getting around to upgrading to 20.4, but I seem to be
> stuck on step one. I went to Ubuntu for instructions and read the
> instructions and watched the video. The first thing that one is to do
> is create a bootable USB stick. I got a couple of brand new 8 GB USB
> sticks from Frys. I opened Startup Disk Creator from the
> applications. It detected the USB stick I had inserted and asked if I
> wanted to erase it. I clicked on that. The program did its thing and
> opened the USB stick in "Files" as a blank disk. When I went back to
> "Make a Startup Disk," the selection of how much space to allocate to
> documents and settings was greyed out. More importantly, the box for
> "Make Startup Disk" was greyed out.

You gravitated towards a Desktop Environment aka DE (GNOME, as packaged
by Ubuntu Linux) that includes an automounter. I rather dislike
automounters, because (by definition & by design) they autodetect
whenever you've attached a device with mountable filesystems and mount
them in background at mountpoints of their choosing, without bothering
to ask you whether you want them mounted or where you wish the
mountpoint to be.

So, it is common for novices like you, having been subtly pushed towards
(IMO) overly complicated DEs and distributions, end up thereafter taken
by surprise by being inexplicably prevented from doing partition
operations, because of the device already having been mounted in
background, e.g., busy.

My _own_ response to this situation would be to say 'The hell with
GNOME, and let's in particular lose its automounter daemon.' Since it's
unlikely you'll make the same decision, I guess the alternative will be
to cultivate awareness of what the automounter is doing in background,
and prepare to intervene to unmount filesystems when, e.g., you want to
overwrite the device in question.

Web-searching about GNOME suggests you should check the 'Disks' applet
to check the mounted/unmounted status of connected removable drives --
and to toggle that status. Apparently, the GNOME automounter uses
/media/[blahblah] as the mountpoint directories, for such removable
drives.

--
Cheers, You must rise or sink / You must conquer or win,
Rick Moen Or serve and lose. / Suffer or triumph, / Be anvil or hammer.
ri...@linuxmafia.com
McQ! (4x80) -- Johann Wolfgang von Goethe, Gesellige Lieder, Ein Anderes

[Christian Peeples]

Rick:

      Thanks. Jitsied into SFlug and got a great deal of help Tom and Michael (including error checking from the terminal), found out one of the thumb drives I had bought was defective.  We got it done and am now using 20.4 LTS.  The only problem I have had is getting my scanner to work.   I called HP and they say they no longer support Linux.  I am working on it.

-- Chris Peeples --

--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20200804020910.GF29756%40linuxmafia.com.

[Rick Moen]

Quoting 'Christian Peeples' via BerkeleyLUG (berke...@googlegroups.com):

> Thanks. Jitsied into SFlug and got a great deal of help Tom and
> Michael (including error checking from the terminal), found out one of
> the thumb drives I had bought was defective.  We got it done and am
> now using 20.4 LTS.

Excellent. Yes, defective media would do that, too. Echoing Michael's
point, the GUIfied tools tend to hide / discard debugging information,
with the result that when something goes wrong, you have little idea
what happened and why. This is one of a number of reasons to gravitate
towards standard command-line tools, at least when doing debugging.
E.g., first check in the dammned GNOME 'Disks' app to make sure the
device's filesystem(s) hasn't (/haven't) been mounted, and unmount if
mounted. Then, open a terminal and do:

$ dmesg | tail #Notice whether it's /dev/sdb or whatever, and
#whether it has partitions, e.g., /dev/sdb1
$ sudo su - #Be wary that you're using root authority after
#this, which is why the prompt changes from $ to #,
#so that you're aware and on-guard.
# fdisk -l /dev/sdb #Or whatever the device is.
#This command displays its partition table, if any.

If worried that the flash drive (or other external drive might be a dud,
you can experimentally format it. Like this would format flash drive
/dev/sdb's first partition /dev/sdb1 as FAT32:

# mkfs -t vfat /dev/sdc1
# exit #Stop wielding root-user authority, now.
$

Old-school tools for overwriting a disk from a disk-image file includ
dd.

> The only problem I have had is getting my scanner to work.   I called
> HP and they say they no longer support Linux.  I am working on it.

You're probably overreading the phrase 'we no longer support'. That
merely means they refuse to assist you. It tells you nothing whatsoever
about whether there is easy support in open source (or proprietary
alternative software, notably VueScan) for your scanner.

If you would please look carefully at your scanner and tell this mailing
list its model number, it'll then be possible to assist you.

So, do that, please.

It's highly likely that there is good open source driver support for
your scanner, though at any given time some models are problem children,
typically models that are either extremely new, or are all-in-one
devices, or both.

--
Cheers, "Maybe the law ain't perfect, but it's the only
Rick Moen one we got, and without it we got nuthin'."
ri...@linuxmafia.com -- U.S. Deputy Marshal Bass Reeves, circa 1875
McQ! (4x80)

[Michael Paoli]

> From: "Rick Moen" <ri...@linuxmafia.com>

> Subject: Re: Help Creating A Bootable USB Disk

> Date: Mon, 3 Aug 2020 19:09:10 -0700

> Quoting 'Christian Peeples' via BerkeleyLUG (berke...@googlegroups.com):
>
> You gravitated towards a Desktop Environment aka DE (GNOME, as packaged
> by Ubuntu Linux) that includes an automounter. I rather dislike
> automounters, because (by definition & by design) they autodetect
> whenever you've attached a device with mountable filesystems and mount
> them in background at mountpoints of their choosing, without bothering
> to ask you whether you want them mounted or where you wish the
> mountpoint to be.

Yeah, ... I generally find such things rather to quite annoying.
Been a long time since I'd looked at it on *buntu (or the like),
so ... was curious ... fired one up (created VM & launched
Ubuntu 20.4 on it).

Last I'd poked at it, some several years or more ago, the offending bit was
nautilus - the Ewey GUI file manager utility thingy.
If it saw new device that looked mountable or like it had mountable
bits, it would automagically mount 'em. I found that highly
annoying. I also found, with some poking around at the Ewey GUI bits,
at least as I seem to recall at the time, somewhere in "Options" or
"Preferences" or the like, it was also quite easy to disable the
automounting thing ... so one could still have all that file management
Ewey GUIness, but without it automounting. Okay, whatever ... that was
some years ago, as I recall it, and was nautilus at the time, and as
Ubuntu (or *buntu or whatever it was) had it configured, at least by
default.

Anyway, decided to look at the current. Similar, but different.
The good news - by default - it seems (at least the bit I tried)
to not automagically mount.
But the bad (and/or not so good) news.
A mere double-click (or click) and - boom mounted.
More annoyingly, was finding the preferences/options settings, and
once found, how to disable the mounting, or make it "harder", like
at least requiring some confirmation dialog. No such option.
No mount related options at all - nothing to say mount automagically
without me doing anything, or don't mount no matter what, no matter
how many times I click on the device icon or how or whatever, no
setting between to ask/force some confirmation dialog, ... no
mount related settings at all - at least in the GUI.
And ... I think most users using GUI to manage files, generally won't
go much beyond options made available in/via the GUI. So, I didn't
bother to dig deeper. Oh, and yes, checked the processes and such,
again, it's nautilus. So, ... I don't know what the DE du jour is
for Ubuntu, but if one might like that ... but nautilus or its
behavior, might be feasible to just disable (or uninstall)
nautilus ... or install/use some other Ewey GUI file manager instead.
Anyway, I didn't poke at it further ... not of that much interest to
me. But I'm sure there must be other folks that know all about
various Ewey GUI file managers, which are available on which distros,
default behaviors of each on various distros, and how each
can/can't be customized, how, and to what degree, and how
easily and/or not.

> So, it is common for novices like you, having been subtly pushed towards
> (IMO) overly complicated DEs and distributions, end up thereafter taken
> by surprise by being inexplicably prevented from doing partition
> operations, because of the device already having been mounted in
> background, e.g., busy.

> My _own_ response to this situation would be to say 'The hell with
> GNOME, and let's in particular lose its automounter daemon.' Since it's

There may be lots of (other) good reasons/arguments to say 'The hell with
GNOME', but I don't know that the Ewey GUI file manager thingy is one
of them. Heck, I don't even know what DE Ubuntu 20.4 is using.
And, might be very feasible to disable/remove/replace/reconfigure
the Ewey GUI file manager thingy ... and not otherwise need to change
DE ... though there may be (many!) other reasons for going to a
different DE or much lighter weight one, or just a simple
window manager.

http://www.shoutfactorytv.com/the-prisoner/the-prisoner-s1-e10-hammer-into-anvil/5b7c40d614edba1447009ac2
https://archive.org/download/The_Prisoner/ThePrisoner10HammerIntoAnvil.mp4
https://archive.org/download/The_Prisoner/ThePrisoner10HammerIntoAnvil.ogv

[Rick Moen]

Quoting Michael Paoli (Michae...@cal.berkeley.edu):

> There may be lots of (other) good reasons/arguments to say 'The hell with
> GNOME', but I don't know that the Ewey GUI file manager thingy is one
> of them. Heck, I don't even know what DE Ubuntu 20.4 is using.

For the record, Canonical-tweaked GNOME3 with gnome-shell 3.36.4,
nautilus 3.36.3, openbox 3.6.1, and gtk 3.24.21.

> And, might be very feasible to disable/remove/replace/reconfigure
> the Ewey GUI file manager thingy ... and not otherwise need to change
> DE ... though there may be (many!) other reasons for going to a
> different DE or much lighter weight one, or just a simple
> window manager.

Obviously, Views Differ[tm], but when I found out how easy a good Linux
distribution makes it to choose what you want to install & run on an 'a
la carte' basis, I lost all interest in installing & running the
figurative kitchen sink.

> http://www.shoutfactorytv.com/the-prisoner/the-prisoner-s1-e10-hammer-into-anvil/5b7c40d614edba1447009ac2
> https://archive.org/download/The_Prisoner/ThePrisoner10HammerIntoAnvil.mp4
> https://archive.org/download/The_Prisoner/ThePrisoner10HammerIntoAnvil.ogv

Ah, yes, that's indeed where I first encountered the Goethe quotation.
In the episode, the sadistic Number Two quotes it unironically, but
there is good reason to think Goethe intend it the opposite way.

Number Two: You are too strong. We'll see. "Du mußt Amboß oder Hammer sein."
Number Six: "You must be anvil or hammer."
Number Two: I see you know your Goethe.
Number Six: And you see me as the anvil?
Number Two: Precisely. I am going to hammer you.

I comment on this in http://linuxmafia.com/pub/humour/sigs-rickmoen.html:


(Archivist's note: This is an English translation of the concluding
lines from one of two poetic songs Goethe wrote in 1787 for a now-lost
opera, Die Mystificierten (The Mystified), both of which he fancifully
pretended to have translated from the Coptic language, thus their names
Kophtisches Lied (Coptic Song) and Ein Anderes (Another, as in another
Coptic song). Long after the opera was abandoned, Goethe saved the two
songs and published them in his collection Gesellige Lieder (Convivial
Songs), in 1814.

My best guess, along with that of an actual contemporary Coptic
scholar, is that the narrator's view, that one must be either ruthless
oppressor or ruthlessly oppressed, is being voiced by Goethe ironically,
as the songs were part of a commentary on contemporaneous European
politics that had turned ruthlessly transformative (particularly the
French Revolution), an approach to life Goethe very much did not share.

The line "You must be hammer or anvil" is well known to fans of
Patrick McGoohan's unique 1960s television programme The Prisoner, as
the thematic quotation cited by the new and particularly ruthless Number
2 in the episode Hammer into Anvil.)

[Michael Paoli]

> From: "Rick Moen" <ri...@linuxmafia.com>
> Subject: Re: Help Creating A Bootable USB Disk

> Date: Mon, 3 Aug 2020 20:36:36 -0700

Yeah ... in our particular case, with the first flash media, it was
only upon reading it back (cmp(1)) that the error was apparent. It
gave no indications of errors upon write. But in both attempts
to verify it after each of two attempts to write then verify,
in both cases that media failed at the same location, so probable that
USB flash is defective.
I suggested badblocks(8) for further/subsequent testing/analysis.
The other USB flash media worked perfectly fine, and verified 100%
correctly after the first write. Also, in each case, to
thwart any side-effects of operating system buffering of the media,
after flash had finished writing and fully flushed out (checked
both by completion of
sync; sync
and also LED activity on the USB flash stopped), we disconnected the
flash, waited a bit, then reinserted it - only after that did
we proceed to do the cmp(1) to do a verification check of the data.

So ... in this particular case, dmesg and the like wouldn't have
caught the error ... but cmp(1) and (presumably) badblocks(8) would.

Sometimes flash fails in funky way. I myself have one USB flash drive
that likewise failed very oddly - writes would appear to go okay,
but reading the data back ... yeah, ... data read would not be at all
like the data written ... egad.

[Michael Paoli]

> From: "Rick Moen" <ri...@linuxmafia.com>
> Subject: Re: Help Creating A Bootable USB Disk
> Date: Mon, 3 Aug 2020 20:36:36 -0700

Yep, very likely it's well supported ... even if not - or not so much,
by HP, very likely well supported by relevant Linux distro(s) and
software thereupon.

And I'm sure, HP also doesn't want to spend time/money on support calls
dealing with Linux - especially without a paid support contract on it,
so I'm quite sure in general they're more inclined to shove/push off
such calls as quickly as feasible. After all, the folks answering
their phones for support and the related infrastructure ... not just
some volunteers and volunteered resources ... so of course they want
to generally be making money off of it ... or at least minimizing
costs thereof. And, speaking of HP, might also have much better results
with their on-line resources ... including also their "community" forum
or whatever they're calling it (that cost 'em a lot less to operate).

[Christian Peeples]

Rick & Michael:

Thank you.

-- Chris Peeples --

--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20200808110930.19585nrvg558drsw%40webmail.rawbw.com.

[Rick Moen]

Quoting Michael Paoli (Michae...@cal.berkeley.edu):

> So ... in this particular case, dmesg and the like wouldn't have
> caught the error ... but cmp(1) and (presumably) badblocks(8) would.

I'm sure cmp(1) (to explain: a utility to compare files byte by byte,
and don't forget that in Unix a filesystem or entire storage device can
if so desired be treated as a file) made troubleshooting easier. In the
event of _gross_ failure, the errors or pathological output from 'fdisk
-l /dev/sdc' or 'mkfs -t vfat /dev/sdc1' would be a dead giveway.

Running badblocks(8) on a suspect flash drive would certainly be
(more than) justified before giving it up and consigning it to the
landfill -- even though I'm so used to conserving flash devices' limited
supply of erase-write cycles that I instictively shy away from using the
likes of mkfs or badblocks on them. Heh. Obviously, in a situation
where the next step is landfill, you do it anyway.

> Sometimes flash fails in funky way.

Yeah, notoriously. One's instincts from spining-rust experience are no
good, and maybe actively misleading. That's one aspect in which the
lack of sounds and of moving parts is actually a problem, i.e., you
keep expecting to hear repeated re-seeks if there is impending disk
failure, and it's easy to forget that there isn't seeking in the first
place.

[Rick Moen]

Quoting 'Christian Peeples' via BerkeleyLUG (berke...@googlegroups.com):

> Rick & Michael:
>
> Thank you.

You're welcome -- but are you going to post the model number of this
scanner?

If you do, it's pretty easy to determine if and how it has Linux support
(via a SANE back-end, or worst case, via VueScan). But if you never
tell us the model number, we can't help.

The implicit ulterior motive we of the Linux community have, in giving
such assistance (he says, confessing his dark aims) is to teach others
_how to solve that problem_ -- so that others, reading this mailing list
or searching for answers on the Web and finding this discussion, will
say 'Oh, I get it. _That's_ how you determine how to enable and test
driver support for a scanner on a Linux distribution.'

We're waiting with antici...

[Christian Peeples]

Rick:

The model number is in the subject line. I'll look at the resources you suggested.

-- Chris Peeples --

--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20200808211344.GS29756%40linuxmafia.com.

[Rick Moen]

Quoting 'Christian Peeples' via BerkeleyLUG (berke...@googlegroups.com):

> Rick & Michael:
>
> Thank you.

Christian, I failed to notice that you put the requested datum as a
(changed) Subject header. (Sorry about missing that.)

Subject: HP MFP m29w Scaning

Not a complaint, but switching out the Subject header isn't the most
efficient way to answer a question. At least speaking for myself, I
normally expect the conversation to keep going on in the message body
text.


[This is going to be a long post, but at the end, the answer near the
end will be 'Good news, you should have easy scanner support, provided
your version of the hplip package, provided by Ubuntu, is 3.18.4 or
later. You should not need anything else (other than the standard Linux
SANE software), and you don't need to download anything from HP.']



Anyway, taking that by fragments:

HP

As you said, a Hewlett-Packard.

MFP

I warned upthread that multifunction devices (combined scanner, printer,
fax, and/or modem) are one of two categories of scanners most likely to
be problematic for driver support.

I'm not 100% sure why this is the case, but I have a theory. Even
though there's absolutely nothing wrong with the _concept_ of such an
all-in-one device, in practice the manufacturers seem to see it as an
opportunity to cut corners, i.e., you get an underpowered and rickety
inkjet (usually) printer, and an underpowered scanner engine that often
has design limitations like offloading a bunch of the scanning work to
an obligatory piece of proprietary (aka 'non-free') software required to
run on an attached PC, just in order to save $2 of electronics in the
scanner itself. According to my theory, partly in consquence, these
all-in-one devices suffer consequences of failing Moen's Law of
Hardware.

http://linuxmafia.com/~rick/lexicon.html#moenslaw-hardware

Moen's Law of Hardware

"Use what the programmers use."

After years of helping people with hapless computer-hardware woes,
especially trouble-prone categories such as Linux on laptops, exotic
peripheral interfaces, etc., it occurred to me to wonder why I never had
such problems. It was mainly because of instinctive avoidance of dodgy,
exotic, new, and/or badly designed components — which happens to track
strongly with programmers' characteristic prejudices. There's a logic to
that, which may not be immediately apparent to many:

Drivers for hardware don't emerge like Athena from the head of Zeus:
Someone has to create them. Especially for open-source OSes such as
Linux, this involves a chipset being brought to market, for it to be out
long enough for coders to buy and start using it, and for them to (if
necessary, absent manufacturer cooperation) do the hard work of
reverse-engineering required to write and debug hardware support. Then,
the resulting code filters out to various OS distributions' next
releases, and thus eventually to users.

It follows that, if you blithely buy what's new and shiny, or so badly
designed or built that coders eschew it, or so exotic that coders tend
not to own it, it will probably have sucky software support, especially
in open source. (Proprietary drivers can be written under NDA, often
before the hardware's release, while manufacturer help is routinely
denied to the open source world.) Conversely, if you buy equipment
that's been out for a while, doesn't suffer the (e.g., Diamond
Multimedia in the early 2000s) problem of chip-du-jour, is bog-standard
and of good but not exotically expensive quality, it will probably have
stellar driver quality, because coders who rely on that hardware will
make sure of that.

Thus, it's very common for slightly ageing but good-quality gear to
outperform and be more reliable than the latest gee-whiz equipment,
because of radically better software support — not to mention the price
advantage.
[...]

The point being, in the case of all-in-one multifunction
printer/scanner/fax/modem hardware, the coders who'd need to write the
SANE (Scanner Acceess Now Easy) back-end open-source drivers are
probably slow to want and acquire such hardware, hence slow to write
drivers for them.

As mentioned upthread, in the event of scant support from SANE, you have
as fallback the proprietary VueScan drivers. The VueScan coders are
willing to sign NDAs to get detailed hardware information from the
manufacturers (because their code will not disclose confidential
hardware information under GPL), and get rewarded for writing driver
software even for lackluster scanners because you as a customer need to
pay them money.

Personally, I avoid the problem by eschewing hardware that transgresses
Moen's Law of Hardware, and favouring using what the programmers use.

But getting back to your scanner:

m29w

The 'w' designates 'wireless'. Which FWIW is a Bad Word where printing
and scanning is concerned. Why? Because hardware manufacturers tend
invent bizarre proprietary WiFi-networking protocol extensions as
transport methods for scanning and printing over WiFi.

So, when you see 'wireless' for such a thing, you basically pray to the
gods that the gadget also supports an access method less problematic,
like (usually) USB (which gets thrown in because it's dirt-cheap).



HP MPW m29w

Just a fine-point, not a big thing: It's in your interest to try to
cite model numbers exactly the way the manufacturer says them.
Lettercase doesn't matter, but people searching for information in
order to help you can do so best if they have the identifying string
exactly as the manufactuer states it. In this case:

HP LaserJet Pro MFP M29w

Specs say it includes an USB 2.0 port, not just wireless
(https://store.hp.com/us/en/pdp/hp-laserjet-pro-mfp-m29w-printer).


So, the next thing I do is Web-search for

"HP LaserJet Pro MFP M29w" "sane"

This guy sought help with an M29's (not M29W) scanning on Linux Mint 19:
https://forums.linuxmint.com/viewtopic.php?t=291127
He stressed that he was trying using USB.

One detail that emerges is that the M29 & M29W are part of a set of
near-identical all-in-one models that require (all) the same printing
and scanner drivers, referred to in driver sofware as
'HP_LaserJet_MFP_M28-M31', i.e., HP LaserJet MFP models M28 through M31,
as they are basically all the same thing.

Pretty much all of the discussion in that Linux Mint forum thread is
about downloading and trying to make work the scanner portion of an
HPLIP (Hewlett-Packard Linux Imaging and Printing) driver that the user
downloaded directly from HP's download site, and trying to get a
proprietary plugin to do its magic.

And here, I must again explain a couple of things.

(1) We of the Linux community go to hardware manufacturers' download
sites for drivers only as an absolute last resort. Those drivers, where
they exist, tend to be really bad, really fragile, and proprietary.
They are almost never provided as distro packages, so, long-term,
relying on them does violence to the software management on your distro.
It is unclear whether this user (in the forum thread) knew this and
tried the Linux Mint packages of HPLIP _first_.

(2) HP pulled off something of a propaganda coup when they created the
HPLIP software architecture, something like 20 years ago. They got
kudos for contributing open source -- which they kind of did, and kind
of not. The core HPLIP set _is_ open source under GPL -- which is why
distros can and do package it.

But HPLIP by itself supports some HP hardware fully, other models
partially, and other models unsatisfactorily or not at all, because
HP consigns 'secret sauce' driver code for some models to proprietary,
binary-only software that cannot by law be even _distributed_ with
HPLIP, but must be fetched directly from HP's site as a separate
download. For both scanners and printers, this functionality gets
implemented as a proprietary 'plugin' for the otherwise open source
HPLIP driver.

_And_, the cheaper you go in HP's model lines, the more likely the
hardware model is to require special 'secret sauce' plugins to do much,
or in some cases to be anything but a paperweight. When you get down to
HP's 'MFP' all-in-ones, it's _totally_ non-surprising that HP expects
you to download proprietary, binary-only crud -- because that's
(frankly) the bottom of the barrel.


The Linux Mint forum thread was active from 2011 to 2017. Following
links from the 2017 comments, one eventually ends up at this Ubuntu bug
for their HPLIP package:
https://bugs.launchpad.net/ubuntu/+source/hplip/+bug/1811504 Quoting:

From my point of view the bug should be closed somehow. As I
understand MFP M28 uses closed source protocols from Apple (AirScan) and
Microsoft (WP-scan) for scanning, therefore the scanning plugin form HP
is closed source and Ubuntu developers can do little with it, otherwise
MFP M28 linux users, who can push HP directly to fix the problem. The
workaround solution (sane-airscan/current SANE) are open source and
based on reverse engineering specification of the protocol. My opinion
that the proper way of fixing the issue is not to do something in the
plugin, but asap include a new version of SANE to next Ubuntu release
and/or backported SANE airscan to current Ubuntu if possible.
P.S. As many other users of HP, I absolutely disappointed of such "good
quality" drivers from the vendor. I hope it will change in the future.

Hey! Wacky proprietary network protocol extensions for cheap-ass
all-in-one hardware. Who knew? ;->

Same guy earlier in the bug:

I was also affected by this problem. The MFP support AirScan,
therefore it is possible to scan with such interface. Very recent
version of SANE support this, but for old SANE versions you can find
a driver here: https://github.com/alexpevzner/sane-airscan
I've install it from the repository and it works! I can scan any
modes with hi resolution (600 dpi).

Oh joy. Extra, weird-ass software to retrofit reverse-engineered
support for a manufacturer's wacky proprietary network protocol
extensions for cheap-ass all-in-one hardware.


Web-searching
MFP_M28-M31 sane
I find among other things this 2019 thread at HP support:
https://h30434.www3.hp.com/t5/Scanning-Faxing-and-Copying/linux-impossible-to-scan/td-p/6969861
That user never found a solution.


Web-searching
MFP_M28-M31 scanner linux
got the same story here:
https://h30434.www3.hp.com/t5/LaserJet-Printing/LaserJet-MFP-M28w-amp-Linux/td-p/6667058



Let's switch tactics for a moment, and attack the driver thing head-on
via the SANE Project pages about scanner-model support by manufacturer:
http://www.sane-project.org/sane-mfgs.html#Z-HEWLETT-PACKARD

Uh-oh, nothing for the entire M28-M31 series.

Let's try the page for the _development_ (beta) SANE code, linked as
such from http://www.sane-project.org/sane-supported-devices.html,
leading to
http://www.sane-project.org/lists/sane-mfgs-cvs.html#Z-HP

There's an entry for the M28w, which appears to be basically the same
unit:

Model: LaserJet MFP M28w
Interface: WiFi
USB ID: n/a
Status: Untested
Comment: Testers needed!
Backend: escl
Man page: sane-escl

The man page says:

NAME: sane-escl - SANE backend for eSCL scanners

DESCRIPTION: The sane-escl library implements a SANE (Scanner Access
Now Easy) back-end that provides access to eSCL protocol scanners.

The "escl" backend for SANE supports AirScan/eSCL devices that
announce themselves on mDNS as _uscan._utcp or _uscans._utcp. If the
device is available, the "escl" backend recovers these capacities.
The user configures and starts scanning. A list of devices that use
the eSCL protocol can be found at
https://support.apple.com/en-us/HT201311. While these devices are
expected to work, your mileage may vary.

So, surprisingly, at least using the official SANE (development version)
software, the dodgy WiFi access is supported, but SANE says nothing
encouraging about USB access.

Last, also on http://www.sane-project.org/sane-supported-devices.html,
there's a link to a list of _external_ SANE backends, i.e., ones that
aren't officially part of the SANE project, but maintained elsewhere.
That list is at:
http://www.sane-project.org/lists/sane-backends-external.html#S-HPAIO

The 'HP LaserJet Pro MFP m29a' (but not the m29w) has a specific entry
under the 'hpaio' SANE back-end. It says that the status of support is
listed as 'Good', but that 'this backend [hpaio] isn't included in SANE
because it is included in the HPLIP software'.


Let's slightly switch tactics again, and look on HP's site for HPLIP.
https://developers.hp.com/hp-linux-imaging-and-printing/supported_devices/index
has an entry for the m29w, as follows:

Model: HP LaserJet Pro MFP M29w
Min. HPLIP Version: 3.18.4
Chrome OS Support: No
Driver Plug-in: No [RM: this mean you do _not_ need a proprietary plugin.]
Support Level: Full
Print Mode: Mono
Scan to PC: Yes
PC Send Fax: No
Connectivity: USB,Network


So, hey, why don't you try installing the needed software?

$ sudo apt install hplip hplip-gui libsane-hpaio

Make sure the installed version of HPLIP is at _least_ 3.18.4, as
mentioned above.[1] If not, there's a deal-breaker, right there, and this
is yet another reason to not use old and poorly supported distro
releases as I remember you were recently.

Then (assuming you have a modern version of HPLIP), as described on
https://www.cyberciti.biz/faq/how-to-install-networked-hp-printer-and-scanner-on-ubuntu-linux/
, do:

$ hp-setup

or, to set up the printer/scaner in the HPLIP framework to use USB:

$ hp-setup -b usb

Then just do the obvious things in the GUI. This'll let you set up
printing if you haven't already. Exit, then do:

$ hp-toolbox

This lets you check on the setup of printing and scanning.



Christian, I hope that helps. I belatedly realised that your request for
help did not include which Ubuntu _release_ (which would be something
you should always include in such requests for help), nor did you state
what you've already tried, and what happened when you did.



[1] Checking at Distrowatch.com, I find:

Ubuntu release HPLIP version New enough?
20.04 LTS 'focal' 3.20.3 Yes
19.10 'eoan' 3.19.6 Yes
19.04 'disco' 3.19.6 Yes
19.10 'cosmic' 3.18.7 Yes
18.04 LTS 'bionic' 3.17.10 No
all earlier <=3.17.10 No

So, if you are still on Ubuntu prior to 19.10, you need to upgrade your
distro.