Greets!
One source of potential confusion is that there are two ways to do this:
Case 1: You download a LiveCD image and write it over the thumb drive in
question. When you boot this up, it will load the system entirely into
RAM and never write any changes back to the drive. So you might boot it
up, create some files, set a password, install bsd-games, whatever, but
the minute you turn off the machine, all those changes disappear. This
is useful if you want an essentially immutable system that will only
ever be used for troubleshooting/installing Linux on other machines and
will never itself change (and in particular will never break).
Case 2: You properly install Linux on a USB drive just as you would
install linux on any other hard drive. There are a few potholes to
avoid when executing this, but upon success, it ends up the same as
running off a hard drive: you can update the installed system, edit
files, etc. as usual, except that this hard drive is not tied to a
particular computer (even if just by a couple screws). In this case, if
the distro didn't originally come with some package that you end up
needing to do some diagnostics, you can merrily install it on your USB
drive system for future use. The risk here is that updating things, if
not done with some care, can [1] render your system unbootable, and if
this in turn means you've broken your recovery system, then, well...
There are of course hybrid approaches, where the system is essentially
immutable but you can store changes to at least your user files. Or if
you have a big enough flash drive you can install a liveCD Linux and a
normal Linux on separate partitions of the drive. Or really anything
else you can imagine.
I use Arch, whose procedure for setting up case 1 is at
https://wiki.archlinux.org/index.php/USB_Installation_Media, and is
pretty much cat in a box. The procedure for setting up case 2 is at
https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_a_USB_key
and is more involved. Having said that, every trouble I had with this
latter process turned out to be a result of my ignoring/missing
something in the directions. If you'd be initially less happy being
presented with nothing but a terminal prompt, I'd hop on the Ubuntu
bandwagon--it isn't fundamentally less capable or anything.
[1] This paranoia might be an artifact of my Arch habit--I've not had
this happen with any other distro.