Clone a FBSD system with something in the likes of ghost

[Alejandro Imass]

Greetings fellow daemons,

I have a beautiful running server (dozens of jails and intricate
configuration) on a single small and aging hard drive. I bought 2 new hard
drives and want to migrate to a ZFS mirror.

Pardon my ignorance but would there be a way to just copy my system from
the old drive to the new ZFS array?

I was thinking of something like this: 1) install the two new drives in the
server and boot with old drive via an USB enclosure. 2) Create a booteable
ZFS array and somehow copy an identical image of my current system onto the
array.

Am I dreaming or are there actual ways of doing this? I really don't want
to re-install and configure everything. Just want to move an identical copy
of my system to the new hard drives on a ZFS mirror.

Any ideas greatly appreciated!

TIA,
Alejandro Imass
_______________________________________________
freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Robert Huff]

On 9/28/2016 10:43 PM, Alejandro Imass wrote:

> I have a beautiful running server (dozens of jails and intricate
> configuration) on a single small and aging hard drive. I bought 2 new hard
> drives and want to migrate to a ZFS mirror.
>
> Pardon my ignorance but would there be a way to just copy my system from
> the old drive to the new ZFS array?
>
> I was thinking of something like this: 1) install the two new drives in the
> server and boot with old drive via an USB enclosure. 2) Create a booteable
> ZFS array and somehow copy an identical image of my current system onto the
> array.
>
> Am I dreaming or are there actual ways of doing this? I really don't want
> to re-install and configure everything. Just want to move an identical copy
> of my system to the new hard drives on a ZFS mirror.

The canonical - and correct - method involves dump piped to restore;
there may be an example in the Handbook.


Respectfully,


Robert Huff

[Warren Block]

On Thu, 29 Sep 2016, Robert Huff wrote:

> On 9/28/2016 10:43 PM, Alejandro Imass wrote:
>
>> I have a beautiful running server (dozens of jails and intricate
>> configuration) on a single small and aging hard drive. I bought 2 new hard
>> drives and want to migrate to a ZFS mirror.
>>
>> Pardon my ignorance but would there be a way to just copy my system from
>> the old drive to the new ZFS array?
>>
>> I was thinking of something like this: 1) install the two new drives in the
>> server and boot with old drive via an USB enclosure. 2) Create a booteable
>> ZFS array and somehow copy an identical image of my current system onto the
>> array.
>>
>> Am I dreaming or are there actual ways of doing this? I really don't want
>> to re-install and configure everything. Just want to move an identical copy
>> of my system to the new hard drives on a ZFS mirror.
>
> The canonical - and correct - method involves dump piped to restore;
> there may be an example in the Handbook.

I have not tried it, but I think restore(8) should work when writing to
a ZFS system.

So dump(8) on the original UFS piped to restore(8) on ZFS, presumably in
a dataset or multiple datasets.

http://www.wonkity.com/~wblock/docs/html/backup.html

After it is on ZFS, dump(8) cannot be used, but zfs send and zfs recv
are similar. Or rsync, or tar, or clonehd, or other things. The
options with them are the trick. It takes a lot to get rsync to make a
serious copy of a non-trivial filesystem with links and flags.

[Alejandro Imass]

On Thu, Sep 29, 2016 at 1:10 AM, Warren Block <wbl...@wonkity.com> wrote:
>
> On Thu, 29 Sep 2016, Robert Huff wrote:
>
>> On 9/28/2016 10:43 PM, Alejandro Imass wrote:
>>
>>> I have a beautiful running server (dozens of jails and intricate
>>> configuration) on a single small and aging hard drive. I bought 2 new hard
>>> drives and want to migrate to a ZFS mirror.
>>>
>>> Pardon my ignorance but would there be a way to just copy my system from
>>> the old drive to the new ZFS array?
>>>
>>> I was thinking of something like this: 1) install the two new drives in the
>>> server and boot with old drive via an USB enclosure. 2) Create a booteable
>>> ZFS array and somehow copy an identical image of my current system onto the
>>> array.
>>>
>>> Am I dreaming or are there actual ways of doing this? I really don't want
>>> to re-install and configure everything. Just want to move an identical copy
>>> of my system to the new hard drives on a ZFS mirror.
>>
>>
>> The canonical - and correct - method involves dump piped to restore; there may be an example in the Handbook.
>
>
> I have not tried it, but I think restore(8) should work when writing to a ZFS system.
>
> So dump(8) on the original UFS piped to restore(8) on ZFS, presumably in a dataset or multiple datasets.
>
> http://www.wonkity.com/~wblock/docs/html/backup.html
>
> After it is on ZFS, dump(8) cannot be used, but zfs send and zfs recv are similar. Or rsync, or tar, or clonehd, or other things. The options with them are the trick. It takes a lot to get rsync to make a serious copy of a non-trivial filesystem with links and flags.

Thanks for your suggestions!
I will look into dump and restore.

Thanks again!

Alejandro Imass

[JD]

By clone, you mean create a new disk with identical
content to the cloned disk? Including the boot blocks
of the cloned disk?
If so, dump and restore will only perform like a full backup.
The target (copy-to) disk will not be bootable.
If you want it bootable, then there is a simlpe procedure
1. Obtain a new disk with sufficient capacity as close as possible
to size of the source disk. Not less than, though!!
2. dd if=</dev/..... i.e. the source disk> of=/dev/..... i.e. the new disk>
bs=128M conv=notrunc conv=fdatasync

When finished, without error,
the new disk will be a bootable mirror of the original.

HTH


On Thu, Sep 29, 2016 at 6:37 PM, Alejandro Imass <aim...@yabarana.com>
wrote:

> To unsubscribe, send any mail to "freebsd-questions-
> unsub...@freebsd.org"

[loka...@gmx.de]

At this point there are some similar products like norton ghost. One is
http://www.netbsd.org/gallery/products.html#g4u
The problem is always, that is can't boot, when it come from a hdd to a ssd.

http://www.cyberciti.biz/datacenter/5-awesome-open-source-cloning-software/
and looking at goole for "clone disk"

greetings, there are more things, that can but not must help you. ;(

[Warren Block]

On Sun, 2 Oct 2016, JD wrote:

> By clone, you mean create a new disk with identical
> content to the cloned disk? Including the boot blocks
> of the cloned disk?
> If so, dump and restore will only perform like a full backup.
> The target (copy-to) disk will not be bootable.
> If you want it bootable, then there is a simlpe procedure
> 1. Obtain a new disk with sufficient capacity as close as possible
>     to size of the source disk. Not less than, though!!
> 2. dd if=</dev/..... i.e. the source disk> of=/dev/..... i.e. the new disk> bs=128M conv=notrunc conv=fdatasync
>
> When finished, without error,
> the new disk will be a bootable mirror of the original.

There can be problems with that. GPT secondary partitions will be in
the wrong place, and this is really bad on SSDs. It also takes longer
than necessary because it copies every block, many of which are unused
and don't need to be copied.

[Warren Block]

On Sun, 2 Oct 2016, loka...@gmx.de wrote:

> At this point there are some similar products like norton ghost. One is
> http://www.netbsd.org/gallery/products.html#g4u

These are mostly going to do the same thing as dd(1) when they don't
understand the filesystem. Clonezilla does understand UFS, and I have
used it once or twice to clone a FreeBSD disk. I still prefer
dump/restore.

In the old MBR days, there was no serious problem with partition tables,
because there was only one partition table at the start of the disk.
GPT puts one at the start of the disk and the end of the disk, so binary
copying a smaller disk to a larger one puts the secondary partition
table somewhere before the end of the larger disk. It's not really
possible to copy a larger disk to a smaller one with dd(1), so that
problem does not come up.

I recommend backing up first, then setting up the target disk partitions
and bootcode:
http://www.wonkity.com/~wblock/docs/html/disksetup.html

Then use dump/restore to copy from the original:
http://www.wonkity.com/~wblock/docs/html/backup.html

> The problem is always, that is can't boot, when it come from a hdd to a ssd.

I have not experienced this, but my first guesses would be that bootcode
was not installed, or that it is one of the earlier FreeBSD versions
that did not boot at all if the secondary GPT table was not correct.

[Polytropon]

On Mon, 3 Oct 2016 10:49:06 -0600 (MDT), Warren Block wrote:
> In the old MBR days, there was no serious problem with partition tables,
> because there was only one partition table at the start of the disk.
> GPT puts one at the start of the disk and the end of the disk, so binary
> copying a smaller disk to a larger one puts the secondary partition
> table somewhere before the end of the larger disk. It's not really
> possible to copy a larger disk to a smaller one with dd(1), so that
> problem does not come up.

Isn't it _technically_ possible to capture the 2nd GPT table (at the
end of the source disk) and calculate its proper position in relation
to the size of the target disk, and then put it there?

Initial state: empty disk:

/dev/da0 --------------------

Image smaller than this disk, including two GPT tables (1 and 2):

source.img 1#######2

Copy the image:

# dd if=source.img of=/dev/da0

/dev/da0 1#######2-----------

Get the 2nd GPT table

# dd if=source.img of=gpt2.img skip=1000 bs=1M count=1

gpt2.img 2

Put it at the end of the disk (the skip= value has to be calculated
from the disk size):

# dd if=gpt2.img of=/dev/da0 skip=1000

/dev/da0 1#######2----------2

Optional step: remove (useless or confusing) 2nd GPT table left
behind by the image:

# dd if=/dev/zero of=/dev/da0 skip=1000 bs=1M count=1

/dev/da0 1#######-----------2

Tadaa!

I assume that a 2nd GPT table somewhere inside the free space is
not a problem (otherwise I'd suggest to add the additional step to
overwrite it with zero bytes).

However, even if this might work, it's a very ugly thing. Putting
things "at the end of the disk" in the age of virtualized and easily
resizable disks doesn't look very clever...

> I recommend backing up first, then setting up the target disk partitions
> and bootcode:
> http://www.wonkity.com/~wblock/docs/html/disksetup.html
>
> Then use dump/restore to copy from the original:
> http://www.wonkity.com/~wblock/docs/html/backup.html

That's the easier approach, as it makes sure that all GPT data is
going to be located at where it belings, but it involves more steps
than the desired "clone" (in _one_ step).



> [...] or that it is one of the earlier FreeBSD versions

> that did not boot at all if the secondary GPT table was not correct.

And _this_ is exactly where the oh so modern and comfortable GPT
is going to shoot users in the foot. ;-)



--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...

[Warren Block]

On Mon, 3 Oct 2016, Polytropon wrote:

> On Mon, 3 Oct 2016 10:49:06 -0600 (MDT), Warren Block wrote:
>> In the old MBR days, there was no serious problem with partition tables,
>> because there was only one partition table at the start of the disk.
>> GPT puts one at the start of the disk and the end of the disk, so binary
>> copying a smaller disk to a larger one puts the secondary partition
>> table somewhere before the end of the larger disk. It's not really
>> possible to copy a larger disk to a smaller one with dd(1), so that
>> problem does not come up.
>
> Isn't it _technically_ possible to capture the 2nd GPT table (at the
> end of the source disk) and calculate its proper position in relation
> to the size of the target disk, and then put it there?

Sure. 'gpart recover' does that, although it rebuilds the secondary
table from the primary table rather than copying it.

> Initial state: empty disk:
>
> /dev/da0 --------------------
>
> Image smaller than this disk, including two GPT tables (1 and 2):
>
> source.img 1#######2
>
> Copy the image:
>
> # dd if=source.img of=/dev/da0
>
> /dev/da0 1#######2-----------
>
> Get the 2nd GPT table
>
> # dd if=source.img of=gpt2.img skip=1000 bs=1M count=1
>
> gpt2.img 2
>
> Put it at the end of the disk (the skip= value has to be calculated
> from the disk size):
>
> # dd if=gpt2.img of=/dev/da0 skip=1000
>
> /dev/da0 1#######2----------2
>
> Optional step: remove (useless or confusing) 2nd GPT table left
> behind by the image:
>
> # dd if=/dev/zero of=/dev/da0 skip=1000 bs=1M count=1
>
> /dev/da0 1#######-----------2
>
> Tadaa!

It looks like a lot of work to do to use a tool that is not well suited
for copying modern disks, though.

> I assume that a 2nd GPT table somewhere inside the free space is
> not a problem (otherwise I'd suggest to add the additional step to
> overwrite it with zero bytes).
>
> However, even if this might work, it's a very ugly thing. Putting
> things "at the end of the disk" in the age of virtualized and easily
> resizable disks doesn't look very clever...

Think about it for a bit, and you realize there are only two places on a
disk that are known for any size of disk, the start and end. If you put
a partition table somewhere aribtrary between those, it breaks up the
space for the user. A backup copy of the partition table could be put
next to the main one, but then both could get wiped out easily.

>> I recommend backing up first, then setting up the target disk partitions
>> and bootcode:
>> http://www.wonkity.com/~wblock/docs/html/disksetup.html
>>
>> Then use dump/restore to copy from the original:
>> http://www.wonkity.com/~wblock/docs/html/backup.html
>
> That's the easier approach, as it makes sure that all GPT data is
> going to be located at where it belings, but it involves more steps
> than the desired "clone" (in _one_ step).

What you showed above wasn't one step, either. :)

Sometimes brute-force is fine. I prefer to save that as a last resort
for disks.

>> [...] or that it is one of the earlier FreeBSD versions
>> that did not boot at all if the secondary GPT table was not correct.
>
> And _this_ is exactly where the oh so modern and comfortable GPT
> is going to shoot users in the foot. ;-)

There are BIOS implementations that will not boot unless the MBR is just
right, either.

All of this is kind of implying there is sort of a fixed cost to doing
many things, a "pay me now or pay me later" thing.

[Ernie Luzar]

JD wrote:
> By clone, you mean create a new disk with identical
> content to the cloned disk? Including the boot blocks
> of the cloned disk?
> If so, dump and restore will only perform like a full backup.
> The target (copy-to) disk will not be bootable.
> If you want it bootable, then there is a simlpe procedure
> 1. Obtain a new disk with sufficient capacity as close as possible
> to size of the source disk. Not less than, though!!
> 2. dd if=</dev/..... i.e. the source disk> of=/dev/..... i.e. the new disk>
> bs=128M conv=notrunc conv=fdatasync
>
> When finished, without error,
> the new disk will be a bootable mirror of the original.
>
> HTH
>

> snip
>

I have a 10.3 box using a MBR 1.5 TB hard drive. This Box acts as the
LANs front door firewall to the public network. The 1.5 TB disk is being
waisted in this situation. What would you recommend as the quickest
method to clone the running system from the 1.5TB disk to a 40MB disk?

Thanks

[Polytropon]

On Tue, 04 Oct 2016 09:52:11 -0400, Ernie Luzar wrote:
> I have a 10.3 box using a MBR 1.5 TB hard drive. This Box acts as the
> LANs front door firewall to the public network. The 1.5 TB disk is being
> waisted in this situation. What would you recommend as the quickest
> method to clone the running system from the 1.5TB disk to a 40MB disk?

Preparing the target disk with MBR or GPT or Dedicated (use GPT
except you have a good reason not to), then use dump + restore.
You can find more information here:

http://www.wonkity.com/~wblock/docs/html/disksetup.html#_the_new_standard_gpt

http://www.wonkity.com/~wblock/docs/html/backup.html#_copying_filesystems

The advantage of dump | restore is that only blocks in use will
be copied, and it works on file system level (instead of disk
block level like dd). Initializing the target disk prior to
starting the copy operation will make sure the partition data
is being written to the correct places.

--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...

[Ernie Luzar]

Polytropon wrote:
> On Tue, 04 Oct 2016 09:52:11 -0400, Ernie Luzar wrote:
>> I have a 10.3 box using a MBR 1.5 TB hard drive. This Box acts as the
>> LANs front door firewall to the public network. The 1.5 TB disk is being
>> waisted in this situation. What would you recommend as the quickest
>> method to clone the running system from the 1.5TB disk to a 40MB disk?
>
> Preparing the target disk with MBR or GPT or Dedicated (use GPT
> except you have a good reason not to), then use dump + restore.
> You can find more information here:
>
> http://www.wonkity.com/~wblock/docs/html/disksetup.html#_the_new_standard_gpt
>
> http://www.wonkity.com/~wblock/docs/html/backup.html#_copying_filesystems
>
> The advantage of dump | restore is that only blocks in use will
> be copied, and it works on file system level (instead of disk
> block level like dd). Initializing the target disk prior to
> starting the copy operation will make sure the partition data
> is being written to the correct places.
>
>

Thanks for your quick reply.

I would like to do the same thing with a win7 disk as source.
But on my 10.3 and 11.0-rc2 systems the mount_ntfs command is missing.
How do I mount a win7 disk in read mode so I can back it up?

[Polytropon]

On Tue, 04 Oct 2016 11:19:51 -0400, Ernie Luzar wrote:
> Polytropon wrote:
> > On Tue, 04 Oct 2016 09:52:11 -0400, Ernie Luzar wrote:
> >> I have a 10.3 box using a MBR 1.5 TB hard drive. This Box acts as the
> >> LANs front door firewall to the public network. The 1.5 TB disk is being
> >> waisted in this situation. What would you recommend as the quickest
> >> method to clone the running system from the 1.5TB disk to a 40MB disk?
> >
> > Preparing the target disk with MBR or GPT or Dedicated (use GPT
> > except you have a good reason not to), then use dump + restore.
> > You can find more information here:
> >
> > http://www.wonkity.com/~wblock/docs/html/disksetup.html#_the_new_standard_gpt
> >
> > http://www.wonkity.com/~wblock/docs/html/backup.html#_copying_filesystems
> >
> > The advantage of dump | restore is that only blocks in use will
> > be copied, and it works on file system level (instead of disk
> > block level like dd). Initializing the target disk prior to
> > starting the copy operation will make sure the partition data
> > is being written to the correct places.
> >
> >
>
> Thanks for your quick reply.
>
> I would like to do the same thing with a win7 disk as source.
> But on my 10.3 and 11.0-rc2 systems the mount_ntfs command is missing.
> How do I mount a win7 disk in read mode so I can back it up?

You need to install FUSE and then use the "ntfs3g <dev> <dir>"
command to mount it. The traditional mount_ntfs "read only"
NTFS mount binary is not part of the system anymore, sadly.

Please note: The dump | restore mechanism mentioned does work
on the filesystem level (here: UFS), not on the file level
(like "copying individual files and directory trees"). I'm
not sure this is the best way to deal with "Windows"
filesystems (no matter if you use cp or rsync) as they are
treated as virtual file systems (like "mapping NTFS into UFS").
So you maybe better DEFRAG and dd. ;-)

--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...

[Jack L.]

have you tried gpart backup src|gpart restore target?

That's usually how I clone my partitions

> To unsubscribe, send any mail to "freebsd-questions-unsubscribe
> @freebsd.org"

[Olivier]

Ernie Luzar <luza...@gmail.com> writes:

> I have a 10.3 box using a MBR 1.5 TB hard drive. This Box acts as the
> LANs front door firewall to the public network. The 1.5 TB disk is being
> waisted in this situation. What would you recommend as the quickest
> method to clone the running system from the 1.5TB disk to a 40MB disk?

You may want to have a look at
https://www.cs.ait.ac.th/~on/technotes/archives/2015/01/30/how-to_clone_a_freebsd_virtual_machine_on_vmware/index.html

Igf you forget the part about virtual machine, the part about cloning is
there and quite complete.

I wrote that as a guide for myself, putting my personal notes one a nice
page, so I can refer to it whenever I need it. But if it can help you.

Best regards,

Olivier

[Alejandro Imass]

On Tue, Oct 4, 2016 at 8:26 PM, Jack L. <xxjac...@gmail.com> wrote:
> have you tried gpart backup src|gpart restore target?
>
> That's usually how I clone my partitions
>
> On Mon, Oct 3, 2016 at 3:41 PM, Warren Block <wbl...@wonkity.com> wrote:
>

Hey guys great discussion. Thanks for following up and all the ideas!

I haven't done anything yet researching and making sure it will work
OK and I end up with a stable syste.

Most of my stuff is in jails anyway with EzJail so I'm thinking that
if I upgrade my old system to exactly the version of the target
installation the ezjail archive and restore feature will work well. I
have done this before and works like a charm so long as both systems
are prtty much the same version and equally updated. So I'm still
undecided as to what the best approach would be at this point. I'm
inclined for the ezjail archive restore right now as I know for a fact
that it works.


Best,
Alejandro Imass

[Warren Block]

On Wed, 5 Oct 2016, Alejandro Imass wrote:

> On Tue, Oct 4, 2016 at 8:26 PM, Jack L. <xxjac...@gmail.com> wrote:
>> have you tried gpart backup src|gpart restore target?
>>
>> That's usually how I clone my partitions
>>
>> On Mon, Oct 3, 2016 at 3:41 PM, Warren Block <wbl...@wonkity.com> wrote:
>>
>
> Hey guys great discussion. Thanks for following up and all the ideas!
>
> I haven't done anything yet researching and making sure it will work
> OK and I end up with a stable syste.
>
> Most of my stuff is in jails anyway with EzJail so I'm thinking that
> if I upgrade my old system to exactly the version of the target
> installation the ezjail archive and restore feature will work well. I
> have done this before and works like a charm so long as both systems
> are prtty much the same version and equally updated. So I'm still
> undecided as to what the best approach would be at this point. I'm
> inclined for the ezjail archive restore right now as I know for a fact
> that it works.

ezjail is filesystem-based, so tools that deal with filesystems are
appropriate. The built-in backup and restore should do fine.