Re: Backshift checkout

[Saint Germain]

On Thu, 31 May 2012 10:51:52 +0200, Saint Germain <sain...@gmail.com>
wrote :
> On 31 May 2012 01:10, Dan Stromberg <stro...@gmail.com> wrote:
> > I've just added a page about ./configure'ing and installing
> > backshift:
> > http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html ...but
> > it's mostly the usual "./configure && make install". �This on the
> > website, and in trunk, but was added after 1.01 was cut.
> >

Hello,

I have some problem building Backshift version 1.01 on Debian Testing
(with current Cpython).

1) it seems that exuberant-ctags and pylint are necessary
dependencies.

2) I have to change this_pylint:
--- this-pylint (r�vision 2361)
+++ this-pylint (copie de travail)
@@ -1,4 +1,4 @@
-#!/usr/local/cpython-3.2/bin/python3
+#!/usr/bin/env python

import os
import re
@@ -80,7 +80,7 @@

significant_found = False
messages_of_interest = []
- for pylint in [ '/usr/local/cpython-2.5/bin/pylint', '/usr/local/cpython-3.2/bin/pylint', ]:
+ for pylint in [ '/usr/bin/pylint',]:
if os.path.exists(pylint):
command = ('%s ' % pylint) + \
('--init-hook=\'import sys; sys.path.append("%s");
sys.path.append(".")\' ' % os.path.expanduser('~/lib')) +

3) I finally managed to get it started, but I got the following error:

./this-pylint --to-pylint backshift backshift_file_mod.py base255.py bloom_filter_mod.py bufsock.py cacher_mod.py chunk_mod_traditional.py compressed_file_mod.py compressed_string_mod.py constants_mod.py dohdbm.py dirops_mod.py escape_mod.py expire_mod.py file_count_mod.py hardlinks_mod.py main.py metadata_mod.py modunits.py py_treap_node.py py_treap.py readline0.py repo_mod.py rolling_checksum_py_mod.py saveset_files_mod.py saveset_summary_mod.py try_psyco.py xz_mod.py
************* Module backshift_file_mod
I:102,0: Locally disabling R0903
************* Module bloom_filter_mod
I: 48,0: Locally disabling W0232
I:240,0: Locally disabling R0902
I:500,0: Locally disabling R0913
************* Module bufsock
I:123,0: Locally disabling R0902
W:236,0: FIXME: This could be sped up a bit using slicing
************* Module chunk_mod_traditional
W:148,0: FIXME: On Jython 2.5.2 r7288, os.utime fails to raise an OSError if the file doesn't exist
************* Module compressed_file_mod
I: 22,0: Locally disabling R0913
W: 27,0: FIXME: We ignore perms - that's pretty safe for backshift, but should be examined if reused in other projects
************* Module constants_mod
I: 9,0: Locally disabling W0232
I: 9,0: Locally disabling R0903
************* Module dohdbm
I: 38,0: Locally disabling W0622
I: 61,0: Locally disabling R0902
I: 61,0: Locally disabling R0912
************* Module dirops_mod
I: 14,0: Locally disabling W0142
************* Module file_count_mod
I: 8,0: Locally disabling W0232
I: 8,0: Locally disabling R0903
************* Module hardlinks_mod
I: 29,0: Locally disabling R0903
************* Module main
I: 74,0: Locally disabling R0903
I:137,0: Locally disabling R0913
I:137,0: Locally disabling R0902
I:187,0: Locally disabling R0912
************* Module metadata_mod
I: 21,0: Locally disabling R0903
I:167,0: Locally disabling W0232
************* Module modunits
I: 40,0: Locally disabling R0903
I: 54,0: Locally disabling R0913
I:231,0: Locally disabling R0912
I:340,0: Locally disabling R0902
************* Module py_treap_node
I:146,0: Locally disabling R0201
W: 9,0: FIXME: nice for debugging, bad for real life
************* Module py_treap
I:198,0: Locally disabling R0913
I:249,0: Locally disabling R0903
************* Module readline0
I: 20,0: Locally disabling R0902
************* Module repo_mod
I: 35,0: Locally disabling E1101
I: 62,0: Locally disabling W0603
I:112,0: Locally disabling R0903
I:119,0: Locally disabling R0201
I:140,0: Locally disabling W0142
I:376,0: Locally disabling R0902
I:510,0: Locally disabling R0914
W: 79,0: FIXME: This might actually be erring too much on the safe side.
W:488,0: FIXME: Now we should optionally go through hardlink_data and warn about any hardlinks that don't show an ideal count
W:499,0: FIXME: Now we should optionally go through tar_state.hardlink_data and warn about any hardlinks that
W:605,0: FIXME: This 0.01 should probably be a user-specifiable parameter or something
************* Module rolling_checksum_py_mod
I: 51,0: Locally disabling R0912
I: 51,0: Locally disabling R0914
I:147,0: Locally disabling R0902
I:156,0: Locally disabling R0913
W: 12,0: Found indentation with tabs instead of spaces
W: 14,0: Found indentation with tabs instead of spaces
W: 15,0: Found indentation with tabs instead of spaces
W: 16,0: Found indentation with tabs instead of spaces
W: 18,0: Found indentation with tabs instead of spaces
W: 19,0: Found indentation with tabs instead of spaces
W: 20,0: Found indentation with tabs instead of spaces
W: 21,0: Found indentation with tabs instead of spaces
W: 22,0: Found indentation with tabs instead of spaces
W: 23,0: Found indentation with tabs instead of spaces
W: 25,0: Found indentation with tabs instead of spaces
W: 27,0: Found indentation with tabs instead of spaces
W: 29,0: Found indentation with tabs instead of spaces
W: 31,0: Found indentation with tabs instead of spaces
W: 33,0: Found indentation with tabs instead of spaces
W: 34,0: Found indentation with tabs instead of spaces
W: 35,0: Found indentation with tabs instead of spaces
W: 37,0: Found indentation with tabs instead of spaces
W: 39,0: Found indentation with tabs instead of spaces
W: 40,0: Found indentation with tabs instead of spaces
W: 41,0: Found indentation with tabs instead of spaces
W: 43,0: Found indentation with tabs instead of spaces
W: 45,0: Found indentation with tabs instead of spaces
W: 46,0: Found indentation with tabs instead of spaces
W: 55,0: Found indentation with tabs instead of spaces
W: 65,0: Found indentation with tabs instead of spaces
W: 68,0: Found indentation with tabs instead of spaces
W: 69,0: Found indentation with tabs instead of spaces
W: 71,0: Found indentation with tabs instead of spaces
W: 73,0: Found indentation with tabs instead of spaces
W: 74,0: Found indentation with tabs instead of spaces
W: 76,0: Found indentation with tabs instead of spaces
W: 78,0: Found indentation with tabs instead of spaces
W: 80,0: Found indentation with tabs instead of spaces
W: 81,0: Found indentation with tabs instead of spaces
W: 82,0: Found indentation with tabs instead of spaces
W: 84,0: Found indentation with tabs instead of spaces
W: 85,0: Found indentation with tabs instead of spaces
W: 87,0: Found indentation with tabs instead of spaces
W: 89,0: Found indentation with tabs instead of spaces
W: 90,0: Found indentation with tabs instead of spaces
W: 91,0: Found indentation with tabs instead of spaces
W: 92,0: Found indentation with tabs instead of spaces
W: 93,0: Found indentation with tabs instead of spaces
W: 94,0: Found indentation with tabs instead of spaces
W: 96,0: Found indentation with tabs instead of spaces
W: 97,0: Found indentation with tabs instead of spaces
W: 98,0: Found indentation with tabs instead of spaces
W: 99,0: Found indentation with tabs instead of spaces
W:101,0: Found indentation with tabs instead of spaces
W:103,0: Found indentation with tabs instead of spaces
W:104,0: Found indentation with tabs instead of spaces
W:105,0: Found indentation with tabs instead of spaces
W:106,0: Found indentation with tabs instead of spaces
W:107,0: Found indentation with tabs instead of spaces
W:109,0: Found indentation with tabs instead of spaces
W:110,0: Found indentation with tabs instead of spaces
W:111,0: Found indentation with tabs instead of spaces
W:112,0: Found indentation with tabs instead of spaces
W:114,0: Found indentation with tabs instead of spaces
W:116,0: Found indentation with tabs instead of spaces
W:118,0: Found indentation with tabs instead of spaces
W:119,0: Found indentation with tabs instead of spaces
W:121,0: Found indentation with tabs instead of spaces
W:122,0: Found indentation with tabs instead of spaces
W:124,0: Found indentation with tabs instead of spaces
W:127,0: Found indentation with tabs instead of spaces
W:129,0: Found indentation with tabs instead of spaces
W:130,0: Found indentation with tabs instead of spaces
W:131,0: Found indentation with tabs instead of spaces
W:150,0: Found indentation with tabs instead of spaces
W:155,0: Found indentation with tabs instead of spaces
W:158,0: Found indentation with tabs instead of spaces
W:159,0: Found indentation with tabs instead of spaces
W:160,0: Found indentation with tabs instead of spaces
W:161,0: Found indentation with tabs instead of spaces
W:162,0: Found indentation with tabs instead of spaces
W:163,0: Found indentation with tabs instead of spaces
W:164,0: Found indentation with tabs instead of spaces
W:166,0: Found indentation with tabs instead of spaces
W:167,0: Found indentation with tabs instead of spaces
W:168,0: Found indentation with tabs instead of spaces
W:169,0: Found indentation with tabs instead of spaces
W:171,0: Found indentation with tabs instead of spaces
W:172,0: Found indentation with tabs instead of spaces
W:173,0: Found indentation with tabs instead of spaces
W:174,0: Found indentation with tabs instead of spaces
W:175,0: Found indentation with tabs instead of spaces
W:177,0: Found indentation with tabs instead of spaces
W:178,0: Found indentation with tabs instead of spaces
W:179,0: Found indentation with tabs instead of spaces
************* Module saveset_files_mod
I: 28,0: Locally disabling W0142
I: 54,0: Locally disabling R0903
************* Module saveset_summary_mod
I:183,0: Locally disabling R0902
I:188,0: Locally disabling R0913
W:264,0: FIXME: Is this method needed for anything?
************* Module try_psyco
I: 4,0: Locally disabling F0401
************* Module xz_mod
I: 3,0: Locally disabling W0404
I: 3,0: Locally disabling F0401
I:256,0: Locally disabling R0914
I:349,0: Locally disabling W0142
E:339,42:compress: Module 'lzma' has no 'FORMAT_XZ' member
E:339,65:compress: Module 'lzma' has no 'PRESET_DEFAULT' member
E:339,92:compress: Module 'lzma' has no 'CHECK_CRC32' member
E:344,44:decompress: Module 'lzma' has no 'FORMAT_XZ' member
make: *** [static-analysis] Erreur 1

Hope that it helps,

[Dan Stromberg]

Hi.

On Thu, May 31, 2012 at 11:46 AM, Saint Germain <sain...@gmail.com> wrote:

On Thu, 31 May 2012 10:51:52 +0200, Saint Germain <sain...@gmail.com>
wrote :

> On 31 May 2012 01:10, Dan Stromberg <stro...@gmail.com> wrote:

> > I've just added a page about ./configure'ing and installing
> > backshift:
> > http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html ...but
> > it's mostly the usual "./configure && make install".  This on the
> > website, and in trunk, but was added after 1.01 was cut.
> >

Hello,

I have some problem building Backshift version 1.01 on Debian Testing
(with current Cpython).

1) it seems that exuberant-ctags and pylint are necessary
dependencies.

2) I have to change this_pylint:

--- this-pylint (révision 2361)

+++ this-pylint (copie de travail)
@@ -1,4 +1,4 @@
-#!/usr/local/cpython-3.2/bin/python3
+#!/usr/bin/env python

 import os
 import re
@@ -80,7 +80,7 @@

    significant_found = False
    messages_of_interest = []
-    for pylint in [ '/usr/local/cpython-2.5/bin/pylint', '/usr/local/cpython-3.2/bin/pylint', ]:
+    for pylint in [ '/usr/bin/pylint',]:
        if os.path.exists(pylint):
            command = ('%s ' % pylint) + \
                ('--init-hook=\'import sys; sys.path.append("%s");
sys.path.append(".")\' ' % os.path.expanduser('~/lib')) +

backshift actually doesn't have much of a build process, unless you want to use CPython with Cython - and even then there's not much to build.  The default make rule runs lots of tests.  You might be assured things are working by running this rule, but things should "just work" if you get the latest version from http://stromberg.dnsalias.org/~strombrg/backshift/documentation/ with the tests - look for "Downloads" at the URL.

1.01 had some lingering tab->spaces issues (I'm finally getting with the program and respecting pep8), but I believe 1.03 eliminates these.

Please try again with 1.03 at the above URL.

Thanks!

 

--
Dan Stromberg

[Saint Germain]

On Thu, 31 May 2012 12:08:12 -0700, Dan Stromberg <stro...@gmail.com>
wrote :

> Hi.
>
> On Thu, May 31, 2012 at 11:46 AM, Saint Germain <sain...@gmail.com>
> wrote:
>
> > On Thu, 31 May 2012 10:51:52 +0200, Saint Germain
> > <sain...@gmail.com> wrote :
> > > On 31 May 2012 01:10, Dan Stromberg <stro...@gmail.com> wrote:
> > > > I've just added a page about ./configure'ing and installing
> > > > backshift:
> > > >
> > http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html...but
> > > > it's mostly the usual "./configure && make install". This on
> > > > the website, and in trunk, but was added after 1.01 was cut.
> > > >
> >
> > Hello,
> >
> > I have some problem building Backshift version 1.01 on Debian
> > Testing (with current Cpython).
> >
> > 1) it seems that exuberant-ctags and pylint are necessary
> > dependencies.
> >
> > 2) I have to change this_pylint:

> > --- this-pylint (r�vision 2361)

Ok I skipped the build process and managed to install v1.03 directly.
I ran a backup of my home directory with minimal progress report (17 GB
and 80000 files) and I had to interrupt Backshift after 50 mn and only
1.1 GB in the backup directory and 8000 files processed.
I'm currently making some kind of benchmark among backup softwares with
deduplication and almost all managed to backup the 17 GB between 20 and
30 minutes (with a resulting backup directory of average 14 GB because
of compression).

Have you also noticed this "slow" behavior ?

By the way it seems that I have some kind of conflict because of the
python-lzma version.

I have to change like this:
diff ./xz_mod.py /home/test/local/lib/backshift/xz_mod.py
339c339
< return lzma.compress(data, format=lzma.FORMAT_XZ, preset=lzma.PRESET_DEFAULT, check=lzma.CHECK_CRC32)
---
> return lzma.compress(data)
344c344
< return lzma.decompress(data, format=lzma.FORMAT_XZ)
---
> return lzma.decompress(data)

I am using python-lzma 0.5.3-2+b1.

Hope that helps !

[Dan Stromberg]

On Thu, May 31, 2012 at 1:45 PM, Saint Germain <sain...@gmail.com> wrote:

On Thu, 31 May 2012 12:08:12 -0700, Dan Stromberg <stro...@gmail.com>
wrote :

> Hi.
>
> On Thu, May 31, 2012 at 11:46 AM, Saint Germain <sain...@gmail.com>
> wrote:
>
> > On Thu, 31 May 2012 10:51:52 +0200, Saint Germain
> > <sain...@gmail.com> wrote :
> > > On 31 May 2012 01:10, Dan Stromberg <stro...@gmail.com> wrote:
> > > > I've just added a page about ./configure'ing and installing
> > > > backshift:
> > > >
> > http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html...but
> > > > it's mostly the usual "./configure && make install".  This on
> > > > the website, and in trunk, but was added after 1.01 was cut.
> > > >
> >
> > Hello,
> >
> > I have some problem building Backshift version 1.01 on Debian
> > Testing (with current Cpython).
> >
> > 1) it seems that exuberant-ctags and pylint are necessary
> > dependencies.
> >
> > 2) I have to change this_pylint:

> > --- this-pylint (révision 2361)

Yes, backshift can be slow for any given host.

It's pretty good at doing multiple hosts in parallel (they even see each other's recent changes), but doing a single host will be kind of slow, especially if it has scads of tiny files.

This is in part because it's creating a lot of directories and files for the first fullsave, and partly because xz is a slow compression algorithm - one that happens to compress to a smaller end result than most other compressors.

Another likely cause of slowness would be using CPython without the Cython augmentation.  But Pypy will blow the doors off even the Cython, so if performance is a concern, I'd suggest downloading Pypy 1.8 and configuring to use that; I do all my backups with Pypy.

By the way it seems that I have some kind of conflict because of the
python-lzma version.

I have to change like this:
diff ./xz_mod.py /home/test/local/lib/backshift/xz_mod.py
339c339
<         return lzma.compress(data, format=lzma.FORMAT_XZ, preset=lzma.PRESET_DEFAULT, check=lzma.CHECK_CRC32)
---
>         return lzma.compress(data)
344c344
<         return lzma.decompress(data, format=lzma.FORMAT_XZ)
---
>         return lzma.decompress(data)

I am using python-lzma 0.5.3-2+b1.

Interesting.  It sounds like the new lzma module in CPython 3.3 is incompatible with python-lzma...  If you're not using python-lzma, you could try deleting it.

I mostly use liblzma via ctypes.

Thanks!

--
Dan Stromberg

[Dan Stromberg]

On Thu, May 31, 2012 at 3:53 PM, Dan Stromberg <stro...@gmail.com> wrote:

Have you also noticed this "slow" behavior ?

Yes, backshift can be slow for any given host.

I thought I'd run these past you, just to make sure they don't get missed:

1. http://stromberg.dnsalias.org/~strombrg/backshift/documentation/performance/
2. http://stromberg.dnsalias.org/~strombrg/backshift/documentation/interpreters/index.html
3. http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html

But even if you go with pypy for speed, the xz compression will still make your first backup kinda slow.  Small, but slow.

--
Dan Stromberg

[Saint Germain]

On Thu, 31 May 2012 19:11:48 -0700, Dan Stromberg <stro...@gmail.com>
wrote :

> On Thu, May 31, 2012 at 3:53 PM, Dan Stromberg <stro...@gmail.com>
> wrote:
> >
> > Have you also noticed this "slow" behavior ?
> >>
> > Yes, backshift can be slow for any given host.
> >
>
> I thought I'd run these past you, just to make sure they don't get
> missed:
>

> 1.
> http://stromberg.dnsalias.org/~strombrg/backshift/documentation/performance/
> 2.
> http://stromberg.dnsalias.org/~strombrg/backshift/documentation/interpreters/index.html
> 3.

> http://stromberg.dnsalias.org/~strombrg/backshift/documentation/for-all/installation.html
>
> But even if you go with pypy for speed, the xz compression will still
> make your first backup kinda slow. Small, but slow.
>

I tried with Pypy 1.7 (available on Debian) and I got an error after
2h40 and 3.9 GB (directory to backup is 17 GB and 80000 files):

Traceback (most recent call last):
File "app_main.py", line 51, in run_toplevel
File "/home/test/local/lib/backshift/backshift", line 29, in <module>
top()
File "/home/test/local/lib/backshift/backshift", line 25, in top
main.main()
File "/home/test/local/lib/backshift/main.py", line 545, in main
backup_minimal_stats(repo, directory_at_outset)
File "/home/test/local/lib/backshift/main.py", line 465, in backup_minimal_stats
repo.put_filename(filename)
File "/home/test/local/lib/backshift/repo_mod.py", line 640, in put_filename
backshift_file = backshift_file_mod.Backshift_file(repo=self, file_=lstat_result, filename=filename, verbose=verbose)
File "/home/test/local/lib/backshift/backshift_file_mod.py", line 134, in __init__
speed = self.init_from_lstat_result(lstat_result=file_)
File "/home/test/local/lib/backshift/backshift_file_mod.py", line 170, in init_from_lstat_result
speed = self.process_from_lstat(lstat_result)
File "/home/test/local/lib/backshift/backshift_file_mod.py", line 182, in process_from_lstat
speed = self.repo.save_chunks(lstat_result, self.filename, self.dict_)
File "/home/test/local/lib/backshift/repo_mod.py", line 620, in save_chunks
return '%*s' % (constants_mod.Constants.file_type_width, generate_and_save_chunks(lstat_result, filename, dict_))
File "/home/test/local/lib/backshift/repo_mod.py", line 302, in generate_and_save_chunks
digest, length_written = save_chunk_at_hash(chunk)
File "/home/test/local/lib/backshift/repo_mod.py", line 272, in save_chunk_at_hash
length_written = chunk.write_or_touch_chunk(chunk_bytes)
File "/home/test/local/lib/backshift/chunk_mod_traditional.py", line 113, in write_or_touch_chunk
dirops_mod.my_mkdir(self.all_dirs_list)
File "/home/test/local/lib/backshift/dirops_mod.py", line 45, in my_mkdir
os.mkdir(partial_relative_path, 7 * 64 + 5 * 8 + 5)
OSError: [Errno 31] Too many links: 'chunks/5f7c0'
Command exited with non-zero status 1

Hope that it will help you.

Regards,

[Dan Stromberg]

It sure does.

What filesystem are you using for your backshift repo?  Backshift can be a bit demanding of a filesystem. 

I personally use ext4, and I believe I once used XFS over NFS (on a fileserver someone else was maintaining).

ext3 probably won't work - I don't think it allows enough directory entries in a single directory.

--
Dan Stromberg

[Saint Germain]

On Fri, 1 Jun 2012 10:14:49 -0700, Dan Stromberg <stro...@gmail.com>
wrote :

> What filesystem are you using for your backshift repo? Backshift can
> be a bit demanding of a filesystem.
>
> I personally use ext4, and I believe I once used XFS over NFS (on a
> fileserver someone else was maintaining).
>
> ext3 probably won't work - I don't think it allows enough directory
> entries in a single directory.
>

I am using ext3. I'll try on an ext4 partition, but that will take at
least one week (very busy this we).

Thanks,

[Saint Germain]

On Fri, 1 Jun 2012 10:14:49 -0700, Dan Stromberg <stro...@gmail.com>

wrote :

> On Fri, Jun 1, 2012 at 9:31 AM, Saint Germain <sain...@gmail.com>
> wrote:
>
> > On Thu, 31 May 2012 19:11:48 -0700, Dan Stromberg
> > <stro...@gmail.com> wrote :
> >
> > > On Thu, May 31, 2012 at 3:53 PM, Dan Stromberg
> > > <stro...@gmail.com> wrote:
> > > >
> > > > Have you also noticed this "slow" behavior ?
> > > >>
> > > > Yes, backshift can be slow for any given host.
> > > >
> > >
> > > I thought I'd run these past you, just to make sure they don't get
> > > missed:
> > >

> > > [snip]

> > >
> > > But even if you go with pypy for speed, the xz compression will
> > > still make your first backup kinda slow. Small, but slow.
> > >
> >
> > I tried with Pypy 1.7 (available on Debian) and I got an error after
> > 2h40 and 3.9 GB (directory to backup is 17 GB and 80000 files):
> >

> > [snip]

> >
> > Hope that it will help you
> >
> It sure does.
>
> What filesystem are you using for your backshift repo? Backshift can
> be a bit demanding of a filesystem.
>
> I personally use ext4, and I believe I once used XFS over NFS (on a
> fileserver someone else was maintaining).
>
> ext3 probably won't work - I don't think it allows enough directory
> entries in a single directory.
>

Hello,

I finally have some time to continue. Sorry for not having get back to
you sooner.
So I ended up creating an artificial home directory in order to have a
reproductible setup by everybody (24 GB, 10 000 files).

I configured Backshift (1.03) with Pypy 1.7 on an ext4 partition and I
still don't manage to backup in a reasonable time (with LZMA I
interrupted after 20 hours and 2000 files, with BZIP2 I interrupted
after 9 hours and 2000 files). The data were on an internal disk and
the backup directory on a USB2 external hard drive.

Do you think that I can improve the speed (maybe I've not installed
correctly ?) or is it currently a more or less expected behavior ?

If you want to test it yourself, here is the code generating the "home
directory" to backup (I'm a bit ashamed of the code, but I needed
something done quickly).
To give you an idea, bup (currently the faster) managed to backup in 33
mn.

Best regards,

import random
import os

random.seed(0)

def random_line():
return ("%1.10f" % random.random())[2:]

def random_file(path, size):
fp = open(path,"wb")
for i in xrange(size/10):
fp.write("%s\n" % random_line())
fp.close()

def random_filename():
return random_line()

def generate_random_files(sequence, size):
if len(sequence) == 1:
for i in xrange(sequence[0]):
path = random_filename()
random_file(path, size)
else:
for i in xrange(sequence[0]):
path = random_filename()
os.mkdir(path)
os.chdir(path)
generate_random_files(sequence[1:], size)
os.chdir('..')

generate_random_files([10, 10, 10, 10], 1000000)
generate_random_files([10, 1], 500000000)
generate_random_files([ 3, 1], 2000000000)

[Dan Stromberg]

It may well turn out that backshift just isn't as fast as bup, but I'm looking into this.

What commands have you used for your bup backup and repo creation?

What commands have you used for your backshift backup and repo creation?

Thanks!

--
Dan Stromberg

[Saint Germain]

On Sat, 9 Jun 2012 07:18:23 -0700, Dan Stromberg <stro...@gmail.com>

It is not just bup, but also BURP, Obnam and BackupPC.
They all are in the same range (30mn-1h30), and Backshift is the only
one having 40h-100h.
So I suspect some kind of misconfiguration.

> What commands have you used for your bup backup and repo creation?

For bup:
BUP_DIR=/media/usb0/backups/bup bup index -uv /media/usb0/random
BUP_DIR=/media/usb0/backups/bup bup save -n test /media/usb0/random

> What commands have you used for your backshift backup and repo
> creation?
>

For Backshift (in /media/usb0/random):
find . -xdev -print0 | backshift --save-directory /media/usb1/backshift --backup --subset slash --init-savedir --progress-report minimal

Let me know if I can help in any way to solve this.

Regards,

[Dan Stromberg]

I'm investigating, but I think I'll reiterate up front: I didn't particularly design backshift for single-backup speed...

--
Dan Stromberg