bup remote reports error under cygwin

[Patrick Rouleau]

Hi,

When I use a samba share, bup runs without problem. However, it reports two kinds of errors when run with the remote option (-r).

bup reports a "permission denied" when I use a empty local repo:

Receiving index from server: 3756236/3756236, done.

bloom: creating from 5 files (541468 objects).

Traceback (most recent call last):

  File "/usr/lib/bup/cmd/bup-bloom", line 150, in <module>

    do_bloom(path, outfilename)

  File "/usr/lib/bup/cmd/bup-bloom", line 125, in do_bloom

    os.rename(tfname, outfilename)

OSError: [Errno 13] Permission denied

['/usr/bin/bup', 'bloom', '--dir', '/cygdrive/t/bup.remote/Fantom.bup/index-cache/tf02_bup_Storage_BUP_Fantom']: returned 1

bup continues to split after that error, but it complains again each time it receives an index. I do not see why. I am the sole user and the UAC is disabled (Vista x64). Cygwin created all the directories and the files seems to be fine. The remote is a Linux server (Fedora 17 x64).

Maybe the second error is a side-effect of the first. If I do a second save with the same data, bup fail at the end with this error:

Receiving index from server: 3786672/3786672, done.

bloom: adding 1 file (135200 objects).

Traceback (most recent call last):

  File "/usr/lib/bup/cmd/bup-save", line 298, in <module>

    keep_boundaries=False)

  File "/usr/lib/bup/bup/hashsplit.py", line 172, in split_to_blob_or_tree

    files, keep_boundaries))

  File "/usr/lib/bup/bup/hashsplit.py", line 160, in split_to_shalist

    for (sha,size,level) in sl:

  File "/usr/lib/bup/bup/hashsplit.py", line 114, in split_to_blobs

    sha = makeblob(blob)

  File "/usr/lib/bup/bup/git.py", line 584, in new_blob

    return self.maybe_write('blob', blob)

  File "/usr/lib/bup/bup/git.py", line 577, in maybe_write

    self._write(sha, type, content)

  File "/usr/lib/bup/bup/git.py", line 550, in _write

    sha=sha)

  File "/usr/lib/bup/bup/client.py", line 345, in _raw_write

    self.objcache.refresh()

  File "/usr/lib/bup/bup/git.py", line 402, in refresh

    mx = midx.PackMidx(full)

  File "/usr/lib/bup/bup/midx.py", line 22, in __init__

    self.map = mmap_read(open(filename))

  File "/usr/lib/bup/bup/helpers.py", line 527, in mmap_read

    return _mmap_do(f, sz, mmap.MAP_PRIVATE, mmap.PROT_READ, close)

  File "/usr/lib/bup/bup/helpers.py", line 517, in _mmap_do

    map = mmap.mmap(f.fileno(), sz, flags, prot)

mmap.error: [Errno 12] Cannot allocate memory

bloom: adding 1 file (12 objects).

Receiving index from server: 1408/1408, done.

bloom: adding 1 file (12 objects).

In both cases, "git log" does not show a commit. A new pack exist, but it is really small. I believe it only contain the tree since all the data has been saved by the first run. The data size is 352G (mostly archives, videos, mp3). The repo size if 335G total, with 360 idx for 1.3G.

I am running bup-v0.25-rc1-70-g4b6c8f0 on both side. Is there something I can look for?

Regards,

P.Rouleau

[Avery Pennarun]

On Sat, Jan 19, 2013 at 10:12 PM, Patrick Rouleau <proul...@gmail.com> wrote:
> When I use a samba share, bup runs without problem. However, it reports two
> kinds of errors when run with the remote option (-r).
>
> bup reports a "permission denied" when I use a empty local repo:
>
> Receiving index from server: 3756236/3756236, done.
> bloom: creating from 5 files (541468 objects).
> Traceback (most recent call last):
> File "/usr/lib/bup/cmd/bup-bloom", line 150, in <module>
> do_bloom(path, outfilename)
> File "/usr/lib/bup/cmd/bup-bloom", line 125, in do_bloom
> os.rename(tfname, outfilename)
> OSError: [Errno 13] Permission denied
> ['/usr/bin/bup', 'bloom', '--dir',
> '/cygdrive/t/bup.remote/Fantom.bup/index-cache/tf02_bup_Storage_BUP_Fantom']:
> returned 1

Generally this sort of problem occurs on Windows when you try to
rename or delete a file that's already in use. In Unix, that's a
valid operation (and very useful for atomically updating a file) but
in Windows it's not allowed. In this case it's probably an accident,
so look around in the bup code to see if it's remembering to close the
file in question before renaming it. (Remember, files sometimes are
closed because a variable goes out of scope, not due to an explicit
close() call.)

Have fun,

Avery

[Rob Browning]

Avery Pennarun <apen...@gmail.com> writes:

> Remember, files sometimes are closed because a variable goes out of
> scope, not due to an explicit close() call.

...and as an aside, if we come across any code like that, I'd like to
change it to explicitly close the files.

Though if I recall correctly, we'll need to use a finally block rather
than "with open() as x:" to remain compatible with older versions of
python.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Avery Pennarun]

On Mon, Jan 21, 2013 at 8:08 PM, Rob Browning <r...@defaultvalue.org> wrote:
> Avery Pennarun <apen...@gmail.com> writes:
>> Remember, files sometimes are closed because a variable goes out of
>> scope, not due to an explicit close() call.
>
> ...and as an aside, if we come across any code like that, I'd like to
> change it to explicitly close the files.

Yeah, people keep saying that, but I remain unconvinced. Explicitly
closing your file objects is just like admitting you never really
wanted a garbage collected language anyway :)

Seriously though, writing all your code with the scope of your
variables at the forefront of your thinking makes your code better -
it helps avoid circular references (which are disastrous for any GC),
and encourages writing "right sized" functions: if you're doing
something after a file has been opened and then closed, then your
function is probably too complicated.

Relatedly, keeping object references around when the object is
"closed" just strikes me as icky. The memory doesn't get released,
but the object itself is just hanging around wasting space.

So in the end, it's not explicitly closing files that bugs me... it's
the attitude that leads to *needing* to close the files. :) If you
let people explicitly close the files, it makes it easy for people to
retain that attitude, which makes things worse in more subtle (but
equally bad) ways.

(Aside to the aside: explicitly closing the files probably would not
have helped to avoid the problem with cygwin, because it's just as
likely people would have explicitly closed the file *after* doing the
rename, etc.)

Have fun,

Avery

[Simon Sapin]

Le 22/01/2013 06:27, Avery Pennarun a �crit :

> On Mon, Jan 21, 2013 at 8:08 PM, Rob Browning<r...@defaultvalue.org> wrote:
>> >Avery Pennarun<apen...@gmail.com> writes:
>>> >>Remember, files sometimes are closed because a variable goes out of
>>> >>scope, not due to an explicit close() call.
>> >
>> >...and as an aside, if we come across any code like that, I'd like to
>> >change it to explicitly close the files.
> Yeah, people keep saying that, but I remain unconvinced. Explicitly
> closing your file objects is just like admitting you never really
> wanted a garbage collected language anyway:)

Closing files explicitly is especially useful with non-refcounted
garbage collectors.

I�ve code had relying on CPython closing a file (and flushing write
operations) as soon as the variable goes out of scope, which became
buggy on PyPy where the writes were only done when the GC felt like it.

--
Simon Sapin

[Patrick Rouleau]

On Tuesday, January 22, 2013 12:27:48 AM UTC-5, apenwarr wrote:

On Mon, Jan 21, 2013 at 8:08 PM, Rob Browning <r...@defaultvalue.org> wrote:
> Avery Pennarun <apen...@gmail.com> writes:
>> Remember, files sometimes are closed because a variable goes out of
>> scope, not due to an explicit close() call.
>
> ...and as an aside, if we come across any code like that, I'd like to
> change it to explicitly close the files.

Yeah, people keep saying that, but I remain unconvinced.  Explicitly
closing your file objects is just like admitting you never really
wanted a garbage collected language anyway :)

 

Thank for the tips and recommendations. I am a pro in C* languages (C/C++, C# ;), but I'm new to Python, so they will really help to understand the code faster. 

Regards,

P.Rouleau

[Avery Pennarun]

On Tue, Jan 22, 2013 at 5:34 AM, Simon Sapin <simon...@exyr.org> wrote:

> Le 22/01/2013 06:27, Avery Pennarun a écrit :
>> On Mon, Jan 21, 2013 at 8:08 PM, Rob Browning<r...@defaultvalue.org>
>> wrote:
>>>
>>> >Avery Pennarun<apen...@gmail.com> writes:
>>>>
>>>> >>Remember, files sometimes are closed because a variable goes out of
>>>> >>scope, not due to an explicit close() call.
>>> >
>>> >...and as an aside, if we come across any code like that, I'd like to
>>> >change it to explicitly close the files.
>>
>> Yeah, people keep saying that, but I remain unconvinced. Explicitly
>> closing your file objects is just like admitting you never really
>> wanted a garbage collected language anyway:)
>
> Closing files explicitly is especially useful with non-refcounted garbage
> collectors.
>

> I’ve code had relying on CPython closing a file (and flushing write

> operations) as soon as the variable goes out of scope, which became buggy on
> PyPy where the writes were only done when the GC felt like it.

It's true. Personally though, I wish even more people would
"misbehave" and intentionally rely on the totally wonderful
deterministic GC in CPython, so that PyPy couldn't get away with
breaking it. Non-deterministic destructors are a pox on humanity and
one of the main reasons GCed languages that aren't python are sucky.

Extended edition of my rant: http://apenwarr.ca/log/?m=201008#10

Have fun,

Avery

[Rob Browning]

[Much of this goes in the category of things Avery probably already
knows...]

Avery Pennarun <apen...@gmail.com> writes:

> Yeah, people keep saying that, but I remain unconvinced. Explicitly
> closing your file objects is just like admitting you never really
> wanted a garbage collected language anyway :)

Well, I doubt I'm likely to be the one to convince you, and I'm not
familiar enough with Python's GC to say much there, but in other
languages, there's no guarantee that the GC will ever run at all[1].

That can lead to surprising (and sometimes catastrophic) results if
you're expecting the GC to close network ports, files, database
connections, windows etc. And at a minimum, you may end up with far
more debugging excitement than you'd like.

But perhaps Python has more explicit guarantees.

[1] ...and threaded or generational GCs add their own bits of
unpredictability.

> it helps avoid circular references (which are disastrous for any GC)

Hmm, in my experience circular references aren't all that uncommon in
(say) graph processing in the Lisp family, and are sometimes the most
convenient way to represent the problem. And in those languages, cycles
aren't considered unusual -- the GCs are designed to expect them.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> [Much of this goes in the category of things Avery probably already
> knows...]

Hah, so perhaps I understated that a bit. Fair enough.

> Avery Pennarun <apen...@gmail.com> writes:

> Extended edition of my rant: http://apenwarr.ca/log/?m=201008#10