TortoiseHG - Large File Commit (only 40Meg is size)

[mutface]

Dear sirs,

I have looked for 3 hours online before making this posting and please
forgive me if I have missed something really simple but please advise what
I'm doing wrong.

I'm on
Windows XP 32-bit
1.5 Gb of RAM
TortoiseHg (version 1.1.7)
with Mercurial-1.7.2, Python-2.6.4, PyGTK-2.16.0, GTK-2.16.6

I have been trying to commit a .psd of only 40Meg through TortoiseHg. The
command that seems to be run is hg commit --verbose --user xxxx

However it keeps giving me an error message:-
trouble committing content/maps/xyz.psd!
transaction abort!
rollback completed
[Errno 22] Invalid argument

[command interrupted]

Can anyone please advise whether TortoiseHg has a limit on file size and how
I can get rid of this. I have tried changing "Max files" under "web server"
within settings and other parameters but to no avail and moreover it seems
there really isn't that much documentation out there .

Thanks in advance.

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2140178.html
Sent from the General mailing list archive at Nabble.com.
_______________________________________________
Mercurial mailing list
Merc...@selenic.com
http://selenic.com/mailman/listinfo/mercurial

[Matt Mackall]

On Thu, 2010-12-23 at 22:22 -0800, mutface wrote:
> Dear sirs,
>
> I have looked for 3 hours online before making this posting and please
> forgive me if I have missed something really simple but please advise what
> I'm doing wrong.
>
> I'm on
> Windows XP 32-bit
> 1.5 Gb of RAM
> TortoiseHg (version 1.1.7)
> with Mercurial-1.7.2, Python-2.6.4, PyGTK-2.16.0, GTK-2.16.6
>
> I have been trying to commit a .psd of only 40Meg through TortoiseHg. The
> command that seems to be run is hg commit --verbose --user xxxx
>
> However it keeps giving me an error message:-
> trouble committing content/maps/xyz.psd!
> transaction abort!
> rollback completed
> [Errno 22] Invalid argument

Trying to commit to a network share rather than a local disk?

Windows' network I/O handling here is.. not exactly enterprise-class.

--
Mathematics is the supreme nostalgia of our time.

[Steve Borho]

On Fri, Dec 24, 2010 at 12:22 AM, mutface <mutfa...@gmail.com> wrote:
>
> Dear sirs,
>
> I have looked for 3 hours online before making this posting and please
> forgive me if I have missed something really simple but please advise what
> I'm doing wrong.
>
> I'm on
> Windows XP 32-bit
> 1.5 Gb of RAM
> TortoiseHg (version 1.1.7)
> with Mercurial-1.7.2, Python-2.6.4, PyGTK-2.16.0, GTK-2.16.6
>
> I have been trying to commit a .psd of only 40Meg through TortoiseHg. The
> command that seems to be run is hg commit --verbose --user xxxx
>
> However it keeps giving me an error message:-
> trouble committing content/maps/xyz.psd!
> transaction abort!
> rollback completed
> [Errno 22] Invalid argument
>
> [command interrupted]
>
>
> Can anyone please advise whether TortoiseHg has a limit on file size and how
> I can get rid of this. I have tried changing "Max files" under "web server"
> within settings and other parameters but to no avail and moreover it seems
> there really isn't that much documentation out there .

There is no limit to the size of files being committed. There is a
max diff size, we will not try to show diffs for files larger than a
certain size, but it has no effect on commit.

Performing the commit on the command line with --traceback would give
us some more useful data.

--
Steve Borho

[Matt Mackall]

We've seen this "invalid argument" traceback a few times. It's almost
certainly going to be at a write() in the revlog code, which is being
given a large string buffer that Windows' CIFS code then chokes on
because it's under-engineered.

--
Mathematics is the supreme nostalgia of our time.

[mutface]

Thanks you experts for writing. It is a local commit - not a push to the
server - so I get the above whilst I'm not even connected to the Internet.

What's the solution? 40Mb doesn't seem like a very large file... Any
pointers?
--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2142076.html

[Mads Kiilerich]

mutface wrote, On 12/24/2010 08:39 PM:
> Thanks you experts for writing. It is a local commit - not a push to the
> server - so I get the above whilst I'm not even connected to the Internet.
>
> What's the solution? 40Mb doesn't seem like a very large file... Any
> pointers?

You could try to add --traceback to find out exactly where it fails.

/Mads

[mutface]

Thanks Mads!

I ran it through the command line for the "hg add ....." and then the error
message was

"... up to 125 Mb RAM may be required to manage this file... "

I have 1.5Gb of RAM on my XP - how come it can't use more of the RAM?

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2146895.html

Sent from the General mailing list archive at Nabble.com.

[Mads Kiilerich]

mutface wrote, On 12/26/2010 09:45 AM:
> Thanks Mads!
>
> I ran it through the command line for the "hg add ....." and then the error
> message was
>
> "... up to 125 Mb RAM may be required to manage this file..."
>
> I have 1.5Gb of RAM on my XP - how come it can't use more of the RAM?

Mercurial can use more memory, and it just warns that future operations
will need that much memory. It is just a warning, not an error. The
message must have been there all the time.

It there is an issue then please paste the full command and output as it
appeared in your console window.

/Mads

[Matt Mackall]

On Sun, 2010-12-26 at 00:45 -0800, mutface wrote:
> Thanks Mads!
>
> I ran it through the command line for the "hg add ....." and then the error
> message was
>
> "... up to 125 Mb RAM may be required to manage this file... "

This is NOT the issue. Repeat: NOT the issue. If we were out of memory,
the error message would be "abort: out of memory".

As I said before, the issue is likely that some versions of the Windows
kernel sometimes chokes on large I/O requests, despite there being
plenty of available memory.

If an application has a 1G buffer and it asks the kernel to write it all
to a file, the kernel should internally break it into a number of small
requests the drive can handle in a loop (a typical older ATA drive can
handle no more than 128K per request). No more than one I/O request's
worth of extra memory is required here.

But in some circumstances, usually associated with writing to shared
disks, Windows is lazy and doesn't do this loop. Instead, if it gets a
request larger than its max internal I/O request size, it gives up and
returns an "invalid argument" return code. This return code is not
documented as a valid return code, nor is any size limit documented for
write(), nor is there any excuse for this kind of lazy coding in the
implementation of write(). This should never happen and is a bug in the
Windows kernel.

The traceback we've asked you now twice for (run your failing commit
command again with --traceback) will confirm this. But we won't know for
sure until you actually do that.

--
Mathematics is the supreme nostalgia of our time.

[mutface]

Hi Matt,

I'm using TortoiseHg through which I haven't worked out how to do
--traceback.

I've tried several times to run commit through command line on windows XP
with:-

hg commit --traceback --verbose --user "xxx" <em...@address.com> --message
abc --content c:\xxx\xxx\xxxx\

I get messages "The system cannot find file specified."

It seems to be a related issue with:-
http://efreedom.com/Question/1-3172303/TortoiseHg-Commit-System-Find-File-Specified

I'm sorry for the inconvenience but there doesn't seem to be much resource
on Google.

Apologies again for the stupidity

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2171495.html

Sent from the General mailing list archive at Nabble.com.

[Mads Kiilerich]

mutface wrote, On 12/31/2010 08:23 AM:
> Hi Matt,
>
> I'm using TortoiseHg through which I haven't worked out how to do
> --traceback.
>
> I've tried several times to run commit through command line on windows XP
> with:-
>
> hg commit --traceback --verbose --user "xxx"<em...@address.com> --message
> abc --content c:\xxx\xxx\xxxx\
>
> I get messages "The system cannot find file specified."

Please post the exact command you used and show us the full output. Just
cut'n'paste from your command prompt.

It seems like your quoting of the --user parameter is incorrect, but we
can't know if that error was introduced in your copy editing.

The name of the user is however usually configured globally and not
specified for each commit.

Mercurial also don't know any --content option. Where do that come from?

/Mads

[mutface]

Hi Mads,

I'm using TortoiseHg as my client and I do not type a single command.
Everything is done through TortoiseHg. TortoiseHg has a GUI and all commits,
push, etc are effected through it.

Anyways, I just tried using command prompt and here is what I got:-

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

s:\xyz\>hg commit --traceback -m "upload maps"
trouble committing content/maps/kwai_chung1.psd!
transaction abort!
rollback completed
Traceback (most recent call last):
File "mercurial\dispatch.pyo", line 58, in _runcatch
File "mercurial\dispatch.pyo", line 590, in _dispatch
File "mercurial\dispatch.pyo", line 401, in runcommand
File "mercurial\dispatch.pyo", line 641, in _runcommand
File "mercurial\dispatch.pyo", line 595, in checkargs
File "mercurial\dispatch.pyo", line 588, in <lambda>
File "mercurial\util.pyo", line 427, in check
File "mercurial\commands.pyo", line 780, in commit
File "mercurial\cmdutil.pyo", line 1333, in commit
File "mercurial\commands.pyo", line 775, in commitfunc
File "mercurial\localrepo.pyo", line 962, in commit
File "mercurial\localrepo.pyo", line 1009, in commitctx
File "mercurial\localrepo.pyo", line 781, in _filecommit
File "mercurial\context.pyo", line 906, in data
File "mercurial\localrepo.pyo", line 615, in wread
IOError: [Errno 22] Invalid argument
abort: Invalid argument

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2175506.html

Sent from the General mailing list archive at Nabble.com.

[Adrian Buehlmann]

On 2011-01-01 10:46, mutface wrote:
>
> Hi Mads,
>
> I'm using TortoiseHg as my client and I do not type a single command.
> Everything is done through TortoiseHg. TortoiseHg has a GUI and all commits,
> push, etc are effected through it.

Yes. I think we got that by now. But most Mercurial developers don't
know much about the TortoiseHg GUI and if they ask you to try something
on the command line, then please do so. Using the command line hg
command makes sure that the smallest possible amount of software is
used, which helps to nail down problems.

As you demonstrate below, the traceback can be produced with the hg
command line command alone.

(note: I assume this is still with Mercurial-1.7.2 on Windows XP 32-bit
with 1.5 Gb of RAM, as mutface noted in their initial post).

mutface: perhaps you can tell us a bit more about your drive s:?

Is s: a local drive? If yes, what file system is it using? NTFS? On what
hardware is that? Harddisk? Have you installed any special device drivers?

Or is s: a Windows share? If yes, what software is s: serving? Windows
or Samba? What versions exactly?

Are you using any on-access antivirus scanners? If yes what's the
product name?

[mutface]

Thanks Adrian.

Please understand that laymen like myself will invariably rely on GUIs
rather than type commands so please forgive me if I may seem stupid. We
intend to use Hg for predominantly binary files so TortoiseHg is quite
useful for our purposes.

As for the question regarding s: drive.

The XP box is running on Virtual Machine (Virtual Box) with an Opensuse
host; the s: drive is a mapped drive which maps to a "Shared Folder" that is
set up within Virtual Box and maps to something like \\vboxsvr\hg\

I don't know whether XP running under VirtualBox will cause this problem;
however I have been able to commit plenty of other files of around 5Meg
(within the s: drive).

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2183219.html

Sent from the General mailing list archive at Nabble.com.

[Adrian Buehlmann]

On 2011-01-03 06:09, mutface wrote:
>
> Thanks Adrian.
>
> Please understand that laymen like myself will invariably rely on GUIs
> rather than type commands so please forgive me if I may seem stupid. We
> intend to use Hg for predominantly binary files so TortoiseHg is quite
> useful for our purposes.

Sure. But debugging problems is a different thing.

> As for the question regarding s: drive.
>
> The XP box is running on Virtual Machine (Virtual Box) with an Opensuse
> host; the s: drive is a mapped drive which maps to a "Shared Folder" that is
> set up within Virtual Box and maps to something like \\vboxsvr\hg\
>
> I don't know whether XP running under VirtualBox will cause this problem;
> however I have been able to commit plenty of other files of around 5Meg
> (within the s: drive).

Just for the record: what versions of VirtualBox and openSUSE are you using?

I recommend not to place Mercurial repositories on such unusual drives.
Use normal local NTFS volumes to commit to.

UNC drives (paths of the form '\\foo\', aka Windows shares) are
problematic anyway, even without exotic parts like VirtualBox and
openSUSE in the loop (see [1]).

Make sure to check repository integrity by using 'hg verify' when moving
away from that drive. I hope you do have backups.

(As a side note: run hg verify when making a backups, so you know that
what you back up is ok. There's hardly anything more frustrating than
noticing your backup doesn't pass verify when you try to use it to
restore a lost repository.)

You probably want to test then if you can read and write big files from
your drive s: without TortoiseHg or Mercurial, since it looks like that
drive (read: the softwares servicing it) on itself might have problems
(I'd try testfiles >100 MB and use some hash software like MD5 to check
integrity of the files before and after each test copy action).

[1] http://mercurial.selenic.com/wiki/HardlinkedClones

[mutface]

Adrian Buehlmann wrote:
>
> On 2011-01-03 06:09, mutface wrote:
>>
>> Thanks Adrian.
>>
>> Please understand that laymen like myself will invariably rely on GUIs
>> rather than type commands so please forgive me if I may seem stupid. We
>> intend to use Hg for predominantly binary files so TortoiseHg is quite
>> useful for our purposes.
>
> Sure. But debugging problems is a different thing.
>
>> As for the question regarding s: drive.
>>
>> The XP box is running on Virtual Machine (Virtual Box) with an Opensuse
>> host; the s: drive is a mapped drive which maps to a "Shared Folder" that
>> is
>> set up within Virtual Box and maps to something like \\vboxsvr\hg\
>>
>> I don't know whether XP running under VirtualBox will cause this problem;
>> however I have been able to commit plenty of other files of around 5Meg
>> (within the s: drive).
>
> Just for the record: what versions of VirtualBox and openSUSE are you
> using?
>

>>> I'm currently using - Opensuse 11.3 (X86_64), VirtualBox - 3.2.8,
>>> running software RAID 1

>
> I recommend not to place Mercurial repositories on such unusual drives.
> Use normal local NTFS volumes to commit to.
>
> UNC drives (paths of the form '\\foo\', aka Windows shares) are
> problematic anyway, even without exotic parts like VirtualBox and
> openSUSE in the loop (see [1]).
>
> Make sure to check repository integrity by using 'hg verify' when moving
> away from that drive. I hope you do have backups.
>
> (As a side note: run hg verify when making a backups, so you know that
> what you back up is ok. There's hardly anything more frustrating than
> noticing your backup doesn't pass verify when you try to use it to
> restore a lost repository.)
>
> You probably want to test then if you can read and write big files from
> your drive s: without TortoiseHg or Mercurial, since it looks like that
> drive (read: the softwares servicing it) on itself might have problems
> (I'd try testfiles >100 MB and use some hash software like MD5 to check
> integrity of the files before and after each test copy action).
>

>>> I will try testing on normal NTFS drive to see if the issue will go
>>> away. At the moment probably no need >> to back up since I'm only using
>>> test data; nonetheless thanks for the advice regarding hg verify which
>>> >> will come in handy one day - I'm sure.

>
> [1] http://mercurial.selenic.com/wiki/HardlinkedClones
> _______________________________________________
> Mercurial mailing list
> Merc...@selenic.com
> http://selenic.com/mailman/listinfo/mercurial
>
>

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2184163.html

Sent from the General mailing list archive at Nabble.com.

[mutface]

Although S: drive may seem a bit unusual - I want to mention that using the
same kind of shared folder I have not had any problems with SVN committing
large binary files. Just mentioning this on an FYI basis.

--
View this message in context: http://mercurial.808500.n3.nabble.com/TortoiseHG-Large-File-Commit-only-40Meg-is-size-tp2140178p2184196.html

[Stefan Rindeskar]

Well it might work a couple of times with SVN but they also discourage
you from using shared folders due to high risk for possible problems
with locking, corrupt repository, etc.

Refs:
http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-repository.html#tsvn-repository-local-share
http://svnbook.red-bean.com/nightly/en/svn.serverconfig.choosing.html

/Stefan

[Matt Mackall]

On Sun, 2011-01-02 at 21:09 -0800, mutface wrote:
> Thanks Adrian.
>
> Please understand that laymen like myself will invariably rely on GUIs
> rather than type commands so please forgive me if I may seem stupid. We
> intend to use Hg for predominantly binary files so TortoiseHg is quite
> useful for our purposes.
>
> As for the question regarding s: drive.
>
> The XP box is running on Virtual Machine (Virtual Box) with an Opensuse
> host; the s: drive is a mapped drive which maps to a "Shared Folder" that is
> set up within Virtual Box and maps to something like \\vboxsvr\hg\

That's the "network share" I asked about on Dec 24th:

http://markmail.org/message/e7tpvqlo7o4r3ixo

It appears my initial analysis stands: your filesystem driver is buggy.
In this case it's probably Virtual Box's sharing code, which is rather
notorious.

Here's a quick Python script that'll find the I/O limit:

--------
import sys

fn = sys.argv[1]
good = 0
size = 2**20

while 1:
f = open(fn, "w")
a = None
a = "a" * size
print size
try:
f.write(a)
oldgood = good
good = size
size += good - oldgood
except IOError:
size = (good + size) / 2
if size == good:
break
f.close()
--------

Run as:

python writebig.py s:\bigfile

(You'll need a copy of Python)

--
Mathematics is the supreme nostalgia of our time.