Errno 31 Too many links in xmlrpclib while uploading tickets
Lukasz Szybalski
I am uploading tickets to trac. 500 a day. After almost a month+ I've
received this error.
Traceback (most recent call last):
File "xxx.py", line 115, in <module>
filename=server.ticket.putAttachment(number,file_name,file_name,xmlrpclib.Binary(open(os.path.join(folder_name,file_name)).read()))
File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
verbose=self.__verbose
File "/usr/lib/python2.5/xmlrpclib.py", line 1201, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
return u.close()
File "/usr/lib/python2.5/xmlrpclib.py", line 787, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 2: "'[Errno 31] Too many links:
'/home/trac/trac/attachments/ticket/32433'' while executing
'ticket.putAttachment()'">
Is there a way to close the xmlrpclib.ServerProxy. I think that is
what caused this error. Every time I upload a file I create a service
that connects to trac. I do that because the filename of the file I
want to upload comes on stdin.
Ideas?
Thanks,
Lucas
--
Using rsync. How to setup rsyncd.
http://lucasmanual.com/mywiki/rsync
OpenLdap - From start to finish.
http://lucasmanual.com/mywiki/OpenLdap
Lukasz Szybalski
> Hello,
> I am uploading tickets to trac. 500 a day. After almost a month+ I've
> received this error.
>
>
> Traceback (most recent call last):
> File "xxx.py", line 115, in <module>
> filename=server.ticket.putAttachment(number,file_name,file_name,xmlrpclib.Binary(open(os.path.join(folder_name,file_name)).read()))
> File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
> return self.__send(self.__name, args)
> File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
> verbose=self.__verbose
> File "/usr/lib/python2.5/xmlrpclib.py", line 1201, in request
> return self._parse_response(h.getfile(), sock)
> File "/usr/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
> return u.close()
> File "/usr/lib/python2.5/xmlrpclib.py", line 787, in close
> raise Fault(**self._stack[0])
> xmlrpclib.Fault: <Fault 2: "'[Errno 31] Too many links:
> '/home/trac/trac/attachments/ticket/32433'' while executing
> 'ticket.putAttachment()'">
>
>
>
> Is there a way to close the xmlrpclib.ServerProxy. I think that is
> what caused this error. Every time I upload a file I create a service
> that connects to trac. I do that because the filename of the file I
> want to upload comes on stdin.
I found this:
http://www.wlug.org.nz/EMLINK
Too Many Links
There are too many hard links to a file or directory on a filesystem.
The exact number allowed is file-system dependent. Eg
/usr/src/linux-2.4.19/include/linux/sysv_fs.h contains
enum {
XENIX_LINK_MAX = 126, /* ?? / SYSV_LINK_MAX = 126, / 127? 251?
/ V7_LINK_MAX = 126, / ?? */ COH_LINK_MAX = 10000,
};
while /usr/src/linux-2.4.19/include/linux/ext3_fs.h has
1. define EXT3_LINK_MAX 32000
and /usr/src/linux-2.4.19/include/linux/reiserfs_fs.h has
1. define REISERFS_LINK_MAX (MAX_US_INT - 1000)
(for maximum unsigned integer on the system).
This could be caused by a directory having too many subdirectories
(each subdirectory has .. as a hardlink to it's parent directory which
causes that directory's hardlink count to be increased by one. So yes,
this does mean that you are limited to 32000 subdirectories in one
directory in ext3, even if you have hashdirs enabled.) As a
consequence of this you can stat(2) a directory and add one (for ..)
and you will know how many directories are in the current directory.
(or subtract one (for .) to find out how many subdirectories there
are).
Some file systems (such as FAT) don't have hardlinks so the hardlink
count can't overflow, and you can't rely on the hardlink count of a
directory to be representive of how many subdirectories it has.
What a confusing world we live in.
I did reach the 32000 tickets. I have exactly:
31999 folder for ticket# on my ext3.
ls -l |wc -l
31999
How do I increase that number on ext3?
Thanks,
Lucas
Remy Blank
> How do I increase that number on ext3?
I'm not sure you can. But I remember reading about a "sharding" scheme
for attachments that would solve the issue. Here's the ticket:
http://trac.edgewall.org/ticket/7397
There's a patch attached, but it didn't get a review yet.
-- Remy
Erik Bray
Well, when lots of SVN users started hitting limits like that for
revisions, they just added a second directory level to handle the
situation. I don't think many Trac instances have, to this point,
been used in a such a way that they've hit this limit.
Leonardo Santagada
On Aug 21, 2009, at 6:32 PM, Erik Bray wrote:
>
> On Thu, Aug 20, 2009 at 6:28 PM, Lukasz
> Szybalski<szyb...@gmail.com> wrote:
>>
>> On Thu, Aug 20, 2009 at 5:13 PM, Lukasz
>> Szybalski<szyb...@gmail.com> wrote
>> I did reach the 32000 tickets. I have exactly:
>> 31999 folder for ticket# on my ext3.
>>
>> ls -l |wc -l
>> 31999
>>
>> How do I increase that number on ext3?
>>
>> Thanks,
>> Lucas
>
> Well, when lots of SVN users started hitting limits like that for
> revisions, they just added a second directory level to handle the
> situation. I don't think many Trac instances have, to this point,
> been used in a such a way that they've hit this limit.
Implementing this two level architecture seems reasonably easy to do
so it is either that or you move to another filesystem, which is
simpler if you don't want to extend trac.
Theres a lot of fs that don't have this problem besides Reiserfs:
ext4 with an extension has no limits (64k otherwise)
http://en.wikipedia.org/wiki/Ext4#Break_32.2C000_subdirectory_limit
jfs and xfs probably don't have this problem either
http://en.wikipedia.org/wiki/XFS
http://en.wikipedia.org/wiki/JFS_file_system
--
Leonardo Santagada
santagada at gmail.com
Olemis Lang
>
> Hello,
> I am uploading tickets to trac. 500 a day. After almost a month+ I've
> received this error.
>
> Traceback (most recent call last):
> xmlrpclib.Fault: <Fault 2: "'[Errno 31] Too many links:
> '/home/trac/trac/attachments/ticket/32433'' while executing
> 'ticket.putAttachment()'">
>
This is not a failure in the client but run time error in the
server-side instead (otherwise `xmlrpclib.Fault` wouldn't be the exc
type being raised ;o) . Details are in the log file, and well possible
root causes in the other msgs in this thread ;o)
> Is there a way to close the xmlrpclib.ServerProxy. I think that is
> what caused this error.
>
So your client code (i.e. using ServerProxy) should be just fine ;o)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Lukasz Szybalski
On Aug 31, 9:28 am, Olemis Lang <ole...@gmail.com> wrote:
>
> > Hello,
> > I am uploading tickets to trac. 500 a day. After almost a month+ I've
> > received this error.
>
> > Traceback (most recent call last):
> [...]
> > xmlrpclib.Fault: <Fault 2: "'[Errno 31] Too many links:
> > '/home/trac/trac/attachments/ticket/32433'' while executing
> > 'ticket.putAttachment()'">
>
> This is not a failure in the client but run time error in the
> server-side instead (otherwise `xmlrpclib.Fault` wouldn't be the exc
> type being raised ;o) . Details are in the log file, and well possible
> root causes in the other msgs in this thread ;o)
>
> > Is there a way to close the xmlrpclib.ServerProxy. I think that is
> > what caused this error.
>
> So your client code (i.e. using ServerProxy) should be just fine ;o)
connection problem but it turned out to be ext3 problem.
In this case most of attaching is done via xmlrpc. We have reached the
ext3 filesystem limit, so nobody was able to attach any new
attachments to new tickets. First got the xmlrpc error, then google to
find it to be an ext3 limit. We were at ticket 31999 after 3 months.
We will be at 128,000 tickets after a year, so this is somewhat a
limitation that needs to be mentioned.
Since I was going on a week long vacation day after this error, I had
to act fast, and at this point the fastest solution was to switch to
xfs based fs. I've reinstalled debian on a separate pc with xfs. Now
that I'm back I need to resize ext3 filesystem on server, resize the
lvm and assign part of the space to xfs based partition, then move
back trac to the server.
This limitation should be mentioned int he "production installation
instructions". I've looked over the ticket that suggested the
structure should be changed for attachments, but I believe the
simplicity of having folder for ticket# and attachment in it is worth
switching over to xfs or ext4.
Thanks,
Lucas