ext3 filesize problem

[Greg Russell]

Hello, we have an ext3 filesystem as follows, but the file size seems
limited to 2GB for some reason, causing the "motion" program to stop when
an mpeg file of that size is reached.

I'd appreciate some help in determining why an ext3 file size is limited
to 2GB, please. It's difficult to imagine that "motion" would limit the
size, but I don't have sufficient talent to examine the source code for
such a restriction.

$ grep hdb1 /etc/fstab
/dev/hdb1 /home/grussell/mnt ext3 defaults,users 0 0

$ mount
...
/dev/hdb1 on /home/grussell/mnt type ext3 (rw,noexec,nosuid,nodev)

$ ls -l /home/grussell/mnt/webcam/motion/*
-rw-r--r-- 1 grussell grussell 2147483647 Jan 21 15:29 /home/grussell/mnt/
webcam/motion/20120121-Sat-timelapse.mpg

$ ps auxw | grep motion
grussell 9865 0.0 0.0 4016 660 pts/1 R+ 15:37 0:00 grep motion

$ motion -s
[0] Processing thread 0 - config file /etc/motion/motion.conf
[0] Motion 3.2.12 Started
[0] ffmpeg LIBAVCODEC_BUILD 3424768 LIBAVFORMAT_BUILD 3423232
[0] Motion running in setup mode.
[0] Thread 1 is from /etc/motion/motion.conf
[0] Thread 1 is device: /dev/video0 input 8
...
[1] File of type 32 saved to: /home/grussell/mnt/webcam/motion/20120121-
Sat-timelapse.mpg
File size limit exceeded

$ df -k /dev/hdb1
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hdb1 480721604 400331204 55971076 88% /home/grussell/mnt

$ find /home/grussell/mnt -type f -size +2G
... (numerous iso and dv files listed)

[J G Miller]

On Saturday, January 21st, 2012, at 23:53:02h +0000, Greg Russell asked:

> Hello, we have an ext3 filesystem as follows, but the file size seems
> limited to 2GB for some reason, causing the "motion" program to stop
> when an mpeg file of that size is reached.

I had never heard of the motion program before today.

A web search took me to the home page which then provided a link
to documentation at

<http://www.lavrsen.DK/foswiki/bin/view/Motion/MotionGuideBasicFeatures>

I then use the browser "text search" feature with the key word "size".

This eventually takes me down the page to

QUOTE

Motion simply creates the timelapse film with a standard mpeg1 framerate.

Note : maximum size for timelapse files is 2GB.

UNQUOTE

> It's difficult to imagine that "motion" would limit the size

Oddly enough I find it quite easy to imagine that. ;)

[Stan Bischof]

J G Miller <mil...@yoyo.org> wrote:
>
> QUOTE
>
> Motion simply creates the timelapse film with a standard mpeg1 framerate.
>
> Note : maximum size for timelapse files is 2GB.
>
> UNQUOTE
>
>> It's difficult to imagine that "motion" would limit the size
>
> Oddly enough I find it quite easy to imagine that. ;)

indeed. 32-bit limit is VERY common even if you have a 64-bit OS.

There's lots of software out there that is going to be 32-bit
for quite a while still since the transition to 64-bits is
taking a long long time.

Stan

[J G Miller]

On Monday, January 23rd, 2012, at 16:24:54h +0000, Stan Bischof wrote:

> There's lots of software out there that is going to be 32-bit for quite
> a while still since the transition to 64-bits is taking a long long
> time.

Good point!

Quite a lot of people, including myself, are still using 32-bit
machines on the desktop.

[Greg Russell]

On Sun, 22 Jan 2012 00:43:47 +0000, J G Miller wrote in
jffm43$ieq$2...@dont-email.me :

...

> QUOTE
>
> Motion simply creates the timelapse film with a standard mpeg1
> framerate.
>
> Note : maximum size for timelapse files is 2GB.
>
> UNQUOTE
>
>> It's difficult to imagine that "motion" would limit the size
>
> Oddly enough I find it quite easy to imagine that. ;)

You do? Thanks for the detective work as that explains the behaviour, but
why motion should limit an mpeg file to 2GB is still not understandable
to me, so I'll speak to the author(s) about it.

Other video files on the same filesystem are about 4.4GB, but motion's
2GB filesize limit kills the program, destroying its usefulness for
remote security documentation.

[J G Miller]

On Friday, February 3rd, 2012, at 19:39:13h +0000, Greg Russell wrote:

> but why motion should limit an mpeg file to 2GB is still not
> understandable to me

Well in another post, Stan has suggested the reason is due to
the 32 bit limit if you are using a 32 bit version of motion.

[Greg Russell]

On Fri, 03 Feb 2012 20:55:59 +0000, J G Miller wrote in
jghhkv$on1$1...@dont-email.me :

Yes, ... and I find that reasoning to be inadequate and unconvincing. 2
GB is not a limit on a 32-bit ext3 filesystem.

Other programs such as ffmpeg have no such limits when creating mpeg
files on the same filesystem.

[Aragorn]

On Saturday 04 February 2012 05:40, Greg Russell conveyed the following
to comp.os.linux.misc...

> On Fri, 03 Feb 2012 20:55:59 +0000, J G Miller wrote in
> jghhkv$on1$1...@dont-email.me :
>
>> On Friday, February 3rd, 2012, at 19:39:13h +0000, Greg Russell
>> wrote:
>>
>>> but why motion should limit an mpeg file to 2GB is still not
>>> understandable to me
>>
>> Well in another post, Stan has suggested the reason is due to the 32
>> bit limit if you are using a 32 bit version of motion.
>
> Yes, ... and I find that reasoning to be inadequate and unconvincing.
> 2 GB is not a limit on a 32-bit ext3 filesystem.

No, but it may be and often is the limit of /certain/ 32-bit
_applications_ [*]. It has nothing to do with the ext3 filesystem.


[*] The following pertains to web browsers and download utilities, but
it's a useful bit of information...:
http://www.kernel.org/faq/#largefiles

--
= Aragorn =
(registered GNU/Linux user #223157)

[Greg Russell]

On Sat, 04 Feb 2012 05:47:06 +0100, Aragorn wrote in
jgid8b$vrm$1...@dont-email.me :

No, it's not useful at all in explaining why motion limits itself to a
2GB output filesize on a filesystem that isn't 2GB-limited; it has no
relation to the topic under discussion.

[Aragorn]

On Saturday 04 February 2012 06:23, Greg Russell conveyed the following

to comp.os.linux.misc...

> On Sat, 04 Feb 2012 05:47:06 +0100, Aragorn wrote in
> jgid8b$vrm$1...@dont-email.me :
>
>> On Saturday 04 February 2012 05:40, Greg Russell conveyed the
>> following to comp.os.linux.misc...
>>
>>> On Fri, 03 Feb 2012 20:55:59 +0000, J G Miller wrote in
>>> jghhkv$on1$1...@dont-email.me :
>>>
>>>> On Friday, February 3rd, 2012, at 19:39:13h +0000, Greg Russell
>>>> wrote:
>>>>
>>>>> but why motion should limit an mpeg file to 2GB is still not
>>>>> understandable to me
>>>>
>>>> Well in another post, Stan has suggested the reason is due to the
>>>> 32 bit limit if you are using a 32 bit version of motion.
>>>
>>> Yes, ... and I find that reasoning to be inadequate and
>>> unconvincing. 2 GB is not a limit on a 32-bit ext3 filesystem.
>>
>> No, but it may be and often is the limit of /certain/ 32-bit
>> _applications_ [*]. It has nothing to do with the ext3 filesystem.
>>
>>
>> [*] The following pertains to web browsers and download utilities,
>> but it's a useful bit of information...:
>> http://www.kernel.org/faq/#largefiles
>
> No, it's not useful at all in explaining why motion limits itself to a
> 2GB output filesize on a filesystem that isn't 2GB-limited; it has no
> relation to the topic under discussion.

It explains why certain applications are known to limit the size of a
downloaded file to 2 GB even though the underlying filesystem supports
larger files. It has to do with the amount of data the application can
process at once.

Apparently "motion" - or at least, the 32-bit version of it - cannot
handle files larger than 2 GB. Buffer size, checkpoints, et al. It is
not - repeat: not - a limitation of the filesystem, but instead it is a
limitation of the application.

There are editors which suffer from the same limitation. Try opening up
that mpeg file in emacs and see how much data you can get into the
buffer before emacs says "Ho buddy, that's enough!"

[Greg Russell]

On Sat, 04 Feb 2012 06:57:04 +0100, Aragorn wrote in
jgihbg$dov$1...@dont-email.me :

> downloaded file to 2 GB ...

No, it certainly doesn't.

> Apparently "motion" - or at least, the 32-bit version of it - cannot
> handle files larger than 2 GB. Buffer size, checkpoints, et al. It is
> not - repeat: not - a limitation of the filesystem, but instead it is a
> limitation of the application.

<sigh> ... good, good, you're starting to address the issue, albeit quite
obliquely.

> There are editors which suffer from the same limitation. ...

<sigh> ... and what limitation is that wrt "motion"?

[Aragorn]

On Saturday 04 February 2012 08:14, Greg Russell conveyed the following

to comp.os.linux.misc...

> On Sat, 04 Feb 2012 06:57:04 +0100, Aragorn wrote in
> jgihbg$dov$1...@dont-email.me :
>
>> On Saturday 04 February 2012 06:23, Greg Russell conveyed the
>> following to comp.os.linux.misc...
>>
>>> On Sat, 04 Feb 2012 05:47:06 +0100, Aragorn wrote in
>>> jgid8b$vrm$1...@dont-email.me :
>>>
>>>> On Saturday 04 February 2012 05:40, Greg Russell conveyed the
>>>> following to comp.os.linux.misc...
>>>>

>>>>> Yes, ... and I find that reasoning to be inadequate and
>>>>> unconvincing. 2 GB is not a limit on a 32-bit ext3 filesystem.
>>>>
>>>> No, but it may be and often is the limit of /certain/ 32-bit
>>>> _applications_ [*]. It has nothing to do with the ext3 filesystem.
>>>>
>>>> [*] The following pertains to web browsers and download utilities,
>>>> but it's a useful bit of information...:
>>>> http://www.kernel.org/faq/#largefiles
>>>
>>> No, it's not useful at all in explaining why motion limits itself to
>>> a 2GB output filesize on a filesystem that isn't 2GB-limited; it has
>>> no relation to the topic under discussion.
>>
>> It explains why certain applications are known to limit the size of a
>> downloaded file to 2 GB ...
>
> No, it certainly doesn't.

Sheesh man, it literally says that some applications are limited to 2 GB
only. That's the part you had to retain. How difficult is that?

>> Apparently "motion" - or at least, the 32-bit version of it - cannot
>> handle files larger than 2 GB. Buffer size, checkpoints, et al. It
>> is not - repeat: not - a limitation of the filesystem, but instead it
>> is a limitation of the application.
>
> <sigh> ... good, good, you're starting to address the issue, albeit
> quite obliquely.

It's what I have been saying all along. I'll repeat the quote...

>>>> No, but it may be and often is the limit of /certain/ 32-bit
>>>> _applications_ [*]. It has nothing to do with the ext3 filesystem.

See?

>> There are editors which suffer from the same limitation. ...
>
> <sigh> ... and what limitation is that wrt "motion"?

I don't know why "motion" has that limitation, but apparently it does.
If you want to know why it happens to have this limitation or how to
circumvent it, talk to the developer(s).

[Peter Köhlmann]

It probably uses a signed integer for filesize.
And in 32bits that is 2GB

Linux had in the past a filesize limit of 2GB, and that was lifted with
kernel 2.2 and 2.4.
That does not mean that applications have removed that limit also, though

[Harold Stevens]

In <jgimui$ga$1...@dont-email.me> Aragorn:

[Snip...]

> Sheesh man, it literally says that some applications are limited to 2 GB

FWIW, I have "Greg Russell" scored BozoBin twice. Jez sayin' ... :)

--
Regards, Weird (Harold Stevens) * IMPORTANT EMAIL INFO FOLLOWS *
Pardon any bogus email addresses (wookie) in place for spambots.
Really, it's (wyrd) at airmail, dotted with net. DO NOT SPAM IT.
I toss GoogleGroup (http://twovoyagers.com/improve-usenet.org/).

[Robert Heller]

Has *anyone* looked at the actual code? The specific limitation might
even be (shudder) *documented* in the code. And there might even be an
explaination as to why the limitation is there. The limitation might be
stupid or might be a design flaw, but it is obviously there.

>
>

--
Robert Heller -- 978-544-6933 / hel...@deepsoft.com
Deepwoods Software -- http://www.deepsoft.com/
() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments

[Richard Kettlewell]

Robert Heller <hel...@deepsoft.com> writes:
> Has *anyone* looked at the actual code? The specific limitation might
> even be (shudder) *documented* in the code.

It does not appear to be, I'm afraid.

> And there might even be an explaination as to why the limitation is
> there. The limitation might be stupid or might be a design flaw, but
> it is obviously there.

The error string reported is "File size limit exceeded" which
corresponds to SIGXFSZ. This signal is sent as a result of breaching
RLIMIT_FSIZE (i.e. ulimit -f).

There's a per-filesystem bound that can produce the same outcome but
that should not be an issue here.

The usual reason for a 2GB limit in 32-bit executables is failure to
compile with -D_FILE_OFFSET_BITS=64. I don't see anything in the
'motion' package that addresses this, so it's a strong possibility.
However the result of this should be the EFBIG error, for which the
expected output would be "File too large".

It's also possible to acquire a 2G limitation (even in 64-bit
executables) by poor choices of data type to reflect file offsets and
sizes. I've skimmed the relevant bits of 'motion' and don't see
anything fitting the bill.

Finally it could be a bug in the ffmpeg libraries used. Upstream knows
about _FILE_OFFSET_BITS and the API is obviously designed with 64-bit
offsets in mind so I don't think this is likely to be the cause.

Things the OP can do include:
- see what 'ulimit -a' says prior to running motion
- see what 'echo $?' says immediately after it fails
(i.e. see if the exit status reflects the signal)
- use strace to get a lower-level idea of what's going on
- if they aren't using a 64-bit platform, try recompiling motion with
-D_FILE_OFFSET_BITS=64 on the compiler command line

--
http://www.greenend.org.uk/rjk/

[Aragorn]

On Saturday 04 February 2012 12:07, Harold Stevens conveyed the
following to comp.os.linux.misc...

> In <jgimui$ga$1...@dont-email.me> Aragorn:
>
> [Snip...]
>
>> Sheesh man, it literally says that some applications are limited to 2
>> GB
>
> FWIW, I have "Greg Russell" scored BozoBin twice. Jez sayin' ... :)

It is always useful to get feedback from other posters. ;-) Yet I
myself generally only bin people when they turn abusive/insulting -
whether it is to me or to others. ;-)

[Greg Russell]

On Sat, 04 Feb 2012 07:47:39 -0600, Robert Heller wrote in
rPCdnUh1sJpmq7DS...@posted.localnet :

...

> Has *anyone* looked at the actual code? The specific limitation might
> even be (shudder) *documented* in the code. And there might even be an
> explaination as to why the limitation is there. The limitation might be
> stupid or might be a design flaw, but it is obviously there.

Thank you, that expresses the issue well.

My efforts to download the source code from sourceforge meet with a debian
package only.

[Greg Russell]

On Sat, 04 Feb 2012 08:32:34 +0100, Aragorn wrote in
jgimui$ga$1...@dont-email.me :

...

>>> It explains why certain applications are known to limit the size of a
>>> downloaded file to 2 GB ...
>>
>> No, it certainly doesn't.
>
> Sheesh man, it literally says that some applications are limited to 2 GB
> only. That's the part you had to retain. How difficult is that?

Obviously we're not communicating well. I'm asking a specific question
about "motion", and you're repeatedly responding allegorically with arm-
waving replies.

Others have addressed the question adequately, so I wish you good-day.

[J G Miller]

On Sunday, February 5th, 2012, at 01:22:38h +0000, Greg Russell complained:

> My efforts to download the source code from sourceforge meet with a
> debian package only.

<http://sourceforge.NET/projects/motion/files/motion%20-%203.2/3.2.12/motion-3.2.12.tar.gz/download>

[Norman Peelman]

But then

http://www.lavrsen.dk/foswiki/bin/view/Motion/MotionGuideBasicFeatures#max_mpeg_time

would seem to contradict that unless I'm confusing issues.

--
Norman
Registered Linux user #461062
AMD64X2 6400+ Ubuntu 10.04 64bit