see file size limits

[Kepner, Jeremy - 0553 - MITLL]

When I try and run sed on files larger than 2GB the resulting file is empty.
-u flag doesn't help.

System is 32 core, 96 GB, Linux 2.6.32
sed version is 4.2.1

[Paul Eggert]

On 02/04/2013 05:03 PM, Kepner, Jeremy - 0553 - MITLL wrote:
> When I try and run sed on files larger than 2GB the resulting file is empty.

I suggest running 'truss' and seeing what the system calls look like.

[Bob Proulx]

Kepner, Jeremy - 0553 - MITLL wrote:
> When I try and run sed on files larger than 2GB the resulting file is empty.

> -u flag doesn't help.
>
> System is 32 core, 96 GB, Linux 2.6.32
> sed version is 4.2.1

I cannot recreate your result. Your system resembles my 64-bit amd64
Debian Squeeze 6.0 system where I performed this quick and dirty test
to verify the operation.

echo now is the time $(seq 1 300) > testdata1
: > testdata2
while [ $(stat --format "%s" testdata1) -lt $((2*1024*1024*1024)) ]; do
cat testdata1 >> testdata2
cat testdata2 >> testdata1
done
sed 's/^now/Now/' testdata1 > testdata2

ls -log
-rw-rw-r-- 1 3905232424 Feb 5 11:26 testdata1
-rw-rw-r-- 1 3905232424 Feb 5 11:29 testdata2

The first part was just the first way I thought of to create a large
file. It creates a text file just under 4G in size called testdata1.
I am sure there are more clever ways to do this.

The second part makes a simple sed operation on that file and writes
testdata2. Both files are large, nonzero, and appear to be correct.
Therefore I believe your problem to be elsewhere.

What filesystem are you trying to use for this operation?

Are there file size quotas limiting total space in operation?

Bob

[Jeremy Kepner]

Hi Bob,
I believe the issue is that my text file had no newline characters
and so appeared as a line > 2^31 characters. It appears that this is fixed
in 4.2.2, which I hope to test soon.

Regards. -Jeremy

[Bob Proulx]

Jeremy Kepner wrote:
> I believe the issue is that my text file had no newline characters
> and so appeared as a line > 2^31 characters. It appears that this is fixed
> in 4.2.2, which I hope to test soon.

Hmm... Okay. That is a rather important detail to not have mentioned!

But I would hope it would have at least one newline or it isn't a text
file. Text files are files with lines of text and a line of text by
definition ends with a newline character.

I know that 4.2.2 has this new feature and if that works better for
you then great. But it does seem to be stretching things.

* don't misbehave (truncate input) for lines of length 2^31 and longer

Bob