What is the largest file I can edit using vim?

[cellurl]

I have a 4GB text file.
Q: Can I use vim to edit it?
If not, any suggestions?

thanks for publishing vim ;-)

-cellurl

[Adam]

For a file that large you'll need to make sure you have a lot of ram and are using a 64 bit version of vim.  You should also consider using the LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)

~Adam~

[james pruett]

ok, I dont have a 64bit system.
Thanks for the reply.

--------------------------
[Speederaser] may be the best automotive accessory since the radar detector.

On Wed, Jun 6, 2012 at 11:09 PM, Adam <les...@gmail.com> wrote:

For a file that large you'll need to make sure you have a lot of ram and are using a 64 bit version of vim.  You should also consider using the LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)

~Adam~

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Kevin O'Gorman]

Reorganized to bottom-posted replies.

> On Wed, Jun 6, 2012 at 11:09 PM, Adam <les...@gmail.com> wrote:
>>
>> For a file that large you'll need to make sure you have a lot of ram and
>> are using a 64 bit version of vim.  You should also consider using the
>> LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)
>>
>> ~Adam~
>>
>> --
>> You received this message from the "vim_use" maillist.
>> Do not top-post! Type your reply below the text you are replying to.
>> For more information, visit http://www.vim.org/maillist.php

On Wed, Jun 6, 2012 at 9:22 PM, james pruett <gpsc...@gmail.com> wrote:
> ok, I dont have a 64bit system.
> Thanks for the reply.

In which case, you could chop the file into segments and edit those.
Or, depending on what you're trying to do, you might be able to write
a program to make your changes.

I have to say that even editors that *can* handle huge files tend to
do so very slowly. There may be exceptions, but I wouldn't count on
it.

--
Kevin O'Gorman, PhD

[Marc Weber]

Talk about what you want to do with such a big file an people will try
to find a solution. Editing 4GB files in a text editor is no joy
anyway.

Marc Weber

[james pruett]

I have an *.sql file.

Typically I chop it, but I am tired of chopping it.

I want to remove the first 10 lines and the last 10 lines of this 4GB file.

I have had no success with UltraEdit and VI, gedit, wordpad, VIM.

I may try google-docs.... 

Thanks for publishing vim!

-cellurl

--------------------------
[Speederaser] may be the best automotive accessory since the radar detector.

[Amit Agarwal]

You can possibly use sed to do that.

--
Sent from my Nokia N900

----- Original message -----
> I have an *.sql file.
> Typically I chop it, but I am tired of chopping it.
> I want to remove the first 10 lines and the last 10 lines of this 4GB
> file. I have had no success with UltraEdit and VI, gedit, wordpad, VIM.
> I may try google-docs....
>
> Thanks for publishing vim!
> -cellurl
>
>
>
>
>
> --------------------------
> [Speederaser] may be the best automotive accessory since the radar

> detector<http://www.motorauthority.com/news/1068165_an-android-app-for-cruise-control#>
> .

[Ben Fritz]

On Thursday, June 7, 2012 8:56:19 AM UTC-5, cellurl wrote:
> I have an *.sql file.
> Typically I chop it, but I am tired of chopping it.
> I want to remove the first 10 lines and the last 10 lines of this 4GB file.

The "head" and "tail" unix commands seem designed exactly for such things.

[Reid Thompson]

On Thu, 2012-06-07 at 19:42 +0530, Amit Agarwal wrote:
> You can possibly use sed to do that.

from the sed one liners page http://sed.sourceforge.net/sed1line.txt

# delete the first 10 lines of a file
sed '1,10d'

# delete the last line of a file
sed '$d'

# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'

# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2

[Marc Weber]

forgett about vim, on linux just do:
tail -n +10 file.sql | head -n +10 > trimmed.sql

Marc Weber

[Erik Christiansen]

On 07.06.12 08:56, james pruett wrote:
> I have an *.sql file.
> Typically I chop it, but I am tired of chopping it.
> I want to remove the first 10 lines and the last 10 lines of this 4GB file.

$ tail --lines=+11 fred.sql | head --lines=-10

(Tested with /etc/passwd, it cuts at the right line numbers, as far as I
can see.)

Erik

--
"When the only tool you have is a hammer, you tend to treat everything as if
it were a nail."
- Abraham Maslow

[Tony Mechelynck]

There are two kinds of limits:

1. Your processor must be able to address a single data-memory space
larger than the file.

See ":help limits" for 16- and 32-bit processors. 64-bit processors can
typically address all your installed RAM and swap memory as one linear
address space: see 2 below. (N.B. 2^63 = approx. 9.22e18)

2. Your total installed RAM and swap space must be larger than the file
+ the resident Vim code + whatever else you're running in parallel.

The largest file you can edit depends on the smaller of the limits
defined by 1 and 2 above.

Notes:
- It's no use having more installed memory than your processor can address;
- It's usually recommended not to have more swap than twice your
installed RAM. (Using swap too much might bring the system to the point
where all it's doing is waiting for disk heads to move to the right
record of your swapfile or swap partition. I've had the case, for
instance when linking the libxul of a Mozilla application at the end of
an own-compile in order to test a bugfix before landing it.)


Best regards,
Tony.
--
FATHER: Make sure the Prince doesn't leave this room until I come and
get him.
FIRST GUARD: Not ... to leave the room ... even if you come and get him.
FATHER: No. Until I come and get him.
SECOND GUARD: Hic.
"Monty Python and the Holy Grail" PYTHON (MONTY)
PICTURES LTD

[Dominique Pellé]

Marc Weber <marco-...@gmx.de> wrote:

> forgett about vim, on linux just do:
> tail -n +10 file.sql  | head -n +10 > trimmed.sql

Many people posted solutions with head and tail that don't work.

Here is one that works:

$ sed 1,10d < input.txt | tac | sed 1,10d | tac > output.txt

Here is another one, faster but uglier:

$ perl -ne 'print $l[$.%10] if ($. >= 10*2); $l[$.%10] = $_' input.txt
> output.txt

-- Dominique

[Ben Fritz]

On Thursday, June 7, 2012 11:49:02 AM UTC-5, Dominique Pelle wrote:
> Marc Weber <marco-...@gmx.de> wrote:
>
> > forgett about vim, on linux just do:
> > tail -n +10 file.sql  | head -n +10 > trimmed.sql
>
> Many people posted solutions with head and tail that don't work.
>

Just curious...I've never used head or tail. But why won't they work in this situation? This seems like exactly what they're meant to accomplish; retrieving only a specific part of a file.

[Kevin O'Gorman]

On Thu, Jun 7, 2012 at 7:30 AM, Erik Christiansen
<dva...@internode.on.net> wrote:
> On 07.06.12 08:56, james pruett wrote:
>> I have an *.sql file.
>> Typically I chop it, but I am tired of chopping it.
>> I want to remove the first 10 lines and the last 10 lines of this 4GB file.

I'm not sure what you mean by chopping it, or why it would be
tiresome. There are utilities for such things, like split(1).

> $ tail --lines=+11 fred.sql | head --lines=-10
>
> (Tested with /etc/passwd, it cuts at the right line numbers, as far as I
> can see.)

Also a skillful technique.

[Gary Johnson]

On 2012-06-07, Dominique Pell� wrote:

> Marc Weber wrote:
>
> > forgett about vim, on linux just do:
> > tail -n +10 file.sql �| head -n +10 > trimmed.sql
>
> Many people posted solutions with head and tail that don't work.
>
> Here is one that works:
>
> $ sed 1,10d < input.txt | tac | sed 1,10d | tac > output.txt

Are you sure that tac works in this case? I thought that tac pushed
all the input lines onto a stack in memory, then popped each line as
it was output. That means having to put the entire file into
memory, which we were trying to avoid.

Regards,
Gary

[Dominique Pellé]

Because tail outputs the last n lines of a file, but what user
wanted is to remove the last lines. So unless you know the
number of lines, you can't use tail. But perhaps someone
can prove me wrong.

-- Dominique

[Tim Chase]

I think you're reading it backwards, as head/tail (at least GNU
versions; for other flavors, YMMV) allow for a "+" in front of the
number so

tail -n +20

chops off the first 19 lines in the file; similarly, "-" in front of
the number with head does all but the N last lines of the file. The
example above should likely read something like

tail -n +11 file.sql | head -n -10 > trimmed.sql

(using a "-" instead of a "+" for head and incrementing the starting
point on tail). There might be a fenceposting error there, but
that's the general gist.

-tim

[Dominique Pellé]

Gary Johnson <gary...@spocom.com> wrote:

Yes, I was wondering about that too. But I checked and
tac only uses very little memory even on huge files.

Actually, tac behaves differently with an input file and with
an input stream:

* with an input file, it outputs results immediately. I suppose
tac reads the file by block from the end of the file and reverse
each blocks.
* with an input stream it can't do that, so it has to read the
full stream before it can output. Yet it uses very little memory.
It uses temporary files (confirmed by looking at /prof/<pid>/fd)

So tac solution works. But given that tac is more efficient on
an input file than on an input stream, changing the order
should be better. In other words, this...

$ tac input.txt | sed 1,10d | tac | sed 1,10d > output.txt

... should be faster than this:

$ sed 1,10d input.txt | tac | sed 1,10d | tac > output.txt

I measured it on a big file to confirm it:

first solution took 9.2 sec, second solution took 12.2 sec

In any case, the Perl solution that I gave and which use a
rotating buffer does one pass only, does not use much memory
and does not use temporary files either.

$ perl -ne 'print $l[$.%10] if ($. >= 10*2); $l[$.%10] = $_' input.txt
>output.txt

Yet this Perl solution is slower than tac. It takes 14.8 sec on
the same input file.

The strange looking solution...

$ sed -e :a -e '$d;N;2,10ba' -e 'P;D' input.txt > output.txt

... takes 6.0 sec.

Tim Chase wrote:

> I think you're reading it backwards, as head/tail (at least GNU
> versions; for other flavors, YMMV) allow for a "+" in front of the
> number so
>
> tail -n +20
>
> chops off the first 19 lines in the file; similarly, "-" in front of
> the number with head does all but the N last lines of the file. The
> example above should likely read something like
>
> tail -n +11 file.sql | head -n -10 > trimmed.sql

Right. My apologies. That works indeed and it's much faster.
It does not have to parse line by line with this solution I suppose.
With the same large input as above, it only took 2.6 sec.

-- Dominique

[james pruett]

Google-docs won't handle a 4GB file...

So I will have to try unix tools...

http://support.google.com/docs/bin/answer.py?hl=en&answer=37603 

[Tim Chase]

On 06/08/12 09:57, james pruett wrote:
> Google-docs won't handle a 4GB file...
> So I will have to try unix tools...

Then I recommend using "split" or "csplit". The former can split on
bytes/lines, while the latter can split on patterns.

-tim