For a file that large you'll need to make sure you have a lot of ram and
are using a 64 bit version of vim. You should also consider using the
LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)
On Wed, Jun 6, 2012 at 11:09 PM, Adam <les...@gmail.com> wrote:
> For a file that large you'll need to make sure you have a lot of ram and
> are using a 64 bit version of vim. You should also consider using the
> LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)
> ~Adam~
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
> On Wed, Jun 6, 2012 at 11:09 PM, Adam <les...@gmail.com> wrote:
>> For a file that large you'll need to make sure you have a lot of ram and
>> are using a 64 bit version of vim. You should also consider using the
>> LargeFile plugin (http://www.vim.org/scripts/script.php?script_id=1506)
>> ~Adam~
>> --
>> You received this message from the "vim_use" maillist.
>> Do not top-post! Type your reply below the text you are replying to.
>> For more information, visit http://www.vim.org/maillist.php On Wed, Jun 6, 2012 at 9:22 PM, james pruett <gpscru...@gmail.com> wrote:
> ok, I dont have a 64bit system.
> Thanks for the reply.
In which case, you could chop the file into segments and edit those.
Or, depending on what you're trying to do, you might be able to write
a program to make your changes.
I have to say that even editors that *can* handle huge files tend to
do so very slowly. There may be exceptions, but I wouldn't count on
it.
I have an *.sql file.
Typically I chop it, but I am tired of chopping it.
I want to remove the first 10 lines and the last 10 lines of this 4GB file.
I have had no success with UltraEdit and VI, gedit, wordpad, VIM.
I may try google-docs....
On Thu, Jun 7, 2012 at 1:37 AM, Marc Weber <marco-owe...@gmx.de> wrote:
> Talk about what you want to do with such a big file an people will try
> to find a solution. Editing 4GB files in a text editor is no joy
> anyway.
> Marc Weber
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
----- Original message -----
> I have an *.sql file.
> Typically I chop it, but I am tired of chopping it.
> I want to remove the first 10 lines and the last 10 lines of this 4GB
> file. I have had no success with UltraEdit and VI, gedit, wordpad, VIM.
> I may try google-docs....
> On Thu, Jun 7, 2012 at 1:37 AM, Marc Weber <marco-owe...@gmx.de> wrote:
> > Talk about what you want to do with such a big file an people will try
> > to find a solution. Editing 4GB files in a text editor is no joy
> > anyway.
> > Marc Weber
> > --
> > You received this message from the "vim_use" maillist.
> > Do not top-post! Type your reply below the text you are replying to.
> > For more information, visit http://www.vim.org/maillist.php
> -- > You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
On Thursday, June 7, 2012 8:56:19 AM UTC-5, cellurl wrote: > I have an *.sql file. > Typically I chop it, but I am tired of chopping it. > I want to remove the first 10 lines and the last 10 lines of this 4GB file.
The "head" and "tail" unix commands seem designed exactly for such things.
> I have an *.sql file.
> Typically I chop it, but I am tired of chopping it.
> I want to remove the first 10 lines and the last 10 lines of this 4GB file.
$ tail --lines=+11 fred.sql | head --lines=-10
(Tested with /etc/passwd, it cuts at the right line numbers, as far as I
can see.)
Erik
-- "When the only tool you have is a hammer, you tend to treat everything as if
it were a nail."
- Abraham Maslow
> I have a 4GB text file.
> Q: Can I use vim to edit it?
> If not, any suggestions?
> thanks for publishing vim ;-)
> -cellurl
There are two kinds of limits:
1. Your processor must be able to address a single data-memory space larger than the file.
See ":help limits" for 16- and 32-bit processors. 64-bit processors can typically address all your installed RAM and swap memory as one linear address space: see 2 below. (N.B. 2^63 = approx. 9.22e18)
2. Your total installed RAM and swap space must be larger than the file + the resident Vim code + whatever else you're running in parallel.
The largest file you can edit depends on the smaller of the limits defined by 1 and 2 above.
Notes:
- It's no use having more installed memory than your processor can address;
- It's usually recommended not to have more swap than twice your installed RAM. (Using swap too much might bring the system to the point where all it's doing is waiting for disk heads to move to the right record of your swapfile or swap partition. I've had the case, for instance when linking the libxul of a Mozilla application at the end of an own-compile in order to test a bugfix before landing it.)
Best regards,
Tony.
-- FATHER: Make sure the Prince doesn't leave this room until I come and
get him.
FIRST GUARD: Not ... to leave the room ... even if you come and get him.
FATHER: No. Until I come and get him.
SECOND GUARD: Hic.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
On Thursday, June 7, 2012 11:49:02 AM UTC-5, Dominique Pelle wrote:
> Marc Weber <marco-owe...@gmx.de> wrote:
> > forgett about vim, on linux just do:
> > tail -n +10 file.sql | head -n +10 > trimmed.sql
> Many people posted solutions with head and tail that don't work.
Just curious...I've never used head or tail. But why won't they work in this situation? This seems like exactly what they're meant to accomplish; retrieving only a specific part of a file.
<dva...@internode.on.net> wrote:
> On 07.06.12 08:56, james pruett wrote:
>> I have an *.sql file.
>> Typically I chop it, but I am tired of chopping it.
>> I want to remove the first 10 lines and the last 10 lines of this 4GB file.
I'm not sure what you mean by chopping it, or why it would be
tiresome. There are utilities for such things, like split(1).
> $ tail --lines=+11 fred.sql | head --lines=-10
> (Tested with /etc/passwd, it cuts at the right line numbers, as far as I
> can see.)
> > forgett about vim, on linux just do:
> > tail -n +10 file.sql | head -n +10 > trimmed.sql
> Many people posted solutions with head and tail that don't work.
> Here is one that works:
> $ sed 1,10d < input.txt | tac | sed 1,10d | tac > output.txt
Are you sure that tac works in this case? I thought that tac pushed
all the input lines onto a stack in memory, then popped each line as
it was output. That means having to put the entire file into
memory, which we were trying to avoid.
On Thu, Jun 7, 2012 at 6:57 PM, Ben Fritz <fritzophre...@gmail.com> wrote:
> On Thursday, June 7, 2012 11:49:02 AM UTC-5, Dominique Pelle wrote:
>> Marc Weber <marco-owe...@gmx.de> wrote:
>> > forgett about vim, on linux just do:
>> > tail -n +10 file.sql | head -n +10 > trimmed.sql
>> Many people posted solutions with head and tail that don't work.
> Just curious...I've never used head or tail. But why won't they
> work in this situation? This seems like exactly what they're
> meant to accomplish; retrieving only a specific part of a file.
Because tail outputs the last n lines of a file, but what user
wanted is to remove the last lines. So unless you know the
number of lines, you can't use tail. But perhaps someone
can prove me wrong.
> On Thu, Jun 7, 2012 at 6:57 PM, Ben Fritz <fritzophre...@gmail.com> wrote:
>> On Thursday, June 7, 2012 11:49:02 AM UTC-5, Dominique Pelle wrote:
>>> Marc Weber <marco-owe...@gmx.de> wrote:
>>>> forgett about vim, on linux just do:
>>>> tail -n +10 file.sql | head -n +10 > trimmed.sql
>>> Many people posted solutions with head and tail that don't work.
>> Just curious...I've never used head or tail. But why won't they
>> work in this situation? This seems like exactly what they're
>> meant to accomplish; retrieving only a specific part of a file.
> Because tail outputs the last n lines of a file, but what user
> wanted is to remove the last lines. So unless you know the
> number of lines, you can't use tail. But perhaps someone
> can prove me wrong.
I think you're reading it backwards, as head/tail (at least GNU
versions; for other flavors, YMMV) allow for a "+" in front of the
number so
tail -n +20
chops off the first 19 lines in the file; similarly, "-" in front of
the number with head does all but the N last lines of the file. The
example above should likely read something like
tail -n +11 file.sql | head -n -10 > trimmed.sql
(using a "-" instead of a "+" for head and incrementing the starting
point on tail). There might be a fenceposting error there, but
that's the general gist.
Gary Johnson <garyj...@spocom.com> wrote:
> On 2012-06-07, Dominique Pellé wrote:
>> Marc Weber wrote:
>> > forgett about vim, on linux just do:
>> > tail -n +10 file.sql | head -n +10 > trimmed.sql
>> Many people posted solutions with head and tail that don't work.
>> Here is one that works:
>> $ sed 1,10d < input.txt | tac | sed 1,10d | tac > output.txt
> Are you sure that tac works in this case? I thought that tac pushed
> all the input lines onto a stack in memory, then popped each line as
> it was output. That means having to put the entire file into
> memory, which we were trying to avoid.
Yes, I was wondering about that too. But I checked and
tac only uses very little memory even on huge files.
Actually, tac behaves differently with an input file and with
an input stream:
* with an input file, it outputs results immediately. I suppose
tac reads the file by block from the end of the file and reverse
each blocks.
* with an input stream it can't do that, so it has to read the
full stream before it can output. Yet it uses very little memory.
It uses temporary files (confirmed by looking at /prof/<pid>/fd)
So tac solution works. But given that tac is more efficient on
an input file than on an input stream, changing the order
should be better. In other words, this...
$ tac input.txt | sed 1,10d | tac | sed 1,10d > output.txt
... should be faster than this:
$ sed 1,10d input.txt | tac | sed 1,10d | tac > output.txt
I measured it on a big file to confirm it:
first solution took 9.2 sec, second solution took 12.2 sec
In any case, the Perl solution that I gave and which use a
rotating buffer does one pass only, does not use much memory
and does not use temporary files either.
Tim Chase wrote:
> I think you're reading it backwards, as head/tail (at least GNU
> versions; for other flavors, YMMV) allow for a "+" in front of the
> number so
> tail -n +20
> chops off the first 19 lines in the file; similarly, "-" in front of
> the number with head does all but the N last lines of the file. The
> example above should likely read something like
Right. My apologies. That works indeed and it's much faster.
It does not have to parse line by line with this solution I suppose.
With the same large input as above, it only took 2.6 sec.
> > On 2012-06-07, Dominique Pellé wrote:
> >> Marc Weber wrote:
> >> > forgett about vim, on linux just do:
> >> > tail -n +10 file.sql | head -n +10 > trimmed.sql
> >> Many people posted solutions with head and tail that don't work.
> >> Here is one that works:
> >> $ sed 1,10d < input.txt | tac | sed 1,10d | tac > output.txt
> > Are you sure that tac works in this case? I thought that tac pushed
> > all the input lines onto a stack in memory, then popped each line as
> > it was output. That means having to put the entire file into
> > memory, which we were trying to avoid.
> Yes, I was wondering about that too. But I checked and
> tac only uses very little memory even on huge files.
> Actually, tac behaves differently with an input file and with
> an input stream:
> * with an input file, it outputs results immediately. I suppose
> tac reads the file by block from the end of the file and reverse
> each blocks.
> * with an input stream it can't do that, so it has to read the
> full stream before it can output. Yet it uses very little memory.
> It uses temporary files (confirmed by looking at /prof/<pid>/fd)
> So tac solution works. But given that tac is more efficient on
> an input file than on an input stream, changing the order
> should be better. In other words, this...
> $ tac input.txt | sed 1,10d | tac | sed 1,10d > output.txt
> ... should be faster than this:
> $ sed 1,10d input.txt | tac | sed 1,10d | tac > output.txt
> I measured it on a big file to confirm it:
> first solution took 9.2 sec, second solution took 12.2 sec
> In any case, the Perl solution that I gave and which use a
> rotating buffer does one pass only, does not use much memory
> and does not use temporary files either.
> > I think you're reading it backwards, as head/tail (at least GNU
> > versions; for other flavors, YMMV) allow for a "+" in front of the
> > number so
> > tail -n +20
> > chops off the first 19 lines in the file; similarly, "-" in front of
> > the number with head does all but the N last lines of the file. The
> > example above should likely read something like
> Right. My apologies. That works indeed and it's much faster.
> It does not have to parse line by line with this solution I suppose.
> With the same large input as above, it only took 2.6 sec.
> -- Dominique
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
