Open big file in Gvim

[lmhelp2]

Hi everyone,

Thank you for reading my post :) .

I have a very big XML file (which is actually a Wikipedia dump)...
its size is approximately 5.9 GB.
I want to open it with "gvim"...
but it looks like it can't open...

Have you ever had such a problem?
How can I edit it with "gvim" all the same?
Do I have to use another editor? If so, which one?

Why can't I open such big files?
Is it because I do not have enough RAM (I have 1.87 GB of RAM)?

Thank you for your help :) .
Sincerely,
--
Lmhelp
--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28013165.html
Sent from the Vim - General mailing list archive at Nabble.com.

[Tim Chase]

lmhelp2 wrote:
> I have a very big XML file (which is actually a Wikipedia dump)...
> its size is approximately 5.9 GB.

I don't know whether you're running under a 32-bit or 64-bit OS.
You can read up on Vim's limitations at

:help limits

particularly the "Maximum file size" section.

If you're not exceeding those limits (which would mean you're on
a 64-bit system), then this script:

http://www.vim.org/scripts/script.php?script_id=1506

might help you trim out some of the fat involved in editing large
files.

> How can I edit it with "gvim" all the same?

If you can't open it (either because you're in 32-bit vim or
because it's just so agonizingly painful because you don't have
6+ GB of RAM), I'd use an external tool like csplit to hack your
file into smaller & more manageable bits based on some split
text/regex, edit the sub-portions and then stitch them all back
together with "cat"

> Do I have to use another editor? If so, which one?

Depending on the changes you're making (you don't detail), you
can use a stream editor such as "sed" to process the file into an
output file in one pass, but it takes a little more thinking
about the problem.

However, I suspect most other visual (non-stream) editors will
have the same OS/processor-level limitations.

-tim

[Tim Chase]

lmhelp2 wrote:
> I have a very big XML file (which is actually a Wikipedia dump)...
> its size is approximately 5.9 GB.

I don't know whether you're running under a 32-bit or 64-bit OS.

You can read up on Vim's limitations at

:help limits

particularly the "Maximum file size" section.

If you're not exceeding those limits (which would mean you're on
a 64-bit system), then this script:

http://www.vim.org/scripts/script.php?script_id=1506

might help you trim out some of the fat involved in editing large
files.

> How can I edit it with "gvim" all the same?

If you can't open it (either because you're in 32-bit vim or

because it's just so agonizingly painful because you don't have
6+ GB of RAM), I'd use an external tool like csplit to hack your
file into smaller & more manageable bits based on some split
text/regex, edit the sub-portions and then stitch them all back
together with "cat"

> Do I have to use another editor? If so, which one?

Depending on the changes you're making (you don't detail), you

[Tony Mechelynck]

On 24/03/10 11:29, lmhelp2 wrote:
>
> Hi everyone,
>
> Thank you for reading my post :) .
>
> I have a very big XML file (which is actually a Wikipedia dump)...
> its size is approximately 5.9 GB.
> I want to open it with "gvim"...
> but it looks like it can't open...
>
> Have you ever had such a problem?
> How can I edit it with "gvim" all the same?
> Do I have to use another editor? If so, which one?
>
> Why can't I open such big files?
> Is it because I do not have enough RAM (I have 1.87 GB of RAM)?
>
> Thank you for your help :) .
> Sincerely,
> --
> Lmhelp

Any file being edited is loaded in its entirety into Vim memory. Vim
could use the OS swapfile (aka virtual memory) but normally doesn't
exceed half the available RAM. Also, Vim keeps track of what you do in
order to be able to undo, and to restore your file in case the system
crashes (a circuit breaker pops open, let's say) before you save your work.

This can lead to two kind of problems:
- A memory problem, see
:help 'maxmem'
:help 'maxmemtot'
:help limits

- A performance problem, see
:help 'backup'
I recommend ":set nobackup writebackup" for normal use
if compiled with +writebackup. You may prefer
":set nobackup nowritebackup" (at your own risk) for
huge files.
:help 'undolevels'
:help 'swapfile'

Best regards,
Tony.
--
DINGO: You must spank her well and after you have spanked her you
may deal with her as you like and then ... spank me.
AMAZING: And spank me!
STUNNER: And me.
LOVELY: And me.
"Monty Python and the Holy Grail" PYTHON (MONTY)
PICTURES LTD

[lmhelp2]

Hi Tim,

Thank you for your answer :) .

> I don't know whether you're running under a 32-bit or 64-bit OS.

I am running a 32-bits OS.

> :help limits
> Maximum file size

2147483647 bytes (2 Gbyte) when a long integer is
32 bits. Much more for 64 bit longs.

> Depending on the changes you're making (you don't detail)

For the moment, I am just trying to view the file's contents...
some other program is supposed to perform things afterwards
I haven't yet faced that matter.

I have just downloaded "csplit".
Thanks for the "cat" and "sed" tips.

> However, I suspect most other visual (non-stream) editors will
> have the same OS/processor-level limitations.

Yes, probably.
"Firefox" won't open it.
"Internet Explorer" shows the beginning of the file...

==========================================================================

Hi Tony,

Thank you for your answer too :) .
Thank you for explaining things clearly.
It is very instructive to read Vim's help pages
especially when people recommend the good ones to read!

Thanks to both of you for your precious help.
I think I'll have to split the file into pieces to be
able to humanly read it.

Best regards,
--
Lmhelp
--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28014784.html

[Christian Brabandt]

On Wed, March 24, 2010 2:23 pm, lmhelp2 wrote:
> For the moment, I am just trying to view the file's contents...
> some other program is supposed to perform things afterwards
> I haven't yet faced that matter.

I think less and probably also more/most only load that part
of the data that they display.

regards,
Christian

[lmhelp2]

Hi again :) ,

I forgot I had a little additional question (sorry):
how can I view "Vim"'s "parameters" current values
like: what are the current values for the "enc" and "fenc" parameters?
I use my ".vimrc" to set some of these parameters but where can I see
all the implicit or default values of all the parameters I do not deal with
directly?
In a file? With a command?

(Please correct me if the proper world is not "parameters" :) ).

Thanks,
sincerely,
--
Lmhelp

--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28015192.html

[Benjamin R. Haskell]

On Wed, 24 Mar 2010, lmhelp2 wrote:

> For the moment, I am just trying to view the file's contents... some
> other program is supposed to perform things afterwards I haven't yet
> faced that matter.
>
> I have just downloaded "csplit".
> Thanks for the "cat" and "sed" tips.
>

For actually processing an XML file that enormous, I'm a huge fan of the
Perl library XML::Twig[1]. And even if you're not planning to process
it with Perl, XML::Twig has a pair of very useful utilities 'xml_split'
(and 'xml_merge'), which are tailored to the task of splitting (and
reässembling) large XML files into manageable chunks. Discussion of the
tool started at perlmonks.org[2], where you might find some useful usage
information (or 'perldoc xml_split' after installed).

--
Best,
Ben

[1] http://search.cpan.org/dist/XML-Twig/
[2] http://www.perlmonks.org/index.pl?node_id=429707

[Charles Campbell]

lmhelp2 wrote:
> Hi again :) ,
>
> I forgot I had a little additional question (sorry):
> how can I view "Vim"'s "parameters" current values
> like: what are the current values for the "enc" and "fenc" parameters?
> I use my ".vimrc" to set some of these parameters but where can I see
> all the implicit or default values of all the parameters I do not deal with
> directly?
> In a file? With a command?
>
> (Please correct me if the proper world is not "parameters" :) ).
>

(you probably should start a new thread for new questions -- more
visibility to potential respondents)

:echo &enc
:echo &fenc

will do the trick.

Regards,
Chip Campbell

[Gary Johnson]

On 2010-03-24, lmhelp2 wrote:
> Hi again :) ,
>
> I forgot I had a little additional question (sorry):
> how can I view "Vim"'s "parameters" current values
> like: what are the current values for the "enc" and "fenc" parameters?
> I use my ".vimrc" to set some of these parameters but where can I see
> all the implicit or default values of all the parameters I do not deal with
> directly?
> In a file? With a command?

To see the values of particular options, such as 'enc' and 'fenc',

:set enc?
:set fenc?

To see the values of all options whose values are different from the
default values,

:set

To see the values of all options except terminal options,

:set all

or organized by topic,

:options

For more on this, see

:help set-option
:help :options

HTH,
Gary

[lmhelp2]

-----------------------------------------------------------------
Hi Ben,

Thank you for the tip.
If I've got it right, "xml_split" and "xml_merge" would
then be an alternative to "csplit"...
Looks great!

-----------------------------------------------------------------
Hi Gary,

It totally helps.
It is exactly what I needed.

Thanks :) ,
all the best,
--
Lmhelp
--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28015933.html

[Joan Miquel Torres Rigo]

2010/3/24 Christian Brabandt <cbl...@256bit.org>

I just went to say that. But less will probably probably spent much more memory than 'more' (it's not a joke ;-)) because it can transparently handle streams (but I think remember that it uses intermediate file to do that).

...Of course. But it could be interesting to use the option -B (and, optionally, -b).

By the way, you can try it and, if goes bad, then use more instead.


Also, with sed, you may use -u option to avoid buffering problems.


Regards.

--
Joan Miquel Torres__________________________________
Linux Registered User #164872
http://www.mallorcaweb.net/joanmiquel
BULMA: http://bulma.net http://breu.bulma.net/?l2301

[lmhelp2]

Hi Ben,

I wanted to use your tool ("Perl XML Twig")
but I am really NOT familiar with "Perl".

I first downloaded "Perl":
strawberry-perl-5.10.1.1

Then I downloaded "XML Twig Perl Library":
XML-Twig-3.35

I have tried to put the "XML-Twig-3.35"
directory in the
"strawberry-perl-5.10.1.1\perl\lib" directory.

I also copied "Twig.pm" and "Twig.pm.slow" to
"strawberry-perl-5.10.1.1\perl\lib".

Then I tried to launch in command line:
"cmd>> perl xml_split"

And here is what I got:
"Can't open perl script 'xml_split': No such
file or directory"

Can you tell me how to proceed to be able to use
the "splitter"?

Also, at the beginning of "xml_split" there is:
"#!/usr/bin/perl -w"
but I am unfortunately not working with a Unix machine
I am working with a Windows one!

Please help me.
Thanks and all the best,
--
Lmhelp
--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28043336.html

[Benjamin R. Haskell]

On Fri, 26 Mar 2010, lmhelp2 wrote:

>
> Hi Ben,
>
> I wanted to use your tool ("Perl XML Twig")
> but I am really NOT familiar with "Perl".
>
> I first downloaded "Perl":
> strawberry-perl-5.10.1.1
>
> Then I downloaded "XML Twig Perl Library":
> XML-Twig-3.35
>
> I have tried to put the "XML-Twig-3.35"
> directory in the
> "strawberry-perl-5.10.1.1\perl\lib" directory.
>
> I also copied "Twig.pm" and "Twig.pm.slow" to
> "strawberry-perl-5.10.1.1\perl\lib".

Hmm. That's probably missing a step or two. I *think* XML::Twig has a
compiled component. I've heard of, but not used, Strawberry Perl. It's
been a while since I used Perl on Windows (especially non-Cygwin), but
I'd be surprised if Strawberry Perl didn't have some kind of package
manager.

From a quick glance at its site, you might follow "standard" Perl
installation procedure:

cmd>> perl -MCPAN -eshell

Then:

cpan[1]> install XML::Twig

If it fails, you might need:

cpan[2]> notest install XML::Twig

> Then I tried to launch in command line:
> "cmd>> perl xml_split"
>
> And here is what I got:
> "Can't open perl script 'xml_split': No such
> file or directory"
>
> Can you tell me how to proceed to be able to use
> the "splitter"?
>
> Also, at the beginning of "xml_split" there is:
> "#!/usr/bin/perl -w"
> but I am unfortunately not working with a Unix machine
> I am working with a Windows one!

If the following returns without error, the XML::Twig module is at least
loadable.

cmd>> perl -MXML::Twig -e 1

To run the xml_split file, if XML::Twig is loadable, you may just need
to pass the full path:

e.g. (not sure of the correct path)

cmd>> perl c:\strawberry-perl-5.10.1.1\perl\bin\xml_split

--
Best,
Ben

[mirod]

On Mar 26, 3:35 pm, lmhelp2 <lea.mass...@ign.fr> wrote:
> Hi Ben,
>
> I wanted to use your tool ("Perl XML Twig")
> but I am really NOT familiar with "Perl".
>
> I first downloaded "Perl":
> strawberry-perl-5.10.1.1

Your next step at this point is to use CPAN, which comes pre-
configured in Strawberry Perl. So 'cpan XML::Twig' from the command
line should be OK.

--
mirod

[lmhelp2]

Hi Ben, hi mirod,

Thank you for your answers.
It worked the way you said.
:-)

1) I installed Perl (Strawberry) for Windows.
- I fetched it on the following website:
http://strawberryperl.com/releases.html
- I chose: strawberry-perl-5.10.1.1-portable.zip

2) I unzipped it to "C:\perl_strawberry_5_10_1_1_portable"
=> because you can't have white spaces in your file path...
like with "C:\Program Files"...

3) Then, in command line, I launched:
C:\perl_strawberry_5_10_1_1_portable# cpan XML::Twig
=> The "XML::Twig" module was correctly installed.

4) And then I could run "xml_split" like that:
<where_the_xml_file_to_split_is>#
perl C:\perl_strawberry_5_10_1_1_portable\perl\bin\xml_split
<the_xml_file_to_split>.xml

Thanks again :-),
sincerely,
--
Lmhelp
--
View this message in context: http://old.nabble.com/Open-big-file-in-Gvim-tp28013165p28058274.html