Lost file using Vim (first time ever in 15 years)

[Toddintr]

For the first time ever since I started using Vim, I lost a file. I had a modeline that specified an "encoding" setting for a Python script. Vim complained when opening the file. I googled for a solution, decided to try fileencoding instead of encoding. When saving the file, I saw some error/warning messages, but went ahead with a ZZ or w! (don't quiet remember). Result: File length of zero. (Had backups disabled.)

My point: I *LOVE* Vim. However, this sort of design decision is unacceptable. It seems to be a design decision because it was not a bug or a crash.

My two-cents' worth.

Todd

[Toddintr]

Version: Vim 7.3, 2010 Aug 15, Compiled Oct 27 2010, on Windows 7.

[Alessandro Antonello]

First of all, not encoding nor fileencoding will work on modelines because the file was already loaded. See ':h encoding' and ':h fileencoding' for that.

In my opinion, there is no bad design. You disabled write backups. That is the way Vim keeps your original file safe.

Sorry for your lost.

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

[Christian Brabandt]

Hi Toddintr!

Can you reproduce it?

regards,
Christian
--

[Toddintr]

I have been able to reproduce it. I have "vim_error.py" file with followoing contents:

# vim: set fileencoding=cp857:

ı

The file has three lines, the modeline, the second line, which is blank, and the third line, where there is a dotless lowercase i from the 857 code page.

File loads fine in Vim. Then I change 'fileencoding' in the modeline to 'encoding' (i.e. just delete the four characters 'file' from 'fileencoding'), everything else remaining the same.

Result:

"vim_error.py" (in yellow)
"vim_error.py" E513: write error, conversion failed (make 'fenc' empty to override)
WARNING: Original file may be lost or damaged
don't quit the editor until the file is successfully written! (In orange)
Press ENTER or type command to continue (in green)

What I meant by bad design decision: When the conversion fails, why not simply restore the previous buffer? The unacceptable behaviour is that even if I do a "q!", I still lose the file.

Thanks -

Todd

[shawn wilson]

On May 13, 2012 7:53 AM, "Toddintr" <amsa...@gmail.com> wrote:
>

> WARNING: Original file may be lost or damaged
> don't quit the editor until the file is successfully written! (In orange)
> Press ENTER or type command to continue (in green)
>
> What I meant by bad design decision: When the conversion fails, why not simply restore the previous buffer?  The unacceptable behaviour is that even if I do a "q!", I still lose the file.
>

So you get a warning about loosing data if you quit, quit anyway, and complain that you lost data? Interesting.

[Christian Brabandt]

Hi Toddintr!

On So, 13 Mai 2012, Toddintr wrote:

> I have been able to reproduce it. I have "vim_error.py" file with followoing contents:
>
> # vim: set fileencoding=cp857:
>
> ı
>
> The file has three lines, the modeline, the second line, which is blank, and the third line, where there is a dotless lowercase i from the 857 code page.
>
> File loads fine in Vim. Then I change 'fileencoding' in the modeline to 'encoding' (i.e. just delete the four characters 'file' from 'fileencoding'), everything else remaining the same.
>
> Result:
>
> "vim_error.py" (in yellow)
> "vim_error.py" E513: write error, conversion failed (make 'fenc' empty to override)
> WARNING: Original file may be lost or damaged
> don't quit the editor until the file is successfully written! (In orange)
> Press ENTER or type command to continue (in green)
>
> What I meant by bad design decision: When the conversion fails, why not simply restore the previous buffer? The unacceptable behaviour is that even if I do a "q!", I still lose the file.

You get a warning, that writing failed and the file may possibly be
corrupt. Vim even tells you, what you should do to write without
converting the content. What else should Vim do?

I don't understand, what Vim should possibly restore?

regards,
Christian

[A.M. Sabuncu]

I never read the warning.  Much like no one reads the manual.  I never thought I would *ever* encounter anything with Vim that would lead to losing a file. 

> ... complain that you lost data?  Interesting.

Please spare me the attitude.  I am not complaining, but trying to provide feedback to help improve the product.

When you lose a file someday, you just might understand such detached, indifferent approaches to usability mean little to the end-user.

--

[Toddintr]

I am assuming that prior to attempting to write the file and failing, Vim had a valid copy in the buffer. It should restore that copy.

Otherwise, if it is going to attempt something that is not reversible, it should never attempt it. It is one of the fundamentals of software systems design as I know it. Losing a file is the worst thing that an editor can do.

[Toddintr]

Also, I did not think that q! would result in any change to the file since the last write. This is against user expectations.

[shawn wilson]

On May 13, 2012 8:53 AM, "A.M. Sabuncu" <amsa...@gmail.com> wrote:
>
> I never read the warning.  Much like no one reads the manual.  I never thought I would *ever* encounter anything with Vim that would lead to losing a file. 
>

Ok, I don't run windows so when I get a message I *read* it. If the message happens to repeat, I figure out how to disable it.

You've never encountered anything in VI that makes you loose a file? Huh, I do it all the time. Process: vim somefile, start typing, ah I don't like that filename, :q! and that data is gone. I don't even get a warning.
/me omits sarcastic thoughts

Also, you use VI without reading the manual? That's impressive.

[shawn wilson]

On May 13, 2012 9:00 AM, "Toddintr" <amsa...@gmail.com> wrote:
>
> Also, I did not think that q! would result in any change to the file since the last write.  This is against user expectations.
>

If this is true, this is quite odd...

[Christian Brabandt]

Hi Toddintr!

(Please don't top poste and please wrap to 75 columns)

On So, 13 Mai 2012, Toddintr wrote:

> I am assuming that prior to attempting to write the file and failing,
> Vim had a valid copy in the buffer. It should restore that copy.
>
> Otherwise, if it is going to attempt something that is not reversible,
> it should never attempt it. It is one of the fundamentals of software
> systems design as I know it. Losing a file is the worst thing that an
> editor can do.

The copy is still in the buffer. It is valid. The problem is, that
writing failed because converting to your desired fileencoding failed.

The problem is probably, that Vim opened the file for writing, truncated
it and when writing noticed the error. But at this time, the old
contents are already gone.

Besides, big red warning messages should always be read, because they
don't usually happen.

regards,
Christian
--

[Toddintr]

On Sunday, May 13, 2012 4:33:40 PM UTC+3, shawn wilson wrote:

> Process: vim somefile, start typing, ah I don't like that filename, :q! and
> that data is gone. I don't even get a warning.

I see. I was not aware that you are a beginner, in grammar school.

[Erik Christiansen]

On 13.05.12 15:52, A.M. Sabuncu wrote:
> I never read the warning. Much like no one reads the manual. I never
> thought I would *ever* encounter anything with Vim that would lead to
> losing a file.

Ah, but Vim is only _largely_ foolproof. It is not proof against
deliberate self harm. It might be worth reading enough to understand
that "q!" is "Forced Quit", and so you didn't "lose" a file, but
deliberately discarded it by choice. The "Forced Quit" is quite useful
when used, if not intelligently, then at least consciously.

However, a lesson learnt the hard way is usually a lesson well learned.

Happy reading!

Erik

--
"Unix was not designed to stop you from doing stupid things, because
that would also stop you from doing clever things." - Doug Gwyn

[Toddintr]

Christian:

> The copy is still in the buffer. It is valid. The problem is, that
> writing failed because converting to your desired fileencoding failed.
>
> The problem is probably, that Vim opened the file for writing, truncated
> it and when writing noticed the error. But at this time, the old
> contents are already gone.

This explains the current way Vim handles the process. It does not have to be this way. This is what I mean by bad design. Loss of data should only happen in an unforeseen situation (a crash, etc). If you are truncating a file, especially prior to a conversion that can fail, you should take extra precautions.

> Besides, big red warning messages should always be read, because they
> don't usually happen.

This is the "Soviet" style of user interface design: it dictates *how* people should use a product, and ignores the real-life scenarios. (Much like the opening line of your message - "Please don't top poste and please wrap to 75 columns.") If I *really* care about customer feedback, I don't care how the customer sends me feedback / it can be top-posted, bottom-posted, sideways. What is important is that I am getting feedback about the worst thing that can happen w/ my product.

I am done w/ this issue, thank you all for your responses.

[Toddintr]

On Sunday, May 13, 2012 4:55:12 PM UTC+3, Erik Christiansen wrote:

> Ah, but Vim is only _largely_ foolproof. It is not proof against
> deliberate self harm. It might be worth reading enough to understand
> that "q!" is "Forced Quit", and so you didn't "lose" a file, but
> deliberately discarded it by choice. The "Forced Quit" is quite useful
> when used, if not intelligently, then at least consciously.
>
> However, a lesson learnt the hard way is usually a lesson well learned.

Typical engineer mentality, lack of any experience with real-life scenarios.

Furthermore, you are wrong. The q! should take me to the prior state of the file, which was an intact buffer before the conversion attempt. Prior to the q!, I did not do a 1,$d.

You are a good example of this entire community - looking down on others with snide remarks. Good luck to you. (Thank God there is stackoverflow where such bulls**t is not tolerated.)

[Jan Larres]

Christian Brabandt <cbl...@256bit.org>:

> The problem is probably, that Vim opened the file for writing,
> truncated it and when writing noticed the error. But at this time, the
> old contents are already gone.

But doesn't Vim first write the contents to a temporary file and then
rename it, at least in cases where the file isn't a symlink? In that
case the temporary file could be deleted instead of renamed if Vim
detects that the writing failed. It would be interesting to know the
value of 'backupcopy' here.

Jan

--
-[ OpenPGP key ID: 00A0FD5F ]-
If Jesus had been killed 20 years ago, Catholic school children would be
wearing little Electric Chairs around their necks instead of crosses.
-- Lenny Bruce

[meino....@gmx.de]

Toddintr <amsa...@gmail.com> [12-05-13 16:00]:

> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>

A software which warns users about potentially data loss and kills the
data, when the user ignores the warning is better than a software,
which only kills the data.
No doubt.
The better software dont kills the data first and tries then to
(re-)store it with a newer version of the data. The better software
keeps the old data, tries to write the new one, and - if it fails -
restores the old version. If it does not fail, it is safe to kill
the old version. Better to be safe than to be sorry ...

Think of makeing backups of a system: Would you delete the old backups
before preparing the new one?

Only my two cents,...,your software may vary...

--
mcc

[Sergey Khorev]

> If I *really* care about customer feedback, I don't care how the customer sends me
> feedback / it can be top-posted, bottom-posted, sideways.  What is important is that
> I am getting feedback about the worst thing that can happen w/ my product.

I doubt the Vim community owes you anything.
Anyway, Vim follows standard Unix way: provide reasonably safe
defaults but allow a users go and shoot himself in the leg if he wants
to. You disabled 'backup' and 'writebackup' without realising the
consequences, you got what you wanted.

[Taylor Hedberg]

Toddintr, Sun 2012-05-13 @ 06:59:42-0700:
> Christian:

> > Besides, big red warning messages should always be read, because
> > they don't usually happen.
>
> This is the "Soviet" style of user interface design: it dictates *how*
> people should use a product, and ignores the real-life scenarios.
> (Much like the opening line of your message - "Please don't top poste
> and please wrap to 75 columns.") If I *really* care about customer
> feedback, I don't care how the customer sends me feedback / it can be
> top-posted, bottom-posted, sideways. What is important is that I am
> getting feedback about the worst thing that can happen w/ my product.

I agree with you that Vim could be improved in this regard. Certainly,
not losing data at all is preferable to losing data after a warning.

Understand, though, that contrary to what your message suggests, Vim is
not a product and you are not a customer. It is developed entirely by a
community of volunteers, who are not paid anything for their work. (If
you donated, then great! But that money went to charity; it is not
lining the pockets of any developers, not even Bram's.) So in short, the
community on this list does not owe you anything at all, and whatever
help is offered is offered out of good will alone.

So, when you show up on the list and start bashing the design of the
software (perhaps understandable in this case) and admit to ignoring
bold, red error messages which explained exactly what was about to
happen to your data, you aren't likely to win much support. What's more,
you defiantly refuse to adhere to the posting conventions of the mailing
list, which are essentially the same conventions followed by nearly
every FOSS list, despite the fact that "do not top-post" appears in the
signature of every posting here, and even after being explicitly asked
to by another contributor. Exactly what kind of response do you expect
to receive?

The most constructive way to improve open-source software of this type
is to make the improvements yourself and submit patches. That's how the
real work gets done. If no one else is independently motivated to work
on the problematic behavior, and you are unable or unwilling to do it
yourself, then the only way you are going to get support is by
interacting with the community in a way that makes other people want to
help you. Only then is anyone likely to consider spending a portion of
their own personal time and energy addressing your complaint. With a
less entitled attitude, you would probably find that most people here
are very friendly and eager to help.

[Christian Brabandt]

Hi Toddintr!

On So, 13 Mai 2012, Toddintr wrote:

> > The copy is still in the buffer. It is valid. The problem is, that
> > writing failed because converting to your desired fileencoding failed.
> >
> > The problem is probably, that Vim opened the file for writing, truncated
> > it and when writing noticed the error. But at this time, the old
> > contents are already gone.
>
> This explains the current way Vim handles the process. It does not
> have to be this way. This is what I mean by bad design. Loss of data
> should only happen in an unforeseen situation (a crash, etc). If you
> are truncating a file, especially prior to a conversion that can fail,
> you should take extra precautions.

Yes true. I didn't look it up in the code, may be, Vim can indeed be
improved in this area. Where is your patch?

>
> > Besides, big red warning messages should always be read, because they
> > don't usually happen.
>

> This is the "Soviet" style of user interface design: it dictates *how*
> people should use a product, and ignores the real-life scenarios.

You just have yourself disqualified for a serious discussion. I bet, you
know nothing about the Soviet Union, and I doubt you can compare. BTW: I
used to live in a country, that was occupied by them, so I know how it
was back then.

> (Much like the opening line of your message - "Please don't top poste
> and please wrap to 75 columns.") If I *really* care about customer
> feedback, I don't care how the customer sends me feedback / it can be
> top-posted, bottom-posted, sideways. What is important is that I am
> getting feedback about the worst thing that can happen w/ my product.

I am sorry, I must have missed the fact that you are going to pay us.
You are not? Oh, well, than you shouldn't be acting like we owe you
something. Pay me, and I listen to everything you are saying and in any
style. I won't complain.

Even better, donate to the ICCF!

> I am done w/ this issue, thank you all for your responses.

So am I.

regards,
Christian

[Paul]

On Sunday, 13 May, 2012 at 13:31:38 BST, Christian Brabandt wrote:
>You get a warning, that writing failed and the file may possibly be
>corrupt. Vim even tells you, what you should do to write without
>converting the content. What else should Vim do?
>
>I don't understand, what Vim should possibly restore?

I think you're both right. He should have read the warning message, but also I'm surprised that vim modified the file before being given any instruction to write it. Usually, when you open a file in vim (or any editor) and start making changes, you can cat the file and see that it hasn't been modified - until told to save it (manually or autosave) - so I'm a bit surprised that vim actually modified the file in this scenario. I'd have expected it to read it into memory, convert it (in memory) if necessary, and not save it until told to do so. So it's not a matter of restoring anything, but rather one of not overwriting in the first place, because no write command was issued.

--

.

[John Beckett]

Paul wrote:
> I think you're both right. He should have read the warning
> message, but also I'm surprised that vim modified the file
> before being given any instruction to write it.

The discussion got very ranty earlier (folks, there is no need
to kick the guy, just be glad it wasn't you), and I ignored it.

Can someone write a clean statement of the problem (the file
contents and the commands that led to the behaviour that was
unexpected). Just the facts, without opinions on Vim or users.

John

[Ben Fritz]

On Sunday, May 13, 2012 3:10:33 AM UTC-5, Toddintr wrote:
> For the first time ever since I started using Vim, I lost a file. I had
> a modeline that specified an "encoding" setting for a Python script.
> Vim complained when opening the file. I googled for a solution, decided
> to try fileencoding instead of encoding. When saving the file, I saw
> some error/warning messages, but went ahead with a ZZ or w! (don't quiet
> remember). Result: File length of zero. (Had backups disabled.)
>

Later you say you used :q! and lost data. Here I see you actually saved the
file with a "force save and quit" and lost data. That's a big difference,
but I admit somewhat irrelevant to the discussion, which has gotten quite
out of hand. I understand you're frustrated. Let's all settle down, we may
have a real issue here (which may have been fixed since 7.3.0, of course).
Regardless, YOU (the original poster) are not going to get anywhere by
outright insulting long-time and very respected members of the Vim
community.

I think I gather from later posts that the problem occurs when you set
'encoding' but not 'fileencoding' from a modeline.

:help 'encoding' says:

This option cannot be set from a |modeline|. It would most likely
corrupt the text.

:help changed-7.3 indicates this protection was added sometime between 7.2
and 7.3, but I don't see it in the changelog for 7.2 patches, so apparently
sometime in the run-up for 7.3 somebody realized setting Vim's internal
encoding via modeline is a very bad idea and fixed this issue. However, your
followup post with version information indicates that you should not have
been able to set the encoding via modeline.

On my Windows Vim, if I successfully save a file with a modeline containing
"encoding=cp857", I get an error when loading the file:

E520: Not allowed in a madeline: encoding=cp857

Setting fileencoding in a modeline, by the way, will not do what you
probably intend. A modeline is processed only AFTER a file is read into Vim,
so setting it in the modeline can only affect how the file is written, it
will NOT affect how Vim interprets the bytes in the file while reading. To
do this, you would need to properly set 'fileencodings' (note the final 's'
in the option name) to detect your desired encoding, PRIOR to reading the
file. A BufReadPre autocmd is a good place to do this, or just have it
always set properly. Another option, if you know the file's encoding, is to
use a ++enc=cp857 argument when editing the file, e.g.

:edit ++enc=cp857 myfile.txt

Also see the AutoFenc plugin:

http://www.vim.org/scripts/script.php?script_id=2721

Anyway, that was slightly off topic. I tried reproducing the error with your
example file, and could not. No matter what I did, without changing text
within the buffer, I was unable to wind up with an empty file by saving
and/or quitting, with or without a !. :wq! doesn't even work for me with a
character in the file invalid for the current 'fileencoding' (e.g. when I
load the file containing the undotted 'i' in your example, without Vim
detecting the correct encoding, and then try to write in the correct
encoding without correcting the character first), it fails to write and
fails to quit, since Vim doesn't know HOW to write. Can you give the exact
commands you execute to result in an empty file? And, what is the initial
value of 'encoding'? And your 'fileencodings' option? Finally, this is
possibly a stupid question, but are you sure you lost your data in the 7.3.0
version of Vim you say you are using? Maybe you accidentally loaded the file
in an older version you still have installed alongside somewhere?

[Sergey Khorev]

> Can someone write a clean statement of the problem (the file
> contents and the commands that led to the behaviour that was
> unexpected). Just the facts, without opinions on Vim or users.

This is how I managed to reproduce. Create a file:
----------- snip -----------

# vim: set fileencoding=cp857:

ı

----------- snip -----------

Open it in Vim, edit, then
:set nobackup nowritebackup
:w!

Get the warning and then check the file got truncated.

[Christian Brabandt]

Bram,

To make this reproducible, I needed to set fenc=ascii

This patch fixes a data conversion error, that leads to a data loss.
Problem is, that open() is done, even though it is not clear, conversion
will work. So this patch runs buf_write_bytes() twice, first time is to
check that each character can be converted successfully and only if this
does not return, does it really open the file and write the result.

Problem is, this will probably slow down writing speed, as
buf_write_bytes needs to be called twice.

I have tested it with the given sample and it works here.

regards,
Christian

[Gary Johnson]

On 2012-05-14, Christian Brabandt wrote:

> This patch fixes a data conversion error, that leads to a data loss.
> Problem is, that open() is done, even though it is not clear, conversion
> will work. So this patch runs buf_write_bytes() twice, first time is to
> check that each character can be converted successfully and only if this
> does not return, does it really open the file and write the result.
>
> Problem is, this will probably slow down writing speed, as
> buf_write_bytes needs to be called twice.
>
> I have tested it with the given sample and it works here.

How much slower is it for a file large enough that the write times
are more than a second?

I don't think we want to noticeably slow down Vim for everyone all
the time in order to more-conveniently avoid an error that is
observed once in 15 years.

Regards,
Gary

[Ben Fritz]

On Monday, May 14, 2012 1:34:52 PM UTC-5, Gary Johnson wrote:
>
> How much slower is it for a file large enough that the write times
> are more than a second?
>
> I don't think we want to noticeably slow down Vim for everyone all
> the time in order to more-conveniently avoid an error that is
> observed once in 15 years.
>

We have this situation:

1. user opens a file
2. (user does some edits)
3. user tries to write the file, gets a conversion error
4. user uses :q!, attempting to abandon all changes
5. user loses ENTIRE CONTENTS of file

I think I'd accept a slow-down in this case!

Granted, the user should have paid attention to the big message saying "don't quit the editor until the file is successfully written!", however, it may not be clear in the slightest how to successfully write the file. What if somebody is editing a huge file (and thus has undolevel set really low), pastes several hundred lines of text from outside Vim, within which a single character is invalid for their current encoding, and then realizes they cannot write the buffer? My first instinct would be to abandon all changes, but apparently that's not going to work here.

[Andy Wokula]

Am 14.05.2012 20:21, schrieb Christian Brabandt:
> Bram,
> On Mo, 14 Mai 2012, Sergey Khorev wrote:
>>> Can someone write a clean statement of the problem (the file
>>> contents and the commands that led to the behaviour that was
>>> unexpected). Just the facts, without opinions on Vim or users.
>>
>> This is how I managed to reproduce. Create a file:
>> ----------- snip -----------
>> # vim: set fileencoding=cp857:
>>

>> i

>> ----------- snip -----------
>>
>> Open it in Vim, edit, then
>> :set nobackup nowritebackup
>> :w!
>>
>> Get the warning and then check the file got truncated.

Thanks, was hard enough to get this error message working ^^

btw, s/warning/error/

> To make this reproducible, I needed to set fenc=ascii
>
> This patch fixes a data conversion error, that leads to a data loss.
> Problem is, that open() is done, even though it is not clear, conversion
> will work. So this patch runs buf_write_bytes() twice, first time is to
> check that each character can be converted successfully and only if this
> does not return, does it really open the file and write the result.
>
> Problem is, this will probably slow down writing speed, as
> buf_write_bytes needs to be called twice.
>
> I have tested it with the given sample and it works here.
>
> regards,
> Christian

Setting 'writebackup' (enabled per default) indeed protects the file
(except in the temp folder, where 'backupskip' applies).

This option creates temporary files only, thus doesn't waste disk space.

Isn't this safe enough?

Am 13.05.2012 10:10, schrieb Toddintr:
> (Had backups disabled.)

Bad luck ... next time, keep 'writebackup' at the default and all should
be fine.

--
Andy

[Gary Johnson]

On 2012-05-14, Ben Fritz wrote:
> On Monday, May 14, 2012 1:34:52 PM UTC-5, Gary Johnson wrote:
> >
> > How much slower is it for a file large enough that the write times
> > are more than a second?
> >
> > I don't think we want to noticeably slow down Vim for everyone all
> > the time in order to more-conveniently avoid an error that is
> > observed once in 15 years.
> >
>
> We have this situation:
>
> 1. user opens a file
> 2. (user does some edits)
> 3. user tries to write the file, gets a conversion error
> 4. user uses :q!, attempting to abandon all changes
> 5. user loses ENTIRE CONTENTS of file
>
> I think I'd accept a slow-down in this case!

If a slow-down was the only solution, sure. I had the impression
from the earlier discussion that the problem was avoidable by
reading the error message and being careful. Hence my description
of the fix as one that would allow the error to be more conveniently
avoided.

> Granted, the user should have paid attention to the big message
> saying "don't quit the editor until the file is successfully
> written!", however, it may not be clear in the slightest how to
> successfully write the file. What if somebody is editing a huge
> file (and thus has undolevel set really low), pastes several
> hundred lines of text from outside Vim, within which a single
> character is invalid for their current encoding, and then realizes
> they cannot write the buffer? My first instinct would be to
> abandon all changes, but apparently that's not going to work here.

As I understand the problem, they also had to have 'nobackup' and
'nowritebackup' set, the latter which is not the default. So the
user threw away the safety net, disregarded the warning sign, and
because of this I have to wait longer each time I write a file?

Of course, if the slowdown is negligible, then there's probably no
reason not to go with Christian's change.

Regards,
Gary

[Christian Brabandt]

Hi Gary!

A small test suggests, that for a 15M file, writing happens here within
0,44s while the patched version takes 1,08s. For smaller files, the time
is negligible, but I fear, for larger files, it is quite noticeable.

Here is another test with a 40B, 1K, 4K, 50K, 500K, 1M, 15M, 70M files:

Size Run Patched Non-patched
40B 1 0,096s 0,089s
2 0,084s 0,114s
3 0,108s 0,099s
� 0,096s 0,101s
1K 1 0,085s 0,080s
2 0,108s 0,079s
3 0,107s 0,099s
� 0,100s 0,086s
4K 1 0,096s 0,072s
2 0,096s 0,089s
3 0,087s 0,089s
� 0,093s 0,083s
50K 1 0,117s 0,082s
2 0,148s 0,114s
3 0,164s 0,088s
� 0,143s 0,095s
100K 1 0,138s 0,102s
2 0,160s 0,079s
3 0,137s 0,090s
� 0,145s 0,090s
500K 1 0,159s 0,102s
2 0,185s 0,083s
3 0,186s 0,113s
� 0,177s 0,099s
1M 1 0,216s 0,089s
2 0,233s 0,108s
3 0,206s 0,113s
� 0,218s 0,103s
15M 1 1,098s 0,212s
2 1,089s 0,110s
3 1,081s 0,097s
� 1,089s 0,140s
70M 1 6,809s 0,097s
2 7,160s 0,193s
3 0,699s 0,097s
� 4,889s 0,129s

Here the non-patched versions are a lot faster, because the bug is
triggered at the beginning of the filewrite, and writing is aborted
afterwards, While the patched version try twice.

Here is a small update to the patch: break in case of errors.
But perhaps, it isn't worth the trouble, since 'writebackup' prevents
that bug.

regards,
Christian

[John Beckett]

To reproduce problem:

1. Create file bad.tmp by executing the following line.
:call writefile(["Very important file \x8D"], 'bad.tmp')

2. Start Vim editing the file and enter commands, as below.
vim -N -u NONE bad.tmp
:set fenc=cp857
:set nobackup nowritebackup
:w

3. Observe message which you have probably never seen before:
"bad.tmp" E513: write error, conversion failed

(make 'fenc' empty to override)
WARNING: Original file may be lost or damaged

don't quit the editor until the file is successfully written!

4. Panic and do what has always worked in the past:
:q!

Result:
- Step 2 replaces bad.tmp with an empty file.
- The original contents have been lost.

The "no backup" options cause Vim to overwrite the existing
file. However, the write aborts when the conversion fails,
resulting in lost data.

While a user should not turn off backups on an important file,
it is a horrible trap that one incorrect byte can effectively
delete a file.

A message like the following might help:

"bad.tmp" E513: write error, conversion failed
WARNING: The file on disk has probably been corrupted.
Do not quit until the file is successfully written!
Try saving it again after entering (nothing after "="):
:set fenc=

I haven't done more than glance at Christian Brabandt's patch
so I don't know what is achievable. Would something as simple
as checking the number of bytes in the file on disk be
sufficient to decide if corrective action is needed?

John

[Ben Fritz]

On Tuesday, May 15, 2012 2:49:38 AM UTC-5, JohnBeckett wrote:
> To reproduce problem:
>
> 1. Create file bad.tmp by executing the following line.
> :call writefile(["Very important file \x8D"], 'bad.tmp')
>
> 2. Start Vim editing the file and enter commands, as below.
> vim -N -u NONE bad.tmp
> :set fenc=cp857

What if, right here, Vim were to check the buffer contents to see if all characters in the file can actually be converted to the new fenc setting, and issue a warning if not? It's not something Vim does currently...but I don't see why it couldn't. And I'd like to know if I do something stupid *when I do it*, not when I try to write the file and end up killing my file.

> :set nobackup nowritebackup
> :w
>
> 3. Observe message which you have probably never seen before:
> "bad.tmp" E513: write error, conversion failed
> (make 'fenc' empty to override)
> WARNING: Original file may be lost or damaged
> don't quit the editor until the file is successfully written!
>
> 4. Panic and do what has always worked in the past:
> :q!
>

Right, to me, :q! means "oh crap I didn't mean to do that, GET ME OUT!"

[Ben Fritz]

On Monday, May 14, 2012 2:39:54 PM UTC-5, Gary Johnson wrote:
>
> As I understand the problem, they also had to have 'nobackup' and
> 'nowritebackup' set, the latter which is not the default. So the
> user threw away the safety net, disregarded the warning sign, and
> because of this I have to wait longer each time I write a file?
>
> Of course, if the slowdown is negligible, then there's probably no
> reason not to go with Christian's change.
>

Since the error only occurs if 'backup' and 'writebackup' are both off, maybe we could only do the "dry run" conversion in this situation? I.e., if either 'backup' or 'writebackup' are set, do what Vim has always done; but if neither are set, do the extra check first. We could even put a nice notice in :help 'writebackup' saying "if this option is off, writing the file will take longer due to extra safety checks" to discourage turning it off in the first place.

[shawn wilson]

This sounds sane.

[Tim Chase]

On 05/15/12 10:20, Ben Fritz wrote:
> On Tuesday, May 15, 2012 2:49:38 AM UTC-5, JohnBeckett wrote:
>> 1. Create file bad.tmp by executing the following line.
>> :call writefile(["Very important file \x8D"], 'bad.tmp')
>>
>> 2. Start Vim editing the file and enter commands, as below.
>> vim -N -u NONE bad.tmp
>> :set fenc=cp857
>
> What if, right here, Vim were to check the buffer contents to see
> if all characters in the file can actually be converted to the
> new fenc setting, and issue a warning if not? It's not something
> Vim does currently...but I don't see why it couldn't. And I'd
> like to know if I do something stupid *when I do it*, not when I
> try to write the file and end up killing my file.

I like Ben's suggestion on this one. Concerns that come to mind:

- warnings might interrupt scripts that run

- how it would interact with the "let &setting=" syntax

- would this just be a message (that gets ignored?) or a prompted
"do you want to undo this catastrophically dangerous action?"

>> 4. Panic and do what has always worked in the past:
>> :q!
>>
>
> Right, to me, :q! means "oh crap I didn't mean to do that, GET ME OUT!"

The caveat here is that, if you have performed an action that might
have fouled up the file-on-disk, your only valid source might be
what is in Vim's buffer. So if you get a warning, think hard before
attempting to quit. E.g.

:e important.txt
:!rm %

get "E211: File "important.txt" no longer available" message. Panic and

:q!

Clearly, vim shouldn't try to override what you expressly did (I
wouldn't want Vim to re-save the file to the disk). In this case,
even ":q" abandon the content without complaint.

-tim

[Christian Brabandt]

On Tue, May 15, 2012 17:20, Ben Fritz wrote:
> What if, right here, Vim were to check the buffer contents to see if all
> characters in the file can actually be converted to the new fenc setting,
> and issue a warning if not? It's not something Vim does currently...but I
> don't see why it couldn't. And I'd like to know if I do something stupid
> *when I do it*, not when I try to write the file and end up killing my
> file.

That doesn't work, you could still afterwards enter glyphs in utf8, that
can't be converted into the fileencoding.

regards,
Christian

[Benjamin Fritz]

This is true, but I'm more likely to remember not to enter glyphs
which can't be encoded in the current fenc, than I am to know that
these glyphs already exist in the buffer when changing the encoding. I
think I'd still like to be warned when I change file encoding.

Then again...since you point this out, it won't help in preventing the
error that started this thread, if the "user does something" step is
not to change file encoding, but rather to insert characters which
cannot be encoded.

I think the better idea is to do what your patch does, but only if
'writebackup' is disabled to prevent a slow-down when there is already
an existing mechanism to prevent data loss.

[Chris Jones]

On Tue, May 15, 2012 at 11:23:57AM EDT, Ben Fritz wrote:

> Since the error only occurs if 'backup' and 'writebackup' are both
> off, maybe we could only do the "dry run" conversion in this
> situation? I.e., if either 'backup' or 'writebackup' are set, do what
> Vim has always done; but if neither are set, do the extra check first.
> We could even put a nice notice in :help 'writebackup' saying "if this
> option is off, writing the file will take longer due to extra safety
> checks" to discourage turning it off in the first place.

So you would penalize those of us who turned off this backup/writebackup
bloat..?

Or would you force us to reinstate it so as not to be penalized..?

Aha... you turned off backup and writebackup.. you wicked little
so-and-so.. You seriously thought you were going get away with it..?

I made the decision to turn off this stuff years ago because when I'm
bleary-eyed not wide-awake or fully in control getting back to what
I was doing the night before, the very last thing I want to see when
I fire up Vim is a confusing message about some .swp file and do I want
to recover (r).. etc. The _only_ time I came close to losing _anything_
in Vim was when I almost gave the wrong answer.

I don't believe in a text editor backing up the the data that it
manipulates anyway. I prefer relying on important data being backed up
by a program whose job it is to back up data. Not perfect I grant you,
but it does have the merit of doing its stuff while I sleep and let me
do my editing unmolested.

Now suppose for a minute Christians's (or s/o else's..) patch causes
another data loss issue down the road.. Or any other problem.? Is the
next guy in line going to add another defensive patch on top of it..?
and another.. and another..? Where does it stop..?

No big deal anyway.. What does bother me is what on earth was in that
precious file that the OP (obnoxious poster) lost..?

AFAICT, World War III hasn't broken out yet but I'm holding my breath.

CJ

--
Focus follow mouse users will burn in hell!!!

[Ben Fritz]

On Tuesday, May 15, 2012 12:24:21 PM UTC-5, Chris Jones wrote:
> On Tue, May 15, 2012 at 11:23:57AM EDT, Ben Fritz wrote:
>
> > Since the error only occurs if 'backup' and 'writebackup' are both
> > off, maybe we could only do the "dry run" conversion in this
> > situation? I.e., if either 'backup' or 'writebackup' are set, do what
> > Vim has always done; but if neither are set, do the extra check first.
> > We could even put a nice notice in :help 'writebackup' saying "if this
> > option is off, writing the file will take longer due to extra safety
> > checks" to discourage turning it off in the first place.
>
> So you would penalize those of us who turned off this backup/writebackup
> bloat..?
>

From Christian's test data, it looked like the slow-down was on the order of a few seconds for a write of a big file. I think that would be a small price to pay to know that the entire content of your file isn't going to be lost because Vim decided it could not modify them. I don't consider it a penalty. I consider it giving users some of the security of 'writebackup' even if they don't have it turned on.

> Or would you force us to reinstate it so as not to be penalized..?
>
> Aha... you turned off backup and writebackup.. you wicked little
> so-and-so.. You seriously thought you were going get away with it..?
>

It's risky, but it's also a valid use scenario, so that's why I want to see the file contents protected against being cleared for no good reason, even if the option is turned off.

[Bram Moolenaar]

The 'writebackup' option is to catch cases where writing fails. It
means that Vim can restore the original file. There is no point in
additional work.

If a user switches off 'writebackup' he must be prepared for losing
work. Perhaps this is not sufficiently clear to the user?

Considering how very few times this problem occurs I don't think it is
justified to do any work on this. Except perhaps a better explanation
in the docs in some place.

--
SOLDIER: Where did you get the coconuts?
ARTHUR: Through ... We found them.
SOLDIER: Found them? In Mercea. The coconut's tropical!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Christian Brabandt]

Hi Ben!

On Di, 15 Mai 2012, Ben Fritz wrote:
> Since the error only occurs if 'backup' and 'writebackup' are both
> off, maybe we could only do the "dry run" conversion in this
> situation? I.e., if either 'backup' or 'writebackup' are set, do what
> Vim has always done; but if neither are set, do the extra check first.
> We could even put a nice notice in :help 'writebackup' saying "if this
> option is off, writing the file will take longer due to extra safety
> checks" to discourage turning it off in the first place.

I think this is a good compromise¹. First off all, a successful write
should have priority, so one needs to be very cautious, in case an
encoding from 'enc' to 'fenc' needs to be done.

That small write penalty should be ok, since it is only noticeable when
writing larger files (e.g. files with a size of several megabytes) and
although I usually use Vim with large csv files of several hundred
megabytes, I only seldom save files with that size, so I think editing
so large files is not done by the majority of vim users.

I have done some more statistics and in the standard case (when
writebackup is set), writing is as fast as usually, while only when
'writebackup' is turned off, write performance is impacted.


¹) updated patch at https://gist.github.com/2704019

regards,
Christian
--
Glück ist ein Stuhl, der plötzlich dasteht, wenn man sich zwischen
zwei andere setzen will.
-- George Bernard Shaw

[Tim Chase]

On 05/15/12 13:46, Christian Brabandt wrote:
> That small write penalty should be ok, since it is only
> noticeable when writing larger files (e.g. files with a size of
> several megabytes) and although I usually use Vim with large csv
> files of several hundred megabytes,

I also use Vim on CSV files of 10-100 megs on occasion. I'd also
like to make sure the tests are also performed on slow network
connections. Occasionally those CSV files come from a SMB share
over wireless which is *ehem* less than performant. I don't know
how netrw would tie into this testing as well.

-tim

[Ben Fritz]

On Tuesday, May 15, 2012 1:33:39 PM UTC-5, Bram Moolenaar wrote:
> Ben Fritz wrote:
>
> > On Monday, May 14, 2012 2:39:54 PM UTC-5, Gary Johnson wrote:
> > >
> > > As I understand the problem, they also had to have 'nobackup' and
> > > 'nowritebackup' set, the latter which is not the default. So the
> > > user threw away the safety net, disregarded the warning sign, and
> > > because of this I have to wait longer each time I write a file?
> > >
> > > Of course, if the slowdown is negligible, then there's probably no
> > > reason not to go with Christian's change.
> > >
> >
> > Since the error only occurs if 'backup' and 'writebackup' are both
> > off, maybe we could only do the "dry run" conversion in this
> > situation? I.e., if either 'backup' or 'writebackup' are set, do what
> > Vim has always done; but if neither are set, do the extra check first.
> > We could even put a nice notice in :help 'writebackup' saying "if this
> > option is off, writing the file will take longer due to extra safety
> > checks" to discourage turning it off in the first place.
>
> The 'writebackup' option is to catch cases where writing fails. It
> means that Vim can restore the original file. There is no point in
> additional work.
>
> If a user switches off 'writebackup' he must be prepared for losing
> work. Perhaps this is not sufficiently clear to the user?
>
> Considering how very few times this problem occurs I don't think it is
> justified to do any work on this. Except perhaps a better explanation
> in the docs in some place.
>

I don't think it's clear at all. The help for 'writebackup' gives no indication that this option is important to leave on to prevent losing not just your edits in progress, but the entire file you're editing. This is especially surprising when the write error has nothing to do with the ability to write to the filesystem, only with Vim's ability to convert characters between encodings. Additionally, it's not just a matter of 'writebackup'. The 'backupskip' option prevents 'writebackup' from taking effect if it matches. So if a user is editing a file in the 'backupskip' pattern, potentially in a directory covered by this pattern BY DEFAULT, like /tmp, they risk losing data.

I don't think that default Vim settings should ever let Vim delete an entire file without being asked to do so.

At least this thread prompted me to double-check and make sure 'writebackup' is safely set on my system :-)

[John Beckett]

Bram Moolenaar wrote:
> If a user switches off 'writebackup' he must be prepared for
> losing work. Perhaps this is not sufficiently clear to the
> user?
>
> Considering how very few times this problem occurs I don't
> think it is justified to do any work on this. Except perhaps
> a better explanation in the docs in some place.

How about twweaking the error message?

Vim currently shows:

"bad.tmp" E513: write error, conversion failed
(make 'fenc' empty to override)
WARNING: Original file may be lost or damaged don't
quit the editor until the file is successfully written!

I suggest:

"bad.tmp" E513: write error, conversion failed
WARNING: The file on disk has probably been corrupted.
Do not quit until the file is successfully written!
Try saving it again after entering (nothing after "="):
:set fenc=

Or, when writebackup is switched off, issue a clear warning?

John

[Gary Johnson]

That is better.

In this scenario, the user has edited the contents of the file to
his liking and wants to save those contents to disk. He has also
chosen the file-encoding that he wants. He doesn't know that the
two are incompatible. He may not even know that it's possible for
the two to be incompatible.

The error messages above tell the user how to safely save the file
contents. Neither tell him what the problem is, where the problem
is, or how to get to the desired state of those contents saved with
the desired file-encoding. That is, after a successful sequence of
":set fenc=", ":w", the user is still left without the desired
file-encoding and no clue as how to change to it.

It would be helpful to the user if the error message included a
statement like, "Character at line 77, column 14 cannot be encoded
with the specified 'fileencoding'."

It that is too much work to implement, then at least ":help E513"
should be more explicit about that particular error, the cause, and
possible solutions. The error message might even suggest reading
":help E513" since that doesn't seem to occur to some users.

If I have time this evening and no one beats me to it, I'll try to
write up something.

> Or, when writebackup is switched off, issue a clear warning?

That would be really annoying, especially if the user knew what he
was doing and did it often.

Regards,
Gary

[Michael Henry]

On 05/15/2012 08:42 PM, Gary Johnson wrote:
> In this scenario, the user has edited the contents of the file to
> his liking and wants to save those contents to disk. He has also
> chosen the file-encoding that he wants. He doesn't know that the
> two are incompatible. He may not even know that it's possible for
> the two to be incompatible.

How does the following sound for damage control? If Vim
encounters an encoding error during a write, it could
automatically re-try the write with utf-8 (or whatever fail-safe
encoding makes sense) to prevent leaving the user with an empty
file. This shouldn't cost extra time for users with valid
settings. Since the clobbered file was of zero length anyway,
it seems that nothing further would be lost by overwriting the
empty file with data in some default encoding. Then the error
message could be changed to mean "your encoding failed; I've
replaced your file with an encoding that will work; change your
encoding and try again if you don't like this choice". Then, a
user with invalid settings who issues :q! will not find his file
clobbered, and users with valid settings won't be penalized with
slower writes.

Michael Henry

[Marko Mahnič]

On May 16, 4:59 am, Michael Henry wrote:
>
> How does the following sound for damage control?  If Vim
> encounters an encoding error during a write, it could
> automatically re-try the write with utf-8 (or whatever fail-safe
> encoding makes sense) to prevent leaving the user with an empty
> file.

Wouldn't it be better if Vim would ask the user what to do when
encoding
with 'fenc' fails? Some of the options could be:
- save with 'enc' encoding
- save with 'fenc' encoding and replace invalid characters with '?'
- don't save anything, I'll fix it

This options should be give no matter what the value of 'writebackup'
is.
Until the user selects an option, nothing should be written to disk.
Only
then the user can be blamed if he looses data.

On May 15, 1:33pm, Bram Moolenaar wrote:
>
> If a user switches off 'writebackup' he must be prepared for losing
> work. Perhaps this is not sufficiently clear to the user?

The file should never be truncated because of some 'error' in the
settings.
The user that removed the 'writebackup' may not be the same user that
set
'fenc'. It is not fair for the second user to pay the price of
someone
else's 'mistake'. The software is there to handle such cases without
loosing any data.

Marko

[Benjamin Fritz]

On Wed, May 16, 2012 at 2:29 AM, Marko Mahnič <marko....@gmail.com> wrote:
> On May 16, 4:59 am, Michael Henry wrote:
>>
>> How does the following sound for damage control?  If Vim
>> encounters an encoding error during a write, it could
>> automatically re-try the write with utf-8 (or whatever fail-safe
>> encoding makes sense) to prevent leaving the user with an empty
>> file.

I don't like it, because a failed write will still modify the file. I
think nobody would expect that a failed write would modify the file.
I'd rather check beforehand whether the write can succeed before even
trying. I do like it better than destroying the file. I'd rather have
the file encoding change than the file contents be deleted. A warning
message should be displayed, that says, "Encoding conversion error
corrupted file contents during write. File encoding on disk is now
xxxxx."

>
> Wouldn't it be better if Vim would ask the user what to do when
> encoding
> with 'fenc' fails?  Some of the options could be:
>   - save with 'enc' encoding
>   - save with 'fenc' encoding and replace invalid characters with '?'
>   - don't save anything, I'll fix it
>
> This options should be give no matter what the value of 'writebackup'
> is.

This would be nice.

> Until the user selects an option, nothing should be written to disk.

The problem is, something IS written to disk before the conversion
error occurs. The file is truncated in preparation for writing, but
then the write fails, so we abandon the write. But the damage has
already been done.

Is there a reason we couldn't do the encoding conversion in memory,
and if it succeeds, only then open the file to write the data? That
seems like the sanest way to do things. Only open the file when you
know the write will succeed. IIUC, that's the gist of Christian's
patch, except the patch does the conversion twice to determine whether
the write will succeed. Why can't we just store off and re-use the
already converted text instead of converting again?

[Christian Brabandt]

Hi Benjamin!

On Mi, 16 Mai 2012, Benjamin Fritz wrote:

> Is there a reason we couldn't do the encoding conversion in memory,
> and if it succeeds, only then open the file to write the data? That
> seems like the sanest way to do things. Only open the file when you

Well, I wouldn't rather not have 2 copies of the buffer in memory.

regards,
Christian
--

[Benjamin R. Haskell]

Surely there's a way to "do the encoding conversion in memory" without
creating a full copy of the buffer. Just running through the buffer
character-by-character, or line-wise, and ensuring that all characters
can be encoded using &fenc should work.

--
Best,
Ben H

[Alessandro Antonello]

But, this way, how can Vim rollback if the conversion fails? Undoing everything in reverse order? IMHO, the 'writebackup' option should be ignored when a character encoding conversion is needed. Warning the user when that fails.

[Benjamin R. Haskell]

[Reversing the top-posting]

On Thu, 17 May 2012, Alessandro Antonello wrote:

> Em 17/05/2012 08:09, "Benjamin R. Haskell" escreveu:
>
>> On Thu, 17 May 2012, Christian Brabandt wrote:
>>
>> Hi Benjamin!
>>>
>>> On Mi, 16 Mai 2012, Benjamin Fritz wrote:
>>>
>>>> Is there a reason we couldn't do the encoding conversion in memory,
>>>> and if it succeeds, only then open the file to write the data? That
>>>> seems like the sanest way to do things.
>>>>
>>>

>>> Well, I wouldn't rather not have 2 copies of the buffer in memory.
>>>
>>
>> Surely there's a way to "do the encoding conversion in memory"
>> without creating a full copy of the buffer. Just running through the
>> buffer character-by-character, or line-wise, and ensuring that all
>> characters can be encoded using &fenc should work.
>

> But, this way, how can Vim rollback if the conversion fails? Undoing
> everything in reverse order?

Sorry for the imprecision: there's nothing to rollback... It's a test
conversion. More fully:

If encoding conversion is necessary for the write, there are two passes
through the buffer:

First pass: for every character in the buffer, ensure that it can be
converted to the target encoding. (This is in memory, so it's not as
time-consuming as a full "practice write" to disk.)

Second pass: write the file to disk, using the target encoding
(converting on-the-fly, or however the current process works).


Not sure how this would work for BufWriteCmd hackery. (But I'm not sure
how any of the proposals would work w.r.t. BufWriteCmd)

> IMHO, the 'writebackup' option should be ignored when a character
> encoding conversion is needed. Warning the user when that fails.

IMHO, the 'writebackup' option should be kept to whatever the user sets
it.

Vim simply shouldn't destroy the user's data due to a failed character
conversion. Despite its rarity, it's a completely-preventable condition
that Vim can detect before the original file is truncated.

--
Best,
Ben H

[Christian Brabandt]

Hi Benjamin!

On Do, 17 Mai 2012, Benjamin R. Haskell wrote:

> If encoding conversion is necessary for the write, there are two
> passes through the buffer:
>
> First pass: for every character in the buffer, ensure that it can be
> converted to the target encoding. (This is in memory, so it's not
> as time-consuming as a full "practice write" to disk.)
>
> Second pass: write the file to disk, using the target encoding
> (converting on-the-fly, or however the current process works).

That is what my patch does.

> Vim simply shouldn't destroy the user's data due to a failed
> character conversion. Despite its rarity, it's a
> completely-preventable condition that Vim can detect before the
> original file is truncated.

Yes.

regards,
Christian
--

[Benjamin R. Haskell]

On Thu, 17 May 2012, Christian Brabandt wrote:

> Hi Benjamin!
>
> On Do, 17 Mai 2012, Benjamin R. Haskell wrote:
>
>> If encoding conversion is necessary for the write, there are two
>> passes through the buffer:
>>
>> First pass: for every character in the buffer, ensure that it can be
>> converted to the target encoding. (This is in memory, so it's not as
>> time-consuming as a full "practice write" to disk.)
>>
>> Second pass: write the file to disk, using the target encoding
>> (converting on-the-fly, or however the current process works).
>
> That is what my patch does.

The updated one? Or did I misinterpret the test results from this
thread? It sounded as though the change doubled the time it would take
to write a large file.

>> Vim simply shouldn't destroy the user's data due to a failed
>> character conversion. Despite its rarity, it's a
>> completely-preventable condition that Vim can detect before the
>> original file is truncated.
>
> Yes.

If your patch doesn't double the time it takes to write large files, and
it prevents data loss, then it seems clearly to be the right thing to
do.

--
Best,
Ben H

[Christian Brabandt]

Hi Benjamin!

On Do, 17 Mai 2012, Benjamin R. Haskell wrote:

> On Thu, 17 May 2012, Christian Brabandt wrote:
>
> >Hi Benjamin!
> >
> >On Do, 17 Mai 2012, Benjamin R. Haskell wrote:
> >
> >>If encoding conversion is necessary for the write, there are two
> >>passes through the buffer:
> >>
> >>First pass: for every character in the buffer, ensure that it
> >>can be converted to the target encoding. (This is in memory, so
> >>it's not as time-consuming as a full "practice write" to disk.)
> >>
> >>Second pass: write the file to disk, using the target encoding
> >>(converting on-the-fly, or however the current process works).
> >
> >That is what my patch does.
>
> The updated one? Or did I misinterpret the test results from this
> thread? It sounded as though the change doubled the time it would
> take to write a large file.

Both did. I was also surprised, that just converting the buffer content
without writing almost doubled the write speed. The updated one just
kicked in, when no backup was done.


regards,
Christian
--

[Benjamin Fritz]

Why not? I doubt very much somebody will be so low on memory that they
can't add another buffer to Vim. It would only be there temporarily as
well. And it wouldn't be an exact copy, unless fenc==enc, in which
case the conversion cannot fail and we can skip the conversion step.