Hex viewer/editor with vim

[Peter de Ridder]

Hi,

Using vim do edit binary files isn't always that easy. With
display+=uhex you can make unprintable characters show as hex, but the
printable characters are still shown as normal characters. Using xxd
so convert a file to hex gives a nice view of the file in both hex and
printable characters. But editing is still a bit difficult even with
hex editing helper scripts. e.g. insering or removing data isn't easy
since all the xxd offsets number need to be renumbered.

To make it easier for hex editing in vim I try the split hex editing
in different parts which could be added to vim and be setup with a
command e.g. :hexthis (like :diffthis for diffing). The parts I came
up with are:
- Show all characters (printable or unprintable) as hex.
- Show unprintable characters as a dot.
- Number lines with byte offsets.
- Columnize every x bytes.
- Enter characters as hex.
- Join lines over eol and make eol editable as if it was a character.
- Other features like vsplit and scrollbind are already present.

I know some if these options can be either impossible or to hard to be
reasonable but I put them in the list to be complete.
I started on implementing a few of the parts. I added preliminary
patches for them.

I'm providing the preliminary patches so other people can test them,
tell what is missing, give pointers in to right direction or provide
ideas on further improvements. I'm interested in using these feature
myself maybe others are interested too and maybe these can become part
of the newer versions of vim, when completed.

I'll explain the different parts a bit further:
- Show all characters (printable or unprintable) as hex.
Preliminary patch: hex-displaymode.patch
A per window setting to display all characters as hex, making all
characters 2 display cells wide. This can be used as main hex editing
window. Like the hex data part of xxd. The patch adds window option
'displaymode'='hex'.
- Show unprintable characters as a dot.
Preliminary patch: hex-displaymode.patch
A per window setting to display all unprintable characters as dots,
making all characters 1 display cell wide. This way this can be setup
with scrollbinding to the main hex editing window and have half the
width. Since the main window will have characters of 2 wide and this
window will have characters of 1 wide, so the wrapping and such will
match and scrollbinding can be used. This setting is like the right
printable character side of xxd. The patch adds window option
'displaymode'='dot'.
- Number lines with byte offsets.
Preliminary patch: hex-hexnumber.patch
A setting like 'number' and the recent 'relativenumber' but instead of
line number it shows line byte offsets. The +byte_offset feature
provides the internals for this. Line the first column if xxd. The
patch adds window option 'hexnumber'.
- Columnize every x bytes.
Experimental patch: hex-columnview.patch
Columnize characters in groups of x bytes or multiple groups. And
break a line at x bytes neither by inserting eol or dependant on the
window width. Gives a better for of data in (mostly) the
'displaymode'='hex' window. The patch adds
'columnview'={break},{count},{count},... ('columnize' might be a
better name?)
- Enter characters as hex.
Type only [0-9a-fA-F] to enter data. This could be handled by a script
and doesn't need to be part of the c code base.
- Join lines over eol and make eol editable as if it was a character.
Make the eol act like it as a normal character and let the line breaks
be fully handled by either 'wrap' or 'columnview'. As for now I don't
see any option in vim which does something similar and it seems very
hard to me if not impossible without breaking the core structure. I
could also live without this part since the other parts would be a
good enough improvement already. vim is still a text editor and not a
full featured hex editor.

Patches:
hex-displaymode.patch: Base patch adding the +hex feature and the
'displaymode' window option. This should work on vim7.3 (It is based
on 7.3-112).
hex-hexnumber.patch: Aditional patch to add 'hexnumber' window option.
hex-displaymode.patch needs to be applied.
hex-columnview.patch: Aditional patch adding experimental window
option 'columnview', this is know to break. hex-hexnumber.patch needs
to be applied. I might have started in the wrong direction with this
implementation, so I might need to start over with this one.

Regards,
Peter

[Tony Mechelynck]

Maybe you don't need to reinvent the wheel: see :help hex-editing

Best regards,
Tony.
--
During a grouse hunt in North Carolina two intrepid sportsmen
were blasting away at a clump of trees near a stone wall. Suddenly a
red-faced country squire popped his head over the wall and shouted,
"Hey, you almost hit my wife."
"Did I?" cried the hunter, aghast. "Terribly sorry. Have a
shot at mine, over there."

[James Vega]

On Wed, Feb 09, 2011 at 01:45:23AM +0100, Tony Mechelynck wrote:
> On 08/02/11 23:25, Peter de Ridder wrote:
> >Using vim do edit binary files isn't always that easy. With
> >display+=uhex you can make unprintable characters show as hex, but the
> >printable characters are still shown as normal characters. Using xxd
> >so convert a file to hex gives a nice view of the file in both hex and
> >printable characters. But editing is still a bit difficult even with
> >hex editing helper scripts. e.g. insering or removing data isn't easy
> >since all the xxd offsets number need to be renumbered.
>

> Maybe you don't need to reinvent the wheel: see :help hex-editing

He addressed that in the first paragraph.

--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@jamessan.com>

[Bee]

>  hex-displaymode.patch
> 22KViewDownload
>
>  hex-hexnumber.patch
> 31KViewDownload
>
>  hex-columnview.patch
> 22KViewDownload

Editing in vim is difficult, so I chose to edit in vim style:

http://bvi.sourceforge.net/

I have compiled this for linux and mac, works well.

-Bill

[Xavier Wang]

2011/2/9 Peter de Ridder <caval...@gmail.com>

--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

thank you for you patch! I desired editing hex text in Vim in a long time.

hex-columnview.patch can't compile in VC2008, you add preprocessing routine within a macro, and VC can not hand this.

change all

RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col

#ifdef FEAT_HEX

, sep

#endif

)

like macro to:

# ifndef FEAT_HEX

        RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col)

# else

        RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col, sep)

# endif

may resolve this problem.

but when I set columnview, Vim behaves strange. it usually only show one or two line of text, and the nest of line are "@", I don't know why it happens.

and, maybe you can close the syntax highlights or add new highlights groups. and, how to show the original text after the hex text, so as in a normal hex editor?

[Xavier Wang]

2011/2/9 Bee <beey...@gmail.com>

--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

has it have windows version? has it support all Vim features? do things in Vim is easy than in another tools just *like* Vim.

[Peter de Ridder]

On Wed, Feb 9, 2011 at 4:10 AM, Bee <beey...@gmail.com> wrote:
>
> Editing in vim is difficult, so I chose to edit in vim style:
>
> http://bvi.sourceforge.net/
>
> I have compiled this for linux and mac, works well.
>
> -Bill

I didn't know that one yet. That is one of the few hex editor which
has support for inserting tekst in an easy way. I see there is a
cygwin verion but no windows version.

So to clarify my motivations, a bit further, for patching vim:
I would like to be able to use it on windows too. So a benefit of vim
is the support for large files with inserting and a wide range of
platform support.
Editing in vim isn't as dedicated to hex editing as bvi and it is not
my intention to make it that way. But scripts could easy the way of
hex editing.
Also at times, I like to see a file, I'm editing, in hex, so it would
be easy to use the same editor for that.

If you do a lot of hex editing/manipulations bvi seems like a nice
editor. And I don't think my patches will replace it.

regards,
Peter

[Peter de Ridder]

On Wed, Feb 9, 2011 at 11:00 AM, Xavier Wang <weasl...@gmail.com> wrote:
>
> thank you for you patch! I desired editing hex text in Vim in a long time.
> hex-columnview.patch can't compile in VC2008, you add preprocessing routine
> within a macro, and VC can not hand this.
> change all
> RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col
> #ifdef FEAT_HEX
> , sep
> #endif
> )
> like macro to:
> # ifndef FEAT_HEX
>         RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col)
> # else
>         RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col, sep)
> # endif
> may resolve this problem.
> but when I set columnview, Vim behaves strange. it usually only show one or
> two line of text, and the nest of line are "@", I don't know why it happens.
> and, maybe you can close the syntax highlights or add new highlights groups.
> and, how to show the original text after the hex text, so as in a normal hex
> editor?

I'll take that suggestion for the define in mind.

The columnview patch is very experimental. I noticed that strange
behaviour too. It seems to be related to line numbering being enabled.
I think I started in the wrong direction with this patch. So I'll
start over and try to base my code on how 'linebreak' works.

The settings for 'displaymode' and 'hexnumber' are pretty straight
forward. But the 'columnview' setting is a bit more difficult. For
example the 'wrap' setting must be enabled. I require that for now
because of the way I implemented the line breaking. I wonder if it
makes sense to keep this or to see if the 'columnview' implementation
should work around the 'wrap' mode. I hope the rest of the
'columnview' setting makes sense.
If I remove the dependency of 'columnview' to 'wrap' I could see
'columnview' be used like 'textwidth' so the line wrapping can be
setup independent of the window width.

Regards,
Peter

[Bee]

On Feb 9, 11:29 am, Peter de Ridder <pe...@xfce.org> wrote:

I would also like to have hex editing in vim and on all platforms that
I work. But at this point vim is not so good for hex editing.

If/until vim is a good hex editor... on Linux and Mac I use bvi, on
Windows I use HxD.
<http://mh-nexus.de/en/hxd/>

-Bill

[Xavier Wang]

2011/2/10 Peter de Ridder <caval...@gmail.com>

Regards,
Peter

--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

the problem is Vim haven't a good abstract paint layer. has anybody try to refactoring the render engine of Vim?

now win_line function in screen.c has 2670 loc, if Vim can split render info several different module:

memline -> layout -> symtax -> show

 then we can replace some of these module to make Vim more flexible, e.g. hex editing.

but the first difficult task is to refactoring Vim, it does need a experienced expert.......

[Tony Mechelynck]

Oops! �Les pi�ges de la lecture rapide� (The pitfalls of fast reading).
My bad.

Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
20. When looking at a pageful of someone else's links, you notice all of
them
are already highlighted in purple.

[Tony Mechelynck]

On 08/02/11 23:25, Peter de Ridder wrote:

IIUC, when 'encoding' is "utf-8" and you are entering characters as hex,
you would have to type E4B880, not 4E00, for the Chinese horizontal-bar
number one (一 U+4E00, one character, three bytes), and F0A08480, not
20100 or 00020100, for the "additional Chinese character" 𠄀 (U+20100,
one character, four bytes)? Similarly for *any* character above U+007F
when 'encoding' is set to "utf-8"? For instance, E9 when 'encoding' is
latin1, but C3A9 when it is utf-8, for the lowercase e-acute, é, U+00E9,
the most frequent accented letter in the French language? But see also
:help i_CTRL-V_digit.

Best regards,
Tony.
--
It is not enough to succeed. Others must fail.
-- Gore Vidal

[Ben Fritz]

On Feb 8, 4:25 pm, Peter de Ridder <cavalie...@gmail.com> wrote:
> Hi,
>
> Using vim do edit binary files isn't always that easy. With
> display+=uhex you can make unprintable characters show as hex, but the
> printable characters are still shown as normal characters. Using xxd
> so convert a file to hex gives a nice view of the file in both hex and
> printable characters. But editing is still a bit difficult even with
> hex editing helper scripts. e.g. insering or removing data isn't easy
> since all the xxd offsets number need to be renumbered.
>

I noticed the other day that XXD has an option to do a raw dump or
convert back from one, without using offsets. This should in theory
allow easy insertion/deletion. I have toyed with the idea of tweaking
my hex-edit autocmds to use just %!xxd to convert TO hex mode, but
then strip off the addresses after conversion. Then to restore,
additionally strip off the ASCII view and use the xxd option which
only works with raw bytes.

Just some random thoughts, I don't have any opinion on your patches as
of yet.

[Atilla Filiz]

Did anyone check how Cream(http://cream.sf.net/ ) does it? AFAIK its hex conversion is based on Vim's own(via xxd) but extended via Vim scripts.

--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--
Atilla Filiz

[Milan Vancura]

> Editing in vim is difficult, so I chose to edit in vim style:
>
> http://bvi.sourceforge.net/
>
> I have compiled this for linux and mac, works well.

Hi,

bvi looks nice but has some limitations - for example you need to handle the
work with large files manually - ask bvi to open only a part of file and don't
change the length of such edited part etc. This was a reason why a project "bvi
plus" started:

http://bviplus.sourceforge.net/

It also has visual blocks etc.

However, I did not try it yet, just sending an interesting link to try.

Milan
--
Milan Vancura, Prague, Czech Republic, Europe

[Peter de Ridder]

On Thu, Feb 10, 2011 at 4:35 AM, Tony Mechelynck
<antoine.m...@gmail.com> wrote:
>
> IIUC, when 'encoding' is "utf-8" and you are entering characters as hex, you
> would have to type E4B880, not 4E00, for the Chinese horizontal-bar number
> one (一 U+4E00, one character, three bytes), and F0A08480, not 20100 or
> 00020100, for the "additional Chinese character" 𠄀 (U+20100, one character,
> four bytes)? Similarly for *any* character above U+007F when 'encoding' is
> set to "utf-8"? For instance, E9 when 'encoding' is latin1, but C3A9 when it
> is utf-8, for the lowercase e-acute, é, U+00E9, the most frequent accented
> letter in the French language? But see also :help i_CTRL-V_digit.
>
>
> Best regards,
> Tony.

For hex insertion I'm thinking of a script. That would be easier as
using i_CTRL-V_digit for every character. The script would only need
to map [0-9a-zA-Z] to be able to capture the hex input and convert it
to single characters. The script could probably solve the encoding
problems.

As for encoding itself, I think it still posses a problem for my
current 'displaymode' patches. I looked at the documentation bit I
didn't dig into it. I see there are quite a few encoding settings. For
now I disable the encoding routines when displaying text and either
'hex' or 'dot', this was a fast hack and I wonder if it even works
correct. I think it is better to disable encoding entirely when hex
editing but I didn't look deep enough into the encoding settings. An
alternative would be to allow encoding but document that there might
be unexpected (but technically correct) output. The same as the tab
character is still shown as a normal tab in 'hex' mode, 'list' needs
to be set and 'listchars' shouldn't contain tab, for a tab to be
displayed as hex.

Suggestions on how encoding should be shown in 'hex' and 'dot' mode
are welcome. Consider multi byte characters which can take up either 1
or 2 display cells.

Regards,
Peter

[Peter de Ridder]

On Wed, Feb 9, 2011 at 11:00 AM, Xavier Wang <weasl...@gmail.com> wrote:
>

> thank you for you patch! I desired editing hex text in Vim in a long time.
> hex-columnview.patch can't compile in VC2008, you add preprocessing routine
> within a macro, and VC can not hand this.
> change all
> RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col
> #ifdef FEAT_HEX
> , sep
> #endif
> )
> like macro to:
> # ifndef FEAT_HEX
>         RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col)
> # else
>         RET_WIN_BUF_CHARTABSIZE(curwin, curbuf, s, col, sep)
> # endif
> may resolve this problem.
> but when I set columnview, Vim behaves strange. it usually only show one or
> two line of text, and the nest of line are "@", I don't know why it happens.
> and, maybe you can close the syntax highlights or add new highlights groups.
> and, how to show the original text after the hex text, so as in a normal hex
> editor?

I didn't answer all your questions here before.
The part about highlight groups I don't quite follow. I chose to keep
the highlighting as is. It is possible to disable highlighting or
chose another syntax to change the highlighting if you wish.
To show both hex and normal side by side I suggest the following commands.
set cpo-=n
set hnu
set cov=16,8,2
set list
set wrap
gg
set scrollbind
set dym=dot
vsplit
set dym=hex
This should become either a :hexthis internal command or be part of a
hex editing support vim script in the future.

I looked into the 'columnview' @ problem. I fixed the patch so it
should work with the new patch.
I'm still wondering if I'm heading it the right direction with the
columnview. I think I need to get more specific specifications to see
if it can work this way or if I should take another approach.

hex-columnview-fix1.patch: The patch should be applied in stead of the
previous 'columnview' patch. The hex-hexnumber.patch needs to be
applied.

Suggestions on how columnview should work are welcome. Considering
separations and line breaks based on bytes, column or virtual column,
maybe even dependant if it is used for only line breaks or dependant
on the 'diplaymode' setting, etc. Should 'columnview' line breaking
only be active when 'wrap' is set of should it work independent so you
can specify wrapping text on a fixed with which isn't the with of the
window, like 'wrap' does now, going in that directions maybe take
'linebreak' into account. Also when 'columnview' is set to 20,8 should
it place a separation on column 8 and 16 on every wrapped line part or
should it place it at column 8 and 16 and on column 4(24-20) and
12(32-20) on the next wrapped line.

Regards,
Peter