gcc plugin with big-endian and little-endian

[Nguyen Xuan Hoa]

Hi all gcc-plugin expert,

I have this problem, I'd like to share with all of you guys.
Intro:
I have big-endian and little-endian machine,
Network protocol(TCP/IP, UDP protocol ) is byte-order  in big-endian for every machine.
So, when I sending out the package from big-endian machine. It's just send straight to the network card.
But with little-endian machine,  I have to swap byte-order from little to big, then pack the message, and I sent it out.

The problem:
I have network application which compile/run correctly on big-endian.
I'd like to take this source code to run correctly on little-endian, but I'll don't modify source code(don't need to swap-byte-order).
So, I think that I'll base on the GCC-MELT to develop the gcc plugin, with arg input is variables, functions, which I need to do. It'll replace for swapping byte-order.
Output is binary file which include swapping byte-order but we don't need to insert inside of source code. It's just insert at command line compiling such as: CFLAGS= "-fplugin=myplugin.so -fplugin-arg-myplugin-variable=....."

Do you think GCC-MELT can do for this?

Thanks for your help. I'd like really appreciate your help for this.

Thanks so much.

[Basile Starynkevitch]

On Tue, May 21, 2013 at 09:01:21PM -0700, Nguyen Xuan Hoa wrote:
> Hi all gcc-plugin expert,

This is a MELT specific list. For general GCC questions, g...@gcc.gnu.org is a better place.

>
> I have this problem, I'd like to share with all of you guys.

> *Intro*:

> I have big-endian and little-endian machine,
> Network protocol(TCP/IP, UDP protocol ) is byte-order in big-endian for
> every machine.
> So, when I sending out the package from big-endian machine. It's just send
> straight to the network card.
> But with little-endian machine, I have to swap byte-order from little to
> big, then pack the message, and I sent it out.

The correct approach would be to use functions like
htons http://man7.org/linux/man-pages/man3/htons.3.html

On some machines, these functions (which could be macros) are identity.
On other machines (x86) these functions are indeed swapping.
See also http://man7.org/linux/man-pages/man3/endian.3.html

>
> *The problem:*

> I have network application which compile/run correctly on big-endian.

How big is your application? millions of lines of source code?
or only a few dozen of thousands of source code?

> I'd like to take this source code to run correctly on little-endian, but
> I'll don't modify source code(don't need to swap-byte-order).

Are you sure you really don't want to modify source code?
I believe you should (at least in the long run). The point is that some part
of the source code should be known as endian dependent (and this is important to the developers
maintaining the code, so writing that knowledge in the source code is significant).

However, even if you want to change to source code, having a MELT extension which would help you
is a significant help. It could find all the source location which needs fixing (i.e. it would emit
warning with precise source location).

> So, I think that I'll base on the GCC-MELT to develop the gcc plugin, with
> arg input is variables, functions, which I need to do. It'll replace for
> swapping byte-order.
> Output is binary file which include swapping byte-order but we don't need
> to insert inside of source code. It's just insert at command line compiling
> such as: CFLAGS= "-fplugin=myplugin.so -fplugin-arg-myplugin-variable=....."
>
> Do you think GCC-MELT can do for this?

Perhaps yes. But you should rather develop (or subcontract the development of
such MELT extension) a MELT extension to help you modify your source code (it is broken,
since endian specific). Even if you could automatize everything, I don't think it
is convenient: your source code is broken since endian-dependent, and you
have to improve it to make it more portable.

And I actually don't understand exactly what you want your MELT extension to do.
I would suggest to emit warning on each suspicious endian-specific source location.
(or do you really want it to modify the Gimple transparently?)

How do you define the data to be swapped? Do you want to follow each short which
is directly sent to output ports?

Remember that MELT is working on the Gimple representation. You need to understand it first.
The MELT probe should help you, and so is the -fdump-tree-all option of GCC (which dumps
hundreds of files!)

So show us some actual code of your application. I suggest the following way to work

Pick up a small example of the problematic code and (if you want more help here) show it.
Or if you cannot show, think by yourself of what source (or Gimple) transformation you
dream about. Perhaps simplify that code.

Then, define very precisely the property of being endian dependent. (I cannot guess what
that means in your particular application).

Look into the Gimple code produced on your particular example.

Imagine how the Gimple should be instrumented (or transformed).

I cannot help more at the moment, because I need to understand really how is your
particular application endian dependent. What should exactly be fixed inside? Show us some
sample code! How are the network buffers constructed in practice?

Perhaps some other people interested in MELT have some other ideas?

Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

[Nguyen Xuan Hoa]

Hi Basile and MELT guys,

How big is your application? millions of lines of source code? 
or only a few dozen of thousands of source code?

Are you sure you really don't want to modify source code? 

It'll be millions/billion lines of source code.
So, We won't touch to source code, we don't modify it.

How do you define the data to be swapped? Do you want to follow each short which 
is directly sent to output ports?

I cannot help more at the moment, because I need to understand really how is your 
particular application endian dependent. What should exactly be fixed inside? Show us some 
sample code! How are the network buffers constructed in practice?

For example: 
this programming run wel in big-endian machine:	
	---------------------------------------------
	//------example_big-endian.c------
	uint32_t some_long = 10;
	uint16_t some_short = 20;
	uint32_t network_byte_order;
	// convert and send
	network_byte_order = some_long;
	send(s, &network_byte_order, sizeof(uint32_t), 0);
	some_short == ntohs(htons(some_short)); // this expression is true
	----------------------------------------------
To keep running well in little-endian machine we should insert like this: 
	---------------------------------------------
	//------example_litle-endian.c------
	uint32_t some_long = 10;
	uint16_t some_short = 20;
	uint32_t network_byte_order;
	// convert and send
	network_byte_order = htonl(some_long);
	send(s, &network_byte_order, sizeof(uint32_t), 0);
	some_short == ntohs(htons(some_short)); // this expression is true
	----------------------------------------------
What I expect with MELT-extension: 
I don't need to insert/modify inside the source code, I have just only insert in the command line compiling, which run well in little-endian machine:
Such as: gcc -o example.c -fplugin=melt_plugin.so -fplugin-arg-melt_plugin-function=file_contain_variable_function_need_to_swap.txt
	---------------------------------------------
	//------example.c------
	uint32_t some_long = 10;
	uint16_t some_short = 20;
	uint32_t network_byte_order;
	// convert and send
	network_byte_order = some_long;
	send(s, &network_byte_order, sizeof(uint32_t), 0);
	some_short == ntohs(htons(some_short)); // this expression is true
	----------------------------------------------

Thanks you so much, I appreciate all your reply.

[Basile Starynkevitch]

On Wed, May 22, 2013 at 03:13:11PM +0700, Nguyen Xuan Hoa wrote:
>
> Hi Basile and MELT guys,
>
> >How big is your application? millions of lines of source code?
> >or only a few dozen of thousands of source code?
> >Are you sure you really don't want to modify source code?
> It'll be millions/billion lines of source code.

If it is that much code, an automated approach
(which does not modify source code, but change the way it is compiled)
is certainly fit; however, many millions lines of code
are often very heterogeneous in programming style.
And then things become complicated

(you could consider subcontracting to my employer CEA, LIST; it could be a 3 years project)

> So, We won't touch to source code, we don't modify it.
>
> >How do you define the data to be swapped? Do you want to follow each short which
> >is directly sent to output ports?
>
> >I cannot help more at the moment, because I need to understand really how is your
> >particular application endian dependent. What should exactly be fixed inside? Show us some
> >sample code! How are the network buffers constructed in practice?
> For example:
> this programming run wel in big-endian machine:
> ---------------------------------------------
> //------example_big-endian.c------
> uint32_t some_long = 10;
> uint16_t some_short = 20;
> uint32_t network_byte_order;
> // convert and send

> *network_byte_order = some_long;*
> send(s,&network_byte_order, sizeof(uint32_t), 0);

> some_short == ntohs(htons(some_short)); // this expression is true
> ----------------------------------------------
> To keep running well in little-endian machine we should insert like this:
> ---------------------------------------------
> //------example_litle-endian.c------
> uint32_t some_long = 10;
> uint16_t some_short = 20;
> uint32_t network_byte_order;
> // convert and send

> *network_byte_order = htonl(some_long);*
> send(s,&network_byte_order, sizeof(uint32_t), 0);

> some_short == ntohs(htons(some_short)); // this expression is true
> ----------------------------------------------

But this example is too simple to be significant. What about more complex code, like

struct s1_st {
uint32_t i;
uint32_t j;
} st;
st.i = htonl(12345); // this is ok
st.j = x; // this is not ok
send (socket, &st, sizeof(st), 0)

and imagine some code like
st.j = some_condition(x)?x:htonl(x);

> What I expect with MELT-extension:
> I don't need to insert/modify inside the source code, I have just only insert in the command line compiling, which run well in little-endian machine:

> Such as:*gcc -o example.c -fplugin=melt_plugin.so -fplugin-arg-melt_plugin-function=file_contain_variable_function_need_to_swap.txt*

Defining the user interface is the most easy part.

Defining the transformations you want to occur is harder.

Actually, I tend to believe that your problem is a bit ill defined
(it feels impossible to solve in the most general case,
because it might be equivalent to the halting problem
http://en.wikipedia.org/wiki/Halting_problem)

Do you have a precise heuristic in mind to decide when some scalar data has or not to be converted ?
Are you thinking of some type analysis? (with which type inference rules)?
Are you thinking of some abstract interpretation? (with which exact lattice)?

The difficulty is not MELT related (you'll have the same with other technologies, eg LLVM)
it is related to defining your problem and how you'll approach it.

I remain confused, and I feel that you could underestimate the difficulty.

If you have bibliographical references, I am interested.