Absolute addressing on the ARM

[Nils M Holm]

Hi and sorry about butting in out of nowhere.

I have a question about absolute addressing on ARMv6 processors
as used in the Raspi. Recently I have written a back end for said
processor and wondered about the best method for loading a value
from an absolute address into a register when the absolute address
cannot be known at compile time (i.e. cannot be placed in range
for PC-relative addressing).

I came up with the following code to load a value from X:

.data
X: .long 0
/* arbitrary distance here */
.data
L1: .long X
.text
ldr r0,L1
ldr r0,[r0]

which works fine.

Now someone told me that it might be possible to construct absolute
addresses with MOV/MOVT and let the linker fix the gory stuff.

I doubt that because of the limitations the ARM seems to place on
immediate values in MOV and MOVT. If I understand the manual correctly,
immediate operands of MOV and friends must be 8-bit values that can
be shifted to the left by up to eight bits.

Wouldn't this limitation make MOV/MOVT unsuitable for loading
absolute addresses that cannot be known at compile time?

Or am I missing something? Any hints would be welcome!

--
Nils M Holm < n m h @ t 3 x . o r g > www.t3x.org

[Tauno Voipio]

The normal way would be to have the address constant in the code
section, at a place outside the code flow. This implies the the
address can be resolved latest at link time. Otherwise, you need
a pointer in the data (or .bss) section to be filled in at run-
time.

If you are using assembler, there is a literal syntax:

ldr r0,=X @ this will create a PC-relative access
ldr r0,[r0]
.....

.pool @ this has to be within the PC-relative addressing
@ range from the instruction, outside of code flow.

The .pool pseudo-op is not absolutely necessary, if the module
is so small that the literal can be accessed at the end. The assembler
will generate the literal pool anyway at the end, if there are any
unresolved literals at the end of assembly.

----

A different story is that the address is a virtual address: the
address translation in the hardware will change it to the physical
address in the way the kernel feels fit.

If you want to access a physical bus address, e.g. a periperal
register, you need to negotiate the addressing with the kernel.
There is no way you can directly point to an absolute physical
address from user-mode code without getting the kernel to map
it for you.

HTH

--

Tauno Voipio

[Hans-Bernhard Bröker]

On 16.03.2014 21:22, Nils M Holm wrote:
> from an absolute address into a register when the absolute address
> cannot be known at compile time

AFAICS, strictly speaking there's no such thing as absolute addressing
on the ARM.

And that's not even particularly uncommon. When a RISC-like platform
uses fixed-size instruction words (ARM does), and they're the same width
as the CPU's address width (ARM nearly does), that usually means they
can't do true absolute addressing --- an instruction just isn't wide
enough to hold a complete address. Such architectures could only do
zero-page absolute addressing, i.e. absolute addressing for a narrow
subset of the address space: the "zero page". And since that subset may
well be so small as to be useless, they sometimes won't even bother with
it at all.

In a nutshell, all addressing on the ARM is relative. There's just
immediate addressing. I.e. you gan get data from inside the machine
instruction, but no addresses.

[Simon Clubley]

On 2014-03-16, Nils M Holm <news...@t3x.org> wrote:
>
> Now someone told me that it might be possible to construct absolute
> addresses with MOV/MOVT and let the linker fix the gory stuff.
>

MOV/MOVT are absolute addressing works just fine. I've never used this
construct myself (I tend to just use one of the ldr variants in my hand
written ARM assembly code) however the above does show up in generated
code. This is the generated code for some bare metal C code of mine for
the Beaglebone Black:

[The code was compiled with optimisation turned on hence the duplicated
source code in the objdump output.]

==========================================================================
80300290 <board_init_phase1>:
void board_init_phase1(void)
{
/*
* Disable watchdog. Stop sequence is on page 4202 of spruh73j.pdf
*/
BBBB_WDT->WDT_WSPR = 0x0000aaaa;
80300290: e3a03a05 mov r3, #20480 ; 0x5000
80300294: e30a2aaa movw r2, #43690 ; 0xaaaa
80300298: e34434e3 movt r3, #17635 ; 0x44e3
while((BBBB_WDT->WDT_WWPS & 0x10) != 0)
8030029c: e1a01003 mov r1, r3
void board_init_phase1(void)
{
/*
* Disable watchdog. Stop sequence is on page 4202 of spruh73j.pdf
*/
BBBB_WDT->WDT_WSPR = 0x0000aaaa;
803002a0: e5832048 str r2, [r3, #72] ; 0x48
==========================================================================

Here's another more readable example:

==========================================================================
80300260 <default_interrupt_handler>:
* We increment a counter here so a debugger can tell if this routine _is_
* actually been called.
*/
void default_interrupt_handler(void)
{
default_interrupt_handler_count++;
80300260: e3003920 movw r3, #2336 ; 0x920
80300264: e3483030 movt r3, #32816 ; 0x8030
80300268: e5932000 ldr r2, [r3]
8030026c: e2822001 add r2, r2, #1
80300270: e5832000 str r2, [r3]
// exec$abort();
}
==========================================================================

> I doubt that because of the limitations the ARM seems to place on
> immediate values in MOV and MOVT. If I understand the manual correctly,
> immediate operands of MOV and friends must be 8-bit values that can
> be shifted to the left by up to eight bits.
>

The above would seem to suggest otherwise (although as I said, I have
not tried using MOVT manually so there may be a limit I am missing).

If you look at the 32-bit opcode above, you should be able to see
how the encoding of the 16-bit values is broken down.

> Wouldn't this limitation make MOV/MOVT unsuitable for loading
> absolute addresses that cannot be known at compile time?
>
> Or am I missing something? Any hints would be welcome!
>

I hope the above output from objdump helps get you started. :-)

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Tim Wescott]

Are you using assembly because you want to, or because you feel you have
to?

I ask, because you mention "compile".

If you're doing things in C, the way to do it is to just declare your
address as "extern". Then define the actual physical address in an
assembler file or in the linker command file.

I make peripheral definitions that end up with a line in a header file
that looks like

extern volatile SSomePeripheralOrAnotherRegs RALPH;

and a line in the linker command file that's something like

RALPH = 0x40039400;

(Note: of course I don't use silly names: "SomePeripheralOrAnother" would
be too ambiguous).

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

[Dimiter_Popoff]

On 17.3.2014 г. 00:14, Hans-Bernhard Bröker wrote:
> On 16.03.2014 21:22, Nils M Holm wrote:
>> from an absolute address into a register when the absolute address
>> cannot be known at compile time
>
> AFAICS, strictly speaking there's no such thing as absolute addressing
> on the ARM.
>
> And that's not even particularly uncommon. When a RISC-like platform
> uses fixed-size instruction words (ARM does), and they're the same width
> as the CPU's address width (ARM nearly does), that usually means they
> can't do true absolute addressing --- an instruction just isn't wide
> enough to hold a complete address. Such architectures could only do
> zero-page absolute addressing, i.e. absolute addressing for a narrow
> subset of the address space: the "zero page".

Does ARM do that zero page thing? Power does, lower 32k and top 32k of
the 32-bit space.

It also does absolute jumps to I think it was 24 bit addresses (the
lowest) - not so sure about the 24 bits though, it's been a while
since I made use of it in vpa.

> And since that subset may
> well be so small as to be useless, they sometimes won't even bother with
> it at all.

Well I would not go as far as "useless" but it certainly is avoidable
to have it, two opcodes instead of one on the not so frequent
accesses of that type. Well I suppose I could be talked into
your "useless" too, won't take that much, but I still use it.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/

[Tauno Voipio]

On 17.3.14 05:57, Dimiter_Popoff wrote:

> Does ARM do that zero page thing? Power does, lower 32k and
> top 32k of the 32-bit space.

No. All addressing is register-indirect or register-relative.
There are many different ways to get a constant (it can be
an address) into a register. There are different ways to have
a constant in the instruction code. The possible constants
are different in the different instruction sets (32 bit,
Thumb1 and Thumb2). For a general 32 bit pattern, there is
the PC-relative addressing with the constant embedded into
the code section outside of program execution flow.

> It also does absolute jumps to I think it was 24 bit addresses (the
> lowest) - not so sure about the 24 bits though, it's been a while
> since I made use of it in vpa.

The jumps (branch instructions) are relative. An absolute jump
is made by loading a constant into register 15 (PC), or loading
a constant into a register and using the bx instruction.

--

Tauno Voipio

[Nils M Holm]

Tauno Voipio <tauno....@notused.fi.invalid> wrote:
> On 16.3.14 22:22, Nils M Holm wrote:
> > .data
> > X: .long 0
> > /* arbitrary distance here */
> > .data
> > L1: .long X
> > .text
> > ldr r0,L1
> > ldr r0,[r0]
>

> The normal way would be to have the address constant in the code
> section, at a place outside the code flow. This implies the the
> address can be resolved latest at link time. Otherwise, you need
> a pointer in the data (or .bss) section to be filled in at run-
> time.

Yes, this is exactly how I had understood things. In my example
above, L1 really should be in the text section.

> If you are using assembler, there is a literal syntax:
>
> ldr r0,=X @ this will create a PC-relative access
> ldr r0,[r0]
> .....
>
> .pool @ this has to be within the PC-relative addressing
> @ range from the instruction, outside of code flow.
>
> The .pool pseudo-op is not absolutely necessary, if the module
> is so small that the literal can be accessed at the end. The assembler
> will generate the literal pool anyway at the end, if there are any
> unresolved literals at the end of assembly.

This sounds very useful, but unfortunately, the GNU assembler does not
seem to support this syntax (or it uses a different one), so I will
have to manage literals myself.

I currently do not have to deal with virtual addresses, but this is
also interesting to know.

Anyway, thanks for your reply! It basically confirms what I have found
out myself.

[Nils M Holm]

Simon Clubley <clubley@remove_me.eisner.decus.org-earth.ufp> wrote:
> On 2014-03-16, Nils M Holm <news...@t3x.org> wrote:
> >
> > Now someone told me that it might be possible to construct absolute
> > addresses with MOV/MOVT and let the linker fix the gory stuff.
> >
>
> MOV/MOVT are absolute addressing works just fine. I've never used this
> construct myself (I tend to just use one of the ldr variants in my hand
> written ARM assembly code) however the above does show up in generated
> code. This is the generated code for some bare metal C code of mine for
> the Beaglebone Black:

Thank you for your examples! However, I tried MOVW/MOVT with the exact
values you used, and got

Error: selected processor does not support `movw r3,#2336'
Error: selected processor does not support `movt r3,#32816'

I have seen that the Beaglebone Black is based on an ARMv7 core while
the Raspi uses an ARMv6 core. Maybe MOV with 16-bit immediates is only
supported on the ARMv7? The manual says that the ARMv6 supports only
8-bit immediates with a 3-bit shift.

[Nils M Holm]

Tim Wescott <t...@seemywebsite.really> wrote:
> On Sun, 16 Mar 2014 20:22:06 +0000, Nils M Holm wrote:
> > I have a question about absolute addressing on ARMv6 processors as used
> > in the Raspi. Recently I have written a back end for said processor and

> > [...]

>
> Are you using assembly because you want to, or because you feel you have
> to?

Maybe that part was not obvious in my original post. I have written a
compiler back end for the ARMv6, and that back end emits assembly.
So I am actually looking for a template for loading an arbitrary
abolute address or large literal into a register.

I already have a template that works fine, but then someone suggest to
use MOV/MOVT instead of a literal pool. I think that MOV/MOVT will not
work on the ARMv6, because it can only load eight-bit immediates with
3-bit shift, so MOV/MOVT are not suitable for later fixup. So that was
the most important part of my question:

Can MOV/MOVT be used on the ARM *v6* to load *any* 16-bit value?

The manual says no, my experiments with GAS say no, but maybe I have
missed something. I am new to ASM programming.

> I ask, because you mention "compile".

Yes, my compiler compiles a subset of C89 to ARMv6 assembly.

[Nils M Holm]

Nils M Holm <news...@t3x.org> wrote:

> missed something. I am new to ASM programming.

ARM, not ASM. I have 30+ years of experience with loads of processors.

[Tauno Voipio]

I beg to have a different opinion. Below is an abbreviated listing
of ARM7TDMI (AT91R40008) startup code, translated with GNU assembler.

---- clip clip ----

ARM GAS /tmp/ccBojrRY.s page 1


1 # 1 "hwstart.S"
6
.globl main @ main program entry
11 .globl __bss_start__ @ -> .bss area in RAM
12 .globl __bss_length__ @ .bss area byte length
14
28
33 @ Compatibility macros
34 @ --------------------
35
36 #if defined(__thumb__)
37 #define LSR(reg,cnt) lsr reg,reg,cnt
38 #define SUBS(reg,cnt) sub reg,cnt
39 #else
40 #define LSR(reg,cnt) movs reg,reg,lsr cnt
41 #define SUBS(reg,cnt) subs reg,reg,cnt
42 #endif
43
129
130 0000 1548 hwstrt: ldr r0,=WD_OKEY @ get watchdog
write key
131 0002 1649 ldr r1,=at91wd @ -> watchdog
register bank
132 0004 0860 str r0,[r1,#WD_OMR] @ disable watchdog
133
140
141 @ Clear .bss
142
143 0010 0020 mov r0,#0 @ get a zero
144 0012 134A ldr r2,=__bss_start__ @ -> bss area to
zero out
145 0014 134B ldr r3,=__bss_length__ @ bss area length
146 0016 9B08 LSR(r3,#2) @ count in full
words - any?
147 0018 02D0 beq zlpex @ no - skip clearing
148
149 001a 01C2 zloop: stmia r2!,{r0} @ clear a word
150 001c 013B SUBS(r3,#1) @ bump count - done?
151 001e FCD1 bne zloop @ no - loop

154 @ Call main program: main(0)
155
156 zlpex: @mov r0,#0 @ no arguments (argc
= 0)
157 0020 FFF7FEFF bl main @ enter main program
158
159 0024 ECE7 b hwstrt @ loop on return

201 @ Startup data
202 @ ------------
203
204 0056 00004023 .pool
204 00000000
204 00000000
204 00000000
204 0000
205 .end

---- clip clip ----

--

-TV

[Nils M Holm]

Tauno Voipio <tauno....@notused.fi.invalid> wrote:
> On 17.3.14 09:46, Nils M Holm wrote:
> > This sounds very useful, but unfortunately, the GNU assembler does not
> > seem to support this syntax (or it uses a different one), so I will
> > have to manage literals myself.
>

> I beg to have a different opinion. Below is an abbreviated listing
> of ARM7TDMI (AT91R40008) startup code, translated with GNU assembler.

> [...]

Oops, are right! I accidentally tried it with MOV instead of LDR.

[Wouter van Ooijen]

>> ldr r0,=X @ this will create a PC-relative access
>> ldr r0,[r0]
>> .....
>>
>> .pool @ this has to be within the PC-relative addressing
>> @ range from the instruction, outside of code flow.
>>
>

> This sounds very useful, but unfortunately, the GNU assembler does not
> seem to support this syntax (or it uses a different one), so I will
> have to manage literals myself.

Strange, this is the syntax I use all the time. Do you use a separate
asm file, or i-line assembly in C? Which error message do you get?

Wouter van Ooijen

[David Brown]

It's a little off-topic for your question, but would you mind telling us
why you are doing this, and what you aims are here? There are already
several excellent C compilers for the ARM (gcc, llvm, Keil/ARM, IAR,
GHS, CodeWarrior - and probably a few others that I've forgotten). The
main ones here are highly optimising, support a range of ARM cores, and
cover modern C and C++ standards (including C11 and C++11 in some
cases). So I am very curious as to your goals in making a compiler for
a subset of C89 - who will use it? Or is this just for fun or education?


Other than that, my advice here is that when you wonder about code
generation, write a simple example function in C and compiler it with
gcc at different optimisation settings. Use the generated assembly as
inspiration for your own code generator.

[Nils M Holm]

David Brown <david...@hesbynett.no> wrote:
> On 17/03/14 09:00, Nils M Holm wrote:
> > Yes, my compiler compiles a subset of C89 to ARMv6 assembly.
>
> It's a little off-topic for your question, but would you mind telling us
> why you are doing this, and what you aims are here? There are already
> several excellent C compilers for the ARM (gcc, llvm, Keil/ARM, IAR,
> GHS, CodeWarrior - and probably a few others that I've forgotten). The
> main ones here are highly optimising, support a range of ARM cores, and
> cover modern C and C++ standards (including C11 and C++11 in some
> cases). So I am very curious as to your goals in making a compiler for
> a subset of C89 - who will use it? Or is this just for fun or education?

The original version of the compiler was for education and was hosted
on and targeted at FreeBSD/386. Later I kept hacking it for fun and
added back-ends for the x86-64, 8086 and, lately, the ARMv6. I also
added runtime support for Linux, various BSDs and DOS. Support for
Windows and Darwin were added by contributors.

What I like about the compiler is that it is simple, easy to hack, and
boostraps in 4 seconds on a 700 MHz Raspi. Of course, its code generator
is rather limited, and its code runs (on average) almost twice as long
as code generated by GCC -O0, but it is suffient for most stuff I do.

It is mostly intended to be studied, though. Quite a few people who had
given up on other small C compilers told me that my code is quite easy
to follow.

If you want to have a look, see: http://www.t3x.org/subc/index.html

The README on the page summarizes the omissions from The C Programming
Language, 2nd Ed.

In case you have a look, please bear in mind that I have never programmed
in ARM assembly before, so the emitted code may be worse than dictated
by the limitations the compiler.

> Other than that, my advice here is that when you wonder about code
> generation, write a simple example function in C and compiler it with
> gcc at different optimisation settings. Use the generated assembly as
> inspiration for your own code generator.

That is exactly what I have done. The only question that brought me
here was if it was possible to load an absolute address using MOV/MOVT
on the ARMv6. GCC and Clang do not do this, they use a literal pool
instead.

[David Brown]

Thanks - that puts things in a more complete context. "Real" compilers
such as gcc are far from simple or understandable, even for people who
have worked with them for years. llvm is a bit clearer and more
structured, but it too is a huge project. So a limited small compiler
for educational purposes seems like a good idea, even if it cannot be
used for "real" programs.

>> Other than that, my advice here is that when you wonder about code
>> generation, write a simple example function in C and compiler it with
>> gcc at different optimisation settings. Use the generated assembly as
>> inspiration for your own code generator.
>
> That is exactly what I have done. The only question that brought me
> here was if it was possible to load an absolute address using MOV/MOVT
> on the ARMv6. GCC and Clang do not do this, they use a literal pool
> instead.
>

I think - given your aims - the choice should be whatever is easiest and
clearest to implement. You don't need to consider fast or small code,
or compatibility with other tools.

[Simon Clubley]

On 2014-03-17, Nils M Holm <news...@t3x.org> wrote:
>
> Thank you for your examples! However, I tried MOVW/MOVT with the exact
> values you used, and got
>
> Error: selected processor does not support `movw r3,#2336'
> Error: selected processor does not support `movt r3,#32816'
>
> I have seen that the Beaglebone Black is based on an ARMv7 core while
> the Raspi uses an ARMv6 core. Maybe MOV with 16-bit immediates is only
> supported on the ARMv7? The manual says that the ARMv6 supports only
> 8-bit immediates with a 3-bit shift.
>

It looks like you are correct.

I know it doesn't work on the ARMv5 series (ARM 9 and friends) because
the same type of code compiles to the traditional ldr PC relative load
from a literal pool.

I don't have any ARMv6 boards so I have not been looking at code generated
for the v6 MCUs but for some reason I thought this worked on the v6 as
well as the v7 architecture.

Sorry to have wasted your time.

[Nils M Holm]

Simon Clubley <clubley@remove_me.eisner.decus.org-earth.ufp> wrote:
> I know it doesn't work on the ARMv5 series (ARM 9 and friends) because
> the same type of code compiles to the traditional ldr PC relative load
> from a literal pool.
>
> I don't have any ARMv6 boards so I have not been looking at code generated
> for the v6 MCUs but for some reason I thought this worked on the v6 as
> well as the v7 architecture.
>
> Sorry to have wasted your time.

Not at all! It is good to know that this works on the v7 but not
on the earlier cores.

[Tim Wescott]

I'm not familiar with ARMv6. Gnu compiles Arm Cortex M3 code to use movt
and movw. It compiles Arm Cortex M0 code to use a load from a literal
stored in the .text section after the function code.

[Nils M Holm]

As I already said in a different post: my mistake, it works fine.

[Lasse Langwadt Christensen]

Nils M Holm <news...@t3x.org> wrote:

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0489c/Chedabhf.html

-Lasse

[Lasse Langwadt Christensen]

why not:

#define RALPH (*(volatile SomePeripheralOrAnother *) 0x40039400)

evrything in one place

-Lasse

[Simon Clubley]

On 2014-03-18, Lasse Langwadt Christensen <lang...@fonz.dk> wrote:
> Tim Wescott <t...@seemywebsite.really> wrote:
>>

>>I make peripheral definitions that end up with a line in a header file
>>that looks like
>>
>>extern volatile SSomePeripheralOrAnotherRegs RALPH;
>>
>>and a line in the linker command file that's something like
>>
>>RALPH = 0x40039400;
>>
>>(Note: of course I don't use silly names: "SomePeripheralOrAnother" would
>>be too ambiguous).
>>
>
> why not:
>
> #define RALPH (*(volatile SomePeripheralOrAnother *) 0x40039400)
>
> evrything in one place
>

The implicit assumption I am about to make is that at the core of
SSomePeripheralOrAnotherRegs is a struct along the lines of:

struct SSomePeripheralOrAnotherRegs_t
{
dev_reg_1_def;
dev_reg_2_def;
dev_reg_3_def;
};

and _without_ the volatile attribute on each register definition.

I don't understand why the struct based datatype is defined as volatile
in this case instead of the registers within the struct.

OTOH, if the registers _are_ defined as volatile, then I don't see why
it's needed on the struct based datatype itself.

As far as I can see, the struct itself is not volatile; it's just a
language level construct to gather together a set of device specific
registers. It's the registers defined within the struct, and only the
registers within the struct, which need to be marked as volatile in this
case.

This is how I define these structs and it has always worked for me across
a range of architectures.

What am I missing ?

Simon.

PS: BTW, I also use the above #define approach as well, and yes, it nicely
keeps everything in one place.

[David Brown]

The key thing to understand about volatile is that there is no such
thing as a real "volatile" type, struct, variable or other object in C -
it is /accesses/ that are volatile. When you give a variable the
volatile qualifier, you are simply telling the compiler that all
accesses to that object must be volatile accesses.

You can mark a variable as volatile either in the declaration of the
individual variable (including an extern declaration), or by declaring
it to be a type that is volatile qualified. If it is an aggregated type
(struct or union), it does not matter if the individual items are marked
volatile, or the whole type is marked volatile. And it does not matter
how many "volatiles" you have.

Thus the effect is the same if you define the struct to be volatile, or
the individual items as volatile, or both. And you can make the type
volatile, or the declared variable volatile, or both.


Where you prefer to put the volatile is a matter of style (usually
picked for you by the compiler or microcontroller vendor when they make
the header files). Putting it in each field in the struct is a bit
verbose, but gives the possibility of having some fields volatile and
some non-volatile. Putting it on the struct definition is neater, and
means the volatile qualifier cannot be forgotten when the struct is used
- but it means you can't avoid using it. Putting it on the declaration
of the object itself makes it explicit at the point of declaration, and
also means that you can use the struct in a non-volatile way (such as
for a local cached copy of the data in question). Finally, you can put
the volatile on the access of the data, making it explicit when it is
used, but forcing you to remember to use it.


Note that there is a subtle difference between declaring RALPH as an
extern volatile, and making it from a cast from an integer literal to a
pointer-to-volatile. When a variable is declared as volatile, it is
undefined behaviour to use a cast to remove the volatile qualification.
But the pointer cast macro does not declare any volatile data, merely a
pointer - you can legally use casts to remove the volatile aspect.

[Tim Wescott]

Mostly, because I think it looks ugly. I can't really defend it further
-- and I promise that if you show me your code, I won't barf on it.

[George Neuner]

On Wed, 19 Mar 2014 13:02:45 +0000 (UTC), Simon Clubley
<clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:

>On 2014-03-18, Lasse Langwadt Christensen <lang...@fonz.dk> wrote:
>> Tim Wescott <t...@seemywebsite.really> wrote:
>>>
>>>I make peripheral definitions that end up with a line in a header file
>>>that looks like
>>>
>>>extern volatile SSomePeripheralOrAnotherRegs RALPH;
>>>
>>>and a line in the linker command file that's something like
>>>
>>>RALPH = 0x40039400;
>>>
>>>(Note: of course I don't use silly names: "SomePeripheralOrAnother" would
>>>be too ambiguous).
>>>
>>
>> why not:
>>
>> #define RALPH (*(volatile SomePeripheralOrAnother *) 0x40039400)
>>
>> evrything in one place
>>
>
>The implicit assumption I am about to make is that at the core of
>SSomePeripheralOrAnotherRegs is a struct along the lines of:
>
>struct SSomePeripheralOrAnotherRegs_t
> {
> dev_reg_1_def;
> dev_reg_2_def;
> dev_reg_3_def;
> };
>
>and _without_ the volatile attribute on each register definition.
>
>I don't understand why the struct based datatype is defined as volatile
>in this case instead of the registers within the struct.

Hi Simon,

Just a postscript to David's excellent response. Remember that a
struct doesn't really "contain" anything, but rather is just an
organizational template for "viewing" a block of address space [not
necessarily "memory"] in a particular way. That template then is
"placed" over a particular address space to view it correspondingly
with the declaration.

[some people don't like the "view" notion of structs, but consider
that you can place multiple different structs at the same location to
yield different views of the same underlying address space.]

Moreover, a more generic declaration might be useful elsewhere without
the requirement for volatile access.

>OTOH, if the registers _are_ defined as volatile, then I don't see why
>it's needed on the struct based datatype itself.
>
>As far as I can see, the struct itself is not volatile; it's just a
>language level construct to gather together a set of device specific
>registers. It's the registers defined within the struct, and only the
>registers within the struct, which need to be marked as volatile in this
>case.

The handling of volatiles is poorly defined in the C standard [any
version - take your pick]. Access to a volatile variable through a
non-volatile pointer may implicitly cast away the volatile qualifier
on the variable. E.g.,

volatile int v;
int *pv = &v; /* bad */
volatile int *vpv = &v; /* much better */

However, given

struct _s { int i } s;
struct _s *ps = &s;
int *pi = &s.i;
int x;

"x = ps->i" is semantically different from "x = *pi" even though the
result [at least here] is equivalent. The struct reference contains
an implicit cast to int*: i.e. it acts as if you really had written

x = *((int*)((char*)ps + offsetof(_s,i)))

The implicit cast on struct member references is a source of confusion
for compiler implementers regarding volatile members: does it or does
it not cast away a volatile qualifier on the member? IME it is
compiler dependent what happens if you access a volatile struct member
through a non-volatile struct pointer.

>This is how I define these structs and it has always worked for me across
>a range of architectures.

MMV. I would expect that if all the compilers use the same base:
e.g., all are GCC (or whatever) derived. But you shouldn't count on
it across different compilers.

IME the result even may depend on how the definitions are written:
e.g.,

struct peripheral_t { volatile uint8_t reg1, ... }
*RALPH = 0x40039400;

may work, whereas

struct peripheral_t { volatile uint8_t reg1, ... };
:
peripheral_t *RALPH = 0x40039400;

may fail.


You can say that's a stupid compiler, and I would agree ... but you
have to work with what you've got.

>Simon.
George

[Tauno Voipio]

This works, and makes the I/O accessible from GDB, too:

--- clip clip ---

/* Cortex-M3 system tick definitions */
/* Semantics Oy, Tauno Voipio */
/* $Id: cxm3tick.h $ */

#ifndef CXM3TICK_H
#define CXM3TICK_H

#include <inttypes.h>

struct systick
{
uint32_t ctrl; /* control / status */
uint32_t reload; /* reload value */
uint32_t value; /* counter value */
};

extern volatile struct systick cxm3tick; /* 0xe000e010 */

/* .ctrl: */

#define TICK_EN (1 << 0) /* enable tick */
#define TICK_IE (1 << 1) /* interrupt enable */
#define TICK_CLK (1 << 2) /* clock from system (always 1) */
#define TICK_FLG (1 << 16) /* counted to zero flag */

#endif

--- end clip ---

For the I/O port addresses an assembler module:

--- clip clip ---

.psize 40,110

@ Cortex-M3 I/O base address definitions
@ Semantics Oy, Tauno Voipio
@ $Id: lm3s6965io.S $

.globl cxm3nvic @ interrupt controller
.globl cxm3scb @ Cortex-M3 system control
.globl cxm3tick @ system tick


cxm3nvic= 0xe000e100 @ interrupt controller
cxm3scb= 0xe000ed00 @ Cortex-M3 system control
cxm3tick= 0xe000e010 @ system tick
.end

--- end clip ---

The I/O addresses could be defined in the linker script
instead, with a similar result.

--

Tauno Voipio

[David Brown]

That's a useful idea, but you need to be a bit careful - as far as the C
language (and therefore the compiler) is concerned, each piece of memory
you access is part of a unique object with a defined type. The only
legal exceptions are unions, pointers to char, and structs which share
the first few fields.

So if you have a pointer pS1 to struct S1 and a pointer pS2 to struct
S2, the compiler "knows" that they cannot both point to the same object
in memory - this is know as type-based alias analysis. If you use pS1
to modify the object, then read the memory via pS2, you may get
unexpected results - since the compilers "knows" that these point to
different areas of memory, the loads and stores are independent and the
compiler may re-arrange them.

Of course, when you are using volatile accesses, the compiler cannot
re-arrange such accesses.

Other ways to avoid type-based alias analysis (other than disabling it,
which limits other compiler optimisations) include using memory
barriers, accessing data through pointer-to-char (a good compiler should
optimise memcpy and similar functions as inline loops using
larger-than-char accesses if the alignments are suitable), and accessing
data through unions (known as "type punning unions"). Instead of
pointers directly to struct S1 and struct S2 types, you could use a
single pointer to a union of the two structs.

>
> Moreover, a more generic declaration might be useful elsewhere without
> the requirement for volatile access.
>
>
>> OTOH, if the registers _are_ defined as volatile, then I don't see why
>> it's needed on the struct based datatype itself.
>>
>> As far as I can see, the struct itself is not volatile; it's just a
>> language level construct to gather together a set of device specific
>> registers. It's the registers defined within the struct, and only the
>> registers within the struct, which need to be marked as volatile in this
>> case.
>
> The handling of volatiles is poorly defined in the C standard [any
> version - take your pick]. Access to a volatile variable through a
> non-volatile pointer may implicitly cast away the volatile qualifier
> on the variable. E.g.,
>
> volatile int v;
> int *pv = &v; /* bad */
> volatile int *vpv = &v; /* much better */
>

I agree that volatiles are poorly defined in the standard - but the
standards are clear on this point. It is undefined behaviour to cast
away the volatile qualifier. So "int *pv = &v" is not just bad, it is
illegal in C.

> However, given
>
> struct _s { int i } s;
> struct _s *ps = &s;
> int *pi = &s.i;
> int x;
>
> "x = ps->i" is semantically different from "x = *pi" even though the
> result [at least here] is equivalent. The struct reference contains
> an implicit cast to int*: i.e. it acts as if you really had written
>
> x = *((int*)((char*)ps + offsetof(_s,i)))
>

I believe that "x = ps->i" is semantically identical to "x = *pi" - it
is valid to take a pointer to an element of a struct. I can't find the
reference in the C standards, but we could always cross-post to
comp.lang.c for an "official" ruling (at the risk of annoying everyone
else here with the serious pedantry that would be posted).

You are correct in how "x = ps->i" is interpreted - but since "int *pi =
&s.i" is interpreted as

int *pi = ((int*)((char*)ps + offsetof(_s,i)))

then "x = *pi" is exactly the same as "x = ps->i".

> The implicit cast on struct member references is a source of confusion
> for compiler implementers regarding volatile members: does it or does
> it not cast away a volatile qualifier on the member? IME it is
> compiler dependent what happens if you access a volatile struct member
> through a non-volatile struct pointer.
>

It is definitely compiler dependent, because it is undefined behaviour -
you cannot cast away the volatile qualifier (or the const qualifier) and
expect it to work properly. Compilers will generally warn about this.

Given this code:


typedef struct {
volatile int x;
volatile int y;
} S;

S s;

void foo1(void) {
s.x = 1;
}

void foo2(void) {
volatile int *pvi = &s.x;
*pvi = 1;
}

void foo3(void) {
int *pi = &s.x;
*pi = 1;
}

gcc will complain "warning: initialization discards qualifiers from
pointer target type" even if you don't enable warnings. It will still
generate identical code for each function - but it could produce nasal
daemons for foo3().

>
>> This is how I define these structs and it has always worked for me across
>> a range of architectures.
>
> MMV. I would expect that if all the compilers use the same base:
> e.g., all are GCC (or whatever) derived. But you shouldn't count on
> it across different compilers.
>
> IME the result even may depend on how the definitions are written:
> e.g.,
>
> struct peripheral_t { volatile uint8_t reg1, ... }
> *RALPH = 0x40039400;
>
> may work, whereas
>
> struct peripheral_t { volatile uint8_t reg1, ... };
> :
> peripheral_t *RALPH = 0x40039400;
>
> may fail.

Neither of these are valid C - you can't turn an integer literal into a
pointer without a cast. If they were cleaned up appropriately,
including casts, then the results are identical. I can't imagine any
compiler treating them differently (although I /can/ imagine a
poor-quality compiler having bugs in its implementation of volatile
struct fields).

[David Brown]

I won't argue with you about looking ugly, but it's okay to have ugly
code hidden away - the definition of RALPH as a macro here is perfectly
clear.

Using macros like this is far and away the most common method in use
today. It has two main advantages over using an "extern" definition.

First, it keeps everything in C rather than having essential parts of
your code in linker scripts (sometimes you need project-specific linker
scripts, but most people prefer to avoid them), or having assembly
modules (which most people also prefer to avoid).

Secondly, it gives the compiler a lot more opportunity for generating
better code. This is particularly noticeable on architectures like ARM
- with the macro definitions, the compiler can load a pointer to one
peripheral into a register and use register+offset addressing for other
peripherals (within the same function). Using extern definitions, the
compiler has to use large and slow absolute addressing for each
different peripheral used.

[Simon Clubley]

Thinking about it a bit more, I think what _really_ caught my attention
when I saw Tim's code was thinking about what is the scope of the
initial read in a read/modify/write cycle when the struct itself is
the volatile component ?

In the approach I prefer, the struct contains a collection of independent
volatile variables and the compiler does not read another variable when
generating code to update a specific volatile variable within the struct.

(This assumes it's physically possible to perform the requested access to
just the variable within the constraints of the MCU's architecture.)

However, when the struct itself is the volatile component, I can see only
one volatile component, the struct. The struct may have multiple variables
but the volatile attribute applies to the struct itself.

In such a struct, is it 100% guaranteed by the C language standards that,
when referencing a specific variable within the struct, only the memory
allocated to that variable is accessed in the generated code or is the
compiler allowed to access the memory of neighbouring variables ?

(Or is it even allowed to read the whole struct when doing a R/M/W cycle
when the volatile attribute is on the struct itself ?)

>
> Where you prefer to put the volatile is a matter of style (usually
> picked for you by the compiler or microcontroller vendor when they make
> the header files). Putting it in each field in the struct is a bit
> verbose, but gives the possibility of having some fields volatile and
> some non-volatile. Putting it on the struct definition is neater, and
> means the volatile qualifier cannot be forgotten when the struct is used
> - but it means you can't avoid using it. Putting it on the declaration
> of the object itself makes it explicit at the point of declaration, and
> also means that you can use the struct in a non-volatile way (such as
> for a local cached copy of the data in question). Finally, you can put
> the volatile on the access of the data, making it explicit when it is
> used, but forcing you to remember to use it.
>

Quite a bit of the time, I just throw away the manufacturer's headers
and create my own from the MCU's technical reference manual. Sometimes
it's because of the manufacturer's copyrights on the headers (Microchip
for example) or because I think the headers are junk and I can do
better. Doing this as a hobby gives you that level of freedom. :-)

(The TRM register layouts are usually in a form which makes them suitable
for editing with a few emacs keyboard macros after you have pasted them
from the TRM PDF into a emacs buffer.)

I use my own data type, which has the volatile attribute as part of the
type, when defining register variables so being verbose isn't a problem
in this case. (I have a fondness for user defined data types at multiple
levels when creating headers.)

Simon.

[David Brown]

I suppose you could say it would be more consistent if accessing a
volatile struct meant fully reading or writing the whole struct each
time. However, that is not what happens either in theory (i.e., the C
standards) or in practice (real-world compilers).

The C standards consider a "volatile struct" to be a struct in which
each member is volatile (the same applies to unions). See the examples
in section 6.5.2.3 in the C11 standards (the latest draft, which is
virtually identical to the final standard, is freely available online -
look for document N1570).

Thus a volatile struct and a struct of volatiles are identical to C.


When you are looking for guarantees about volatiles, the key sentence in
the standard is "What constitutes an access to an object that
has volatile-qualified type is implementation-defined". There are /no/
guarantees from the standards. So you have no guarantee that a volatile
read will not also read neighbouring data, nor that it will be carried
out as a single read rather than multiple smaller reads. Writes may not
write to neighbouring data (except within bitfield operations, which are
even less clearly specified), but they can use multiple small writes.

Compiler implementations can, of course, give specific guarantees. And
almost all compilers will generate code that does not split the access
into smaller accesses, assuming alignments and sizes match the hardware.
But some compilers will generate larger read accesses than specified -
in particular, it was common in pre-Cortex ARM code to implement 8-bit
and 16-bit reads as a 32-bit read with shifting and masking. Also note
that if you are doing something like assigning one struct to another
struct, the compiler can combine small reads into larger ones even if
they are made of small volatile fields.


_Atomic accesses are better specified, if your compiler supports them -
then there /is/ a difference between an _Atomic struct and a struct of
_Atomic.

>>
>> Where you prefer to put the volatile is a matter of style (usually
>> picked for you by the compiler or microcontroller vendor when they make
>> the header files). Putting it in each field in the struct is a bit
>> verbose, but gives the possibility of having some fields volatile and
>> some non-volatile. Putting it on the struct definition is neater, and
>> means the volatile qualifier cannot be forgotten when the struct is used
>> - but it means you can't avoid using it. Putting it on the declaration
>> of the object itself makes it explicit at the point of declaration, and
>> also means that you can use the struct in a non-volatile way (such as
>> for a local cached copy of the data in question). Finally, you can put
>> the volatile on the access of the data, making it explicit when it is
>> used, but forcing you to remember to use it.
>>
>
> Quite a bit of the time, I just throw away the manufacturer's headers
> and create my own from the MCU's technical reference manual. Sometimes
> it's because of the manufacturer's copyrights on the headers (Microchip
> for example) or because I think the headers are junk and I can do
> better. Doing this as a hobby gives you that level of freedom. :-)

I have sometimes had to correct manufacturer's headers (fixing typos, or
removing the volatile qualifier on struct members because of the
limitations noted above). But I don't write my own if I can avoid it -
doing this as a professional means I can't spend time on such niceties
without good reason.

[Simon Clubley]

On 2014-03-20, David Brown <david...@hesbynett.no> wrote:
> On 19/03/14 22:50, Simon Clubley wrote:
>>
>> Thinking about it a bit more, I think what _really_ caught my attention
>> when I saw Tim's code was thinking about what is the scope of the
>> initial read in a read/modify/write cycle when the struct itself is
>> the volatile component ?
>>
>> In the approach I prefer, the struct contains a collection of independent
>> volatile variables and the compiler does not read another variable when
>> generating code to update a specific volatile variable within the struct.
>>
>> (This assumes it's physically possible to perform the requested access to
>> just the variable within the constraints of the MCU's architecture.)
>>
>> However, when the struct itself is the volatile component, I can see only
>> one volatile component, the struct. The struct may have multiple variables
>> but the volatile attribute applies to the struct itself.
>>
>> In such a struct, is it 100% guaranteed by the C language standards that,
>> when referencing a specific variable within the struct, only the memory
>> allocated to that variable is accessed in the generated code or is the
>> compiler allowed to access the memory of neighbouring variables ?
>>
>> (Or is it even allowed to read the whole struct when doing a R/M/W cycle
>> when the volatile attribute is on the struct itself ?)
>>
>
> I suppose you could say it would be more consistent if accessing a
> volatile struct meant fully reading or writing the whole struct each
> time. However, that is not what happens either in theory (i.e., the C
> standards) or in practice (real-world compilers).
>

And it would be a Very Bad Thing (TM) if it did full struct reads. :-)

> The C standards consider a "volatile struct" to be a struct in which
> each member is volatile (the same applies to unions). See the examples
> in section 6.5.2.3 in the C11 standards (the latest draft, which is
> virtually identical to the final standard, is freely available online -
> look for document N1570).
>

Thank you, I will.

> Thus a volatile struct and a struct of volatiles are identical to C.
>

And that answers my query rather nicely thanks.

>
> When you are looking for guarantees about volatiles, the key sentence in
> the standard is "What constitutes an access to an object that
> has volatile-qualified type is implementation-defined". There are /no/
> guarantees from the standards. So you have no guarantee that a volatile
> read will not also read neighbouring data, nor that it will be carried
> out as a single read rather than multiple smaller reads. Writes may not
> write to neighbouring data (except within bitfield operations, which are
> even less clearly specified), but they can use multiple small writes.
>

Oh, I'm fully aware about the issues surrounding bitfields. :-) I did some
experiments a year or two ago with using bitfields instead of masks in
some ARM header definitions.

I gave up and went back to using masks. Some things worked rather nicely
but I could never 100% guarantee the generated code would do what I expected
it to do with regards to reads/writes to the underlying 32-bit register.

>
>>
>> Quite a bit of the time, I just throw away the manufacturer's headers
>> and create my own from the MCU's technical reference manual. Sometimes
>> it's because of the manufacturer's copyrights on the headers (Microchip
>> for example) or because I think the headers are junk and I can do
>> better. Doing this as a hobby gives you that level of freedom. :-)
>
> I have sometimes had to correct manufacturer's headers (fixing typos, or
> removing the volatile qualifier on struct members because of the
> limitations noted above). But I don't write my own if I can avoid it -
> doing this as a professional means I can't spend time on such niceties
> without good reason.
>

In my day job, I am a commercial systems programmer/sysadmin and it's quite
usual for me to work with high quality code and documentation. For example,
I have a VMS background (and a general DEC background) and while VMS is
dated by today's standards. the documentation and example code is very
good.

It was quite revealing seeing some of things the various MCU manufacturers
put out as code/documentation when examined in light of my day job
experiences.

However, I understand exactly where you are coming from and understand that
you have to work with what you have when someone is paying you for your
time.

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message
news:lg6pe0$l3s$1...@dont-email.me...

> On 17/03/14 11:21, Nils M Holm wrote:

>> The original version of the compiler was for education and was hosted
>> on and targeted at FreeBSD/386. Later I kept hacking it for fun and
>> added back-ends for the x86-64, 8086 and, lately, the ARMv6. I also
>> added runtime support for Linux, various BSDs and DOS. Support for
>> Windows and Darwin were added by contributors.
>>
>> What I like about the compiler is that it is simple, easy to hack, and
>> boostraps in 4 seconds on a 700 MHz Raspi. Of course, its code generator
>> is rather limited, and its code runs (on average) almost twice as long
>> as code generated by GCC -O0, but it is suffient for most stuff I do.

> Thanks - that puts things in a more complete context. "Real" compilers
> such as gcc are far from simple or understandable, even for people who
> have worked with them for years. llvm is a bit clearer and more
> structured, but it too is a huge project.

> So a limited small compiler
> for educational purposes seems like a good idea, even if it cannot be
> used for "real" programs.

Why not? There are considerable advantages to having a simple, small
compiler, some of which the OP has pointed out.

I've always used my own compilers *and* languages, and they have been used
to create real, commercial products too.

I think there is a place for a simple, static compiled language other than
the same boring choice of always using C (or sometimes, C++; same thing
really). It can be simpler and tidier too because it will have less baggage.
(Although my own effort is likely to stay private.)

As for speed, my last working unoptimised compiler, for x86, was on a par
with gcc -O0 (and an experimental optimised version could just match other
non-gcc optimising compilers). However, because typically I make it very,
very easy to have inline asm code, it is a simple matter to optimise
specific routines this way, and approach or surpass gcc -O1.

As for ARM, I haven't had a go at that yet. I also noticed the lack of
absolute addressing. But what puts me off though is, after generating ARM
ASM, I end up having to use gcc anyway! (I assume the gnu assembler that's
been mentioned is the one inside gcc.)

But a funny thing about ARM (specifically the one in the 'Pi') and gcc: the
first C program I tried, even compiled with -O3, ran at about one third the
expected speed. This was because gcc, thinking some pointer values were
misaligned (they weren't misaligned in my program, and this model of ARM
didn't have that issue anyway) was doing byte-at-a-time accesses to load and
store values! It didn't bother to mention this small detail. I can tell you
that any code of mine, no matter how bad, at least wouldn't have done that!

--
Bartc

[David Brown]

Normally that's true - although sometimes people misunderstand volatile
and think it gives them atomic access. Imagine a struct containing a
count of days, and a count of seconds, with an interrupt routine
updating the counters. If the main loop reads this without due
consideration, you get a chance of everything going horribly wrong once
per 24 hours - and volatile on the struct or the fields will not help.
The _Atomic qualifier on the struct will solve the problem - /if/ you've
got a compiler that supports it.

>
>> The C standards consider a "volatile struct" to be a struct in which
>> each member is volatile (the same applies to unions). See the examples
>> in section 6.5.2.3 in the C11 standards (the latest draft, which is
>> virtually identical to the final standard, is freely available online -
>> look for document N1570).
>>
>
> Thank you, I will.

It is not the most exciting bedtime reading, but it can be useful. It
is also a bit depressing how badly some things are specified (or left as
implementation defined), and how many things we take for granted that
are actually not required by the standards.

>
>> Thus a volatile struct and a struct of volatiles are identical to C.
>>
>
> And that answers my query rather nicely thanks.
>
>>
>> When you are looking for guarantees about volatiles, the key sentence in
>> the standard is "What constitutes an access to an object that
>> has volatile-qualified type is implementation-defined". There are /no/
>> guarantees from the standards. So you have no guarantee that a volatile
>> read will not also read neighbouring data, nor that it will be carried
>> out as a single read rather than multiple smaller reads. Writes may not
>> write to neighbouring data (except within bitfield operations, which are
>> even less clearly specified), but they can use multiple small writes.
>>
>
> Oh, I'm fully aware about the issues surrounding bitfields. :-) I did some
> experiments a year or two ago with using bitfields instead of masks in
> some ARM header definitions.
>
> I gave up and went back to using masks. Some things worked rather nicely
> but I could never 100% guarantee the generated code would do what I expected
> it to do with regards to reads/writes to the underlying 32-bit register.

It is all "implementation defined". So if your compiler says it
implements volatile bitfields in a particular way (such as gcc with the
-fvolatile-bitfields flag), you have your guarantees. But things can
change on a different compiler, different versions, or different
targets. As a general rule, if the compiler vendor provides headers
with registers defined using bitfields, you can be confident that it
will implement volatile bitfields in the most user-friendly manner of
using the bitfield specified size for all reads and writes (except when
you read or write the structure as a whole, of course).

>
>>
>>>
>>> Quite a bit of the time, I just throw away the manufacturer's headers
>>> and create my own from the MCU's technical reference manual. Sometimes
>>> it's because of the manufacturer's copyrights on the headers (Microchip
>>> for example) or because I think the headers are junk and I can do
>>> better. Doing this as a hobby gives you that level of freedom. :-)
>>
>> I have sometimes had to correct manufacturer's headers (fixing typos, or
>> removing the volatile qualifier on struct members because of the
>> limitations noted above). But I don't write my own if I can avoid it -
>> doing this as a professional means I can't spend time on such niceties
>> without good reason.
>>
>
> In my day job, I am a commercial systems programmer/sysadmin and it's quite
> usual for me to work with high quality code and documentation. For example,
> I have a VMS background (and a general DEC background) and while VMS is
> dated by today's standards. the documentation and example code is very
> good.
>
> It was quite revealing seeing some of things the various MCU manufacturers
> put out as code/documentation when examined in light of my day job
> experiences.

At the risk of making unfair generalisations, I think a lot of the
example code produced by MCU and compiler vendors seems to be produced
by students doing summer jobs - certainly the code is seldom of the
quality you would expect for serious embedded systems.

[David Brown]

On 20/03/14 15:35, BartC wrote:
>
>
> "David Brown" <david...@hesbynett.no> wrote in message
> news:lg6pe0$l3s$1...@dont-email.me...
>> On 17/03/14 11:21, Nils M Holm wrote:
>
>>> The original version of the compiler was for education and was hosted
>>> on and targeted at FreeBSD/386. Later I kept hacking it for fun and
>>> added back-ends for the x86-64, 8086 and, lately, the ARMv6. I also
>>> added runtime support for Linux, various BSDs and DOS. Support for
>>> Windows and Darwin were added by contributors.
>>>
>>> What I like about the compiler is that it is simple, easy to hack, and
>>> boostraps in 4 seconds on a 700 MHz Raspi. Of course, its code generator
>>> is rather limited, and its code runs (on average) almost twice as long
>>> as code generated by GCC -O0, but it is suffient for most stuff I do.
>
>> Thanks - that puts things in a more complete context. "Real" compilers
>> such as gcc are far from simple or understandable, even for people who
>> have worked with them for years. llvm is a bit clearer and more
>> structured, but it too is a huge project.
>
>> So a limited small compiler
>> for educational purposes seems like a good idea, even if it cannot be
>> used for "real" programs.
>
> Why not? There are considerable advantages to having a simple, small
> compiler, some of which the OP has pointed out.

Yes, there are advantages to having a small and simple compiler in some
contexts. But this particular compiler is very limited in the subset of
C it supports, and that greatly reduces its usefulness of normal work.
And since the generated code is about half the speed of gcc on -O0, it
will be a factor of 5 to 10 times slower than /real/ code generated by
/real/ compilers using /real/ compiler flags. Would you be happy for
all the software on your computer to run at 10-20% of normal speed, just
because the software vendor wanted to use a simpler compiler? Of course
not. So for normal work, you use a normal compiler - for educational
work or other specialised usage, you might want to use a more niche
compiler.

>
> I've always used my own compilers *and* languages, and they have been used
> to create real, commercial products too.

There are vast numbers of programming languages, many of which are
supported by different tools - and there are many good reasons for using
them. Very often you will pick the right programming language for the
task (occasionally making your own language if that's the best
solution), then pick the compiler or interpreter to suit (again,
occasionally writing your own). But if you pick a major language - such
as C - for your program, you do not then pick a very small and very
limited compiler as your development tool unless you have very
specialised needs - such as being able to run it on a tiny host, or
being able to understand the compiler's source code.

>
> I think there is a place for a simple, static compiled language other than
> the same boring choice of always using C (or sometimes, C++; same thing
> really). It can be simpler and tidier too because it will have less
> baggage.

I don't disagree with that (except that C and C++ are not the same thing
at all - and the distance between them has been growing rapidly). I
don't think the complexity or size of C compilers is the reason for
this, however - the aim would be to make a better statically compiled
language than C (for whatever value of "better" suits your purpose).

> (Although my own effort is likely to stay private.)
>
> As for speed, my last working unoptimised compiler, for x86, was on a par
> with gcc -O0 (and an experimental optimised version could just match other
> non-gcc optimising compilers). However, because typically I make it very,
> very easy to have inline asm code, it is a simple matter to optimise
> specific routines this way, and approach or surpass gcc -O1.

With inline assembly, you are no longer working in C (obviously) and so
is of no relevance in a speed contest.

There are many reasons why one might usefully write a C compiler - for
fun, education (either your own or other peoples'), for specialised
processors, for specialised hosts, or as a basis for an "extended C"
language. But you don't write one for a standard processor architecture
and aim to be fast, unless you have the resources to compete with the
big names (gcc, llvm, Intel, MS, and big embedded toolchain vendors) -
unless it is for fun or education. Otherwise you need to do an enormous
amount of work (probably tens of man-years) to get close on speed,
features, and correctness, for a tool that almost no one will ever use.

>
> As for ARM, I haven't had a go at that yet. I also noticed the lack of
> absolute addressing. But what puts me off though is, after generating ARM
> ASM, I end up having to use gcc anyway! (I assume the gnu assembler that's
> been mentioned is the one inside gcc.)

The gnu assembler is part of the gnu binutils project (along with the
linker and librarian) - it is not part of gcc. And of course if you are
writing your own compiler, you can write your own assembler and linker -
it's a lot easier than writing an optimising compiler.

>
> But a funny thing about ARM (specifically the one in the 'Pi') and gcc: the
> first C program I tried, even compiled with -O3, ran at about one third the
> expected speed. This was because gcc, thinking some pointer values were
> misaligned (they weren't misaligned in my program, and this model of ARM
> didn't have that issue anyway) was doing byte-at-a-time accesses to load
> and store values! It didn't bother to mention this small detail. I can
> tell you that any code of mine, no matter how bad, at least wouldn't
> have done that!
>

gcc will assume that pointers are properly aligned according to their
types - you have to go out of your way to lose that (such as by casting
pointers). If you have code that takes a pointer-to-char and you pass
it pointers to 32-bit values, then of course the compiler has to
generate byte-sized code, because that is the only legal choice in C.
There are several ways to tell the compiler about larger alignment and
access sizes, but you have to give the compiler the information before
it can generate such code. (And if your compiler breaks the relevant
rules for C and/or the hardware, that's your choice - but it is not then
a C compiler.)

[Tim Wescott]

That's an interesting point. The peripherals on the part I'm using tend
to be in widely-spaced blocks, so I'm not sure if that really applies
here. But it could. I'll need to think about that.

Most of my customers have low enough volume sales that my goal in
choosing a processor is to get one that's big enough and fast enough so
that I don't have to be sparing with clock ticks or memory when I'm
writing code: shoe-horning always seems to consume lots of expensive
engineering hours; in this age of big fast inexpensive processors, I
don't see the payback unless you're building 100k units at a time.

[Simon Clubley]

On 2014-03-20, David Brown <david...@hesbynett.no> wrote:

> On 20/03/14 15:35, BartC wrote:
>>
>> As for ARM, I haven't had a go at that yet. I also noticed the lack of
>> absolute addressing. But what puts me off though is, after generating ARM
>> ASM, I end up having to use gcc anyway! (I assume the gnu assembler that's
>> been mentioned is the one inside gcc.)
>
> The gnu assembler is part of the gnu binutils project (along with the
> linker and librarian) - it is not part of gcc.
>

Quite true. Don't the Free Pascal people use binutils just as gcc does ?

>>
>> But a funny thing about ARM (specifically the one in the 'Pi') and gcc: the
>> first C program I tried, even compiled with -O3, ran at about one third the
>> expected speed. This was because gcc, thinking some pointer values were
>> misaligned (they weren't misaligned in my program, and this model of ARM
>> didn't have that issue anyway) was doing byte-at-a-time accesses to load
>> and store values! It didn't bother to mention this small detail. I can
>> tell you that any code of mine, no matter how bad, at least wouldn't
>> have done that!
>>
>
> gcc will assume that pointers are properly aligned according to their
> types - you have to go out of your way to lose that (such as by casting
> pointers). If you have code that takes a pointer-to-char and you pass
> it pointers to 32-bit values, then of course the compiler has to
> generate byte-sized code, because that is the only legal choice in C.
> There are several ways to tell the compiler about larger alignment and
> access sizes, but you have to give the compiler the information before
> it can generate such code. (And if your compiler breaks the relevant
> rules for C and/or the hardware, that's your choice - but it is not then
> a C compiler.)
>

I've seen this byte-at-a-time issue before on a ARM target when
__attribute__ ((packed)) was specified for a struct. Even though gcc
had enough information to generate reasonable code, and to guarantee the
struct was aligned correctly, the code it generated was rather stupid.
It even generated byte-at-a-time code for 32-bit volatile variables
(used for device registers).

[Nils M Holm]

David Brown <david...@hesbynett.no> wrote:
> [...] Would you be happy for

> all the software on your computer to run at 10-20% of normal speed, just
> because the software vendor wanted to use a simpler compiler? Of course

> not. [...]

Of course I would. I would be happy to trade execution speed for
compilation speed in many cases. Back in the days we had a project that
took almost an hour to compile using MSC and four minutes to compile
with Turbo C. We were quite happy to use a faster compiler, even if the
resulting executable was a bit larger and slower.

BTW, when comparing my own compiler with GCC -O3, it's more 35% of the
optimal speed, and for many programs that I use, both in private and at
work, that does not make any difference.

[BartC]

"Simon Clubley" <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote in
message news:lgfa7q$gqa$1...@dont-email.me...

I think that was it (iirc): 1-byte alignment was applied to a struct such
as:

{int; double; int;}

so that you had field widths of (4, 8, 4) bytes, and a total size of 16.
Without the packing, I think it would align the double on an 8-byte boundary
(which seemed pointless on the 32-bit ARM) and end up with a 20-byte struct.

In the end I just moved the fields around so that it was {int; int;
double;}. (The actual struct was much more complex.)

--
Bartc

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lgf4j6$4oo$1...@dont-email.me...

> On 20/03/14 15:35, BartC wrote:

>> Why not? There are considerable advantages to having a simple, small
>> compiler, some of which the OP has pointed out.
>
> Yes, there are advantages to having a small and simple compiler in some
> contexts. But this particular compiler is very limited in the subset of
> C it supports, and that greatly reduces its usefulness of normal work.
> And since the generated code is about half the speed of gcc on -O0, it
> will be a factor of 5 to 10 times slower than /real/ code generated by
> /real/ compilers using /real/ compiler flags. Would you be happy for
> all the software on your computer to run at 10-20% of normal speed, just
> because the software vendor wanted to use a simpler compiler?

That 5-10 times would be the absolutely worst case, when you are running
tight integer benchmarks.

Real code would be different. On my interpreter program, the difference
between -O0 and -O3 is less than 2:1. And even *that* is when interpreteing
integer benchmarks!

And unless all the software (OS, drivers, libraries) is recompiled with the
same slower compiler, the slowdown will only apply to the application.

Also, development speed might be more important than the maximum possible
execution speed; you can always run gcc with -O3 when it's finished! It
might also be that a simple optimising pass, if desired, could make an quick
improvement on that 2x slower that -O0.

>> I think there is a place for a simple, static compiled language other
>> than
>> the same boring choice of always using C (or sometimes, C++; same thing
>> really). It can be simpler and tidier too because it will have less
>> baggage.
>
> I don't disagree with that (except that C and C++ are not the same thing
> at all - and the distance between them has been growing rapidly). I
> don't think the complexity or size of C compilers is the reason for
> this, however - the aim would be to make a better statically compiled
> language than C (for whatever value of "better" suits your purpose).

For embedded work especially, being able to express integer constants in
binary (together with separators for readability) is one of many dozens of
simple enhancements that can make life easier. (Maybe gcc has some
non-standard extension, and obscure switch to enable it, for doing that.
That's not the same.) Type-specifiers that you simply read and write
left-to-right like English is another (I doubt there's a gcc switch for
that!).

> However, because typically I make it very,
>> very easy to have inline asm code, it is a simple matter to optimise
>> specific routines this way, and approach or surpass gcc -O1.
>
> With inline assembly, you are no longer working in C (obviously) and so
> is of no relevance in a speed contest.

You will still be working predominantly in C (or whatever comparable high
level is used). The inline asm stuff is just a technique. After all gcc C
compilers seem to depend on a plethora of weird-looking attributes, a
thousand compiler options, and an entirely separate language in make files
in order to do what they do. A few lines of ASM is tame!

And if done properly, you will provide both asm and regular versions of the
code, so that you can switch off the asm when necessary.

> The gnu assembler is part of the gnu binutils project (along with the
> linker and librarian) - it is not part of gcc.

OK, I will take a look. The ARM assembly code posted here looked like that
assembled by gcc when you give it a .s file. (I'm not interested these days
in writing assemblers and linkers...)

--
Bartc

[Simon Clubley]

On 2014-03-20, BartC <b...@freeuk.com> wrote:
> "David Brown" <david...@hesbynett.no> wrote in message
> news:lgf4j6$4oo$1...@dont-email.me...
>

>> The gnu assembler is part of the gnu binutils project (along with the
>> linker and librarian) - it is not part of gcc.
>
> OK, I will take a look. The ARM assembly code posted here looked like that
> assembled by gcc when you give it a .s file. (I'm not interested these days
> in writing assemblers and linkers...)
>

If you are talking about my code, gcc compiled a C program by generating
a file containing ARM assembly source code which it handed off to the
assembler which is a part of binutils. The binutils assembler then turned
that generated source code into a binary .o object file.

The binutils assembler can be used just fine outside of gcc; it's just
that most people, myself included, invoke the binutils assembler by
using the gcc frontend.

objdump, the utility used to disassemble the .elf file in my example
code, is also a part of binutils. It has a option to include the source
code at the correct points in the disassembled output.

However, with various optimisation levels turned on, that means you
usually see the same sections of source code included more than once
which happened in my example.

[David Brown]

On 20/03/14 18:58, Simon Clubley wrote:
> On 2014-03-20, David Brown <david...@hesbynett.no> wrote:
>> On 20/03/14 15:35, BartC wrote:
>>>
>>> As for ARM, I haven't had a go at that yet. I also noticed the lack of
>>> absolute addressing. But what puts me off though is, after generating ARM
>>> ASM, I end up having to use gcc anyway! (I assume the gnu assembler that's
>>> been mentioned is the one inside gcc.)
>>
>> The gnu assembler is part of the gnu binutils project (along with the
>> linker and librarian) - it is not part of gcc.
>>
>
> Quite true. Don't the Free Pascal people use binutils just as gcc does ?

They do indeed, although Free Pascal is a relatively small community.
gcc itself supports several languages in mainline (C, C++, Ada, Java,
Fortran, Go) as well as a number of other out-of-mainline languages (D,
Pascal, Mercury, Cobol, Modula-2, VHDL, PL/1 - and probably many more).
And pretty much any other compiler in the Linux (or *nix) world will
use binutils rather than re-inventing the assembler and linker.

And of course, don't forget assembler programmers!

>
>>>
>>> But a funny thing about ARM (specifically the one in the 'Pi') and gcc: the
>>> first C program I tried, even compiled with -O3, ran at about one third the
>>> expected speed. This was because gcc, thinking some pointer values were
>>> misaligned (they weren't misaligned in my program, and this model of ARM
>>> didn't have that issue anyway) was doing byte-at-a-time accesses to load
>>> and store values! It didn't bother to mention this small detail. I can
>>> tell you that any code of mine, no matter how bad, at least wouldn't
>>> have done that!
>>>
>>
>> gcc will assume that pointers are properly aligned according to their
>> types - you have to go out of your way to lose that (such as by casting
>> pointers). If you have code that takes a pointer-to-char and you pass
>> it pointers to 32-bit values, then of course the compiler has to
>> generate byte-sized code, because that is the only legal choice in C.
>> There are several ways to tell the compiler about larger alignment and
>> access sizes, but you have to give the compiler the information before
>> it can generate such code. (And if your compiler breaks the relevant
>> rules for C and/or the hardware, that's your choice - but it is not then
>> a C compiler.)
>>
>
> I've seen this byte-at-a-time issue before on a ARM target when
> __attribute__ ((packed)) was specified for a struct. Even though gcc
> had enough information to generate reasonable code, and to guarantee the
> struct was aligned correctly, the code it generated was rather stupid.
> It even generated byte-at-a-time code for 32-bit volatile variables
> (used for device registers).
>

Yes, the "packed" attribute can cause this - you really want to avoid
that unless you have very good reason. Mostly it generates good code,
but sometimes you get byte-at-a-time access. There can be several
factors involved here, such as the compiler version, the flags used, and
how the data is being accessed. But it is quite possible to get
slower-than-necessary code for packed struct accesses - and it is also
possible to get misaligned accesses, which are illegal on some hardware,
but you usually have to cheat a little to force that (such as by taking
a pointer to a packed struct member).

In many cases, you can re-order your struct to avoid gaps without
resorting to "packed". Personally, I often like to put in any padding
explicitly (and use the "-Wpadded" flag to check).

[George Neuner]

On Wed, 19 Mar 2014 22:36:43 +0100, David Brown
<david...@hesbynett.no> wrote:

>On 19/03/14 19:34, George Neuner wrote:
>>
>> [some people don't like the "view" notion of structs, but consider
>> that you can place multiple different structs at the same location to
>> yield different views of the same underlying address space.]
>
>That's a useful idea, but you need to be a bit careful - as far as the C
>language (and therefore the compiler) is concerned, each piece of memory
>you access is part of a unique object with a defined type. The only
>legal exceptions are unions, pointers to char, and structs which share
>the first few fields.
>
>So if you have a pointer pS1 to struct S1 and a pointer pS2 to struct
>S2, the compiler "knows" that they cannot both point to the same object
>in memory - this is know as type-based alias analysis. If you use pS1
>to modify the object, then read the memory via pS2, you may get
>unexpected results - since the compilers "knows" that these point to
>different areas of memory, the loads and stores are independent and the
>compiler may re-arrange them.
>
>Of course, when you are using volatile accesses, the compiler cannot
>re-arrange such accesses.

Absolutely.

But my point was more esoteric: namely that a struct is a reification
of a view of the address space and not a "container" as such. You
really have to think about it in this way to correctly (ab)use unions.

>> However, given
>>
>> struct _s { int i } s;
>> struct _s *ps = &s;
>> int *pi = &s.i;
>> int x;
>>
>> "x = ps->i" is semantically different from "x = *pi" even though the
>> result [at least here] is equivalent. The struct reference contains
>> an implicit cast to int*: i.e. it acts as if you really had written
>>
>> x = *((int*)((char*)ps + offsetof(_s,i)))
>>
>
>I believe that "x = ps->i" is semantically identical to "x = *pi" - it
>is valid to take a pointer to an element of a struct.

Yes, you can take the address of a struct member.

But here's a question for you: how does the compiler get an rvalue
from memory? Answer: it computes the corresponding lvalue and then
dereferences it.

Next question: what does the type checking for that look like?


The type "int *pi" is trivially ptr(int) and the type of the rvalue
expression "*pi", also trivially, is deref(ptr(int) => int.

However the type of the rvalue expression "ps->i" is not trivially
int, but rather something like:

member( _s,int(i) ) => int

which is compatible with and substitutable for int at a higher level,
but it is not itself int because a struct defines a separate namespace
that must be qualified to decide the type of the expression. Applying
the address operator to the above, i.e. "&ps->i", makes a pointer and
gives:

ptr( member( _s,int(i) ) => int ) => ptr(int)

which then is assignment compatible with "int *pi".


A different level of semantics.

>> The implicit cast on struct member references is a source of confusion
>> for compiler implementers regarding volatile members: does it or does
>> it not cast away a volatile qualifier on the member? IME it is
>> compiler dependent what happens if you access a volatile struct member
>> through a non-volatile struct pointer.
>>
>
>It is definitely compiler dependent, because it is undefined behaviour -
>you cannot cast away the volatile qualifier (or the const qualifier) and
>expect it to work properly. Compilers will generally warn about this.
>
>Given this code:
>>
>typedef struct {
> volatile int x;
> volatile int y;
>} S;
>

> :

>
>void foo3(void) {
> int *pi = &s.x;
> *pi = 1;
>}
>
>gcc will complain "warning: initialization discards qualifiers from
>pointer target type" even if you don't enable warnings. It will still
>generate identical code for each function - but it could produce nasal
>daemons for foo3().

However, it is trivial to remove the volatile qualifier. E.g.,

void foo4(void)
{
*((int*)((char*)&S + offsetof(S,y))) = 42;
}

foo4() constructs and uses a non-volatile pointer to S.y, but the
compiler won't utter a peep about it.


Btw: this isn't an exercise in fooling the compiler - I've seen this
sort of thing done in the real world to do things like application
level scatter/gather I/O, generic field access to dynamically defined
record structures, etc. E.g.,

size_t read_int_member( FILE* fp, void* base, int offset )
{
int *p = (int*)((char*)base + offset);
return fread( p, sizeof(int), 1, fp );
}
:
read_int_member( file1, &S, offsetof( S, x ) );
read_int_member( file2, &S, offsetof( S, y ) );
:


YMMV,
George

[David Brown]

The alignment of a double is normally determined by the ABI for the
target - gcc follows the standard set down by ARM (as will any other ARM
compiler). I am guessing that on chips with floating point hardware,
the 8-byte alignment here /does/ matter - and for consistency, it is
kept the same across the whole architecture.

Your {int; double; int;} struct would actually be 24 bytes, not 20 bytes
- the end of the struct is padded so that you can use it in an array and
have the correct 8-byte alignment for all elements.

In most cases, optimal code will still be produced even with "packed" if
the compiler knows that the alignment is good - my brief testing of a
packed "int; double; int" struct was fine. But sometimes it can be
messy - you do not want to use "packed" unless you really have to.
(That applies to all targets and all compilers, not just arm and gcc.)

[David Brown]

On 20/03/14 19:20, Nils M Holm wrote:
> David Brown <david...@hesbynett.no> wrote:
>> [...] Would you be happy for
>> all the software on your computer to run at 10-20% of normal speed, just
>> because the software vendor wanted to use a simpler compiler? Of course
>> not. [...]
>
> Of course I would. I would be happy to trade execution speed for
> compilation speed in many cases. Back in the days we had a project that
> took almost an hour to compile using MSC and four minutes to compile
> with Turbo C. We were quite happy to use a faster compiler, even if the
> resulting executable was a bit larger and slower.
>

Any tradeoff has its balance points - when there is such a huge
difference in compilation times, you are going to pick the faster tool.

> BTW, when comparing my own compiler with GCC -O3, it's more 35% of the
> optimal speed, and for many programs that I use, both in private and at
> work, that does not make any difference.
>

For many programs, a slowdown of a factor of 3 is not going to be a big
issue - but for many others, it would be very significant. Different
types of program have different balances between issues like run-time
speed and ease of development.

The biggest reason against using your compiler for real work is not the
speed - it is the features. "SUBC" is such a limited subset of C89
(which itself is 25 years outdated - C90, C99 and C11 have at least a
few useful new features) that you cannot call it a "C compiler", as it
cannot compile normal C code. You might feel it is a big enough subset
to be useful, which is fair enough - and I'm sure it is very useful for
educational purposes (I haven't looked at the source, but I believe you
when you say it is clear and understandable). But it's not a realistic
tool for serious C programming.

[David Brown]

For small and medium sized projects, compiler speed is rarely a problem
- and it is often easily solved by using a faster development PC and
using "make -j" for parallel compilation. For more complicated
projects, decent makefiles (rather than an IDE's project manager) can
make a big difference, and tools like ccache can speed up compilation.

For larger projects, you generally need a mainstream compiler to get the
features you need.


I agree on the principle, of course - the speed and convenience of
development tools is very important. And sometimes that also means your
tools or options vary through the development process - such as using
-O1 for fast compilation and -O2 for fast run-time. I just don't agree
that using a simple, limited C compiler rather than gcc (or llvm, MSVC
for C++, Intel, etc.) is going to be a good choice.


(gcc on -O3 usually produces slower code than -O2, as the larger code
makes less use of caches. Only go above -O2 for particular types of
code - it is usually best to use -O2 with hand-picked optimisations that
match the particular code in use.)

>
>>> I think there is a place for a simple, static compiled language other
>>> than
>>> the same boring choice of always using C (or sometimes, C++; same thing
>>> really). It can be simpler and tidier too because it will have less
>>> baggage.
>>
>> I don't disagree with that (except that C and C++ are not the same thing
>> at all - and the distance between them has been growing rapidly). I
>> don't think the complexity or size of C compilers is the reason for
>> this, however - the aim would be to make a better statically compiled
>> language than C (for whatever value of "better" suits your purpose).
>
> For embedded work especially, being able to express integer constants in
> binary (together with separators for readability) is one of many dozens of
> simple enhancements that can make life easier. (Maybe gcc has some
> non-standard extension, and obscure switch to enable it, for doing that.
> That's not the same.) Type-specifiers that you simply read and write
> left-to-right like English is another (I doubt there's a gcc switch for
> that!).

gcc supports binary constants in the form 0b0010, and has done for quite
some time - without any sort of "obscure switch". It /is/ an extension,
in that it is not valid standard C syntax, but it is very much a "simple
enhancement". It is also planned for the C++17 standard, and hopefully
will one day make it into the C standards.

gcc has many other enhancements to make life easier for its users. It
is not uncommon for them to make their way into newer C and C++
standards - it is easier to get a feature into the standards if you can
point to an existing successful and popular feature from a major toolchain.

Of course, there are many other possible enhancements or extensions that
gcc does not support. And if you have a lot of such features that you
need, there can be some sense in making your own compiler that supports
them. Of course, you could also consider adding them to gcc (it's much
easier these days, due to the plugin support in gcc) - then you get all
the benefits of gcc, and you get your desired enhancements, and you have
avoided the work of writing the rest of the compiler.

>
>> However, because typically I make it very,
>>> very easy to have inline asm code, it is a simple matter to optimise
>>> specific routines this way, and approach or surpass gcc -O1.
>>
>> With inline assembly, you are no longer working in C (obviously) and so
>> is of no relevance in a speed contest.
>
> You will still be working predominantly in C (or whatever comparable high
> level is used). The inline asm stuff is just a technique. After all gcc C
> compilers seem to depend on a plethora of weird-looking attributes, a
> thousand compiler options, and an entirely separate language in make files
> in order to do what they do. A few lines of ASM is tame!

Exaggeration is the root of all evil!

[Nils M Holm]

David Brown <david...@hesbynett.no> wrote:
> On 20/03/14 19:20, Nils M Holm wrote:
> > Of course I would. I would be happy to trade execution speed for
> > compilation speed in many cases. Back in the days we had a project that
> > took almost an hour to compile using MSC and four minutes to compile
> > with Turbo C. We were quite happy to use a faster compiler, even if the
> > resulting executable was a bit larger and slower.
> >
>
> Any tradeoff has its balance points - when there is such a huge
> difference in compilation times, you are going to pick the faster tool.

Absolutely. There are certainly areas where you have to squeeze out
every cycle you can get.

> The biggest reason against using your compiler for real work is not the
> speed - it is the features. "SUBC" is such a limited subset of C89
> (which itself is 25 years outdated - C90, C99 and C11 have at least a
> few useful new features) that you cannot call it a "C compiler", as it
> cannot compile normal C code. You might feel it is a big enough subset
> to be useful, which is fair enough - and I'm sure it is very useful for
> educational purposes (I haven't looked at the source, but I believe you
> when you say it is clear and understandable). But it's not a realistic
> tool for serious C programming.

First of all: I do not intend to propagating SubC for "real"
development.

But then I do not agree with your implied definitions of "real work"
and "serious programming". What is "real" work? Working on complex
problems? Getting paid? Being creative? Having fun?

In the field where I work, algorithms are more important than
using the latest compilers. We all know that quicksort with -O0
beats bubblesort with any fancy optimization flags you could
through at it, hands down, given the input set is big enough.

In fact I am using an interpreter most of the time and a slow one
at that plus a language that predates C89 by almost 20 years. I do
research and I get paid for it. So is this "real" work or not?

I think a narrow definition of "real work" and "serious programming"
(in whatever language) marginalizes people who think outside of the
box and are not afraid to do their own thing. I think this is a pity,
because it propagates uniformity, and uniformity kills creativity
(and joy, and many other things that make us human).

[David Brown]

OK...

> Next question: what does the type checking for that look like?
>
>
> The type "int *pi" is trivially ptr(int) and the type of the rvalue
> expression "*pi", also trivially, is deref(ptr(int) => int.
>

OK...

> However the type of the rvalue expression "ps->i" is not trivially
> int, but rather something like:
>
> member( _s,int(i) ) => int
>
> which is compatible with and substitutable for int at a higher level,
> but it is not itself int because a struct defines a separate namespace
> that must be qualified to decide the type of the expression.

It is true that "ps->i" is not /trivially/ an int in the same way as *pi
was, but it is nonetheless an int - that is how structs work in C.

> Applying
> the address operator to the above, i.e. "&ps->i", makes a pointer and
> gives:
>
> ptr( member( _s,int(i) ) => int ) => ptr(int)
>
> which then is assignment compatible with "int *pi".
>
>
> A different level of semantics.

A struct is a collection of members, which can be of (almost) any type.
Accessing them, and identifying the types of members, is applied
recursively through all layers of struct, union, array, and pointer.
It's a little more complicated than a simple type, but it is not
fundamentally different.

You don't need anything like that sort of "offsetof" mess. A simple
cast is enough "int *pi = (int*) &s.y; *pi = 42;".

The syntax is correct, and the compiler does not warn about it (other
compilers might, of course), and the generated code is as expected. But
it is still undefined behaviour according to the C standards - and no
amount of messing around with offsetof macros will make the slightest
difference. Accessing a volatile qualified object through a
non-volatile lvalue is undefined behaviour - the standards are very
clear on this.

>
>
> Btw: this isn't an exercise in fooling the compiler - I've seen this
> sort of thing done in the real world to do things like application
> level scatter/gather I/O, generic field access to dynamically defined
> record structures, etc. E.g.,
>
> size_t read_int_member( FILE* fp, void* base, int offset )
> {
> int *p = (int*)((char*)base + offset);
> return fread( p, sizeof(int), 1, fp );
> }
> :
> read_int_member( file1, &S, offsetof( S, x ) );
> read_int_member( file2, &S, offsetof( S, y ) );
> :
>

Yes, it /is/ an exercise in fooling the compiler - even if the code
author did not know it.

It is legal to construct and use a pointer-to-int from a pointer-to-char
like this (especially since a pointer-to-char can alias anything else),
as long as you make sure that any alignment requirements are satisfied.

But it is not legal to cast away the volatile qualifier from an object
or type (though perfectly legal to add it). It is the same situation as
you have with "const" qualifiers.

>
> YMMV,
> George
>

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lgh1ua$mlg$1...@dont-email.me...

> On 20/03/14 20:45, BartC wrote:

> (gcc on -O3 usually produces slower code than -O2, as the larger code
> makes less use of caches.

(On x86, -O2 isn't faster, not for my main project anyway. But I'll try it
on the ARM board next time I dig it out.)

>>> I don't think the complexity or size of C compilers is the reason for
>>> this, however - the aim would be to make a better statically compiled
>>> language than C (for whatever value of "better" suits your purpose).
>>
>> For embedded work especially, being able to express integer constants in
>> binary (together with separators for readability) is one of many dozens
>> of
>> simple enhancements that can make life easier. (Maybe gcc has some
>> non-standard extension, and obscure switch to enable it, for doing that.
>> That's not the same.)

> gcc supports binary constants in the form 0b0010, and has done for quite
> some time - without any sort of "obscure switch". It /is/ an extension,
> in that it is not valid standard C syntax, but it is very much a "simple
> enhancement". It is also planned for the C++17 standard, and hopefully
> will one day make it into the C standards.

OK, that's good. Maybe next they will allow separators! (This is more
important in binary, because the numbers are longer, and also the digits
often represent specific patterns or groups.)

But as I said there are very many language enhancements that are easy to
implement even for a static C-class language. At the moment also, I'm
looking at generating C source code from my compiler, so I will get the best
of both worlds: nice feature-rich non-C syntax, and the speed of gcc -O3!

(Example input:
print 11_0001B

output:
printf("%d",49);

which doesn't depend on any C extensions.)

--
Bartc

[David Brown]

It depends of course on the sort of register+offset modes the cpu in
question supports - but it's not uncommon to support a 16-bit signed
offset, and there are often a lot of peripherals within such an area.

In fact, gcc has an optimisation option "-fsection-anchors" to treat
file-level data as though it were a struct, precisely so that it can use
register+offset addressing rather than absolute addressing. And many
targets (like the PPC) have a "small data section" to let all small
global data be accessed in this way.

So using register+offset instead of absolute addressing can make a
significant difference.

>
> Most of my customers have low enough volume sales that my goal in
> choosing a processor is to get one that's big enough and fast enough so
> that I don't have to be sparing with clock ticks or memory when I'm
> writing code: shoe-horning always seems to consume lots of expensive
> engineering hours; in this age of big fast inexpensive processors, I
> don't see the payback unless you're building 100k units at a time.
>

Yes, but optimising code is fun!

Seriously though, I agree with you here. However, I think it is a good
thing to understand these issues - it is better to know that you are
writing code that is easy to write and maintain but sub-optimal at
run-time, rather than guessing about the final result.

[David Brown]

I have been using inverted commas around "real" precisely because I
don't have a good definition - it is a fuzzy concept. Note that I think
having fun or learning is important - these are both perfectly good
reasons for writing or using a compiler. But I think "real work" is
perhaps best thought of in terms of the reliability of the program, the
responsibility of the developer, the quality of the coding (its
correctness, its clarity, the clarity of its correctness, its
maintainability, the value for time, money and effort in development, etc.).

Suppose you are responsible for the software for brake handling in a car
(which would definitely count as "real"), and there is a crash due a
software fault. When you are asked in court about the tools you used,
and you say "a homemade compiler", you will be facing manslaughter
charges for wilful incompetence.

In more realistic scenarios, the limitations of your C subset make the
language unsuitable for complex coding. No typedefs, no support for
different integer sizes, no floating point support, no complex data
structures, no "const", no macros, no conditional compilation, limited
function pointers, limited initialisations, and no post-C89 features
(inline functions, mixed code and declarations, booleans, // comments,
compound literals, designated initialisers, anonymous structs and
unions, static assertions, alignment specification). There are other
differences between your subc and C89, and between the different C
standards - these are just the ones that /I/ would miss in my embedded
programming.

>
> In the field where I work, algorithms are more important than
> using the latest compilers. We all know that quicksort with -O0
> beats bubblesort with any fancy optimization flags you could
> through at it, hands down, given the input set is big enough.

That's true. And in that sort of field, it is common to use higher
level programming languages so that it is easy to express your
algorithms, rather than worrying about the implementation details.
People write such code in functional programming languages like Haskell,
or mixed-paradigm interpreted languages like Python. And if they care
about speed, they use C++ and the STL containers rather than wasting
time and effort reinventing wheels. For more mathematical work, Fortran
is still popular - and there are always people that use more esoteric
languages such as APL. And of course there are times when the best
choice is specialist languages like OpenCL or occam, or when you want a
language that is part of a bigger tool such as Matlab.

In other words, they pick a language and tools that let them work best
at what they want to concentrate on. The best tool for the job depends
on many things - easy of expressing the task in hand, ease of
development, run-time speed, popularity, experience, etc.

What they do not do is pick a relatively difficult and very low-level
language like C, then pick an even lower level subset of it.

>
> In fact I am using an interpreter most of the time and a slow one
> at that plus a language that predates C89 by almost 20 years. I do
> research and I get paid for it. So is this "real" work or not?
>

Out of curiosity, which languages are these? 20 years before C89 could
be Pascal - Fortran was earlier, and Algol 68 is not much used these
days. For the interpreted language my guess would be Python, but there
are many to choose from.

> I think a narrow definition of "real work" and "serious programming"
> (in whatever language) marginalizes people who think outside of the
> box and are not afraid to do their own thing. I think this is a pity,
> because it propagates uniformity, and uniformity kills creativity
> (and joy, and many other things that make us human).
>

I did not want to get into a definition of "real work" - because there
is no fixed definition. But I want to make the point that while
/writing/ and /studying/ a small compiler for a subset of C is an
interesting project and has its good points, it is very rare that
/using/ it will be the best tool for the job.

[David Brown]

On 21/03/14 13:09, BartC wrote:
> "David Brown" <david...@hesbynett.no> wrote in message
> news:lgh1ua$mlg$1...@dont-email.me...
>> On 20/03/14 20:45, BartC wrote:
>
>> (gcc on -O3 usually produces slower code than -O2, as the larger code
>> makes less use of caches.
>
> (On x86, -O2 isn't faster, not for my main project anyway. But I'll try
> it on the ARM board next time I dig it out.)

Your mileage will vary here. Often -O2 or -Os is the fastest, and if
you find that -O3 is faster than -O2 then you could probably get even
faster by using -O2 with some specific optimisation flags, and also by
being careful about the code design (things like making sure that
everything that can be "static" /is/ "static", and using attributes like
"always_inline" and "flatten"). Of course, whether this is worth the
effort is up to you.

>
>>>> I don't think the complexity or size of C compilers is the reason for
>>>> this, however - the aim would be to make a better statically compiled
>>>> language than C (for whatever value of "better" suits your purpose).
>>>
>>> For embedded work especially, being able to express integer constants in
>>> binary (together with separators for readability) is one of many
>>> dozens of
>>> simple enhancements that can make life easier. (Maybe gcc has some
>>> non-standard extension, and obscure switch to enable it, for doing that.
>>> That's not the same.)
>
>> gcc supports binary constants in the form 0b0010, and has done for quite
>> some time - without any sort of "obscure switch". It /is/ an extension,
>> in that it is not valid standard C syntax, but it is very much a "simple
>> enhancement". It is also planned for the C++17 standard, and hopefully
>> will one day make it into the C standards.
>
> OK, that's good. Maybe next they will allow separators! (This is more
> important in binary, because the numbers are longer, and also the digits
> often represent specific patterns or groups.)

I agree 100%.

C++17 will (according to plan) allow a single quote mark (123'457'134)
as a separator within numbers, which is nice. I expect to see that
implemented in gcc for C and C++ as an extension long before C++17 is ready.

>
> But as I said there are very many language enhancements that are easy to
> implement even for a static C-class language. At the moment also, I'm
> looking at generating C source code from my compiler, so I will get the
> best of both worlds: nice feature-rich non-C syntax, and the speed of
> gcc -O3!
>
> (Example input:
> print 11_0001B
>
> output:
> printf("%d",49);
>
> which doesn't depend on any C extensions.)
>

I'd recommend changing your binary syntax to match gcc and the future
C++17 standard - it's consistent with existing 0x for hexadecimal, while
a "B" suffix conflicts with the C use of suffixes indicating types. And
while I would prefer a "_" separator, as used in Ada, the C++ folks have
picked '.


When you are making your own language, it makes a lot of sense to
generate C as the output. When you are targeting a specific compiler,
there is usually little harm in using some extensions, and they can make
a significant difference sometimes.


If I were making my own language, I would write the compiler in Python -
it has lots of libraries for parsing, excellent string handling and
regular expression support, and easy and fast structures such as
dictionaries. I would generate C code as the output, using extensions
where they were useful, targeting gcc and llvm.

If the language because successful and popular, the next step would be
to change the output format into the internal formats for gcc and/or
llvm. This would be harder than generating C, and give both the scope
and the necessity of front-end optimisations. But it would still take
advantage of the middle-end optimisations and back-end code generation
from gcc and/or llvm - thus saving an enormous amount of the work.

The final step would be translating the Python front-end into C++ for
conveniently building it along with the gcc and/or llvm binaries.

That process would let me concentrate on the interesting part - writing
and using the new language - and skip the more tedious and
time-consuming parts, and at the same time give a highly optimised end
result.

[Nils M Holm]

David Brown <david...@hesbynett.no> wrote:
> Suppose you are responsible for the software for brake handling in a car
> (which would definitely count as "real"), and there is a crash due a
> software fault. When you are asked in court about the tools you used,
> and you say "a homemade compiler", you will be facing manslaughter
> charges for wilful incompetence.

Note, again, that I am not here to defend my compiler or my choice
of languages. I just wanted to point out that choosing the latest
greatest is not always a rational choice and that it is by no means
self-evident that the most modern approach is always the best.

An aquaintance of mine owns a company that manufactures medical devices,
heart monitors, defibrillation devices, stuff like that. They are in fact
using a _homebrew_, _non-optimizing_ compiler for a _limited subset_ of a
simple, _low-level_ language for developing the firmware of such devices,
exactly *because* they need to be sure that the software does not endanger
the lifes of people. They can prove formally that the compiler generates
correct output which, I guess, would even be a good defence in court, but
it never came to that. Try this with a modern C compiler.

> In more realistic scenarios, the limitations of your C subset make the
> language unsuitable for complex coding. No typedefs, no support for
> different integer sizes, no floating point support, no complex data
> structures, no "const", no macros, no conditional compilation, limited
> function pointers, limited initialisations, and no post-C89 features
> (inline functions, mixed code and declarations, booleans, // comments,
> compound literals, designated initialisers, anonymous structs and
> unions, static assertions, alignment specification). There are other
> differences between your subc and C89, and between the different C
> standards - these are just the ones that /I/ would miss in my embedded
> programming.

I am not here to defend or endorse my compiler. Not at all. I am
talking about a much greater context.

> > In the field where I work, algorithms are more important than
> > using the latest compilers. We all know that quicksort with -O0
> > beats bubblesort with any fancy optimization flags you could
> > through at it, hands down, given the input set is big enough.
>

> What they do not do is pick a relatively difficult and very low-level
> language like C, then pick an even lower level subset of it.

As outlined above, in fact, sometimes "they" do (even if it was not
C in that case).

> Out of curiosity, which languages are these? 20 years before C89 could
> be Pascal - Fortran was earlier, and Algol 68 is not much used these
> days. For the interpreted language my guess would be Python, but there
> are many to choose from.

I am writing most of my code in Scheme (1975, IIRC), using an interpreter
because it offers faster turn-around.

> I did not want to get into a definition of "real work" - because there
> is no fixed definition. But I want to make the point that while
> /writing/ and /studying/ a small compiler for a subset of C is an
> interesting project and has its good points, it is very rare that
> /using/ it will be the best tool for the job.

I agree that there very probably is no need for a compiler like SubC
in production. And for my part, this is not about my compiler. I just
wanted to point out that there exist cases where you actually do want
to use a very simple tool for the task, *especially* in some very
critical and "serious" areas.

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lghgbb$ja$1...@dont-email.me...

> On 21/03/14 12:45, Nils M Holm wrote:

>> But then I do not agree with your implied definitions of "real work"
>> and "serious programming". What is "real" work? Working on complex
>> problems? Getting paid? Being creative? Having fun?

> Suppose you are responsible for the software for brake handling in a car
> (which would definitely count as "real"), and there is a crash due a
> software fault. When you are asked in court about the tools you used,
> and you say "a homemade compiler", you will be facing manslaughter
> charges for wilful incompetence.

Perhaps even the use of C would be a bad choice for such an application.
Because it's easier to have coding bugs in such a language. Also, C is now
quite complex, and compilers for it you say are also large and complex. That
would be one advantage of a streamlined language and a simple compiler
(whether it's created at 'home' or in a work environment is not relevant).

> In more realistic scenarios, the limitations of your C subset make the
> language unsuitable for complex coding. No typedefs, no support for
> different integer sizes, no floating point support, no complex data
> structures, no "const", no macros, no conditional compilation, limited
> function pointers, limited initialisations, and no post-C89 features
> (inline functions, mixed code and declarations, booleans, // comments,
> compound literals, designated initialisers, anonymous structs and
> unions, static assertions, alignment specification). There are other
> differences between your subc and C89, and between the different C
> standards - these are just the ones that /I/ would miss in my embedded
> programming.

I'd miss 'goto'. This is more important from the point of view of using it
as a target language (some of the other stuff can be emulated).

--
Bartc

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lghh7k$8ki$1...@dont-email.me...

> On 21/03/14 13:09, BartC wrote:
>> "David Brown" <david...@hesbynett.no> wrote in message

>>> gcc supports binary constants in the form 0b0010

> C++17 will (according to plan) allow a single quote mark (123'457'134)
> as a separator within numbers, which is nice. I expect to see that
> implemented in gcc for C and C++ as an extension long before C++17 is
> ready.

>> (Example input:
>> print 11_0001B
>>
>> output:
>> printf("%d",49);

> I'd recommend changing your binary syntax to match gcc and the future
> C++17 standard - it's consistent with existing 0x for hexadecimal, while
> a "B" suffix conflicts with the C use of suffixes indicating types. And
> while I would prefer a "_" separator, as used in Ada, the C++ folks have
> picked '.

For binary I allow 1101B or 2x1101 formats (the latter form allowing other
bases too). But I've now added 0b1101/0B1101, if that's what C will use. (It
took a few minutes; another advantage of a DIY language!)

While for separators, I allow _ ' and `. I like to give a choice...

> When you are making your own language, it makes a lot of sense to
> generate C as the output. When you are targeting a specific compiler,
> there is usually little harm in using some extensions, and they can make
> a significant difference sometimes.

(Actually, I've experienced a *lot* of problems in targeting C. Generating
assembly or machine code is more straightforward, even if the resulting code
is not the best. You don't have an idiosyncratic syntax, type system, and a
bunch of highly pendantic and picky C compilers to get in the way. I've had
to simplify my syntax to make it possible to express it in high-level-ish C,
and a few things can't be expressed at all. However, the portability, and
speed of the result makes it worth the effort!)

--
Bartc

[Paul Rubin]

Nils M Holm <news...@t3x.org> writes:
> They are in fact using a _homebrew_, _non-optimizing_ compiler for a
> _limited subset_ of a simple, _low-level_ language for developing the
> firmware of such devices, exactly *because* they need to be sure that
> the software does not endanger the lifes of people. They can prove
> formally that the compiler generates correct output

Have they published anything about this? What kind of formalization
do they use?

> Try this with a modern C compiler.

* http://compcert.inria.fr

> I agree that there very probably is no need for a compiler like SubC
> in production. And for my part, this is not about my compiler. I just
> wanted to point out that there exist cases where you actually do want
> to use a very simple tool for the task, *especially* in some very
> critical and "serious" areas.

That's a tempting line of reasoning but (at least unless heavy testing
and extensive process is involved) it seems to fall down under
real-world examination. See:

* http://blog.regehr.org/archives/1036

(the simpler gcc versions from the "good old days" were much buggier
than the current versions people complain about more).

* http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

From p.10 of the pdf:

*Compiler simplicity*. For non-bottleneck applications, compiler
optimization adds little end-user value. It would seem possible to
take a simple compiler such as TCC [2], which does not optimize
across statement boundaries, and validate it through code
inspections, heavy use, and other techniques. At present, however,
TCC is much buggier than more heavily-used compilers such as GCC and
LLVM.

[Paul Rubin]

David Brown <david...@hesbynett.no> writes:
> If I were making my own language... I would generate C code as the

> output, using extensions where they were useful, targeting gcc and
> llvm.

See http://www.cminusminus.org/ for the disadvantages of using C as an
intermediate language.

[David Brown]

On 21/03/14 16:03, BartC wrote:
> "David Brown" <david...@hesbynett.no> wrote in message
> news:lghgbb$ja$1...@dont-email.me...
>> On 21/03/14 12:45, Nils M Holm wrote:
>
>>> But then I do not agree with your implied definitions of "real work"
>>> and "serious programming". What is "real" work? Working on complex
>>> problems? Getting paid? Being creative? Having fun?
>
>> Suppose you are responsible for the software for brake handling in a car
>> (which would definitely count as "real"), and there is a crash due a
>> software fault. When you are asked in court about the tools you used,
>> and you say "a homemade compiler", you will be facing manslaughter
>> charges for wilful incompetence.
>
> Perhaps even the use of C would be a bad choice for such an application.

That is true - and there are many who prefer Ada for such high
reliability applications.

> Because it's easier to have coding bugs in such a language. Also, C is now
> quite complex, and compilers for it you say are also large and complex.
> That
> would be one advantage of a streamlined language and a simple compiler
> (whether it's created at 'home' or in a work environment is not relevant).

It is /highly/ relevant whether it is created at work or at home - but
it is /irrelevant/ whether the compiler is simple or complex. What is
important is that you, as the compiler user, have taken appropriate
steps to ensure that the tools you use are solid and reliable, and that
they appropriately translate the source code into object code. Without
that assurance, it doesn't matter how good or bad your source code is,
or how complex or simple it is - if you cannot be sure that the compiler
translates it correctly, then all your effort in writing good source
code is wasted.

So how do you ensure that the compiler works correctly - at least to a
high degree? There are industry-standard compiler test and
certification suites, such as Plum Hall. There is quality control and
good development practices for the toolchain, such as accurate source
code control and bug tracking. There is internal testing, including
automatic test suites, regression testing, and procedures for building
and testing the toolchain. And there is a user base - if large numbers
of people use the toolchain without noticing problems, then it is likely
that the toolchain is working correctly.

This is totally impossible to achieve with a homemade compiler - you
have no choice but to use one of the big, well-known toolchains.


Then there is the choice of language. You are correct that C is often a
less than ideal language - but you can do well by using a slightly
limited subset (such as avoiding malloc and friends) and strict coding
standards, as well as using static error checking tools and perhaps also
dynamic checking tools. It is also vital to use an industry-standard
language so that any weaknesses or problem areas are well understood.

Again, this is totally impossible to achieve with a homemade language.
Your choices are C, C++, and Ada.


Using a custom language and compiler that generated C /might/ be
acceptable, if it were possible to do a good analysis of the generated C
code (i.e., it should be understandable to C experts) and it were
compiled with a suitable C compiler.

>
>> In more realistic scenarios, the limitations of your C subset make the
>> language unsuitable for complex coding. No typedefs, no support for
>> different integer sizes, no floating point support, no complex data
>> structures, no "const", no macros, no conditional compilation, limited
>> function pointers, limited initialisations, and no post-C89 features
>> (inline functions, mixed code and declarations, booleans, // comments,
>> compound literals, designated initialisers, anonymous structs and
>> unions, static assertions, alignment specification). There are other
>> differences between your subc and C89, and between the different C
>> standards - these are just the ones that /I/ would miss in my embedded
>> programming.
>
> I'd miss 'goto'. This is more important from the point of view of using it
> as a target language (some of the other stuff can be emulated).
>

Agreed. I very rarely use "goto" in my own code, but it would be likely
to be more useful in generated code.

[David Brown]

On 21/03/14 16:04, Nils M Holm wrote:
> David Brown <david...@hesbynett.no> wrote:
>> Suppose you are responsible for the software for brake handling in a car
>> (which would definitely count as "real"), and there is a crash due a
>> software fault. When you are asked in court about the tools you used,
>> and you say "a homemade compiler", you will be facing manslaughter
>> charges for wilful incompetence.
>
> Note, again, that I am not here to defend my compiler or my choice
> of languages. I just wanted to point out that choosing the latest
> greatest is not always a rational choice and that it is by no means
> self-evident that the most modern approach is always the best.

Fair enough.

>
> An aquaintance of mine owns a company that manufactures medical devices,
> heart monitors, defibrillation devices, stuff like that. They are in fact
> using a _homebrew_, _non-optimizing_ compiler for a _limited subset_ of a
> simple, _low-level_ language for developing the firmware of such devices,
> exactly *because* they need to be sure that the software does not endanger
> the lifes of people. They can prove formally that the compiler generates
> correct output which, I guess, would even be a good defence in court, but
> it never came to that. Try this with a modern C compiler.

If you can provide a formal proof of correctness for the compiler tool,
then you have a strong case for calling it "safe". But it is certainly
no small feat to prove formally that a compiler is correct, even for a
simple language.

>
>> In more realistic scenarios, the limitations of your C subset make the
>> language unsuitable for complex coding. No typedefs, no support for
>> different integer sizes, no floating point support, no complex data
>> structures, no "const", no macros, no conditional compilation, limited
>> function pointers, limited initialisations, and no post-C89 features
>> (inline functions, mixed code and declarations, booleans, // comments,
>> compound literals, designated initialisers, anonymous structs and
>> unions, static assertions, alignment specification). There are other
>> differences between your subc and C89, and between the different C
>> standards - these are just the ones that /I/ would miss in my embedded
>> programming.
>
> I am not here to defend or endorse my compiler. Not at all. I am
> talking about a much greater context.

OK.

(And I am not trying to beat on your compiler or the work you have done
on it - I am just discussing why a tool like that would have limited usage.)

>
>>> In the field where I work, algorithms are more important than
>>> using the latest compilers. We all know that quicksort with -O0
>>> beats bubblesort with any fancy optimization flags you could
>>> through at it, hands down, given the input set is big enough.
>>
>> What they do not do is pick a relatively difficult and very low-level
>> language like C, then pick an even lower level subset of it.
>
> As outlined above, in fact, sometimes "they" do (even if it was not
> C in that case).

The case above was for a very different situation than you had mentioned
(as far as I understood it). Different jobs require different tools.

>
>> Out of curiosity, which languages are these? 20 years before C89 could
>> be Pascal - Fortran was earlier, and Algol 68 is not much used these
>> days. For the interpreted language my guess would be Python, but there
>> are many to choose from.
>
> I am writing most of my code in Scheme (1975, IIRC), using an interpreter
> because it offers faster turn-around.

I have never been a fan of Lisp - too many brackets to keep track of! I
guess it's a matter of habit.

>
>> I did not want to get into a definition of "real work" - because there
>> is no fixed definition. But I want to make the point that while
>> /writing/ and /studying/ a small compiler for a subset of C is an
>> interesting project and has its good points, it is very rare that
>> /using/ it will be the best tool for the job.
>
> I agree that there very probably is no need for a compiler like SubC
> in production. And for my part, this is not about my compiler. I just
> wanted to point out that there exist cases where you actually do want
> to use a very simple tool for the task, *especially* in some very
> critical and "serious" areas.
>

I can agree with that - and it is certainly common to use subsets of
programming languages for safety-critical systems (such as Spark for
Ada, or Misra C). But those subsets are chosen on the basis of
minimising the risks of errors in programming - not on the basis of
reducing the complexity of the compiler. You compile your Spark code
with Gnat or GHS Ada, despite the complexity of these tools, because you
rely on the experience, maturity and testing of those tools and their
developers. You take a tool that is known to be good for large,
complicated systems and then make it even more reliable by sticking to
small, simple programs.

And my point against homemade tools and languages was because you
normally don't get the kind of quality and testing that you get with
bigger toolchains. But - as you pointed out with your example - there
are exceptions.

[Simon Clubley]

On 2014-03-21, David Brown <david...@hesbynett.no> wrote:
>
> C++17 will (according to plan) allow a single quote mark (123'457'134)
> as a separator within numbers, which is nice. I expect to see that
> implemented in gcc for C and C++ as an extension long before C++17 is ready.
>

Sorry, but that single quote syntax looks utterly naff. :-)

I think Ada style 123_457_134 would be much more readable than 123'457'134.

[snip]

>
> I'd recommend changing your binary syntax to match gcc and the future
> C++17 standard - it's consistent with existing 0x for hexadecimal, while
> a "B" suffix conflicts with the C use of suffixes indicating types. And
> while I would prefer a "_" separator, as used in Ada, the C++ folks have
> picked '.
>

Strongly agree with using Ada's "_"; do you have any idea why they picked
a single quote ?

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lghoj2$6n9$1...@dont-email.me...

> On 21/03/14 16:03, BartC wrote:
>> "David Brown" <david...@hesbynett.no> wrote in message

>>> Suppose you are responsible for the software for brake handling in a car

>> Perhaps even the use of C would be a bad choice for such an application.
>
> That is true - and there are many who prefer Ada for such high
> reliability applications.

>> (whether it's created at 'home' or in a work environment is not
>> relevant).
>
> It is /highly/ relevant whether it is created at work or at home - but
> it is /irrelevant/ whether the compiler is simple or complex. What is
> important is that you, as the compiler user, have taken appropriate
> steps to ensure that the tools you use are solid and reliable, and that
> they appropriately translate the source code into object code. Without
> that assurance, it doesn't matter how good or bad your source code is,
> or how complex or simple it is - if you cannot be sure that the compiler
> translates it correctly, then all your effort in writing good source
> code is wasted.
>
> So how do you ensure that the compiler works correctly - at least to a
> high degree? There are industry-standard compiler test and
> certification suites, such as Plum Hall. There is quality control and
> good development practices for the toolchain, such as accurate source
> code control and bug tracking. There is internal testing, including
> automatic test suites, regression testing, and procedures for building
> and testing the toolchain. And there is a user base - if large numbers
> of people use the toolchain without noticing problems, then it is likely
> that the toolchain is working correctly.
>
> This is totally impossible to achieve with a homemade compiler - you
> have no choice but to use one of the big, well-known toolchains.

It is not essential in every single case to take this stuff so seriously.

There can be a place for small, informal tools that are not going to be used
for mission-critical applications.

When I started creating my own tools (around 1981), there was no viable
alternative. When I just wanted a few lines of code to test out some I/O
signals on a new board, I needed a language that would compile instantly. At
the time, if I wanted to use C (if we could somehow acquire it) it would
probably have taken several minutes of grinding floppy disks to compile the
simplest program. Too long!

But later I found my own tools were more productive anyway (and for much of
the 80s, my naive unoptimised compilers tended to outperform C! At least,
for Z80 and 8086. They could also support newer processors sooner, eg. 80186
(with on-chip peripherals) and 80386, so that gave me an edge too).

Anyway, I see examples of bad design and programming all the time,
especially with consumer devices with embedded code. Just using a mainstream
language doesn't guarantee good results; there's a bit more to it than that.

--
Bartc

[Paul Rubin]

"BartC" <b...@freeuk.com> writes:
> When I started creating my own tools (around 1981), there was no
> viable alternative. When I just wanted a few lines of code to test out
> some I/O signals on a new board, I needed a language that would
> compile instantly.

It was before my time, but FIG-Forth was released in 1978 and it sounds
like exactly what you wanted.

[Walter Banks]

Nils M Holm wrote:

> An aquaintance of mine owns a company that manufactures medical devices,
> heart monitors, defibrillation devices, stuff like that. They are in fact
> using a _homebrew_, _non-optimizing_ compiler for a _limited subset_ of a
> simple, _low-level_ language for developing the firmware of such devices,
> exactly *because* they need to be sure that the software does not endanger
> the lifes of people. They can prove formally that the compiler generates
> correct output which, I guess, would even be a good defence in court, but
> it never came to that. Try this with a modern C compiler.

Your point in its context are well taken. The FDA has rules that
safely allow C (and other languages) to be safely used in these
types of applications. As good as the a provable correct compiler
is, it does not guarantee that the application is also correct.

w..

[Tom Gardner]

On 21/03/14 15:04, Nils M Holm wrote:
> An aquaintance of mine owns a company that manufactures medical devices,
> heart monitors, defibrillation devices, stuff like that. They are in fact
> using a_homebrew_,_non-optimizing_ compiler for a_limited subset_ of a

> simple,_low-level_ language for developing the firmware of such devices,
> exactly*because* they need to be sure that the software does not endanger

> the lifes of people. They can prove formally that the compiler generates
> correct output which, I guess, would even be a good defence in court, but
> it never came to that. Try this with a modern C compiler.

Possibly with an extra advantage: it would cost litigants a lot
of time/money first to learn the environment then to prove them
wrong, whereas with C the problems are well documented!

[Tom Gardner]

(Rats, hit button too soon)

But then wouldn't it be more reasonable to choose a "high integrity"
language/compiler/environment than a bog-standard C one? Spark/Ada
would seem a good starting point.

[upsid...@downunder.com]

On Fri, 21 Mar 2014 20:40:47 +0000 (UTC), Simon Clubley
<clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:

>On 2014-03-21, David Brown <david...@hesbynett.no> wrote:
>>
>> C++17 will (according to plan) allow a single quote mark (123'457'134)
>> as a separator within numbers, which is nice. I expect to see that
>> implemented in gcc for C and C++ as an extension long before C++17 is ready.
>>
>
>Sorry, but that single quote syntax looks utterly naff. :-)
>
>I think Ada style 123_457_134 would be much more readable than 123'457'134.

Or perhaps PL/M style 123$457$134 :-)

[BartC]

"Tom Gardner" <spam...@blueyonder.co.uk> wrote in message
news:d%zXu.59536$Gy5....@fx11.am4...

If my life or safety depended on some essential software, then I'd be
happier if it was written in Ada than C. (So long as someone else does the
actual programming; using Ada seems like wearing a straitjacket.)

But then, there does seem to be a version of Ada that is just a front-end to
gcc, presumably inheriting the latter's complexity and bugs too.

--
Bartc

[Tom Gardner]

That does seem like the worst of all worlds!

The entire environment is important, from requirements gathering,
specification, code control, compiler, static analysis, etc etc.

If I was to investigate this area, I'd start with
http://www.adacore.com/sparkpro/
and then see if there was anything better.

The entire pedigree of the company (Praxis, not Altran) and
personnel (esp Martyn Thomas and Bernard Carre) that created
the toolset was oriented strongly towards high-integrity solutions.

[Niklas Holsti]

On 14-03-23 14:50 , BartC wrote:
>
>
> "Tom Gardner" <spam...@blueyonder.co.uk> wrote in message
> news:d%zXu.59536$Gy5....@fx11.am4...

[snip]

>> But then wouldn't it be more reasonable to choose a "high integrity"
>> language/compiler/environment than a bog-standard C one? Spark/Ada
>> would seem a good starting point.
>
> If my life or safety depended on some essential software, then I'd be
> happier if it was written in Ada than C. (So long as someone else does the
> actual programming; using Ada seems like wearing a straitjacket.)

Programming in C feels to me (a habitual Ada user) like crossing a deep,
rocky ravine on a narrow, swaying plank. I guess it's subjective, and
some people may enjoy it :-).

> But then, there does seem to be a version of Ada that is just a
> front-end to gcc, presumably inheriting the latter's complexity
> and bugs too.

Just to clarify that: the most widely used Ada compiler, GNAT from
AdaCore, is part of the GNU compiler collection. The front-end generates
an intermediate representation (like the other gcc compilers) which goes
into the gcc back-end for code generation. GNAT does not generate C code.

Then there is another Ada compiler (also now from AdaCore) which
generates C source code which can be compiled with various C compilers.
This is useful for targets which GNAT does not support.

But there are two different issues here for critical SW: firstly, is the
programming language suitable; secondly, are the tools/compilers
reliable (i.e. sufficiently correct).

On the first issue, using Ada, and perhaps even SPARK/Ada, inherently
helps avoid many kinds of errors which are common in C code. Moreover,
static analysis works better, because the language gives more
information to the analyser, especially with the "contract" features
added in Ada 2012 and used to good effect by SPARK 2014. This advantage
does not depend on the compiler technology.

On the second issue, if the compiler uses the gcc back-end or generates
intermediate C code and then uses some C compiler, certainly this reuses
the bugs in those tools, not much one can do about that.

I don't know of any empirical study of the reliability of the gcc
back-end compared to the back-ends of other compilers, whether for Ada
or C. Would be interesting.

--
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
. @ .

[Paul Rubin]

Niklas Holsti <niklas...@tidorum.invalid> writes:
> I don't know of any empirical study of the reliability of the gcc
> back-end compared to the back-ends of other compilers, whether for Ada
> or C. Would be interesting.

* http://blog.regehr.org/archives/1036
* http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

[Nils M Holm]

Paul Rubin <no.e...@nospam.invalid> wrote:
> Nils M Holm <news...@t3x.org> writes:
> > They are in fact using a _homebrew_, _non-optimizing_ compiler for a
> > _limited subset_ of a simple, _low-level_ language for developing the
> > firmware of such devices, exactly *because* they need to be sure that
> > the software does not endanger the lifes of people. They can prove
> > formally that the compiler generates correct output
>
> Have they published anything about this? What kind of formalization
> do they use?

Operational semantics defined in terms of a virtual machine. The
compiler emitted code fragments that implemented instructions of
that VM. Easy to formalize, but the generated output was rather
inefficient. I remember that I proposed some "really safe"
optmizations back then, but they were not impressed. Looking back,
I think they made the right decision.

But that was in the late 80's/early 90's, so I do not remember
much of the details. As far as I know, they never published
anything about their approach.

> > Try this with a modern C compiler.
>
> * http://compcert.inria.fr

Interesting! Even if they only handle a subset of C, this is still
quite impressing. Machine-assisted proof has come a long way since
the 90's.

> > I agree that there very probably is no need for a compiler like SubC
> > in production. And for my part, this is not about my compiler. I just
> > wanted to point out that there exist cases where you actually do want
> > to use a very simple tool for the task, *especially* in some very
> > critical and "serious" areas.
>
> That's a tempting line of reasoning but (at least unless heavy testing
> and extensive process is involved) it seems to fall down under
> real-world examination. See:
>
> * http://blog.regehr.org/archives/1036
>
> (the simpler gcc versions from the "good old days" were much buggier
> than the current versions people complain about more).
>
> * http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

Appreciated, thanks!

> From p.10 of the pdf:
>
> *Compiler simplicity*. For non-bottleneck applications, compiler
> optimization adds little end-user value. It would seem possible to
> take a simple compiler such as TCC [2], which does not optimize
> across statement boundaries, and validate it through code
> inspections, heavy use, and other techniques. At present, however,
> TCC is much buggier than more heavily-used compilers such as GCC and
> LLVM.

However, I would argue, that TCC was not really simple in its
days. Simpler than most other C compilers, but still an optimizing
compiler.

I agree that compiler quality goes up, and formal verification,
regression testing, a large user base, etc has all contributed to
the very high standards we have these days. However, for really
critical tasks, I would still prefer something that is *easy* to
understand, formalize and verify. Especially, if I do not need the
power of an optimizing compiler.

Plus, a simple homebrew compiler requires less man power for
maintenance because in the ideal case one persion can understand
both internals of the compiler and its formal model.

[Nils M Holm]

David Brown <david...@hesbynett.no> wrote:
> I have never been a fan of Lisp - too many brackets to keep track of! I
> guess it's a matter of habit.

Brackets? What brackets?

Just kidding, after a while you just don't see them any longer.

[Nils M Holm]

Good point! While a compiler that is known to produce correct code
is a necessity, it does not stop the developers from writing faulty
code. So in an ideal world, we would prove the correctness of the
application, too. However, not every application is equally suitable
for formalization and verification. For a compiler, we have formal
specs, like a grammar and formal or semi-formal semantics. Many
many other problems are underspecified, though (which is, IMHO, a
bad idea) and hence not suitable for the above approach.

As you raise that point, what would be your approach?

[Paul Rubin]

Nils M Holm <news...@t3x.org> writes:

>> it does not guarantee that the application is also correct.

> As you raise that point, what would be your approach?

* http://blog.regehr.org/archives/1106
* http://jlouisramblings.blogspot.com/2012/08/getting-25-megalines-of-code-to-behave.html
* http://hackage.haskell.org/package/improve
* http://hackage.haskell.org/package/atom
* http://betterembsw.blogspot.ca/
* don't use C ;-)

[David Brown]

I agree that there is plenty of use for small "informal" tools - the
case I picked was intentionally an extreme example.

But is there a place for small, informal tools when there are large,
well-tested and powerful mainstream tools available? Sometimes cost is
an issue, but gcc and llvm have reduced that enormously. Yes, I think
there are still situations where small tools are a better choice - but
only for niche areas with particularly good reasons for choosing them.

[Tom Gardner]

On 23/03/14 17:04, David Brown wrote:
> I agree that there is plenty of use for small "informal" tools - the case I picked was intentionally an extreme example.
>
> But is there a place for small, informal tools when there are large, well-tested and powerful mainstream tools available? Sometimes cost is an issue, but gcc and llvm have reduced that enormously.
> Yes, I think there are still situations where small tools are a better choice - but only for niche areas with particularly good reasons for choosing them.

Given that in general there is an "impedance mismatch" between
the problem-being-solved and the language, under what conditions
is it better to create
- a small domain specific language tailored to the problem, or
- a library in a standard language

Yes, I have my own opinions, but I'd like to hear other people's
opinions.

[Paul Rubin]

Tom Gardner <spam...@blueyonder.co.uk> writes:
> Given that in general there is an "impedance mismatch" between
> the problem-being-solved and the language, under what conditions
> is it better to create
> - a small domain specific language tailored to the problem, or
> - a library in a standard language

Some languages are particularly amenable to the creation of embedded
DSL's (EDSL's) inside a general purpose host language. That lets you
leverage the capabilities of the host language. Lisp, Forth, and
Haskell are examples of host languages sometimes used this way. I just
posted some links to Atom and ImProve, both of which are Haskell EDSL's
used for realtime control stuff.

[Wouter van Ooijen]

Tom Gardner schreef op 23-Mar-14 6:13 PM:

> Given that in general there is an "impedance mismatch" between
> the problem-being-solved and the language, under what conditions
> is it better to create
> - a small domain specific language tailored to the problem, or
> - a library in a standard language
>
> Yes, I have my own opinions, but I'd like to hear other people's
> opinions.
>

You should not regard that as a one-or-the-other question, there is a
continuum from solving a problem using standard ways (that require
'more' of them to solve the problem) to writing a dedicated language for
it that solves this one problem trivially. The 'soft spot' in this
continuum varies. When a certain type of problem needs to be solved only
once the 'standard ways' is to be preferred, because it requires less
work, and is easier to read. The more often a type of problem needs to
be solved, the more a 'dedicated' solution is favoured: the amount of
up-front work (both on the implementor and on the reader) can be
averaged out over all applications.

I tend to prefer using an existing programming language that has a
sufficiently high abstraction level (named parameters and overloading
are a plus).

Wouter van Ooijen

[Tom Gardner]

On 23/03/14 17:50, Wouter van Ooijen wrote:
> Tom Gardner schreef op 23-Mar-14 6:13 PM:
>> Given that in general there is an "impedance mismatch" between
>> the problem-being-solved and the language, under what conditions
>> is it better to create
>> - a small domain specific language tailored to the problem, or
>> - a library in a standard language
>>
>> Yes, I have my own opinions, but I'd like to hear other people's
>> opinions.
>>
>
> You should not regard that as a one-or-the-other question, there is a continuum from solving a problem using standard ways (that require 'more' of them to solve the problem) to writing a dedicated
> language for it that solves this one problem trivially. The 'soft spot' in this continuum varies. When a certain type of problem needs to be solved only once the 'standard ways' is to be preferred,
> because it requires less work, and is easier to read. The more often a type of problem needs to be solved, the more a 'dedicated' solution is favoured: the amount of up-front work (both on the
> implementor and on the reader) can be averaged out over all applications.

Well, yes. No argument there.

So what would you regard as useful indicators or preconditions
that would bias the answer one way or the other?

> I tend to prefer using an existing programming language that has a sufficiently high abstraction level (named parameters and overloading are a plus).

OK, so why? :)

[Tom Gardner]

OK, but under what conditions would/wouldn't you use
them to solve an embedded problem via a DSLanguage
or DSLibrary?

I must admit that, while LISP is clearly a sufficient
base in which to create a DSLanguage, I'm not sure I
would rush to use it in an embedded environment.

Haskell? I don't know enough about that to have a valid
opinion, but "lazy evaluation" and "strictly functional"
cause my eyebrows to go upwards in the embedded domain.
(The former because of time/space considerations, the
latter because an embedded system with side effect is
more than a little pointless!)

Forth? I'll discuss my opinions at a later date, if
appropriate.

[Walter Banks]

Paul Rubin has already posted some links related to your question.

This is an area that at has to a greater or lessor extent been part
of my life for 30+ years.

Here are a few places to start.

A unambiguously defined language. In some was SubC has that.
Ada, Pascal and the like are better than C and far from perfect.
Implementation defined is convention but has many side effects
most of which are unintended.

Focus on goals. The reason for my FDA reference was because
they have both goals based on product reliability and safety but
also recognize that the tools and the people who use them are
far from perfect.

A lot more real research on software engineering and implementation
practices.. A simple example of this is tools that walk the generated
code applying normal reliability math to the code with any assumption
about the reliability of an instruction based on pick one ( 1 , size in
bytes, execution cycles) . This will not give you an MTTF that isn't
the goal. The goal is to independently evaluate comparative reliabilities.
Compare two implementations and it will reasonably accurately
predict which will run most reliably. We have created and used
such tools and it has had a remarkable positive effect on changes
in the way we think about and plan software.

The topic lists for software engineering as opposed to
software science is now quite large. Some of the things that are
important that I see our better customers using are.
- Application design documents.
- Ram, rom, and execution cycle budgets
- Formal function interface documentation. (Independent
component for reliably analysis)
- Coding practice document that support the application goals
for a project

That's where I would start

Walter Banks
Byte Craft Limited

[Walter Banks]

Some very bad tools have been used to create some very reliable
application code. It has a lot to do with design, coding and testing
practices. Focus on final product.

One thing I see is companies that standardize on a particular version
of a tool and although the tool has evolved over many years they use
the tools they are familiar with. The trade-off is dealing with a single
set of changes as a product evolves related to their product and not
side effects of new tools and product changes.

w..

[Paul Rubin]

Tom Gardner <spam...@blueyonder.co.uk> writes:
>> posted some links to Atom and ImProve, both of which are Haskell
>> EDSL's used for realtime control stuff.
>
> OK, but under what conditions would/wouldn't you use them to solve an
> embedded problem via a DSLanguage or DSLibrary?

Embedded is a pretty wide space and in more critical applications, there
may be prescribed processes that I'd have to follow regardless of my
preferences. In less critical applications, memory and realtime
constraints can still be determining factors. Another issue is the
skill sets of the programmers likely to work on the code. Using DSL's
can require a level of PL geekery that a typical embedded developer
would probably not have. I could imagine an approach of prototyping
using a DSL, then conferring with the customer about whether they were
comfortable staying with that approach or wanted a more traditional
approach. If I were the main consumer of the code, then as a PL geek
I'd probably use DSL's rather freely.

> I must admit that, while LISP is clearly a sufficient base in which to
> create a DSLanguage, I'm not sure I would rush to use it in an
> embedded environment.

I don't see why not, if the target CPU is powerful enough to run it, and
there aren't realtime constraints that would preclude garbage
collection. I've done embedded stuff in Python which amounts to the
same thing.

> Haskell? I don't know enough about that to have a valid opinion, but
> "lazy evaluation" and "strictly functional" cause my eyebrows to go
> upwards in the embedded domain. (The former because of time/space
> considerations, the latter because an embedded system with side effect
> is more than a little pointless!)

The two Haskell EDSL's that I mentioned (Atom and ImProve) both compile
to C or Ada. I've played with Atom. You write your program in a style
that looks like multitasking (plus you get to use Haskell syntax and
type safety), and then the Atom library converts it to a C-coded state
machine with deterministic timing, that you run through a C compiler to
produce target code. ImProve operates about the same way, though its
target problem set is a bit different.

Strictly functional code can have side effects: they just have to be
known to the type system. So if some function in your program wants to
print something (or call some other function that prints stuff), it has
to have "IO" in its type signature. The function output is e.g. a
"print command" that gets handed off to the runtime library that does
the printing. So the type system lets you segregate the side-effecting
code from the "pure" code, which helps write in a style where most of
the code is pure and easier to reason about.

[Paul Rubin]

Tom Gardner <spam...@blueyonder.co.uk> writes:
> Haskell? I don't know enough about that to have a valid opinion, but
> "lazy evaluation" and "strictly functional" cause my eyebrows to go
> upwards in the embedded domain.

Here's a talk about critical systems experience in Haskell:

(slides)
http://www.scribd.com/doc/19502765/Engineering-Large-Projects-in-Haskell-A-Decade-of-FP-at-Galois

(video)
http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/A-Means-to-Many-Ends-10-Years-of-Haskell-at-Galois

I haven't watched the video and I'm not sure if the slides and the video
are from the exact same presentation, but they're both the same guy
discussing the same subject matter.

[Wouter van Ooijen]

>> You should not regard that as a one-or-the-other question, there is a
>> continuum from solving a problem using standard ways (that require
>> 'more' of them to solve the problem) to writing a dedicated
>> language for it that solves this one problem trivially. The 'soft
>> spot' in this continuum varies. When a certain type of problem needs
>> to be solved only once the 'standard ways' is to be preferred,
>> because it requires less work, and is easier to read. The more often a
>> type of problem needs to be solved, the more a 'dedicated' solution is
>> favoured: the amount of up-front work (both on the
>> implementor and on the reader) can be averaged out over all applications.
>
> Well, yes. No argument there.
>
> So what would you regard as useful indicators or preconditions
> that would bias the answer one way or the other?

Try to minimize the total effort involved. Is the investment in tool
building and learning (not only your learning, but of all people
involved) compensated by the effort saved in using the tool and
writing/reading the (much shorter/readable) solution(s)?

favours existing paradigms in an existing language:
- one-time, small problem
- many people need to understand it, especially if they need to
understand only a small part, and already know the language/paradigms
- semantic difference between programming language/paradigms and problem
domain is small

favours full special-purpose language:
- large recurring problems
- only few people involved, and they all need to understand large
amounts of it
- semantic difference is hughe

>> I tend to prefer using an existing programming language that has a
>> sufficiently high abstraction level (named parameters and overloading
>> are a plus).

IME this is a 'sweet spot' inbetween that is often a good choice. Named
parameters make complex 'specifications' (more) readable. My
professional experience was with Ada, but I also used Python for this
purpose for mopre private projects.

Wouter

[David Brown]

Just remember that people can write bad software in any language. A
good choice of language and tools makes it easier to write good software
and harder to write bad software, but it is no guarantee. And the
language and tools are very much secondary to factors such as
development methods, testing procedures, reviews, documentation,
specifications, and everything else around the actual coding.

But of course there is a strong correlation - a company or organisation
that is willing to invest in Ada development rather than C is likely to
pay attention to the other parts of the development ecosystem.

>
> But then, there does seem to be a version of Ada that is just a
> front-end to gcc, presumably inheriting the latter's complexity and
> bugs too.
>

Compilers are not bug-free - but bugs in the compiler users' code
outweigh the bugs in compilers by many orders of magnitude. Since the
end result is only bug-free if the entirely of the user code is bug
free, and that it does not use any part of the compiler that contains a
bug, it seems obvious that reducing the risk of user error (such as by
using a "safer" language like Ada, or by restricting C to a carefully
chosen subset) is far more important than worrying about the particular
choice of toolchain.

Also, as noted before in this thread, simplicity or complexity of a
toolchain is no indication of the quality or the risk of hitting a bug.
What /does/ increase your chance of meeting a bug is using odd or
complicated structures in your own code - it is not hard to write your C
code in a way that significantly reduces the risk of hitting a compiler
bug (while simultaneously making the code clearer and easier to write
correctly).

[David Brown]

On 21/03/14 21:40, Simon Clubley wrote:
> On 2014-03-21, David Brown <david...@hesbynett.no> wrote:
>>
>> C++17 will (according to plan) allow a single quote mark (123'457'134)
>> as a separator within numbers, which is nice. I expect to see that
>> implemented in gcc for C and C++ as an extension long before C++17 is ready.
>>
>
> Sorry, but that single quote syntax looks utterly naff. :-)
>
> I think Ada style 123_457_134 would be much more readable than 123'457'134.
>

> [snip]
>
>>
>> I'd recommend changing your binary syntax to match gcc and the future
>> C++17 standard - it's consistent with existing 0x for hexadecimal, while
>> a "B" suffix conflicts with the C use of suffixes indicating types. And
>> while I would prefer a "_" separator, as used in Ada, the C++ folks have
>> picked '.
>>
>
> Strongly agree with using Ada's "_"; do you have any idea why they picked
> a single quote ?
>

I don't know, but I guess it might be something to do with the
user-defined literal syntax in newer C++ (that lets you write things
like "2_hours + 45_minutes" by defining a "_hours" operator to convert a
number into a time class).

[BartC]

"David Brown" <david...@hesbynett.no> wrote in message

news:lgnevt$crq$1...@dont-email.me...

(Hey, I had that over twenty years ago! But mine are just scale factors, and
are not connected to the literal:

2 m + 35 cm, 5 miles+375 yds and so on (all these are converted to mm as
that was my basic unit).

As I implement it, the "m", "cm" etc don't interfere with the normal
namespace. Originally used in an application scripting language, these days
I mainly use the feature to be able to write 5 million, 1 billion and so
on.)

But the ' character as separator is not that terrible, at least it's easy to
type with no shift needed.

One extra literal connected thing they might look at, but it's probably too
late, is the 'bug' where writing leading zeros such as 064 actually gives
you the number 52 not 64. Data, especially from external sources, can
sometimes have leading zeros. Perhaps 'avoid leading zeros' should be added
to the lists that people have posted of how to ensure more bug-free code.

--
Bartc

[Tom Gardner]

On 23/03/14 19:53, Wouter van Ooijen wrote:
>>> You should not regard that as a one-or-the-other question, there is a
>>> continuum from solving a problem using standard ways (that require
>>> 'more' of them to solve the problem) to writing a dedicated
>>> language for it that solves this one problem trivially. The 'soft
>>> spot' in this continuum varies. When a certain type of problem needs
>>> to be solved only once the 'standard ways' is to be preferred,
>>> because it requires less work, and is easier to read. The more often a
>>> type of problem needs to be solved, the more a 'dedicated' solution is
>>> favoured: the amount of up-front work (both on the
>>> implementor and on the reader) can be averaged out over all applications.
>>
>> Well, yes. No argument there.
>>
>> So what would you regard as useful indicators or preconditions
>> that would bias the answer one way or the other?
>
> Try to minimize the total effort involved. Is the investment in tool building and learning (not only your learning, but of all people involved) compensated by the effort saved in using the tool and
> writing/reading the (much shorter/readable) solution(s)?

Those are two points that I think are critical: tools and
people.

There are many very good toolsets (i.e. not just compilers)
available for standard languages, and appropriate use can
significantly improve the end result. Re-creating vaguely
equivalent tools for a special-purpose language is an
enormous burden, and I've never seen it happen. End result:
you spend more time on the nitty-gritty crud that good tools
make easy.

If your product is wildly successful then you will need to
recruit more people to get the job done. By definition you
can't hire people with experience, so you have to spend time
and energy training them. Plus it is unlikely that really
good people will want to develop their career around
something that has no transferable skills.

> favours existing paradigms in an existing language:
> - one-time, small problem
> - many people need to understand it, especially if they need to understand only a small part, and already know the language/paradigms
> - semantic difference between programming language/paradigms and problem domain is small
>
> favours full special-purpose language:
> - large recurring problems
> - only few people involved, and they all need to understand large amounts of it
> - semantic difference is hughe

But the key point is that /usually/ the combination of standard
language plus DSLibrary gets /most/ of the benefits of a
DSLanguage. Without the disadvantages of a DSLanguage.

In addition, my experience is that a small DSLanguage for
limited purposes usually grows "organically" like topsy until
nobody understands it any more! ("Organically" is clearly a
euphemism, of course). Unfortunately some mainstream
standard languages also suffer from that problem :(

[Tom Gardner]

On 23/03/14 19:21, Paul Rubin wrote:
> Tom Gardner <spam...@blueyonder.co.uk> writes:
>>> posted some links to Atom and ImProve, both of which are Haskell
>>> EDSL's used for realtime control stuff.
>>
>> OK, but under what conditions would/wouldn't you use them to solve an
>> embedded problem via a DSLanguage or DSLibrary?
>
> Embedded is a pretty wide space and in more critical applications, there
> may be prescribed processes that I'd have to follow regardless of my
> preferences. In less critical applications, memory and realtime
> constraints can still be determining factors. Another issue is the
> skill sets of the programmers likely to work on the code. Using DSL's
> can require a level of PL geekery that a typical embedded developer
> would probably not have.

Agreed.

> I could imagine an approach of prototyping
> using a DSL, then conferring with the customer about whether they were
> comfortable staying with that approach or wanted a more traditional
> approach.

Sounds close to the "executable requirements specification" concept.
I have no problems with that.

The main issue becomes ensuring the "executable requirements" are
fully implemeted in the standard language plus DSLibrary. I know
aerospace companies go to considerable trouble to develop/use
special-purpose tools to ensure such tracability.

> If I were the main consumer of the code, then as a PL geek
> I'd probably use DSL's rather freely.

Nagging doubt: the small, limited-purpose DSLanguages I've
seen become successful all gradually evolved over the years
until they were so large and complex that the designers
didn't really understand them fully - let alone the people
that used them.

Naturally DSLibraries also tend to have that problem, but
at least there should be good tool support for understanding
them.

>> I must admit that, while LISP is clearly a sufficient base in which to
>> create a DSLanguage, I'm not sure I would rush to use it in an
>> embedded environment.
>
> I don't see why not, if the target CPU is powerful enough to run it, and
> there aren't realtime constraints that would preclude garbage
> collection. I've done embedded stuff in Python which amounts to the
> same thing.

Yeah, I'm getting old w.r.t. processor/memory capabilities.
But my embedded projects have all had a significant soft
or hard real-time or low-power elements to them.

>> Haskell? I don't know enough about that to have a valid opinion, but
>> "lazy evaluation" and "strictly functional" cause my eyebrows to go
>> upwards in the embedded domain. (The former because of time/space
>> considerations, the latter because an embedded system with side effect
>> is more than a little pointless!)
>
> The two Haskell EDSL's that I mentioned (Atom and ImProve) both compile
> to C or Ada. I've played with Atom. You write your program in a style
> that looks like multitasking (plus you get to use Haskell syntax and
> type safety), and then the Atom library converts it to a C-coded state
> machine with deterministic timing, that you run through a C compiler to
> produce target code. ImProve operates about the same way, though its
> target problem set is a bit different.

I'd need to glimpse the compelling advantages (vs standard
language plus DSLibrary) of that before I invested my time.

> Strictly functional code can have side effects: they just have to be
> known to the type system. So if some function in your program wants to
> print something (or call some other function that prints stuff), it has
> to have "IO" in its type signature. The function output is e.g. a
> "print command" that gets handed off to the runtime library that does
> the printing. So the type system lets you segregate the side-effecting
> code from the "pure" code, which helps write in a style where most of
> the code is pure and easier to reason about.

When it suits me I become a purist, so I'll claim that's merely
hiding the dirty stuff under the carpet, pretending it isn't there.
But it is there, so it cannot be strictly functional.

[David Brown]

On 23/03/14 22:09, BartC wrote:
> "David Brown" <david...@hesbynett.no> wrote in message
> news:lgnevt$crq$1...@dont-email.me...
>> On 21/03/14 21:40, Simon Clubley wrote:
>
>>> Strongly agree with using Ada's "_"; do you have any idea why they
>>> picked
>>> a single quote ?
>>>
>>
>> I don't know, but I guess it might be something to do with the
>> user-defined literal syntax in newer C++ (that lets you write things like
>> "2_hours + 45_minutes" by defining a "_hours" operator to convert a
>> number
>> into a time class).
>
> (Hey, I had that over twenty years ago! But mine are just scale factors,
> and
> are not connected to the literal:
>
> 2 m + 35 cm, 5 miles+375 yds and so on (all these are converted to mm as
> that was my basic unit).
>
> As I implement it, the "m", "cm" etc don't interfere with the normal
> namespace. Originally used in an application scripting language, these days
> I mainly use the feature to be able to write 5 million, 1 billion and so
> on.)

Just for your interest:

<http://en.wikipedia.org/wiki/C%2B%2B11#User-defined_literals>
<http://en.wikipedia.org/wiki/C%2B%2B14#Standard_user-defined_literals>
<http://en.wikipedia.org/wiki/C%2B%2B14#Binary_literals>

>
> But the ' character as separator is not that terrible, at least it's
> easy to
> type with no shift needed.

I'd rather have a _ as a separator than ', but it could be worse -
someone mentioned another language that uses $.

>
> One extra literal connected thing they might look at, but it's probably too
> late, is the 'bug' where writing leading zeros such as 064 actually gives
> you the number 52 not 64. Data, especially from external sources, can
> sometimes have leading zeros. Perhaps 'avoid leading zeros' should be
> added to the lists that people have posted of how to ensure more
> bug-free code.
>

Octal literals are intentional by design, but I too think that in most
programs they are likely to be a mistake. The only common usage I know
of them is for posix file modes. Personally, I would far rather see
leading zeros ignored and 0o64 being used for "octal 64", or something
akin to Ada like 8#64. But octal in C is well established - the best we
can hope for is optional warning messages added to compilers.

[Simon Clubley]

I've not yet seen anyone bring up the languages designed for various
research purposes which are designed to explore new scenarios for
which the current languages would be considered stale or ossified.

A good example would be Wirth's research with the Oberon range of
languages.

Simon.

PS: And while I am thinking about Wirth, don't forget that Pascal was
originally a small scale specialised language designed only for teaching
before some people decided it was unique enough and brought enough new
ideas to the table to be developed into a successful commercial language.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Paul Rubin]

Tom Gardner <spam...@blueyonder.co.uk> writes:
> When it suits me I become a purist, so I'll claim that's merely
> hiding the dirty stuff under the carpet, pretending it isn't there.
> But it is there, so it cannot be strictly functional.

There's not any really strict definition of functional programming
that's widely accepted. I'd say Haskell does sweep some stuff under the
carpet, but the way it does i/o is purely functional in the sense that
the programs are written as state transformers on the external world.
If you look closely at the type signature of the "print" function, its
input is an abstract data value of type RealWorld (it's actually written
that way) and its output is another such value. So given a real world
that contains a printer on your desk and a blank sheet of paper in the
printer, the function produces a new real world, where the formerly
blank paper now has stuff printed on it. This allows various theorems
about functional programs to keep working, which is what makes it pure.

In the actual implementation, the i/o operations (such as printing) are
done by the runtime environment, which is a separate entity from the
(pure) program evaluator. Your program using the "print" function
doesn't actually print anything. Instead it computes commands that it
returns to the runtime environment, and the runtime environment prints
stuff. It's not just a nomenclature thing--it has a fancy mathematical
underpinning based on category theory, though programmers don't have to
be directly concerned with that.

This is an old but fairly readable explanation:

http://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf

This goes into more (practical) detail:

http://research.microsoft.com/en-us/um/people/simonpj/papers/marktoberdorf/mark.pdf

[Tom Gardner]

On 23/03/14 22:28, Simon Clubley wrote:
> On 2014-03-23, Tom Gardner <spam...@blueyonder.co.uk> wrote:
>> On 23/03/14 17:04, David Brown wrote:
>>> I agree that there is plenty of use for small "informal" tools - the case I picked was intentionally an extreme example.
>>>
>>> But is there a place for small, informal tools when there are large, well-tested and powerful mainstream tools available? Sometimes cost is an issue, but gcc and llvm have reduced that enormously.
>>> Yes, I think there are still situations where small tools are a better choice - but only for niche areas with particularly good reasons for choosing them.
>>
>> Given that in general there is an "impedance mismatch" between
>> the problem-being-solved and the language, under what conditions
>> is it better to create
>> - a small domain specific language tailored to the problem, or
>> - a library in a standard language
>>
>> Yes, I have my own opinions, but I'd like to hear other people's
>> opinions.
>>
>
> I've not yet seen anyone bring up the languages designed for various
> research purposes which are designed to explore new scenarios for
> which the current languages would be considered stale or ossified.

I could probably make an argument along the lines that
most DSLanguages effectively just that - but for a very
narrow domain.

> A good example would be Wirth's research with the Oberon range of
> languages.

Hardly a domain specific language!

> PS: And while I am thinking about Wirth, don't forget that Pascal was
> originally a small scale specialised language designed only for teaching
> before some people decided it was unique enough and brought enough new
> ideas to the table to be developed into a successful commercial language.

Yes indeed. Somewhere I still have the red/silver (2nd edition?)
version of the language definition, probably from c1977.

But it was never intended as a domain specific language either!

[Paul Rubin]

Formalization (in the CompCert style) is still a very specialized topic
that's completely outside the skillset of normal real-world embedded
developers. Even places that use it (e.g. Galois Corp.) I get the
impression that it's a separate area of responsibility than messing with
day to day code resident inside devices.

It's getting to be more accessible. The book "Software Foundations" is
pretty readable and has good exercises:

http://www.cis.upenn.edu/~bcpierce/sf/

I haven't spent much time with it yet, but I want to.

[Robert Wessel]

In many years the only time I've had occasion to deal with octal
constants in C or C++ is when an unwanted leading zero snuck in
somewhere causing a problem. Usually when a table of numbers was
copied from an outside source.

A syntax like 0o123 would have been a far better choice (or an Ada
-like "base#number"), but the leading-zero form of octal number
specification predates hex ("0x") in C, so getting rid of it is likely
to be impossible. It's even propagated into a number of other
languages (Java, for example).

Some lints can warn about any octal usage, but I've not noticed that
feature on any compiler, except as part of MISRA checking (MISRA
disallows octal constants and literals), but that's just too painful
for most uses.

[Robert Wessel]

On Fri, 21 Mar 2014 20:40:47 +0000 (UTC), Simon Clubley
<clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:

>On 2014-03-21, David Brown <david...@hesbynett.no> wrote:
>>
>> C++17 will (according to plan) allow a single quote mark (123'457'134)
>> as a separator within numbers, which is nice. I expect to see that
>> implemented in gcc for C and C++ as an extension long before C++17 is ready.
>>
>
>Sorry, but that single quote syntax looks utterly naff. :-)
>
>I think Ada style 123_457_134 would be much more readable than 123'457'134.

Underscore might be a bit better, but the single quote is perfectly
workable, and even looks a bit like a comma.

[Wouter van Ooijen]

> Those are two points that I think are critical: tools and
> people.
>
> There are many very good toolsets (i.e. not just compilers)
> available for standard languages, and appropriate use can
> significantly improve the end result. Re-creating vaguely
> equivalent tools for a special-purpose language is an
> enormous burden, and I've never seen it happen. End result:
> you spend more time on the nitty-gritty crud that good tools
> make easy.

If your 'vaguely equivalent' predicate is applicable the semantic
distance between the problem domain and the language/paradigm you have
available is small. This of course votes heavily against anything
specific, because the main advantage of something specific is that it
can reduce (the cost associated with) that gap!

Wouter

[Paul Rubin]

Tom Gardner <spam...@blueyonder.co.uk> writes:
> Nagging doubt: the small, limited-purpose DSLanguages I've seen become
> successful all gradually evolved over the years until they were so
> large and complex that the designers didn't really understand them
> fully - let alone the people that used them.

Maybe this is less of an issue with embedded DSL's, where you can
use the features of the host language as well as the EDSL.

> [Lisp] Yeah, I'm getting old w.r.t. processor/memory capabilities.

> But my embedded projects have all had a significant soft or hard
> real-time or low-power elements to them.

I think it's fine to use gc'd languages for soft real time, in the
typical case where deadlines are in the tens of msec and it's ok to miss
one now and then. It's possible to keep GC latency at that level
without using any fancy and inefficient methods. If you look at the
literature on Erlang (which is basically a concurrent Lisp with
Prolog-like syntax glued on), soft real time applications were the
design target from the beginning. Hard real time applications
(microsecond deadlines that must never be missed) are different, of
course.

> I'd need to glimpse the compelling advantages (vs standard
> language plus DSLibrary) of that before I invested my time.

The blurbs for ImProve and Atom at http://tomahawkins.org/ might
interest you. ImProve uses an SMT solver to statically verify that your
code meets assertions that you specify. I guess SPARK/Ada or some of
the tools for the new Ada 2012 design-by-contract stuff does similar
things. Atom transmogrifies your program from one that appears to be
written with multiple concurrent tasks, into one with a single outer
loop that (on a realtime cpu) spends the exact same number of cycles in
each iteration regardless of the data. It gets rid of the need for
locks, semaphores, etc. while doing all task scheduling statically at
compile time.

I found Atom to be reasonably easy to use, given that I was already
familiar with Haskell. Haskell's learning curve is notoriously steep
though, and it's not really possible to use Atom without at least some
Haskell understanding.