Provide custom avrdude.conf with 3rd party boards for 1.6.x series IDE?

[Tony DiCola]

Hi all, I'm looking at adding support for a few AVR-based boards to the new 1.6.x series IDE but am running into trouble getting the IDE to pick up a custom avrdude.conf and am curious if anyone has advice or ideas to resolve the issue.  For some background I'm using Arduino 1.6.2 linux 64 bit and have added a hierarchy of new custom boards following the info from here and here (great resources, thanks for providing them!).

Inside my IDE's hardware folder I have a file structure setup that looks like this:

 ./adafruit/avr/boards.txt = my custom boards configuration

 ./adafruit/avr/platform.txt = my custom platform configuration

 ./adafruit/avr/variants/... = a few custom variants that some of the boards use

 ./adafruit/avr/tools/avrdude.conf = a custom avrdude.conf (more on that below)

The boards I've defined show up as expected in the Arduino IDE board list, but the problem comes when trying to upload to them.  My boards.txt defines a few ATtiny85-based boards (Adafruit Gemma, Trinket, etc.) that look like for example:

gemma.name=Adafruit Gemma 8MHz

gemma.bootloader.low_fuses=0xF1

gemma.bootloader.high_fuses=0xD5

gemma.bootloader.extended_fuses=0xFE

gemma.bootloader.tool=avrdude

gemma.build.mcu=attiny85

gemma.build.f_cpu=8000000L

gemma.build.core=arduino:arduino

gemma.build.variant=tiny8

gemma.build.board=AVR_GEMMA

gemma.upload.tool=avrdude

gemma.upload.maximum_size=5310

To program this board avrdude's configuration needs to be updated slightly for the ATtiny85 to bump up some of the delays, like the chip erase delay increased from 4500 uS to 400 mS.  In the old 1.0.x IDE this was done by providing a completely new avrdude.conf that overwrote the file in the hardware/tools path.  However in the new 1.6.x IDE it looks like the right way to override the avrdude.conf is to do so with some configuration in my platform.txt (but let me know if that's wrong please).  Assuming I'm on the right path, I wrote a platform.txt that overrides the tools.avrdude.config.path value and looks like:

name=Adafruit Boards

version=1.0.0

tools.avrdude.config.path={runtime.ide.path}/hardware/adafruit/avr/tools/avrdude.conf

That adafruit/avr/tools/avrdude.conf path points to my custom avrdude.conf file, which is just a copy I made of the current avrdude 6.0 config that the normal Arduino cores use but with a few tweaks to bump up the necessary ATtiny85 delays.

The problem I'm running into is that the IDE doesn't seem to be using my custom avrdude.conf configuration on upload.  When I turn on verbose upload info I can see avrdude is using the default chip erase delay of 4500 uS instead of my custom 400 mS value and the upload fails.  I can also see the path to the configuration file being used (it's in a hidden folder under my home, .arduino15/packages/arduino/tools/avrdude/6.0.1-arduino2/etc/avrdude.conf) and when I look inside the file it looks like it's just the stock avrdude.conf without any of my changes.  

If I manually edit the hidden config file above and put in my custom delays I can upload the boards just fine, so it definitely looks like the problem is my custom avrdude.conf not being used for some reason.  However for some reason even though my platform.txt overrides the avrdude config path I don't see the custom config being used.

I'm curious has anyone run into this issue before and have a workaround?  Am I doing something totally wrong with trying to override the avrdude.conf in the 1.6.x series IDE?  Let me know if anyone has advice, thanks!

-Tony

[Mikael Patel]

Cosa supports a number of ATtiny MCUs. The default avrdude configuration file does not work very well and gives a bunch of warnings. To workaround this Cosa platform.txt file is updated and there is a local configuration file.

Ref.
https://github.com/mikaelpatel/Cosa/blob/master/platform.txt#L102
https://github.com/mikaelpatel/Cosa/blob/master/build/avrdude.conf

This workaround really only works on "core" level if you do not want to update the default configuration file.

Cheers! Mikael

[Tony DiCola]

Thanks for the reply and suggestion!  Unfortunately when I'm using the stock arduino core it doesn't appear that my platform.txt override to the tools.avrdude.config.path value is being used.  No matter what I set tools.avrdude.config.path to (including garbage values that shouldn't work at all) I always see the -C option to avrdude is pointing at the avrdude.conf included by the Arduino IDE.

For folks that are familiar with how the platform.txt is interpreted by the IDE, does this sound like a bug or am I doing something wrong to override the avrdude configuration path?  Very curious to know the right way to change avrdude.conf, thanks!

[Matthew Ford]

I have a commercial customer using Arduino Mega boards and he is getting seemingly random errors.

While there may be many other possible problems, I came across a discussion of the lack of CLI() calls when handling HardwareSerial head and tail references which are declared volatile  (http://bleaklow.com/2012/02/29/why_im_ditching_the_arduino_software_platform.html)
The Arduino core AVR code in V1.5.8 only has one instance of CLI() in HardwareSerial.cpp in availableForWrite().  This method does not appear to be used anywhere else.

From the http://www.nongnu.org/avr-libc/user-manual/group__util__atomic.html  docs

While declaring such a variable as volatile ensures that the compiler will not optimize accesses to it away, it does not guarantee atomic access to it.

So as I assess the situation, the head and tail values are updated by UART interrupts, both send and receive, but are also updated by the user's code and in the process can be corrupted.

My question is "Is this still the case and is Arduino UART serial handling un-reliable"?

regards
matthew ford

[Brian Cook]

My question is "Is this still the case and is Arduino UART serial handling un-reliable"?

There are tens-of-thousands, possibly hundreds-of-thousands, of people who use Arduino Serial.  With no problems.  Including me.  This leads me to suspect you are barking up the wrong true.

While declaring such a variable as volatile ensures that the compiler will not optimize accesses to it away, it does not guarantee atomic access to it.

Ugh.  The code in (AVR) Serial has been analyzed to death.  I guess we get yet another round.  I'll start...

As long as the values do not exceed what can be stored in a byte (255) and only one side reads while the other side writes the code is safe and reliable.

While there may be many other possible problems, I came across a discussion of the lack of CLI() calls when handling HardwareSerial head and tail references which are declared volatile  (http://bleaklow.com/2012/02/29/why_im_ditching_the_arduino_software_platform.html)

The analysis ignores the fact that the values never exceed 255.  Which makes the analysis incomplete.

The indexes have been changed to 8 bit.  Which makes the analysis out-of-date.


- Brian

[Matthew Ford]

Hi Brian,
Happy to be convinced that everything is OK
Just two points

i)  in some cases, SERIAL_TX_BUFFER_SIZE>256, the head , tail variable are 16bit variables.  In fact the only case where CLI() is used, availableForWrite(), there is the following test
#if (SERIAL_TX_BUFFER_SIZE>256)
  uint8_t oldSREG = SREG;
  cli();
#endif

but there is no such test anywhere else so it would seem the rest of the code is error prone when SERIAL_TX_BUFFER_SIZE>256

ii) atomic read/write of a single variable is not sufficient
HardwareSerial::read() has the following line
_rx_buffer_tail = (rx_buffer_index_t)(_rx_buffer_tail + 1) % SERIAL_RX_BUFFER_SIZE;

i.e. load _rx_buffer_tail,  atomically if 8bit
do some of the calculation, get interrupted by uart
_rx_complete_irq() 
 which, with interrupts disabled,  changes _rx_buffer_tail value
then the read code continues the calculation and stores result of its calculation, overwriting the what the uart interrupt just updated.

So I still think there are problems.
But very happy to be convinced otherwise.
regards
matthew

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Brian Cook]

> i) in some cases, SERIAL_TX_BUFFER_SIZE>256, the head , tail variable
> are 16bit variables.

At what point is SERIAL_TX_BUFFER_SIZE greater than 256? Or even
greater than 64?

> ii) atomic read/write of a single variable is not sufficient
> HardwareSerial::read() has the following line
> _rx_buffer_tail = (rx_buffer_index_t)(_rx_buffer_tail + 1) %
> SERIAL_RX_BUFFER_SIZE;
>
> i.e. load _rx_buffer_tail, atomically if 8bit
> do some of the calculation, get interrupted by uart
> _rx_complete_irq()
> which, with interrupts disabled, changes _rx_buffer_tail value

At what point does the interrupt service routine write to
_rx_buffer_tail value?


- Brian

[Andrew Kroll]

The only way serial sizes would be > 64  is if someone changed the buffer sizes...

256 does not fit into uint8_t, 255 does.

If they are changing stuff, tell them not to exceed 255 and all will be happy.

Otherwise if he does modifications like this, uint16_t is fine, but he needs to tell every other part that it is 16bit, and he has to disable then enable interrupts too during read and write access. Even at 2Mbaud this should be fine, you can execute 64 instructions between characters at that rate.

Personally, I've never had a problem with serial. My bet is that it is his code blocking IRQ too long. I always run my code with IRQ wide open. Then again, I also use recursive ISRs, which do take some know-how.

- Brian

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[Matthew Ford]

Thanks for that Brian,
With your explanation, I can cross off this as a possible problem.

>At what point is SERIAL_TX_BUFFER_SIZE greater than 256?  Or even
>greater than 64?

Fair enough  I agree in the current code the buffer size is 64 or less, but then the
SERIAL_RX_BUFFER_SIZE>256
looks very odd.  Half finished coding?  (see code sample from HardwareSerial.h below)

>At what point does the interrupt service routine write to _rx_buffer_tail value?

OK I see your point,  _rx_buffer_tail value is only read, not written, from the interrupt routine

Thanks again for your assistance.
all the best
matthew


Excerpt from HardwareSerial.h
...
#define SERIAL_TX_BUFFER_SIZE 16
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64
#define SERIAL_RX_BUFFER_SIZE 64
#endif
#endif
#if (SERIAL_TX_BUFFER_SIZE>256)
typedef uint16_t tx_buffer_index_t;
#else
typedef uint8_t tx_buffer_index_t;
#endif
#if  (SERIAL_RX_BUFFER_SIZE>256)
typedef uint16_t rx_buffer_index_t;
#else
typedef uint8_t rx_buffer_index_t;
#endif
...

[Matthijs Kooijman]

Hi Matthew,

> >At what point is SERIAL_TX_BUFFER_SIZE greater than 256? Or even
> >greater than 64?
>
> Fair enough I agree in the current code the buffer size is 64 or
> less, but then the SERIAL_RX_BUFFER_SIZE>256 looks very odd. Half
> finished coding? (see code sample from HardwareSerial.h below)

I guess at some point, modifying SERIAL_TX_BUFFER_SIZE to above 256 was
somewhat implemented, but you are correct that this isn't really
properly supported.

I guess that we should either conditionally (based on the buffer size)
disable/enable interrupts (using the ATOMIC_BLOCK macro seems a concise
way of doing this), or just add a comment stating that a buffer size of
256 or bigger has atomicity issues.

Care to submit a PR for either one? I'm happy to review it if you do.

Gr.

Matthijs

[Brian Cook]

Matthew,

Thanks for that Brian,

You are welcome.

Half finished coding?

Could be.  Could be the author was simply not aware there is an impending multi-byte issue.  In any case it is somewhat difficult to change the buffer size to be greater than 256.  And arguably unnecessary.  I doubt it will cause a problem.

- Brian

[Michael Dreher]

On Tuesday, April 7, 2015 at 11:09:26 AM UTC+2, Brian Cook wrote:

Matthew,

Could be.  Could be the author was simply not aware there is an impending multi-byte issue.  In any case it is somewhat difficult to change the buffer size to be greater than 256.  And arguably unnecessary.  I doubt it will cause a problem.

It caused me real problems some years ago. I made my local fix and used it in a bluetooth ISP programmer, see the files HardwareSerialOpt.h/.cpp here:
https://code.google.com/r/michaeldreher42-bluecontroller/source/browse/bluecontroller#bluecontroller%2Fexamples%2FBlueC_ISP

The base of the code was an probably Arduino 1.0.5.

The most interesting part might be the comments:
// Some methods are not IRQ-safe, that means some of the variables are also modified in the ISR (especially rx_buffer->head)
// Rules for disabling IRQ:
//  _rx_buffer->tail and _rx_buffer->head are 16-bit variables and cannot be accessed atomically on 8-Bit ATmegas. They have
//     to be volatile and access must be protected. This doesn't matter as long as RX_BUFFER_SIZE is less or equal than 0x100.
//  _rx_buffer->tail is only modified by app and never by ISR => IRQs only need to be locked when modifying
//  _rx_buffer->head is only modified by ISR and not by app (there are two exceptions: ctor() and flush()) => always lock IRQ when reading or modifying it
//  the filled part of the ringbuffer belongs to the app and doesn't have to be locked, just make sure to read the values before changing _rx_buffer->tail
//  the emtpy part of the ringbuffer belongs to the ISR and doesn't have to be locked

  Michael

[Tony DiCola]

Do you guys mind moving this thread about serial access out to its own topic?  For some reason I think the conversation about serial access accidentally got replied to my thread that was about extending avrdude with a custom configuration defined by the platform.txt.  I'd like to keep this thread on that topic of extending avrdude if possible, thanks!