Suggested method to allow user to adjust serial buffer size in IDE

[ro...@mail.com]

for some applications, it is necessary (or highly convenient) to have the serial buffer sizes different to the default sizes. to achieve this, the user has to modify the contents of the HardwareSerial.h file in ...\hardware\arduino\avr\cores\arduino. however, it is undesirable to have the (potentially inexperienced) user tinkering with these files.

i would like to propose the following solution be implemented, to allow the user to alter the serial buffer sizes:

 

1. in HardwareSerial.h add in the following single (bolded) line:

#if (RAMEND < 1000)

#define SERIAL_TX_BUFFER_SIZE 16

#define SERIAL_RX_BUFFER_SIZE 16

#else

#define SERIAL_TX_BUFFER_SIZE 64

#define SERIAL_RX_BUFFER_SIZE 64

#endif

#include "OVERRIDE\serial_buffer_size.h"  // ##### override default buffer size #####

#endif

2. create a subdirectory ...\hardware\arduino\avr\cores\arduino\OVERRIDE

3. in that subdirectory create a zero-length file called serial_buffer_size.h

the above three steps provide a safe location where (a) a user can override default settings, which can (b) easily be restored to 'factory default' by truncating the files back to zero length, all without (c) altering existing functionality or compiler output provided step 3. holds. in principal the same method could be used to allow for overriding other settings on other core libraries.

i would then like to suggest that the following IDE changes could be made to facilitate using the above provisions:

4. to the Arduino IDE add an entry under the "Tools" menu called "OVERRIDES" where the user can select any settings that they wish to override, and select amongst safe override values. for instance, there may be Tools -> OVERRIDES -> Serial RX Buffer Size with options of 'default', '16', '64', '256', '1024' (and the same for Serial TX Buffer Size).

what do the developers think of this idea? steps 1 to 3 could initially be carried out with minimal effort (i have tested out steps 1 to 3 and verified they work as desired), with step 4 (or some variation thereof) implemented later on. overrides could be provided as global irrespective of the board selected, or made specific to only one board or type of processor.

i would much appreciate feedback.

cheers,

rob   :-)

[Brian Cook]

#include "OVERRIDE\serial_buffer_size.h"  // ##### override default buffer size #####

Do you have an aversion to the variants folder?

- Brian

[Tyler]

I've been thinking about this a lot as well, and I think we should leverage the preprocessor for this. For instance, just change the code to this:

#ifndef SERIAL_TX_BUFFER_SIZE
#    define SERIAL_TX_BUFFER_SIZE 16
#endif

Then there's several options to change the buffer size from the user's side:

1. Add your own define at the top of your sketch, before Arduino.h gets included:
• #define SERIAL_TX_BUFFER_SIZE 32
• Is <Arduino.h> is now included automatically? If so, how do we put this before it?
  
2. Or, specify it during the build process with gcc's standard flags:
• gcc ... -DSERIAL_TX_BUFFER_SIZE 32
• Perhaps there could be an option in the Preferences to add your own flags to the build process?
  

I think these are pretty simple options and could be implemented with minimal effort. What do you think?

-Tyler

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[ro...@mail.com]

i'm more than happy with any solution that works.

i've been doing some experiments, adding the following line to a sketch:

#define OVERRIDE_

the symbol sort of propagates down into HardwareSerial.h fine, and can trigger the following code i inset there:

#if defined(OVERRIDE_)

#undef SERIAL_TX_BUFFER_SIZE

#undef SERIAL_RX_BUFFER_SIZE

// #define SERIAL_TX_BUFFER_SIZE 512

// #define SERIAL_RX_BUFFER_SIZE 512

#endif

with the '512' lines commented out, compilation fails as expected. but with the lines uncommented, the previous values (64) are reinstated. i feel i must be doing something wrong here - i am very much not a C expert.

does anyone know why this is happening? i have tried defining SERIAL_RX_BUFFER_SIZE in a sketch, but it does not propagate down into HardwareSerial.h unfortunately. it does look like the author had some original intention that it should.

cheers,

rob   :-)

[David Mellis]

BTW, this has been a long-standing discussion. See, for example: https://code.google.com/p/arduino/issues/detail?id=27

[ro...@mail.com]

the fact that it has been discussed for so long is perhaps an indicator that it is something worth addressing with an actual solution. however, i can see the real risks in a carte blanche exposure of the core library internals to users.

how about this as a compromise: an options.xml file contained in the sketch folder, that the IDE is capable of editing. when a compilation/upload is initiated, the IDE parses the xml file into an options.h file that gets included?

in this way, the IDE has control over what can (and more importantly can not) be changed, and can manage how that change is translated into the .h file that is finally included. while the .xml file remains unchanged, between IDE versions the generated .h files may be vastly different.

cheers,

rob   :-)

[Andrew Kroll]

I've always wanted an options header per-sketch.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[ro...@mail.com]

Coding Badly, from the arduino user forums, has offered the following procedure that seems to work flawlessly. might i suggest it could be incorporated in the next release of the IDE - provided it passes testing and the developers can agree to the level of exposure it offers to the innards of the core libraries:

(1) in the file platform.txt change the following line:

        build.extra_flags=

to:

        build.extra_flags=-I "{build.path}" -include options.h

(2) in the Arduino\hardware\arduino\avr\cores\arduino directory

create an empty file called options.h

now in any sketch directory you can create a file called options.h

that contains the things you want to change locally to that sketch.

in my case this is:

#define SERIAL_TX_BUFFER_SIZE 64

#define SERIAL_RX_BUFFER_SIZE 1024

(i suspect that both RX and TX sizes need to be specified as a pair)

when i open a project in the IDE, the options.h file is also opened (if it exists). if i change the contents of options.h those changes are reflected in the next compile [u]without[/u] needing to restart the IDE. all quite seamless.

cheers,

rob   :-)

[Matthew Ford]

Now might be a good time to remind people of the thread

No CLI() to prevent Hardware Serial errors

(https://groups.google.com/a/arduino.cc/forum/#!msg/developers/g1DSarFptS8/cmgdeCrKvQUJ)

"I guess at some point, modifying SERIAL_TX_BUFFER_SIZE to above 256 was
somewhat implemented, but you are correct that this isn't really
properly supported.

I guess that we should either conditionally (based on the buffer size)
disable/enable interrupts (using the ATOMIC_BLOCK macro seems a concise
way of doing this), or just add a comment stating that a buffer size of
256 or bigger has atomicity issues."

[robert rozee]

it is a bit of luck then, that i am keeping my TX buffer size at 64 bytes!   :-)   as an aside, it would be easy enough for HardwareSerial.h to have pre-processor code added to prevent SERIAL_TX_BUFFER_SIZE from exceeding 256, with just a simple range check and error message. i have found that the RX buffer size does have the limitation of needing to be a power of 2 for some reason.

 

but, like Tony DiCola wrote in that other thread, it would be good to not get bogged down in the specific example.

 

cheers,

rob   :-)
 

Sent: Thursday, June 18, 2015 at 2:36 AM
From: "Matthew Ford" <matthe...@forward.com.au>
To: devel...@arduino.cc
Subject: Re: [Developers] Re: Suggested method to allow user to adjust serial buffer size in IDE

You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/4q9k_gvKpLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.

[Georgitzikis Vasilis (tzikis)]

fyi, one of our (codebender) user’s once wanted us to allow this, but he finally ended up just copy-pasting the HardwareSerial.h/.cpp folder in his sketch and claimed it worked. I haven’t done this myself and I wouldn’t expect it to work based on my knowledge of how the Arduino IDE compiles and links, so take this with a grain of salt.

He was also using codebender and not the official IDE, and while we go to extreme lengths to be as close to 100% compatible, there are some small inconsistencies, so it might not work with the IDE.

Cheers,

Vasilis

[Matthijs Kooijman]

Hey Tyler,
> <li>Add your own define at the top of your sketch, before
> Arduino.h gets included:</li>
> <ul>
> <li>#define SERIAL_TX_BUFFER_SIZE 32</li>
This doesn't work, since it only affects the compilation of your sketch.
When HardwareSerial.cpp is compiled, the buffer size will still be the
default.

> <ul>
> <li>gcc ... -DSERIAL_TX_BUFFER_SIZE 32</li>
> <li>Perhaps there could be an option in the Preferences to add
> your own flags to the build process?<br>
This sounds good to me, but it's tricky to get this right. Should this
be a global preference? Stored per-board? Per-sketch? etc.


Gr.

Matthijs

[Andrew Kroll]

Not if the Serial.cpp includes the options file at the top :-)

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Matthijs Kooijman]

> > > <li>Add your own define at the top of your sketch, before
> > > Arduino.h gets included:</li>
> > > <ul>
> > > <li>#define SERIAL_TX_BUFFER_SIZE 32</li>
> > This doesn't work, since it only affects the compilation of your sketch.
> > When HardwareSerial.cpp is compiled, the buffer size will still be the
> > default.

> Not if the Serial.cpp includes the options file at the top :-)

Even then, defining things "at the top of your sketch" won't work. You
are correct that having an options.h that is included is another option
that would work, but that's already discussed in another part of this
thread.

Gr.

Matthijs

[Matthijs Kooijman]

Hi Matthew,

> No CLI() to prevent Hardware Serial errors
>
> (https://groups.google.com/a/arduino.cc/forum/#!msg/developers/g1DSarFptS8/cmgdeCrKvQUJ)

FWIW, the issue is also being tracked at
https://github.com/arduino/Arduino/issues/2405

I just added a small summary of the problem there to clarify (there's
really two separate atomicity issues remaining).

> I guess that we should either conditionally (based on the buffer size)
> disable/enable interrupts (using the ATOMIC_BLOCK macro seems a concise
> way of doing this),

That's also what I was thinking. I proposed it in the issue above, but
won't have the time to actually implement it. If anyone else submits a
PR, I'm happy to free up time for review and some testing, though.

> or just add a comment stating that a buffer size of
> 256 or bigger has atomicity issues."

For now, that seems like the easiest option. Since it's a trivial bit of
work, I just created a PR for this:

https://github.com/arduino/Arduino/pull/3361

Gr.

Matthijs

[Randall Routh]

I long ago recommended and enhancement that would resolve this and may other problems.

The enhancement was to define a directory within your project that would be included in the 'include' path ahead of all other includes that would be used during the compiles.  The way it works is very simple.

1) a directory could be called something like 'Configuration' would be included in your project, if needed.

2) the 'Configuration' directory is always the first one searched for header file for all compiles.

3) any library, core or 3rd party, would put configuration parameters in a special "config" header.

4) to modify the configuration from that of the default, copy the XXXX_conf.h from the library to you 'Configuration' directory and modify it with your desired parameters.

This could be used for so may things.

The hardware serial configuration parameters could be stored in the HardwareSerial_conf.h.  The header would then be loaded at compile time.  If it is not copied and modifed, it will be read from the core headers location.

For 3rd party librarys.  If want to write a library for a Widget that can be access either parallel or I2C then I must provide two libraries or library would need to provide two sets of object, methods or whatever.  This increases maintenance time, could cause code bloat by including code that is not need or used to check which interface is being used.  The solution is to set up in the configuration header the interface that is going to be used.  Then with #ifdef compile only the code needed for that interface.

I first requested this functionality when I was working with Real Time Clocks (RTC).  I wanted to develop a library to read set and manipulate the RTCs.  Rather than define a different library for each RTC I wanted a single library.  Rather than instantiate a different class for each RTC chip I wanted to just instantiate an RTC object.  Given, the RTC object would not have an alarm set method if the chip did not support an alarm but the RTC used would be transparent to the project. 

The change to the IDE should be minimal but I do not have the programming experience to go modifying the IDE.

Gr.

Matthijs

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Randall Routh
404-276-8373

[Mikael Patel]

In Cosa @jeditekunum has provided a very elegant solution to local per sketch configuration. The idea is as follows:

1. An empty Options.h included by Arduino.h
2. Local Option.h when needed in sketch folder

Please see the commit https://github.com/mikaelpatel/Cosa/commit/d911dda4c897b37ee5ffe788be358458dc059f7f
for details on how this works in Cosa.

Cheers! Mikael

BW: Consider adding a static_assert() to check power 2 size of buffers. See Cosa IOBuffer for an example. https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/IOBuffer.hh#L37 The Cosa IOBuffer is used by the UART and nicely separates ring-buffer handling from serial. https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/IOStream/Driver/UART.cpp#L39

[Randall Routh]

How does that work?

If I have "Option.h" in my project how will it be picked up.  If my project include library is searched first then that could cause problems with the "core" compile if I should name one of my headers the same as a "core" library header.

If my project include library is searched last then the empty "Options.h" would mask my version of the file.

Also having a single header file to hold all options for all libraries seems like an issue.  A define of BUFFER_SIZE could have different meaning in different libraries.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Randall Routh
404-276-8373

[Mikael Patel]

Sorry about the typo. It should only be "Options.h" in my example (not "Option.h" and "Options.h"). Translating the Cosa solution to Arduino would be:

1. Update Arduino.h to include Options.h
2. Empty Options.h file in the Arduino core
3. Sketch adds the file Options.h with settings such as BUFFER_SIZE when needed

The compiler options (i.e. include search order) will fix the rest. This works fine as the Arduino core is compiled for each sketch/board setting. The local "Options.h" will be used when compiling the Hardware Serial source (in the example). Please note that it could detect the board type and set configuration accordingly.

Cheers! Mikael

[Paul Stoffregen]

On 06/23/2015 07:29 AM, Randall Routh wrote:

I long ago recommended and enhancement that would resolve this and may other problems.

Yes, you and others have proposed this or something very similar many, many times.

The trouble is may cause many more problems than it solves.  Maybe.  It's easy to propose a change like this, thinking only of the upsides, without really considering how this changes the build process, dependencies, APIs and overall long-term evolution of Arduino as a platform.

I'm not saying this is necessarily a bad idea.  But it is one that needs a lot of careful, forward-looking thought and analysis about what it really means for Arduino as a platform.

[Mikael Patel]

I have done some tests on this method and it turns out that it requires an addition include option with the current sketch directory. The Cosa makefile based build adds "-I." to the compiler options. This is what makes it work for Cosa. For Arduino IDE build preprocessing this would require that the sketch director is added to the "{includes}" in the build patterns.

Cheers! Mikael

[Mikael Patel]

Here is a working solution of the Arduino build.

1. Modified platform.txt (attached)
Added extra build flags with the include search path to the build directory.

build.extra_flags="-I{build.path}"

2. Modified Arduino.h (attached) and empty options.h

#include <options.h>

3. Modified SerialEvent.ino for demo
Check size of buffers with local configuration (options.h).

#include "options.h"

...

void setup() {
  ...
  // print size of buffers
  int availableForWrite = Serial.availableForWrite();
  Serial.print("availableForWrite = ");
  Serial.print(availableForWrite);
  Serial.println();
  Serial.print("SERIAL_TX_BUFFER_SIZE = ");
  Serial.print(SERIAL_RX_BUFFER_SIZE);
  Serial.println();
  Serial.print("SERIAL_RX_BUFFER_SIZE = ");
  Serial.print(SERIAL_RX_BUFFER_SIZE);
}

Cheers!

Mikael

[ro...@mail.com]

Mikael, what is the difference in result between your method and the one (that i posted earlier) suggested by Coding Badly? i genuinely ask this as i have almost no understanding of the details of how files are included and linked! from my perspective, it seems you are using a different (slightly simpler) setting for build.extra_flags, but additionally requiring a change be made to Arduino.h. i'd also be worried if there were a necessity for the user to have at the beginning of their options.h file any specific lines. to be robust, the solution implemented should allow for options.h in the sketch directory to be a zero length file, or for it to not exist at all.

if there is consensus on the way forward, is there a will within the developers to implement one of the suggested mechanisms? if so, what next? i'd love to see this change incorporated in the next release of the IDE, as i do believe it is something that (1) users will find useful, while at the same time (2) keeping users away from any need to tinker with the core files directly.

cheers,

rob   :-)