Serial baud rates

[Angus Gratton]

We've been over this situation a couple of times, but I've been digging around in it again and I have a proposal. Keen to hear other people's responses.

** The situation **

On reset, ESP8266 hardware default baud rate depends on the crystal choice. With a 40MHz crystal you get 115200bps. With a 26MHz crystal you get proportionally less, 74880bps (115200*26/40). Almost everyone (including Espressif) ships hardware with a 26MHz crystal, so almost everyone gets 74880bps by default.

115200bps is a POSIX baud rate, 74880bps is not.

The esp-open-rtos framework currently keeps the 74880 default, same as the Espressif SDKs. However most of the examples have uart_set_baud(0, 115200) as a first step in user_init. This situation is always a bit broken, you either get early serial output or late serial output - you can't get both.

** The problem **

On Linux, very few serial tools support non-POSIX baud rates. Among those that _don't_: screen, minicom, stty, picocom (before 2.1 which is not yet in many distros), getserial, microcom.

When I want to debug at 74880bps I'm currently running a 5 line Python script with pyserial, or I use esptool.py to launch the flash bootloader at 115200bps, which then keeps that baud rate when it "exits" into a normal boot (esptool.py --port /dev/blah flash_id; screen /dev/blah 115200).

I think on OS X or Windows the situation is less terrible, but if you use Linux then 74880bps is really unfriendly and looks like it will stay that way for a while.[1]

** Proposed Solution **

We already use the open source bootloader, rboot, when building images in "OTA" mode. This is similar to the bootloader approach Espressif use on their recent SDKs. The rom bootloader loads the rboot stub into IRAM and executes it, then rboot loads all the other image components and executes those. If the build process was changed to remove the non-OTA mode, we could write a forked rboot that sets the baud rate to 115200 first up before it loads the main image.

This way the only output at 74880 would be the ROM bootloader loading the rboot stub, or an error message if the boot mode pins were wrong or the flash was non-functional. All other failures/diagnostics/info would be at 115200bps.

** Other Benefits **

- Less variables in the build process (no more OTA vs non-OTA mode).

- A standalone bootloader would let us code in additional diagnostic output, and/or pass a flag to the main image telling it to enable or disable diagnostic/verbose output when it starts up.

- Only one firmware image file instead of two.

Thoughts?


Angus



[1] The picocom author wrote about how fiddly it is to set non-POSIX rates on Linux/glibc here: https://github.com/npat-efault/picocom/blob/master/termios2.txt - I tried to write a patch for GNU Screen and I have to agree!

[Alex Stewart]

FWIW, I generally use miniterm.py under Linux to talk to the ESP8266 (installed by default in Ubuntu, not sure about other distros, but it handles nonstandard baud rates fine)

I've been thinking for a little while about maybe creating a custom terminal program (possibly hooking into esptool.py so that you could reflash an image without having to exit the terminal, etc) which could not only handle the nonstandard baud rate, but could actually detect a transition indicator sent by esp-open-rtos (or rboot) indicating a bps change and automagically change along with it..  Of course I haven't really investigated how much of the initial output comes from the ROM vs the bootloader.. if there isn't a lot from the ROM and we can set the baud rate consistently and early, it might not be as big a deal..

I've also been thinking about this in the context of another idea I've had kicking around, though:

I think we should have a standard way to configure various parameters/settings/etc for a given ESP8266 without having to compile them into the firmware image and reflash the whole thing (these could be configured with something similar to esptool.py, and could persist across reflashing, etc).  The most obvious use cases for this would be WiFi SSID and password settings, server hostnames, etc, but another potential use would be for people to choose what baud rate they would prefer esp-open-rtos to start up with..  That way you could have your 115200, and I can use 74880 (or whatever) even when using exactly the same flash code (or even the same prebuilt image).

In any case, getting back to your suggestion, I didn't realize that the bootloader spit out that much output (I'd always assumed that most/all of it came from the ROMs), but I was actually going to suggest that at some point we should switch over to our own bootloader (presumably based on rboot) anyway, so this seems like a logical next step anyway.  I also like building things the same way regardless of whether or not you want OTA functionality.

Part of the reason for this was because I was also thinking that it might be useful to try our hand at improving the overall firmware format.  If we abandon the Espressif flash image format (which has some clunkiness, if nothing else), we could do all kinds of things:

1. Combine everything into a single (compact) file
   
2. Come up with a single, standard format that could be used for both direct flashing and OTA (either by having the OTA flasher ignore the bootloader, or by having the program sending the image just leave it out)
   
3. Include arbitrary other data to be flashed at different locations (for example, "file" data to go in flash in a fixed location above the memory-map region, which could be read by the app using flash routines)
4. Hook in with the "settings" mechanism I was mentioning above to include default values, etc, in the image, which could be overridden during/after flashing, etc.

Anyway, I wandered around a bit there, but I think what it comes down to is "I think your suggestion makes some sense and actually fits in with several other things I've been sorta thinking about doing too"..

--
You received this message because you are subscribed to the Google Groups "esp-open-rtos mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esp-open-rto...@googlegroups.com.
To post to this group, send email to esp-op...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/esp-open-rtos/20160224041032.GJ26205%40ex2.lan.
For more options, visit https://groups.google.com/d/optout.

[Johan Kanflo]

I am all in for 115200.

[Johan Kanflo]

I've been thinking for a little while about maybe creating a custom terminal program (possibly hooking into esptool.py so that you could reflash an image without having to exit the terminal, etc) which could not only handle the nonstandard baud rate, but could actually detect a transition indicator sent by esp-open-rtos (or rboot) indicating a bps change and automagically change along with it..  Of course I haven't really investigated how much of the initial output comes from the ROM vs the bootloader.. if there isn't a lot from the ROM and we can set the baud rate consistently and early, it might not be as big a deal..

That would be nice. My solution is to use two serial boards, one for flashing and one for (read only) debug output, a bit clunky...

 

I think we should have a standard way to configure various parameters/settings/etc for a given ESP8266 without having to compile them into the firmware image and reflash the whole thing (these could be configured with something similar to esptool.py, and could persist across reflashing, etc).  The most obvious use cases for this would be WiFi SSID and password settings, server hostnames, etc, but another potential use would be for people to choose what baud rate they would prefer esp-open-rtos to start up with..  That way you could have your 115200, and I can use 74880 (or whatever) even when using exactly the same flash code (or even the same prebuilt image).

I have been playing with this idea too, have a look at https://github.com/kanflo/eor-past. (past is an acronym for parameter storage). Parameters are specified by an ID number and some IDs are "system defined" (wifi credentials, ...) and some are left to the actual user application to define). There is support for entering a "factory mode" upon boot where the parameters can be read and written. 

I think we should explore this idea further and decide on a flash layout supporting it. Say, the last blocks in flash (determined by reading the flash ID).

Sorry for veering off further Angus, maybe we should move this discussion to a new thread :)

Cheers,

Johan

[Alex Stewart]

Heh, I guess great minds think alike :). I've about half-written something a bit more sophisticated but basically along exactly the same lines as your "past" code for my own "system parameter" proof-of-concept.. I'll see if I can finish it up in the next few days so I can put it up for discussion as well (in a different thread :) )

To view this discussion on the web visit https://groups.google.com/d/msgid/esp-open-rtos/41771c06-356f-4e4c-a5ef-4d83738c52c5%40googlegroups.com.

[Johan Kanflo]

:D

Looking forward to see your implementation.

[Angus Gratton]

Hey folks,

Sorry for the long turnaround on this one, got busy with other things.

On Tue, Feb 23, 2016 at 10:31:23PM -0800, Alex Stewart wrote:
> FWIW, I generally use miniterm.py under Linux to talk to the ESP8266
> (installed by default in Ubuntu, not sure about other distros, but it
> handles nonstandard baud rates fine)

Nice! I had no idea miniterm.py existed. It's part of pyserial, so anyone with esptool.py working has miniterm.py somewhere on their system.

I've updated the esptool.py README to encourage people use it.

This makes the situation a lot more agreeable. I'd consider just changing all the examples to use 74880bps instead of 115200bps and forget about this issue.

Although I still favour working around Espressif-introduced "weirdness" whereever we can, and 74880bps definitely falls under the "weirdness" category for me.

> I've been thinking for a little while about maybe creating a custom
> terminal program (possibly hooking into esptool.py so that you could
> reflash an image without having to exit the terminal, etc) which could not
> only handle the nonstandard baud rate, but could actually detect a
> transition indicator sent by esp-open-rtos (or rboot) indicating a bps
> change and automagically change along with it.. Of course I haven't really
> investigated how much of the initial output comes from the ROM vs the
> bootloader.. if there isn't a lot from the ROM and we can set the baud rate
> consistently and early, it might not be as big a deal..

Here's what comes out from the ROM before rboot starts:

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 1640, room 16
tail 8
chksum 0xc2
load 0x3ffe8000, len 700, room 0
tail 12
chksum 0x8a
csum 0x8a

... the 'rst cause'/'boot mode' can be extracted by the bootloader and re-printed at a different baud rate.

The error cases that wouldn't be caught if you couldn't see this section are (AFAIK):

- Boot mode pins set wrong
- Flashed incorrectly ('qio' mode for 'dio' flash, bad flash, brownout, etc.) This can be mitigated a bit by always flashing the bootloader with conservative flash settings (slow speed dio), and then ramping up to a faster setting in the bootloader itself.

> I've also been thinking about this in the context of another idea I've had
> kicking around, though:
>
> I think we should have a standard way to configure various
> parameters/settings/etc for a given ESP8266 without having to compile them
> into the firmware image and reflash the whole thing (these could be
> configured with something similar to esptool.py, and could persist across
> reflashing, etc). The most obvious use cases for this would be WiFi SSID
> and password settings, server hostnames, etc, but another potential use
> would be for people to choose what baud rate they would prefer
> esp-open-rtos to start up with.. That way you could have your 115200, and
> I can use 74880 (or whatever) even when using exactly the same flash code
> (or even the same prebuilt image).

This is a neat idea that you and kanflo both had. :) I think it would be useful for a lot of things.

On a related note, something I'd like to try (someday) is default runtime auto-detection of flash size and type, instead of having to flash it into the device. Obviously won't work in all cases (would need a manual override), but it'd be nice (and possible, I think) to have defaults that work optimally nearly everywhere.

> In any case, getting back to your suggestion, I didn't realize that the
> bootloader spit out that much output (I'd always assumed that most/all of
> it came from the ROMs), but I was actually going to suggest that at some
> point we should switch over to our own bootloader (presumably based on
> rboot) anyway, so this seems like a logical next step anyway. I also like
> building things the same way regardless of whether or not you want OTA
> functionality.
>
> Part of the reason for this was because I was also thinking that it might
> be useful to try our hand at improving the overall firmware format. If we
> abandon the Espressif flash image format (which has some clunkiness, if
> nothing else), we could do all kinds of things:
>

> 1. Combine everything into a single (compact) file
> 2. Come up with a single, standard format that could be used for both

> direct flashing and OTA (either by having the OTA flasher ignore the
> bootloader, or by having the program sending the image just leave it out)

The current esp-open-rtos OTA with the v2 format can do these two - you can build a single image (including bootloader at offset 0, followed by image data) and flash it at offset 0 regardless of whether you want OTA. Or you can use it as an OTA image and have it flashed at a different 1MB aligned offset, and it still works.

The v2 format is fairly compact, the only "hole" is the 8KB at the beginning (4KB for rboot, 4KB for rboot's config sector.)

> 3. Include arbitrary other data to be flashed at different locations

> (for example, "file" data to go in flash in a fixed location above the
> memory-map region, which could be read by the app using flash routines)

Neat idea, I guess this would require a file format aware flashing tool.

> 4. Hook in with the "settings" mechanism I was mentioning above to

> include default values, etc, in the image, which could be overridden
> during/after flashing, etc.

Also very useful. :)

... this all sounds good.

I'm really in two minds about the baud rate thing now, though. Does anyone else have a strong preference for 74880 or 115200?


Angus

[Alex Stewart]

Although I still favour working around Espressif-introduced "weirdness" wherever we can, and 74880bps definitely falls under the "weirdness" category for me.

I do agree in principle, but really I think if we can make this configurable it'll be the best of all worlds anyway, so that's what I'm most interested in right now.  I'd actually be fine with defaulting to 115200 if there's a way for users to persistently set individual modules to stay at 74906 (or even do other baud rates) if they want to.

(On a side-note, I still don't buy the "crystal change" explanation for that bps rate, because the UART divisor value which they set in the startup code (1068) is the value necessary to divide the 80MHz system clock down to 74906 bps, not the 26MHz crystal clock.  If it was actually based off the crystal clock, then with the divisor they set, we'd actually be getting 24345 bps out (which we're not)..)

... the 'rst cause'/'boot mode' can be extracted by the bootloader and re-printed at a different baud rate.

I think that's a good idea in general, yeah.

 

On a related note, something I'd like to try (someday) is default runtime auto-detection of flash size and type, instead of having to flash it into the device. Obviously won't work in all cases (would need a manual override), but it'd be nice (and possible, I think) to have defaults that work optimally nearly everywhere.

Heh.. funny you should mention that.. that's actually sorta the next thing on my "want to do" list..  The SPI flash protocols actually have a fairly standard way to get this info from the chip already, we just need to write the code to do the query and use the results (and completely ignore the value in the flash header).

 

Neat idea, I guess this would require a file format aware flashing tool.

Yeah, I was kinda figuring this would all involve some modified/new tools..

[Angus Gratton]

Hey Alex,

On Tue, Mar 08, 2016 at 02:59:55PM -0800, Alex Stewart wrote:
> (On a side-note, I still don't buy the "crystal change" explanation for
> that bps rate, because the UART divisor value which they set in the startup
> code (1068) is the value necessary to divide the 80MHz system clock down to
> 74906 bps, not the 26MHz crystal clock. If it was actually based off the
> crystal clock, then with the divisor they set, we'd actually be getting
> 24345 bps out (which we're not)..)

Oh, it is _way_ more wacky than that! :)

Before the SDK's initial crystal change, the divisor register value is 694. Then the SDK (theirs and ours) re-sets it to 1068.

40e6 * 2 / 1068 = 74906 (74880 + 0.03%)
40e6 * 2 / 694 = 115273 (115200 + 0.06%)
26e6 * 2 / 694 = 74927 (74880 + 0.06%)

Here's what I think happens:

1) ROM bootloader initially sets internal "APB" PLL to XTAL*2, sets UART divider to 694. This means nominal 115200bps for 40MHz crystals, 74880 for 26MHz crystals. I'm betting this means the initial CPU speed is also 52MHz on 26MHz crystals.

2) The first few lines printed by the ROM bootloader are printed with these settings, then the SDK starts with these settings.

3) Startup function init_networking() calls register_chipv6_phy(phy_info) which reconfigures the PLL based on the crystal value stored in the "wifi/phy config" sector in the flash (which is where phy_info is read from). It sets them for an 80MHz APB & CPU frequency.

4) If you add uart print statements in between when register_chipv6_phy() returns and uart_set_baud(0, 74906), then you'll get output at nominal 115200bps.

... so I think when Espressif noticed they/everyone was using 26MHz, they hardcoded the UART frequency _back_ to nominal 74880 to match that crystal value, so that the output would stay consistent instead of changing when the PLL reconfigured.

I'm looking forward to (some day) really understanding the PLL registers and how they impact the available ESP8266 clocks.

(On the subject of auto detecting stuff, it should also be possible to use RC frequency to autodetect the crystal speed, rather than having it read from flash.)

> > On a related note, something I'd like to try (someday) is default runtime
> > auto-detection of flash size and type, instead of having to flash it into
> > the device. Obviously won't work in all cases (would need a manual
> > override), but it'd be nice (and possible, I think) to have defaults that
> > work optimally nearly everywhere.
> >
>
> Heh.. funny you should mention that.. that's actually sorta the next thing
> on my "want to do" list.. The SPI flash protocols actually have a fairly
> standard way to get this info from the chip already, we just need to write
> the code to do the query and use the results (and completely ignore the
> value in the flash header).

Awesome!


Angus

[Johan Kanflo]

 

On a related note, something I'd like to try (someday) is default runtime auto-detection of flash size and type, instead of having to flash it into the device. Obviously won't work in all cases (would need a manual override), but it'd be nice (and possible, I think) to have defaults that work optimally nearly everywhere.

Heh.. funny you should mention that.. that's actually sorta the next thing on my "want to do" list..  The SPI flash protocols actually have a fairly standard way to get this info from the chip already, we just need to write the code to do the query and use the results (and completely ignore the value in the flash header).

This one is a must have if you ask me. I recall an SDK function to get the flash JEDEC ID and then all we need to do is make a lookup table to map flash ID with its size. Depending on what the user told the flashing tool seems like something that was used to get things up and running quickly and then, as these things go, became the permanent solution ;)

 

 

Neat idea, I guess this would require a file format aware flashing tool.

Yeah, I was kinda figuring this would all involve some modified/new tools..

And an EOR flash map specifying where the sysparam area goes and so on.

 

[Alex Stewart]

>> Heh.. funny you should mention that.. that's actually sorta the next thing on my "want to do" list..  The SPI flash protocols actually have a fairly standard way to get this info from the chip already, we just need to write the code to do the query and use the results (and completely ignore the value in the flash header).
>
>
> This one is a must have if you ask me. I recall an SDK function to get the flash JEDEC ID and then all we need to do is make a lookup table to map flash ID with its size. Depending on what the user told the flashing tool seems like something that was used to get things up and running quickly and then, as these things go, became the permanent solution ;)

No, this is exactly what we should not do.  I realize that's what everybody else has been doing but it's still error-prone and wrong.  What I was actually trying to point out is that there is a fairly standard command for SPI flash devices (completely separate from the "get id" command) that actually tells you the size and layout of the chip you're talking to, without having to do any hacky JEDEC ID lookups, etc.  I don't know why Espressif didn't use it in the first place, but we should.  That's the correct way to do this sort of thing.

[Johan Kanflo]

Interesting, didn't know that. Seems like a sound way forward.

[Johan Kanflo]

** Proposed Solution **

We already use the open source bootloader, rboot, when building images in "OTA" mode. This is similar to the bootloader approach Espressif use on their recent SDKs. The rom bootloader loads the rboot stub into IRAM and executes it, then rboot loads all the other image components and executes those. If the build process was changed to remove the non-OTA mode, we could write a forked rboot that sets the baud rate to 115200 first up before it loads the main image.

Getting back to the original discussion (sorry Angus :), I like the idea of building all images in OTA mode and use a modified boot loader running at 115200. The verbose output handling could be placed in the parameter storage area that @foogod is working on.

 

- A standalone bootloader would let us code in additional diagnostic output, and/or pass a flag to the main image telling it to enable or disable diagnostic/verbose output when it starts up.

And to tell us why the main image was booted, as in "did the watchdog bite". 

[Johan Kanflo]

1) ROM bootloader initially sets internal "APB" PLL to XTAL*2, sets UART divider to 694. This means nominal 115200bps for 40MHz crystals, 74880 for 26MHz crystals. I'm betting this means the initial CPU speed is also 52MHz on 26MHz crystals.

52Mhz would explain my troubles in getting rboot print at 115200. Setting the baud rate to 115200 in boot.c yielded output at 74880. So I multiplied 115200 with (115200/74880) = 177230 and got output at 115200. 80/52 == 115200/74880 :)

3) Startup function init_networking() calls register_chipv6_phy(phy_info) which reconfigures the PLL based on the crystal value stored in the "wifi/phy config" sector in the flash (which is where phy_info is read from). It sets them for an 80MHz APB & CPU frequency.

Slightly off topic. The "wifi/phy config" sector, is it the same as "SDK config" at 0x3fc000 (for a 32Mbit flash)? Googling makes me think that has not been reverse engineered yet. Is that correct?

[Angus Gratton]

On Mon, Mar 14, 2016 at 11:12:45PM -0700, Johan Kanflo wrote:
> 3) Startup function init_networking() calls register_chipv6_phy(phy_info)
> > which reconfigures the PLL based on the crystal value stored in the
> > "wifi/phy config" sector in the flash (which is where phy_info is read
> > from). It sets them for an 80MHz APB & CPU frequency.
> >
>
> Slightly off topic. The "wifi/phy config" sector, is it the same as "SDK
> config" at 0x3fc000 (for a 32Mbit flash)? Googling makes me think that has
> not been reverse engineered yet. Is that correct?

A little bit is known about this sector.

There wasn't a dedicated page on the re wiki so I added one:
http://esp8266-re.foogod.com/wiki/Phy_info_%28IoT_RTOS_SDK_0.9.9%29

Please add anything else that is missing. :)


Angus