Choice of target SDK type/version?

[Paul Sokolovsky]

Hello,

As
http://esp8266-re.foogod.com/w/index.php/IoT_SDK#SDK_Versions_Used_Here
says, current efforts are centered about FreeRTOS SDK 0.9.9. This
surprisingly matches choice of
https://github.com/SuperHouse/esp-open-rtos project, so I wonder is the
other of esp8266-re independently came to the same conclusion regarding
usage of the above version, or just taking those arguments for granted.

Note that I personally find that choice strange. It's very unlikely
that RE effort will lead to from-scratch implementation of complete
stack required to drive esp8266. More realistic goal is to discover how
to do various useful things in cooperation with existing vendor code.
In this regard, analyzing RTOS SDK is non-rewarding, as "almost nobody"
uses it. It lacks behind in updates, and its support actually can stop
any time (because "nobody uses it").

Note that one of the problem with this RE effort is exactly moving
target. Trying to ride on top of it, instead of sticking with some old
niche version of SDK would be an interesting topic, and something new
in community RE efforts, to the best of my knowledge.


--
Best regards,
Paul mailto:pmi...@gmail.com

[Alex Stewart]

To start with, lest anyone get the wrong impression, I wrote that bit about "current efforts" mainly just because that was the version I've been working on and I wasn't aware of much other work out there based on other versions, but I certainly don't want that to imply that people aren't welcome to do work based on other versions of the SDK as well and post them on the site (and we can change that paragraph to reflect as much)..  I always kinda figured that eventually some stuff would get RE'd from newer versions at least for some of the features/fixes that weren't in the 0.9.9 SDK anyway..

As for why I chose that version for the work I've been doing thus far, it was actually a combination of several things:

1. Frankly, the licensing of the newer versions of the SDK does concern me for the same reason as the esp-open-rtos project mentioned, and it's not just about reimplemening the whole thing from scratch:  If somebody comes up with code based on an Espressif-MIT licensed library, and that code is later determined to legally be a derivative work, and if any portion of that code has been used in any kind of open source project, that open source project then becomes tainted by the proprietary aspect of the Espressif license.  (Hopefully, if we do the RE work well, such things wouldn't be found to be derivative works anyway, but one can never be sure of that, so why not minimize the risk of complications down the road?)

2. I have done a bit of pulling apart the non-FreeRTOS version of the SDK in addition to the FreeRTOS version.  From what I can tell, it looks like the two are actually almost identical except for a few minor changes to use the FreeRTOS APIs for the OS bits it provides.  Given this, I expect just about everything gleaned from the FreeRTOS SDK will be directly applicable to the other one, and at least with the FreeRTOS version, when you get down to the lowest level you encounter known APIs there too, which helps to not have to RE bits of the OS you don't need to..

3. The 0.9.9 version isn't the newest, but it does still provide a fairly complete implementation of all the base functionality, including most of the stuff needed to understand how all the bits of the hardware actually work and talk to things, so there's a lot of information that can be obtained from it already (and a lot of that doesn't appear to have changed much in newer versions of the SDK either).  Given point #1, why not use the open-licensed stuff to figure out all the basics to begin with, and then if necessary people can build on that base by looking at only the bits of the newer SDKs that actually differ from that version?

Frankly, as for whether "nobody uses it" or not, I don't see that it really matters.  It's available, it's legally unencumbered, it's fairly complete, large portions of it are basically the same as the SDK everybody does use, it's not changing out from under us, it contains tons of useful information about how the hardware works, and there's a clear path from it forward to the current stuff when we get to needing that, so it seems to me like a pretty good place to start from..

(I should be clear about my motivations, though:  I actually am interested in trying to implement a hardware-up open-source replacement for the Espressif SDK (and I don't believe that's unrealistic).  Of course, I do also believe that in the process we'll also learn a lot of useful details about the hardware and how to use it as a side-benefit..)

--Alex

--
You received this message because you are subscribed to the Google Groups "esp8266-re" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esp8266-re+...@googlegroups.com.
To post to this group, send email to esp82...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/esp8266-re/20150704154911.4717439f%40x230.
For more options, visit https://groups.google.com/d/optout.

[Paul Sokolovsky]

Hello,

On Sat, 4 Jul 2015 15:49:11 +0300
Paul Sokolovsky <pmi...@gmail.com> wrote:

[]

> Note that one of the problem with this RE effort is exactly moving
> target. Trying to ride on top of it, instead of sticking with some old
> niche version of SDK would be an interesting topic, and something new
> in community RE efforts, to the best of my knowledge.

This idea didn't receive any feedback. But going forward, there're more
and more cases where it is useful. For example, latest unnumbered patch
to 1.2.0 SDK Espressif did was on "optimizing RAM usage". What did
they do? Did they removed that gigantic static buffer in one of
wDev-related .o's, or just threw out all the bloated additions they
made since 0.9.x? Answering such questions "manually" can be quite a
chore. What's needed is a way to "diff" SDK lib by lib, object by
object.

Alex' tool works on the level of individual object files, so I
wondered if he would be interested to implement such feature. But if
not, I guess that's one of the most useful tasks I can add to my
queue.

I'd start it simple though, and that's why I'm writing - do you
guys know a tool to diff map files as produced by GNU ld? It seems
such an obvious feature and need, that it's hard to believe nobody
implemented it before, and yet all my searches were vain (and it's
actually not first time I find it would be useful). I found only
differs/reporters for various proprietary linkers, but not for GNU ld.
I'd really love to avoid reinventing the wheel, so let me know if you
stumbled upon such a tool, or other tools which let compare object
files in some ways.

[Alex Stewart]

What's needed is a way to "diff" SDK lib by lib, object by
object.

I've been thinking about this a bit..  Unfortunately, the more I think about it the harder it sounds..

I don't think you could actually do this purely at the object-code level (for starters, simply locating/identifying all of the static (non-exported) functions is pretty difficult without disassembling everything first, but more importantly, if somebody just added/removed a function or a data object it could potentially change memory offsets for all kinds of other stuff, resulting in different opcodes being produced, and potentially different binary representations for other functions whose code hasn't changed at all)..

You can actually already do some level of this sort of thing by combining xtobjdis with diff, though (using --nohex and doing a little bit of post-processing to avoid false-positives due to addresses and autogenerated label names changing, etc)..  As a quick hack, I've modified a script I made a little bit back (which I'd also been meaning to post for a while) to unpack and disassemble a whole set of SDK libs at once, and now also put together a 'sdk-diff' script which unpacks and disassembles two different sets of SDK libs and then runs them through diff in this manner.  It's not perfect, but for a really quick first pass, it's actually not a bad start for identifying what's different between two versions of things, I think..

I've put the sdk-diff script (and a few other tools I've hacked up during my RE work, including the 'unpack' script, etc) up at https://bitbucket.org/foogod/esp8266-re-tools for anybody who wants to play with it..

--Alex

--
You received this message because you are subscribed to the Google Groups "esp8266-re" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esp8266-re+...@googlegroups.com.
To post to this group, send email to esp82...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/esp8266-re/20150717021227.1343a7a2%40x230.

[Paul Sokolovsky]

Hello,

On Thu, 30 Jul 2015 12:18:42 -0700
Alex Stewart <foo...@gmail.com> wrote:

> >
> > What's needed is a way to "diff" SDK lib by lib, object by
> > object.
>
>
> I've been thinking about this a bit.. Unfortunately, the more I think
> about it the harder it sounds..
>
> I don't think you could actually do this purely at the object-code
> level (for starters, simply locating/identifying all of the static
> (non-exported) functions is pretty difficult without disassembling
> everything first, but more importantly, if somebody just
> added/removed a function or a data object it could potentially change
> memory offsets for all kinds of other stuff, resulting in different
> opcodes being produced, and potentially different binary
> representations for other functions whose code hasn't changed at
> all)..

Yes, that's why I talked about some high-level measure of differences,
using some heuristic, e.g. size of functions. I don't know how well
that would work, but common sense says that on an incrementally
elaborated project it should work. Of course, any "global" change, like
different compiler optimization switches, would throw it off, but then
probably still leaving correlation with the previous results.

Btw, I had a look at libraries supplied with 1.2.0 "RAM optimization"
patch and they feature dramatical decrease in raw object byte size.
Didn't try to figure what may been involved.

I also started writing a "map file diff" utility I mentioned, but so far
that's backlogged.

> You can actually already do some level of this sort of thing by
> combining xtobjdis with diff, though (using --nohex and doing a
> little bit of post-processing to avoid false-positives due to
> addresses and autogenerated label names changing, etc).. As a quick
> hack, I've modified a script I made a little bit back (which I'd also
> been meaning to post for a while) to unpack and disassemble a whole
> set of SDK libs at once, and now also put together a 'sdk-diff'
> script which unpacks and disassembles two different sets of SDK libs
> and then runs them through diff in this manner. It's not perfect,
> but for a really quick first pass, it's actually not a bad start for
> identifying what's different between two versions of things, I think..
>
> I've put the sdk-diff script (and a few other tools I've hacked up
> during my RE work, including the 'unpack' script, etc) up at
> https://bitbucket.org/foogod/esp8266-re-tools for anybody who wants
> to play with it..

Thanks Alex, I wanted to ask actually how you manage various lists in
wiki - it doesn't seem to be practical to do everything by hand, and I
see you have some helper scripts there, thanks for sharing, it helps
understand methodology better.

This sdk-diff script also looks interesting. I'm however backlogged
with other stuff and vacations, so it will take me some time to get
back to this and other esp8266-re matters.

>
> --Alex
>

[]

[Paul Sokolovsky]

Hello,

On Fri, 31 Jul 2015 00:00:23 +0300
Paul Sokolovsky <pmi...@gmail.com> wrote:

[]

> > I don't think you could actually do this purely at the object-code
> > level (for starters, simply locating/identifying all of the static
> > (non-exported) functions is pretty difficult without disassembling
> > everything first, but more importantly, if somebody just
> > added/removed a function or a data object it could potentially
> > change memory offsets for all kinds of other stuff, resulting in
> > different opcodes being produced, and potentially different binary
> > representations for other functions whose code hasn't changed at
> > all)..
>
> Yes, that's why I talked about some high-level measure of differences,
> using some heuristic, e.g. size of functions. I don't know how well
> that would work, but common sense says that on an incrementally
> elaborated project it should work. Of course, any "global" change,
> like different compiler optimization switches, would throw it off,
> but then probably still leaving correlation with the previous results.

Just as a quick update, radare appears to have a radiff2 tool which can
do control flow graph diffs: see
https://github.com/Maijin/Workshop2015/raw/master/slides.pdf , slide
15.