Build noise & Dev Activity

[Rob McDonald]

We use STEPCode in OpenVSP.  We build SC as an ExternalProject_Add in CMake, with GitHub Actions as a CI framework.  We're on our own branch that contains a handful of small build fixes on top of an August 2015 commit 274de2a91de.

It works for us -- but it seems like the progression of compilers since 2015 has resulted in a very noisy build -- almost 7000 lines of output in one case, many are several hundred chars wide.  This makes working with the build logs a pain -- you can't find any signal through the nose.

So, I created a MWE that simulates our OpenVSP STEPCode build system.  This is located here.  If you go there and poke around, you'll see the build output for five platforms.  The Windows builds 'fail' but it is just because there is nothing to do in the final step -- not worth triggering another build to clean that up right now.

I can update this test build to instead build a much newer SC (and if it works, I can move OpenVSP to something newer), but I have no idea which version to target.  Perhaps most of these warnings have already been cleaned up by some of the recent work.

It seems that since 2015 there have been occasional flurries of development on SC -- but I haven't seen any discussion on the mailing list or anything that would resemble a release announcement.

It would be great if some of the folks who are making changes to SC could make a brief post to this list describing their work.  What are the goals, does it work, what is the timeline, etc.  It would also be great to sweep together the desirable changes into a pile and have a plan about how to get to a next release.

Hopefully this test build / CI setup can help some of the folks make that release a bit more quiet.

Best,

Rob

[Clifford Yapp]

Hi Rob - apologies for not posting to the list, the recent flurry came out of some BRL-CAD work and so most of the initial discussions took place on their Zulip chat.

So, background:

I've made (yet another) major change to the way BRL-CAD manages third party dependencies, trying to have my cake and eat it too by using vanilla third party build systems but retaining the RPATH management and bundled install capabilities provided by CMake.  That appears to be (just barely) possible, but in the course of that work I hit some of the same issues you referenced when I tried a vanilla upstream stepcode rather than the copy in BRL-CAD's repo.  Moreover, trying to use stepcode in a "properly third party" manner rather than building it as a subbuild of our own CMake exposed a couple issues that were otherwise suppressed.

I made a series of changes that got the develop branch of stepcode (which was in better shape than master as far as being able to support BRL-CAD's converter) building more cleanly and generating C++ code that avoided some of the more verbose compiler warnings, and pushed that back upstream.  Unfortunately, I had missed some in-progress efforts by Chris Holder and my new changes ended up conflicting with his, so we rolled back to the last state of develop before I began applying my updates and we'll work forward from there more methodically to re-introduce the improvements.

Your SCTestBuild setup would actually make a good starting point for the CI testing I'd like to get in place in the main stepcode repo - I've been doing a lot of work on BRL-CAD with Github Actions and I'm planning to try and incorporate (insofar as the runners can handle it) some actual per-commit functional testing of the BRL-CAD step-g converter and hopefully some other real-world integrations so we can track breakage on an ongoing basis.

The initial (and hopefully trickiest) step of the initial part of this is a clean-up and simplification of the CMake build logic, which also involves simplification of the versioning logic and handling of the parser generator fallback files.  After that there were changes to address deprecated-copy warnings, GCC indentation warnings, and adding a NULL check (among others).  If it's not too disruptive I may retrieve or redo some work I did in a branch on BRL-CAD to eliminate the base library and make header inclusions more explicit in the files; the latter is to simplify which include directories need to be specified (and make it clearer which header is coming from where).

I did get the BRL-CAD version of stepcode to "work", but since that version as it stands isn't workable for upstream merge we don't yet have a version that can be called "working" in the broader sense - some of the bugs I hit (like the one that prompted the simplification of the versioning logic) where quirky, liked to play hide and seek, and were platform specific.  We'll want to hammer on the "final" version of this pretty hard before we are ready to call it a release.

My own goal is to get to an official stepcode release that can support BRL-CAD out of the box, to allow for "proper" installation from distro repositories and such.  Chris can tell you about his efforts on the parser/lexer rewrite - I'm not spun up enough yet on that work to do it justice.

Cheers,

Cliff

--
You received this message because you are subscribed to the Google Groups "STEPcode - Developers Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scl-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scl-dev/CAEppYpE5Yiubg66RnPg-xZMqo4-2xVz%2BBBBiLT8xDz1hdcNkuQ%40mail.gmail.com.

[Clifford Yapp]

My apologies - the new parser work is being done by Chris Horler - https://github.com/cshorler (I fumbled his last name).

[Rob McDonald]

Thanks for the reply Cliff.

I was unaware of Zulip or the BRL-CAD channel there -- I may have to check that out.

I'm thrilled to see some maintenance of SC -- and it looks like you've been going after some of my annoyances.  I look forward to some sort of a release when your and Chris' work is done.

Glad to hear the SCTestBuild might be useful -- I just grabbed what we do in OpenVSP and stripped it down as far as it would go.

A long time ago, I contributed the AP203Min example to SC -- including an odd build system that embedded an ExternalProject_add build of the entire SC library within...

It is probably worth my time to update my SCTestBuild to actually build the AP203Min example and run a test that compares the result to a reference file...  I'll see if I can do that.

We've discussed approaches to bundling dependencies in the past.  Your approaches have always been more sophisticated than mine -- if you have time at some point to write a blog post about the design and philosophy of what you're trying to do, I'm certain I would learn from it.

For OpenVSP, we have a few goals...

On MacOS and Windows, package all the dependencies with OpenVSP, try to use the project's native build system, try to link statically.

On Linux, try to lean on the system's packaging system for all dependencies, use dynamic linking for those.  For anything where that doesn't work, or a user is on a system where they can't get a library installed, fall back to our bundled version and static linking.

While STEPCode is set up like this in OpenVSP, so far I haven't had any expectation that it would ever be a system installed library managed by a distribution.  So, we always use the bundled version.  So long as it works with a FindSTEPCode.cmake script, if that status ever changes, OpenVSP will be fine with it.

We want the CMake on the application side to not care whether it was a bundled library or installed to a standard location with a packaging system.  So, we try to use CMake's Find_Package( Blah ) system -- setting a 'hint' variable just before that will lead it to find the bundled version if we're going that way.

We follow the 'SuperProject' pattern -- this lets us have the Libraries in one project and the App in another -- as well as a SuperProject that bundles both together.

Someone who just wants to build OpenVSP for a particular system can use the SuperProject - one stop shop.  A developer is more likely to build the Libraries separately from the app -- that way, they don't have to constantly re-build the Libraries as they work on the app.

If I find myself needing to develop on a Library while working on OpenVSP, I can tell the build system to use a 'System Installed' version of the library -- and then set the hint variable to point at my development version of the lib.  That way, I can hack on the library, build there (if it isn't header-only) and then re-build OpenVSP and everything works - with just that one library pulled from my 'live' dev version.

I have to admit, I've gotten this far in life not knowing that RPATH was a thing or that I might want to manage it...

Most of our CMake techniques for this stuff are vintage 2013.  I'm generally getting to the point where I need to do an overhaul and cleanup to 'Modern CMake' (while still supporting RHEL 7 users).

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/scl-dev/CAHsgyAS1AD-JeyQ-ue%2BTP9QtN4DQtaChZiDUEdF%3DROX8j6uErQ%40mail.gmail.com.

[C Y]

On Thursday, August 12, 2021 at 9:33:48 PM UTC-4 Rob wrote:

Glad to hear the SCTestBuild might be useful -- I just grabbed what we do in OpenVSP and stripped it down as far as it would go.

Looking at that example, I see you've got Win32 listed - do you still have active users needing a 32 bit Windows build?  I must confess I've not built a 32 bit Windows app in a long time now...

I initially tried the matrix setup with BRL-CAD, but found in the end that enough per-platform custom pieces were needed that it was simpler and clearer to just define what was needed for each platform individually.

The current work-in-progress Github Actions for stepcode can be seen here (not hooked up in the main repo yet, but looking promising in my test setup):

https://github.com/stepcode/stepcode/blob/93e7a135183cce0d9d985f5fe2ea4af5538ecaea/.githhub/workflows/build.yml

Towards the bottom you can see the current candidates for integration testing with BRL-CAD - I don't know if OpenVSP has a command-line based tool that could be similarly used to exercise the stepcode integration, but if so it would also be a good prospect for this type of testing.  The AP203Min example would be another one - I've not looked at that in a while.

[Rob McDonald]

On Tue, Aug 17, 2021 at 12:25 PM C Y <clif...@gmail.com> wrote:

On Thursday, August 12, 2021 at 9:33:48 PM UTC-4 Rob wrote:

Glad to hear the SCTestBuild might be useful -- I just grabbed what we do in OpenVSP and stripped it down as far as it would go.

Looking at that example, I see you've got Win32 listed - do you still have active users needing a 32 bit Windows build?  I must confess I've not built a 32 bit Windows app in a long time now...

I'm not actually sure if we have users who _need_ a Win32 build.  We're coming at it from the other end a bit....

We resisted a Win64 build for a long time (MacOS and Linux were already 64-bit).  At some point, one of our dependencies used some types that didn't directly compile on Win64 without some adjustment.  That slowed us down, but wasn't the end...

For a long time, we simply didn't need to make the additional resources available to the application and it wasn't worth the larger builds.  When we would occasionally have users who thought they needed 64-bit, we generally found that they were actually finding some sort of sub-optimal memory usage in our program and that some changes (that at the time were easier than fixing the 64-bit typing issues) were able to resolve the resource hog and solve their problems.

So, for a long time, the 32-bit build acted as a bit of a gatekeeper that helped keep us honest in terms of memory use.

Only relatively recently did we add an actual Win64 build.  I think we could probably drop Win32 now, but we aren't in a hurry to do so.

The rest of your email will take some time to digest and think about.  I will respond to that after I've had some time to check things out.

Rob