First steps for contributing to the Dedalus project as a Software Engineer

[John Plocharczyk]

Hi there,

I heard about this project from my friend Evan Anders, a graduate student working in Prof. Ben Brown's lab, and am interested in being involved part time as a contributor. I work professionally as a software engineer and do not have much of a astronomy background so my question is: where is the best place to start for someone in my situation? Talking with Evan, it sounds like the project's tools might be a good place for me to dive in, but I am super flexible so please let me know the best opportunity for me to contribute and the next steps for doing so!

Things I know that might be useful: object oriented design, system design, bug fixing, fun variable names, machine learning, unit testing, finding the right technologies for the job, data warehousing, etc.

For a bit more in depth idea of my skill-set, please check out my LinkedIn: https://www.linkedin.com/pub/john-plocharczyk/55/954/770

Thanks,
John

[j s oishi]

Dear John,

Welcome to the Dedalus team! We're excited to have a professional software engineer contributing! We are a lot of things, but professional software engineers are not among them.

Keaton Burns is the architect and primary developer (just run hg churn!!) of the code, though we try to practice knowledge-weighted democracy as much as possible. 

I think there are a lot of interesting places for you to contribute. Let me describe two. First, we are woefully lacking in test coverage. We have no automated test framework, which is a major shortcoming. I envision both unit tests (do basic components like Chebyshev and Fourier bases still work? Does .antidifferentiate() work? Does .average() work? etc) and larger answer tests (does our 3D Rayleigh-Benard convection simulation still give bitwise identical answers after 100 timesteps?). There are loads of issues related to the latter, like "is bitwise identicality even possible? is it too strict a criterion for correctness?" 

Aside from testing, we are also very interested in developing a software as a service system, where we could have a web accessible instance of Dedalus running on AWS or some other back end. This would probably not be for large computations but would be useful for small problems and teaching. I want to use Dedalus in the classroom, but physics undergrads don't usually have the skill or inclination (or Linux/mac hardware) to install our stack.

If either of these sounds interesting, please let me know. If not, I'm sure other team members have more ideas.

Again, welcome to the team! We look forward to your contributions.

Jeff

--
You received this message because you are subscribed to the Google Groups "Dedalus Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-dev...@googlegroups.com.
To post to this group, send email to dedal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/33624ddf-5cc5-4d6e-b088-d0e7d4f774dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Brown]

All,

    Brief note re bitwise accuracy:

On Wed, Jul 1, 2015 at 6:08 AM, j s oishi <jso...@gmail.com> wrote:

<...>

(does our 3D Rayleigh-Benard convection simulation still give bitwise identical answers after 100 timesteps?). There are loads of issues related to the latter, like "is bitwise identicality even possible? is it too strict a criterion for correctness?"

This is actually feasible, though it will take some work.  I watched Bill Paxton drag the MESA community to bitwise identicality for the MESA stellar structure code.  It was a non-trivial effort and driven mostly by Bill's software development background and his demand that bugs be reproducible, rather than by rising naturally out of the community.  It required detailed testing and verification and some key restructuring changes, e.g. tapping into crlibm (http://lipforge.ens-lyon.fr/www/crlibm/) to get same floating point properties on differing architectures. 

In the end, I think it has turned out to be very positive exercise.  Among other things, models are now bitwise identical on differing architectures, and this has greatly simplified bug-reporting and squashing, never mind scientific reproducibility.

If we decide to go down this path now or in the future, I'm sure that Bill would be happy to share his experiences and insights.

Cheers,

--Ben

[j s oishi]

That's VERY interesting, Ben. The enzo team has never been able to achieve this, despite quite a bit of effort. I know Matt Turk and Sam Skillman (and probably others) spent a LOT of time chasing bitwise identicality, and were never able to get it. One potential issue affecting them is the sheer size of the codebase (certainly over 10^5 lines for enzo); Dedalus (and perhaps MESA) do not have this problem. However, we do have a lot of reliance on libraries that are going to play a role here.

j

--

You received this message because you are subscribed to the Google Groups "Dedalus Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-dev...@googlegroups.com.
To post to this group, send email to dedal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAHqBLzw2d2Asgp0rjXUNtGVuf%3D5MFWj%3DhisOpSL9PkDjNsDwbw%40mail.gmail.com.

[Daniel Lecoanet]

My impression is that MESA's codebase is similar in size to enzo's.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAAOZTN89ZcJNsR-_gZp4z1oB23gF9cwX73VzELtMZonwmURKqg%40mail.gmail.com.

[Ben Brown]

It's probably a lot more vertically integrated; in my understanding, essentially everything in MESA itself was coded by Bill.  Many people contributed ideas, but he often carefully re-did the implementation.  That may have contributed to the ability to get uniform results.

I've written to Bill to ask if my memory is right about bitwise identicality, will let you all know his response.

--Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAJoYf%3DjOgEk642m5nJjYPm%3D7wVxpivQ6tr4_YXoo8pW9HwJ4AQ%40mail.gmail.com.

[John Plocharczyk]

Hi Jeff, thanks for the warm welcome!

I think both of the areas you addressed would be a great areas for me to start focusing on. In the coming week I plan to familiarize myself further with the codebase and will look for good opportunities to develop some unit tests as well as start thinking about end to end testing The bitwise identicality testing sounds like a challenging but extremely interesting engineering problem so I will take a deeper dive on that especially. Some of the resources in this thread (the enzo and MESA projects) will be involved in my investigation. Finally, setting something up like that on AWS should be relatively painless. While I am familiarizing myself with the codebase I'll make sure to send questions your way re: the "vision" of Dedalus on AWS as they arise.

Thanks,
John

[Daniel Swid]

Hi John, 

Resurfacing this thread. Did you ever deploy on AWS?

Anyone else using AWS to run workloads? I'd love to learn more as I consider doing the same. 

Thanks!

Daniel 

[Jeffrey S. Oishi]

Hi Daniel,

I can't speak for John, but yes, Katie Manduca was able to get Dedalus to run on AWS. We haven't finished our full web-accessible service yet, but we can run on AWS. I'll let Katie chime in with the details.

j

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/5d45a10a-02c0-432c-8499-cbe3afce5cc6%40googlegroups.com.