Increasingly slow feedback loop for developers, increasingly large WAR files

[Philip Durbin]

Dear Dataverse developers and sysadmins,

Last week in tech hours (Tuesdays at 3) I discussed my concerns about how developer productivity is getting worse because the feedback loop is getting slower and slower. Perhaps you've seen the XKCD cartoon where the developers are having a sword fight while they wait for their code to compile. For us, the feedback loop requires getting changes to the code into Glassfish and waiting the new behavior to be ready to test.

I wrote up my concerns into an issue and I would love to have input in the form of comments or replies on this thread about what you're thinking, what your experiences have been in other code bases (I gave a PHP example): https://github.com/IQSS/dataverse/issues/5593

At this point, I'm looking for confirmation and validation that I've identified an actual problem that's getting worse. I want Dataverse to be a pleasure to hack on.

Toward that end, you'll find in that issue a link to a spreadsheet where I would like as many of us as possible to record the results from a simple test. For me the test took 36 seconds. It goes something like this:

- Make sure http://localhost:8000/api/info/version is working.

- Get your stopwatch ready.

- Make a change to the code (add foo:bar to the output or whatever, the file is Info.java). Save the change. Start the deployment process. Start your stopwatch.

- Keep hitting the http://localhost:8000/api/info/version endpoint with curl until you can see your new output. Consider watching all the activity in server.log while you wait.

- Once http://localhost:8000/api/info/version starts responding with your change, stop your stopwatch and either record the time in seconds in the spreadsheet (request edit access) or reply here with your results. Please note the commit you are on as well.

Again, for me, the process above took 36 seconds. I find this to be rather slow (especially compared to the instant feedback you can get with PHP). I'm curious what others experience.

Assuming we agree that we do have a problem on our hands, the next phase would be to identify and remediate root causes of the problem. Ideas for how to achieve this are very welcome and should be added to the issue as comment or added to this thread as a reply, or both. I'm also happy to chat about this stuff in real time at http://chat.dataverse.org (#dataverse on freenode).

If you have any questions about any of this, please do not hesitate to ask. Thanks!

Happy hacking,

Phil

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Leonid Andreev]

https://xkcd.com/303/