Breakout: Best practices for documenting scientific software development

[MikeJ888]

(apologies, wrong subject line last time :-( )

Best practices for documenting scientific software development

Name of Chair: Aleksandra Pawlik
Name of Scribe: Mike Jackson

What are the five most important things learnt during this discussion:

1. There is not just reader of documentation, each have their own
requirements e.g. users versus developers. Users may start as black
box users but soon want to understand the internals and tweak.
2. Papers describe how it works, software developer manuals describe
what it does, neither are the same as describing how to use it.
3. A 2 page, 10 minute quick start guide is essential which provides
the initial uptake - without even that no uptake.
4. Dynamic approach to documentation e.g. archived mailing lists and
issue trackers, generate documentation on demand.
5. Bad documentation can deter users. Good documentation can persuade
users to make the effort because you have.

What are the problems, and are there solutions?

Some want black box, some want internals e.g. mathematical algorithm.
Two sets of documentation needed for users and user-developers?
Limited time so which do you focus on? 10 minute quick start guide.
Are researcher-developers happy to spend time writing doc at all?
Change culture to make writing doc more of a coding task?
Are manuals or papers enough to understand? Sometimes need to read
papers and contact authors.
System architecture diagrams can help for non-techies.
Ideally, "User (representative) writes the manual"
Often a lack of documentation about _every_ feature and function.
For pure users, embed help in the software e.g. computer games.
Researchers often assumed to have certain level of technical
expertise. State assumed level of knowledge in documentation. Reader
understanding of terms may differ from developer understanding.
Bad documentation can deter potential users. Good, quality,
documentation and its effort can be rewarded by user effort to
download and use it.
Third-party usage experiences. Are they of value? Yes to developers -
point out doc limitations, a tutorial or example of use. Users may
show how software works, but all their assumptions e.g. data cleaning
and the like. Useful to share experiences of how software has been
used by the users themselves. Where to get this? Papers. Wiki page?
Mailing lists or GoogleGroups? Blogs? Comment on online wiki page
doc,
like MySQL.
Documentation standards - checklists of areas to be covered, ... - but
nothing specifically for scientific software. It is an art as much as
a science!
Research on task-oriented manuals, allowing users to explore and make
mistakes but to recognise errors and return to known states.

What further work could be done, and who should do it?

-Identify good examples of best practice in documentation by asking
the researchers what they think are well and poorly documented
software.
-Recommend journal referees look at software doc as well as the
software itself.
-Alternatively, insist that repository managers insist on this
when software is deposited (e.g. at least a start-up guide)

Are there any useful resources that people should know about?

Free software to record interactions e.g. Camtasia.
http://www.techsmith.com/camtasia2-1111.html
Camstudio, http://camstudio.org
Blogs
http://informatics.northwestern.edu/blog/nubic-dev-2/2012/01/why-rtfm-doesnt-work/
http://www.imaginaryplanet.net/weblogs/idiotprogrammer/2007/10/why-users-dont-read-documentation-technical-writing-secrets/