moved to Stan-dev: ODEs and non-observed compartements
Sebastian Weber
The case of having only very few observed states is not so special. Indeed it is the most common one and with the current design Stan cannot solve a large class of problems where it would be very interesting to use. The reason is that oftentimes literature knowledge is available whenever you can write down a large ODE system. The presentation from Thierry shows such a case:
http://page-meeting.org/pdf_assets/5612-PAGE2015_TW.pdf
He had to use Nonmem for his problem which includes N=16 ODE equations and M=15 parameters. So Stan currently calculates 512 ODEs, but since only one compartment is observed you only need to calculate 47! So a reduction in the size of the ODE system by a factor of >10 which is a huge speedup. Ok, this may be an extreme example, but many real-world problems are like this, i.e. many compartments are never observed, they are just created to model a physical process.
Frederic may confirm this case and relevance for practical applications, I guess.
To solve this problem would need to somehow tell the integrate_ode function which sensitivities to track from which compartment - if it is ok to let users specify that, then this appears to me easily to be addressed. The utility would be immense.
Concerning (stiff) ODE solvers: What Stan urgently needs is just a different ODE solver than the boost one! I.e. the cvode non-stiff solver runs very fast on problems where the boost solver just freezes. I suppose other non-stiff solvers (which don't increase precision indefinetly on a bad luck draw like the boost one) will do as well a good job.
Sebastian
@ Michael: If you have some math written up concerning NUTS and ODEs, I would appreciate a copy of that