info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Automatic Differentiation
188 views
Skip to first unread message
Daniel Feenberg
Apr 25, 2022, 8:05:06 AM4/25/22
to
There are a number of hits to "fortran automatic differentiation" but the only one that claims to be available without cost is ADIFOR, from the Argonne Laboratory. On various web pages I have found 5 email addresses and 3 phone numbers. According to the web page, potential users should write to one of the addresses for access to the program itself.
I have writen to Paul Hovland (hov...@mcs.anl.gov) and Alan Carle (ca...@rice.edu) but I have received no response from either. Another page at ANL (https://www.anl.gov/partnerships/adifor-automatic-differentiation-of-fortran-77 ) gives the 2 phone numbers but neither is in service. A webpage at Rice University (http://www.crpc.rice.edu/newsletters/sum95/news.adifor.html ) adds another email address (bis...@mcs.anl.gov ), also no response there. These pages are 30 years old, so no real surprise. part...@anl.gov did respond and was referred to the same web page that proved uninformative above.
It seems like a very well documented and useful program. It seems a shame for it to die. I do have a large F77 program so I do expect it would be useful for me.
Daniel Feenberg
National Bureau of Economic Research
Cambridge MA 02138
I have writen to Paul Hovland (hov...@mcs.anl.gov) and Alan Carle (ca...@rice.edu) but I have received no response from either. Another page at ANL (https://www.anl.gov/partnerships/adifor-automatic-differentiation-of-fortran-77 ) gives the 2 phone numbers but neither is in service. A webpage at Rice University (http://www.crpc.rice.edu/newsletters/sum95/news.adifor.html ) adds another email address (bis...@mcs.anl.gov ), also no response there. These pages are 30 years old, so no real surprise. part...@anl.gov did respond and was referred to the same web page that proved uninformative above.
It seems like a very well documented and useful program. It seems a shame for it to die. I do have a large F77 program so I do expect it would be useful for me.
Daniel Feenberg
National Bureau of Economic Research
Cambridge MA 02138
Ron Shepard
Apr 25, 2022, 12:25:44 PM4/25/22
to
of research for that group. The code works also with f90+ (modules,
operators, defined types, etc.), but I do not know the current status
regarding the most recent language versions.
Chris Bischof moved to Aachen, Germany about 15 years ago, but I think
he is also still active in that area of research.
$.02 -Ron Shepard
Thomas Koenig
Apr 25, 2022, 12:34:39 PM4/25/22
to
Daniel Feenberg <feen...@gmail.com> schrieb:
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
> from the Argonne Laboratory. On various web pages I have found 5
> email addresses and 3 phone numbers. According to the web page,
> potential users should write to one of the addresses for access
> to the program itself.
What exactly do you need differentiated?
If you want fully automated Fortran-to-Fortran conversion, then
I don't have anything.
If you can massage your formulas so Maxima can accept them,
like
(%i1) display2d:false;
(%o1) false
(%i2) diff(sin(x^2),x);
(%o2) 2*x*cos(x^2)
from which you can massage the outputs into valid Fortran.
Maple (which isn't free) can write fixed-form Fortran code.
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
> from the Argonne Laboratory. On various web pages I have found 5
> email addresses and 3 phone numbers. According to the web page,
> potential users should write to one of the addresses for access
> to the program itself.
If you want fully automated Fortran-to-Fortran conversion, then
I don't have anything.
If you can massage your formulas so Maxima can accept them,
like
(%i1) display2d:false;
(%o1) false
(%i2) diff(sin(x^2),x);
(%o2) 2*x*cos(x^2)
from which you can massage the outputs into valid Fortran.
Maple (which isn't free) can write fixed-form Fortran code.
gah4
Apr 25, 2022, 3:52:46 PM4/25/22
to
On Monday, April 25, 2022 at 9:34:39 AM UTC-7, Thomas Koenig wrote:
(snip)
> What exactly do you need differentiated?
> If you want fully automated Fortran-to-Fortran conversion, then
> I don't have anything.
> If you can massage your formulas so Maxima can accept them,
> like
>
(snip)
> (%i2) diff(sin(x^2),x);
>
> (%o2) 2*x*cos(x^2)
I read just a little of the description, which mentions functions
and subroutines.
Yes there are a few ways to differentiate an expression, in Fortran or not,
and get the result. But now consider a whole Fortran function!
You might have a Fortran function with loops and IFs and all, and
desire a similar function returning the derivative of that one.
A program (or you) could go through statement by statement,
adding after each statement, a statement to evaluate the
derivative of that one. Later statements will likely need both
the value of variables, and also (from the product rule and
chain rule) the derivative.
(snip)
> What exactly do you need differentiated?
> If you want fully automated Fortran-to-Fortran conversion, then
> I don't have anything.
> If you can massage your formulas so Maxima can accept them,
> like
>
> (%i2) diff(sin(x^2),x);
>
> (%o2) 2*x*cos(x^2)
I read just a little of the description, which mentions functions
and subroutines.
Yes there are a few ways to differentiate an expression, in Fortran or not,
and get the result. But now consider a whole Fortran function!
You might have a Fortran function with loops and IFs and all, and
desire a similar function returning the derivative of that one.
A program (or you) could go through statement by statement,
adding after each statement, a statement to evaluate the
derivative of that one. Later statements will likely need both
the value of variables, and also (from the product rule and
chain rule) the derivative.
gah4
Apr 25, 2022, 4:03:44 PM4/25/22
to
On Monday, April 25, 2022 at 5:05:06 AM UTC-7, feen...@gmail.com wrote:
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
This reminds me of a system I knew years ago called Prose.
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
It seems to be mentioned here, along with the follow-on: fortranCalculus:
https://goal-driven.net/compiler-intro-&-history/history.html
Prose was developed and available in the 1970's, and it seems from
above gone by 1980. It was 1978 when I knew about it.
It is an interpreted language that includes the ability to keep track
of derivatives along with each variable, and then use them in the
included solvers, or in user-written code.
About that time, it occurred to me, since all Fortran compilers I
knew used library routines for complex multiply and divide, that
you could replace those with routines that return the value and
derivative. That is, that use the calculus product rule and
quotient rule for each operation. In that case, any expression
that you could write with +, -, *, and /, would return the value and
its derivative. I don't remember thinking about it up to **, or
built-in functions, but you could also do it for them.
In any case, the idea for Prose, and I presume fortranCalculus
(that seems the capitalization they use) is to keep derivatives
along with each variable, and use them as needed, but built
into the language.
Otherwise, I have much fun with the TI-92 calculator, which
has a derivative operator, and can generate symbolic derivatives
of expressions entered.
Thomas Koenig
Apr 25, 2022, 4:13:37 PM4/25/22
to
gah4 <ga...@u.washington.edu> schrieb:
With IFs, it is really hard - what should the derivative of
if (x > 0.1) then
foo = 1+x
else
foo = -1-x
end if
be?
Also, you usually want many derivatives for a filling a Jacobi
matrix, which should be well-behaved, and your function should at
least be continuous, or you will get into hot water with whatever
you plan to do with that matrix.
if (x > 0.1) then
foo = 1+x
else
foo = -1-x
end if
be?
Also, you usually want many derivatives for a filling a Jacobi
matrix, which should be well-behaved, and your function should at
least be continuous, or you will get into hot water with whatever
you plan to do with that matrix.
gah4
Apr 25, 2022, 4:33:51 PM4/25/22
to
On Monday, April 25, 2022 at 1:13:37 PM UTC-7, Thomas Koenig wrote:
> gah4 <ga...@u.washington.edu> schrieb:
(snip)
> > You might have a Fortran function with loops and IFs and all, and
> > desire a similar function returning the derivative of that one.
> > A program (or you) could go through statement by statement,
> > adding after each statement, a statement to evaluate the
> > derivative of that one. Later statements will likely need both
> > the value of variables, and also (from the product rule and
> > chain rule) the derivative.
> With IFs, it is really hard - what should the derivative of
> if (x > 0.1) then
> foo = 1+x
> else
> foo = -1-x
> end if
What it should do is easy, but what you do with the result, is where
it gets harder:
if (x > 0.1) then
foo = 1+x
dfoodx = 1
else
foo = -1-x
dfoodx = -1
end if
> Also, you usually want many derivatives for a filling a Jacobi
> matrix, which should be well-behaved, and your function should at
> least be continuous, or you will get into hot water with whatever
> you plan to do with that matrix.
It does help to have a good understanding of the problem you are
working in, and especially where things can go wrong.
But usually the problem is more general, and so outside the derivative problem.
I do remember doing much non-linear least-squares fitting, with the usual
Newton's method solver. You might have a function that can only be evaluated
for positive values, but it tries for negative values anyway.
Now, if it is using sqrt, the function value and its derivative will
have problems when it goes negative.
I do remember, though hand computing some derivatives to put
into least-squares fitting programs. Automating might have
been nice to have.
> gah4 <ga...@u.washington.edu> schrieb:
(snip)
> > You might have a Fortran function with loops and IFs and all, and
> > desire a similar function returning the derivative of that one.
> > A program (or you) could go through statement by statement,
> > adding after each statement, a statement to evaluate the
> > derivative of that one. Later statements will likely need both
> > the value of variables, and also (from the product rule and
> > chain rule) the derivative.
> With IFs, it is really hard - what should the derivative of
> if (x > 0.1) then
> foo = 1+x
> else
> foo = -1-x
> end if
it gets harder:
if (x > 0.1) then
foo = 1+x
else
foo = -1-x
dfoodx = -1
end if
> Also, you usually want many derivatives for a filling a Jacobi
> matrix, which should be well-behaved, and your function should at
> least be continuous, or you will get into hot water with whatever
> you plan to do with that matrix.
working in, and especially where things can go wrong.
But usually the problem is more general, and so outside the derivative problem.
I do remember doing much non-linear least-squares fitting, with the usual
Newton's method solver. You might have a function that can only be evaluated
for positive values, but it tries for negative values anyway.
Now, if it is using sqrt, the function value and its derivative will
have problems when it goes negative.
I do remember, though hand computing some derivatives to put
into least-squares fitting programs. Automating might have
been nice to have.
Thomas Koenig
Apr 25, 2022, 5:05:01 PM4/25/22
to
gah4 <ga...@u.washington.edu> schrieb:
> I do remember, though hand computing some derivatives to put
> into least-squares fitting programs. Automating might have
> been nice to have.
A few decades ago, I started using REDUCE for automatically
calculating derivatives, which I then transferred to FORTRAN to
do some calculations with equations of state, and then transferred
them to F77 using an editor.
It was nice, but the expressions became really big (naming the
program EXPAND might have been more appropriate).
A few years later, Maple sold site-wide licenses to all universities
of the state of Baden-Württemberg, and I started using that.
> I do remember, though hand computing some derivatives to put
> into least-squares fitting programs. Automating might have
> been nice to have.
calculating derivatives, which I then transferred to FORTRAN to
do some calculations with equations of state, and then transferred
them to F77 using an editor.
It was nice, but the expressions became really big (naming the
program EXPAND might have been more appropriate).
A few years later, Maple sold site-wide licenses to all universities
of the state of Baden-Württemberg, and I started using that.
David Duffy
Apr 25, 2022, 7:26:09 PM4/25/22
to
Daniel Feenberg <feen...@gmail.com> wrote:
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
> from the Argonne Laboratory. On various web pages I have found 5 email
> addresses and 3 phone numbers. According to the web page, potential
> users should write to one of the addresses for access to the program
> itself.
http://www.autodiff.org/?module=Tools&language=Fortran77
> There are a number of hits to "fortran automatic differentiation"
> but the only one that claims to be available without cost is ADIFOR,
> from the Argonne Laboratory. On various web pages I have found 5 email
> addresses and 3 phone numbers. According to the web page, potential
> users should write to one of the addresses for access to the program
> itself.
lists quite a few that still seem to be free. I last used TAPENADE,
which still offers a service where you upload your subroutine(s) that
you want to equip.
Some posters upthread were wondering why such a technology has not
been replaced by symbolic differentiation. The autodiff.org FAQ mentions
How big a function can one differentiate using AD? The largest
application to date is a 1.6 million line FEM code written in Fortran 77.
Cheers, David Duffy.
Jeff Ryman
Apr 25, 2022, 7:55:14 PM4/25/22
to
On Monday, April 25, 2022 at 5:05:06 AM UTC-7, feen...@gmail.com wrote:
Arjen Markus
Apr 26, 2022, 5:01:08 AM4/26/22
to
On Tuesday, April 26, 2022 at 1:55:14 AM UTC+2, rym wrote:
Regards,
Arjen
PS I removed the email addresses to keep Google groups happy.
> On Monday, April 25, 2022 at 5:05:06 AM UTC-7, feen wrote:
> > There are a number of hits to "fortran automatic differentiation" but the only one that claims to be available without cost is ADIFOR, from the Argonne Laboratory. On various web pages I have found 5 email addresses and 3 phone numbers. According to the web page, potential users should write to one of the addresses for access to the program itself.
> >
> > I have writen to Paul Hovland and Alan Carle but I have received no response from either. Another page at ANL (https://www.anl.gov/partnerships/adifor-automatic-differentiation-of-fortran-77 ) gives the 2 phone numbers but neither is in service. A webpage at Rice University (http://www.crpc.rice.edu/newsletters/sum95/news.adifor.html ) adds another email address, also no response there. These pages are 30 years old, so no real surprise. part at anl.gov did respond and was referred to the same web page that proved uninformative above.
> > There are a number of hits to "fortran automatic differentiation" but the only one that claims to be available without cost is ADIFOR, from the Argonne Laboratory. On various web pages I have found 5 email addresses and 3 phone numbers. According to the web page, potential users should write to one of the addresses for access to the program itself.
> >
> >
> > It seems like a very well documented and useful program. It seems a shame for it to die. I do have a large F77 program so I do expect it would be useful for me.
> >
> > Daniel Feenberg
> > National Bureau of Economic Research
> > Cambridge MA 02138
> Although it hasn't been worked on in some time there is also GRESS 3.0 from RSICC. See https://rsicc.ornl.gov/codes/psr/psr2/psr-231.html . It was only for Fortran 77. Although software with CCC- and DLC- designations have a fee, the last time I checked PSR- packages were still no cost.
If a commercial solution is somehow acceptable, then NAG may have an alternative. I have worked with it a couple of years and it does what you want. I do not know more about ADIFOR than you.
> > It seems like a very well documented and useful program. It seems a shame for it to die. I do have a large F77 program so I do expect it would be useful for me.
> >
> > Daniel Feenberg
> > National Bureau of Economic Research
> > Cambridge MA 02138
> Although it hasn't been worked on in some time there is also GRESS 3.0 from RSICC. See https://rsicc.ornl.gov/codes/psr/psr2/psr-231.html . It was only for Fortran 77. Although software with CCC- and DLC- designations have a fee, the last time I checked PSR- packages were still no cost.
Regards,
Arjen
PS I removed the email addresses to keep Google groups happy.
Daniel Feenberg
Apr 26, 2022, 8:13:29 AM4/26/22
to
On Monday, April 25, 2022 at 4:13:37 PM UTC-4, Thomas Koenig wrote:
> gah4 <ga...@u.washington.edu> schrieb:
> > On Monday, April 25, 2022 at 9:34:39 AM UTC-7, Thomas Koenig wrote:
> >
> > (snip)
> >
> >> What exactly do you need differentiated?
> >
I have a program that calculates income tax liability for any year from 1960-2023, and for any state. It is used by economists and others to estimate after tax income and prices in survey or administrative data. You can see it at http://taxsim.nber.org/taxsim35 . I have a hundred of so lines of fortran in the program to get the analytic derivative of federal tax with respect to earnings, but it would be too much work (and confusion) to do the states too, or any other types of income or deduction. There are different tax treatments for different types of income and deduction. I do use finite differences but they uncover many discontinuities, which are a problem for users.
> gah4 <ga...@u.washington.edu> schrieb:
> > On Monday, April 25, 2022 at 9:34:39 AM UTC-7, Thomas Koenig wrote:
> >
> > (snip)
> >
> >> What exactly do you need differentiated?
> >
> >> If you want fully automated Fortran-to-Fortran conversion, then
> >> I don't have anything.
> >
> >> If you can massage your formulas so Maxima can accept them,
> >> like
> >>
> > (snip)
> >
> >> (%i2) diff(sin(x^2),x);
> >>
> >> (%o2) 2*x*cos(x^2)
> >
> > I read just a little of the description, which mentions functions
> > and subroutines.
> >
> With IFs, it is really hard - what should the derivative of
>
> if (x > 0.1) then
> foo = 1+x
> else
> foo = -1-x
> end if
>
> be?
>
The AD program is another program, and when it executes the value of x would be a known constant. In my application dfoo/dx would be plus or minus one, depending on the value of x. The computer doesn't know about real numbers, but that doesn't bother me.
>
> if (x > 0.1) then
> foo = 1+x
> else
> foo = -1-x
> end if
>
> be?
>
> Also, you usually want many derivatives for a filling a Jacobi
> matrix, which should be well-behaved, and your function should at
> least be continuous, or you will get into hot water with whatever
> you plan to do with that matrix.
Daniel Feenberg
Spiros Bousbouras
Apr 26, 2022, 8:35:39 AM4/26/22
to
continuous for x = 0.1
Arjen Markus
Apr 26, 2022, 9:18:19 AM4/26/22
to
On Tuesday, April 26, 2022 at 2:35:39 PM UTC+2, Spiros Bousbouras wrote:
> Actually the derivative is not defined for x = 0.1 , foo is not even
> continuous for x = 0.1
The function is continuous and differentiable on intervals not include x = 0.1. And that is exactly how it can be treated. Mind you, you do get results that are geared to precisely the surroundings of the input So, with discontinuous or non-smooth functions, you may need to consider different régimes.
> Actually the derivative is not defined for x = 0.1 , foo is not even
> continuous for x = 0.1
Regards,
Arjen
Arjen Markus
Apr 26, 2022, 9:20:39 AM4/26/22
to
On Tuesday, April 26, 2022 at 2:13:29 PM UTC+2, feen wrote:
> I am familiar with symbolic computation programs, but that isn't what I am looking for. The whole program (excluding the user interface) is more than 20,000 lines of code. (Note for non-US readers: The US income tax system is very complicated compared to European ones, and it changes every year in every state. Sometimes it is just a parameter value that changes, but new provisions are common and old ones expire with regularity). Converting it all to Macsyma would be a chore. The documentation for ADIFOR is very clear, so I am still hoping to get it running.
> Daniel Feenberg
> I am familiar with symbolic computation programs, but that isn't what I am looking for. The whole program (excluding the user interface) is more than 20,000 lines of code. (Note for non-US readers: The US income tax system is very complicated compared to European ones, and it changes every year in every state. Sometimes it is just a parameter value that changes, but new provisions are common and old ones expire with regularity). Converting it all to Macsyma would be a chore. The documentation for ADIFOR is very clear, so I am still hoping to get it running.
So you have been able to find it? I failed to do so.
Regards,
Arjen
Ron Shepard
Apr 26, 2022, 10:45:12 AM4/26/22
to
not. And the function is discontinuous there. This is a common situation
with physical simulations, for example at phase transitions where some
physical properties are discontinuous as a function of say, temperature
or pressure. If you look at a complicated phase diagram of, say water,
with several solid phases, a liquid phase, a gas phase, and a
supercritical region, then you will see many such boundaries along
trajectories in (T,P) phase space.
This situation also occurs artificially when, for example, different
algorithms are used to evaluate a function in its domain. The
mathematical function may not have any boundary regions, but the
piecewise approximations might have them. In practice, even common
functions, e.g. trig functions such as sin(x), cos(x), etc., are often
evaluated in this piecewise way.
$.02 -Ron Shepard
gah4
Apr 26, 2022, 12:18:24 PM4/26/22
to
On Tuesday, April 26, 2022 at 7:45:12 AM UTC-7, Ron Shepard wrote:
(snip on discontinuous derivatives)
> This situation also occurs artificially when, for example, different
> algorithms are used to evaluate a function in its domain. The
> mathematical function may not have any boundary regions, but the
> piecewise approximations might have them. In practice, even common
> functions, e.g. trig functions such as sin(x), cos(x), etc., are often
> evaluated in this piecewise way.
They are. And presumably this would show up in doing numerical
derivatives using those functions, though I don't know that I have
ever seen the problem. Might not be hard to find, though.
(snip on discontinuous derivatives)
> This situation also occurs artificially when, for example, different
> algorithms are used to evaluate a function in its domain. The
> mathematical function may not have any boundary regions, but the
> piecewise approximations might have them. In practice, even common
> functions, e.g. trig functions such as sin(x), cos(x), etc., are often
> evaluated in this piecewise way.
derivatives using those functions, though I don't know that I have
ever seen the problem. Might not be hard to find, though.
gah4
Apr 26, 2022, 1:20:58 PM4/26/22
to
On Tuesday, April 26, 2022 at 5:13:29 AM UTC-7, feen...@gmail.com wrote:
(snip)
> The AD program is another program, and when it executes the value of x would
> be a known constant. In my application dfoo/dx would be plus or minus one,
> depending on the value of x.
Reading this one:
https://en.wikipedia.org/wiki/PROSE_modeling_language
reminded me of the difference between symbolic derivatives and
AD (automatic derivatives). For AD, with each mathematical operation,
and for that matter, statement, you compute the value and its derivatives.
You then propagate them with the chain rule, and other calculus rules,
step by step. Each step uses the values from the previous step.
I suspect in many cases, it will result in less redundant calculations
that you would get from the whole symbolic derivative.
If you want to (and I don't know why you would)
compute y=sin(sin(sin(x)))
and also dy/dx
then with my nearby TI-92 you get:
y=sin(sin(sin(x)))
dydx = cos(x) * cos(sin(x)) * cos(sin(sin(x)))
If instead you expand, in the way that AD might do it:
v=sin(x)
w=sin(v)
y=sin(w)
then:
v=sin(x)
dvdx = cos(x)
w=sin(v)
dwdx = cos(v) * dvdx
y=sin(w)
dydx = cos(w) * dwdx
only six function evaluations instead of nine.
Convenient to do internally in an interpreted language,
(or internally in a compiler), a little harder in Fortran source.
> The computer doesn't know about real numbers, but that doesn't bother me.
(snip)
> It does seem that almost all authors of automatic differentiation software assume
> the application is for a Jacobi matrix but that isn't my application.
At some point, the Jacobi matrix is a convenient way to write down
the derivatives. And then after that, the Hessian matrix.
But yes, the Jacobi matrix, and often Hessian matrix, are used in a variety
of optimization algorithms, and so popular in AD systems.
(snip)
> The AD program is another program, and when it executes the value of x would
> be a known constant. In my application dfoo/dx would be plus or minus one,
> depending on the value of x.
https://en.wikipedia.org/wiki/PROSE_modeling_language
reminded me of the difference between symbolic derivatives and
AD (automatic derivatives). For AD, with each mathematical operation,
and for that matter, statement, you compute the value and its derivatives.
You then propagate them with the chain rule, and other calculus rules,
step by step. Each step uses the values from the previous step.
I suspect in many cases, it will result in less redundant calculations
that you would get from the whole symbolic derivative.
If you want to (and I don't know why you would)
compute y=sin(sin(sin(x)))
and also dy/dx
then with my nearby TI-92 you get:
y=sin(sin(sin(x)))
dydx = cos(x) * cos(sin(x)) * cos(sin(sin(x)))
If instead you expand, in the way that AD might do it:
v=sin(x)
w=sin(v)
y=sin(w)
then:
v=sin(x)
dvdx = cos(x)
w=sin(v)
dwdx = cos(v) * dvdx
y=sin(w)
dydx = cos(w) * dwdx
only six function evaluations instead of nine.
Convenient to do internally in an interpreted language,
(or internally in a compiler), a little harder in Fortran source.
> The computer doesn't know about real numbers, but that doesn't bother me.
> It does seem that almost all authors of automatic differentiation software assume
> the application is for a Jacobi matrix but that isn't my application.
the derivatives. And then after that, the Hessian matrix.
But yes, the Jacobi matrix, and often Hessian matrix, are used in a variety
of optimization algorithms, and so popular in AD systems.
Thomas Koenig
Apr 26, 2022, 2:34:41 PM4/26/22
to
Ron Shepard <nos...@nowhere.org> schrieb:
typical binary floating-point type (which is why I chose it that
way).
The question is: What should automatic differentiation do
with this sort of thing?
>This is a common situation
> with physical simulations, for example at phase transitions where some
> physical properties are discontinuous as a function of say, temperature
> or pressure. If you look at a complicated phase diagram of, say water,
> with several solid phases, a liquid phase, a gas phase, and a
> supercritical region, then you will see many such boundaries along
> trajectories in (T,P) phase space.
Agreed. There is a reason for using, for example, enthalpy
and entropy as variables.
> This situation also occurs artificially when, for example, different
> algorithms are used to evaluate a function in its domain. The
> mathematical function may not have any boundary regions, but the
> piecewise approximations might have them. In practice, even common
> functions, e.g. trig functions such as sin(x), cos(x), etc., are often
> evaluated in this piecewise way.
Getting this right is _very_ hard indeed.
> On 4/26/22 8:18 AM, Arjen Markus wrote:
>> On Tuesday, April 26, 2022 at 2:35:39 PM UTC+2, Spiros Bousbouras wrote:
>>
>>> Actually the derivative is not defined for x = 0.1 , foo is not even
>>> continuous for x = 0.1
>>
>> The function is continuous and differentiable on intervals not include x = 0.1. And that is exactly how it can be treated. Mind you, you do get results that are geared to precisely the surroundings of the input So, with discontinuous or non-smooth functions, you may need to consider different régimes.
>
> The left derivative is defined at x=0.1, but the right derivative is
> not. And the function is discontinuous there.
And, of course, 0.1 is not even exactly representable in your
>> On Tuesday, April 26, 2022 at 2:35:39 PM UTC+2, Spiros Bousbouras wrote:
>>
>>> Actually the derivative is not defined for x = 0.1 , foo is not even
>>> continuous for x = 0.1
>>
>> The function is continuous and differentiable on intervals not include x = 0.1. And that is exactly how it can be treated. Mind you, you do get results that are geared to precisely the surroundings of the input So, with discontinuous or non-smooth functions, you may need to consider different régimes.
>
> The left derivative is defined at x=0.1, but the right derivative is
> not. And the function is discontinuous there.
typical binary floating-point type (which is why I chose it that
way).
The question is: What should automatic differentiation do
with this sort of thing?
>This is a common situation
> with physical simulations, for example at phase transitions where some
> physical properties are discontinuous as a function of say, temperature
> or pressure. If you look at a complicated phase diagram of, say water,
> with several solid phases, a liquid phase, a gas phase, and a
> supercritical region, then you will see many such boundaries along
> trajectories in (T,P) phase space.
and entropy as variables.
> This situation also occurs artificially when, for example, different
> algorithms are used to evaluate a function in its domain. The
> mathematical function may not have any boundary regions, but the
> piecewise approximations might have them. In practice, even common
> functions, e.g. trig functions such as sin(x), cos(x), etc., are often
> evaluated in this piecewise way.
Ian Gay
Apr 26, 2022, 4:24:20 PM4/26/22
to
>
> Now, if it is using sqrt, the function value and its derivative will
> have problems when it goes negative.
>
> I do remember, though hand computing some derivatives to put
> into least-squares fitting programs. Automating might have
> been nice to have.
*********** To reply by e-mail, make w single in address **************
Louis Krupp
Apr 29, 2022, 4:36:11 PM4/29/22
to
adi...@mcs.anl.gov in what looks like a screwed-up link.
Louis
Thomas Koenig
May 4, 2022, 1:28:31 PM5/4/22
to
Stefan Ram <r...@zedat.fu-berlin.de> schrieb:
>
> Numerically, one can easily get an approximation for anything
> as a difference quotient for a small difference, but needs
> to take care, because some results might not be correct
> (when the function changes too fast or is not differentiable
> at that point).
>
> Symbolically, one can also differentiate anything by taking
> the steps a mathematician would take for manual symbolic
> differentiation. Since FORTRAN was designed for numerical
> mathematics, writing symbolic differentiation in FORTRAN
> might be a tad more difficult than in a language like LISP
> that was made for list processing. What is more difficult
> actually is /simplifying/ the raw result of the symbolic
> differentiation to some reasonable and human-readable
> expression (in the case of large and complex expressions).
>
> Now, to your question: My first idea would be
>
> if (x > 0.1) then
> result = 1
> else
> result = -1
> end if
>
> as a first coarse approximation. (Some thoughts about
> the behavior at exactly x=0.1 might have to be added.)
There is no "exactly x=0.1" in binary floating point (and I'm not
sure that even a single Fortran compiler supports decimal floating
point, despite radix being present in selected_real_kind), which
is why I chose that particular number, to add a bit of difficulty.
However, when you differentiate, there are usually certain
assumptions, which are violated in this case - people sort of expect
that integrating what you differentiated gets the same result
(plus a constant). Since there is no way to specify the Dirac
delta function (ok, so it's a distribution) in floating point,
that's not given.
Or people my want to use it for something useful like evaluating
sensitivies, or for root-finding, or... all of that is likely
to fall down in the face of discontinuity.
> Thomas Koenig <tko...@netcologne.de> writes:
>>With IFs, it is really hard - what should the derivative of
>> if (x > 0.1) then
>> foo = 1+x
>> else
>> foo = -1-x
>> end if
>>be?
>
> There is /numeric/ and /symbolic/ differentiation.
>>With IFs, it is really hard - what should the derivative of
>> if (x > 0.1) then
>> foo = 1+x
>> else
>> foo = -1-x
>> end if
>>be?
>
>
> Numerically, one can easily get an approximation for anything
> as a difference quotient for a small difference, but needs
> to take care, because some results might not be correct
> (when the function changes too fast or is not differentiable
> at that point).
>
> Symbolically, one can also differentiate anything by taking
> the steps a mathematician would take for manual symbolic
> differentiation. Since FORTRAN was designed for numerical
> mathematics, writing symbolic differentiation in FORTRAN
> might be a tad more difficult than in a language like LISP
> that was made for list processing. What is more difficult
> actually is /simplifying/ the raw result of the symbolic
> differentiation to some reasonable and human-readable
> expression (in the case of large and complex expressions).
>
> Now, to your question: My first idea would be
>
> if (x > 0.1) then
> result = 1
> else
> result = -1
> end if
>
> as a first coarse approximation. (Some thoughts about
> the behavior at exactly x=0.1 might have to be added.)
There is no "exactly x=0.1" in binary floating point (and I'm not
sure that even a single Fortran compiler supports decimal floating
point, despite radix being present in selected_real_kind), which
is why I chose that particular number, to add a bit of difficulty.
However, when you differentiate, there are usually certain
assumptions, which are violated in this case - people sort of expect
that integrating what you differentiated gets the same result
(plus a constant). Since there is no way to specify the Dirac
delta function (ok, so it's a distribution) in floating point,
that's not given.
Or people my want to use it for something useful like evaluating
sensitivies, or for root-finding, or... all of that is likely
to fall down in the face of discontinuity.
gah4
May 5, 2022, 12:34:00 AM5/5/22
to
On Wednesday, May 4, 2022 at 10:28:31 AM UTC-7, Thomas Koenig wrote:
(snip)
(snip)
> There is no "exactly x=0.1" in binary floating point (and I'm not
> sure that even a single Fortran compiler supports decimal floating
> point, despite radix being present in selected_real_kind), which
> is why I chose that particular number, to add a bit of difficulty.
I now have an IBM Power7 machines, which (like Power6) has decimal
> sure that even a single Fortran compiler supports decimal floating
> point, despite radix being present in selected_real_kind), which
> is why I chose that particular number, to add a bit of difficulty.
floating point. I haven't gotten to install an OS yet, with AIX and Linux
being two choices, and don't know which compilers have support for it.
Ron Shepard
May 5, 2022, 2:37:07 AM5/5/22
to
whether a particular processor supports it or not.
In the original code, there was an if statement that compared to the
literal value 0.1. That literal value is translated to floating point,
and in floating point, it has a definite value. That numerical value
might be different between the two, but that doesn't matter. Just like
it doesn't matter if the value is represented in 32-bit floating point
or 46-bit floating point or 64-bit floating point, however it is
translated, it has a value. What matters is that some value exists. Then
expressions such as "x > 0.1" have a meaning given the values of x and
the values of 0.1. Those values can compare to be equal, regardless of
whether binary or floating point arithmetic is being done.
Or stated a different way, when the programmer writes "x > 0.1", he is
telling the processor to compare the value of x to another value that it
can represent, he is not telling the processor to compare x to some
hypothetical value that it might not be able represent.
As for fortran support of decimal floating point, I am curious how this
is done too. Presumably there are some KIND values for those REALs? What
happens with mixed-kind arithmetic? Is one kind converted to the other,
or is there hardware support for the mixed-kind operations?
$.02 -Ron Shepard
0 new messages