How to avoid distributing a constant factor after differentiation?

[Matthias Geier]

Hi all.

I have a simple expression:

>>> import sympy as sp
>>> a, b, t, t0 = sp.symbols('a b t t0')
>>> expr = a*(t - t0)**3 + b*(t - t0)**2

And I would like to differentiate it with respect to t:

>>> expr.diff(t)
3*a*(t - t0)**2 + b*(2*t - 2*t0)

Why is the constant "2" distributed in the second term?
It seems like an additional step that SymPy does, which doesn't really
"improve" the situation in this case.
Maybe there is a more general advantage that's just not visible in
this simple case?
But if that is so, would it be possible to tell SymPy to skip the distributing?

To be clear, this is the result I was expecting:

>>> expr.diff(t)
3*a*(t - t0)**2 + 2*b*(t - t0)

For context, this question came up in a slightly more complicated
situation: https://github.com/AudioSceneDescriptionFormat/splines/issues/31

cheers,
Matthias

[Chris Smith]

Autodistribution of Number into an Add is how SymPy works and there is no flag for differentiation (or for many functions) that would prevent it. Simply pass the expression to `factor_terms` to get it cleaned up. (But that will extract a factor of `t-t0`, too, which you might not want so you could use `Add(*[factor_terms(i) for i in expr.diff(t).args])` in this case.)

Some day autodistribution will go away and I expect that we will then ask how to get constants to distribute into simple expressions.

/c

[Aaron Meurer]

There is a distribute() context manager which lets you disable
automatic distribution, though it's not pretty:

>>> from sympy.core.parameters import distribute
>>> with distribute(False):
... print(expr.diff(t))

3*a*(t - t0)**2 + 2*b*(t - t0)

While this is less dangerous than the similar evaluate() context
manager, it is possible this could break something if you put too much
under the context.

As Chris said, we do want to eventually remove this automatic
behavior, but it hasn't been easy to do as a lot of things depend on
it currently. Rearranging things after the fact as Chris suggests is
probably the better solution. There's really no guarantees about what
the form of an expression from diff() will look like.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/389f5dd9-1498-455a-b6cc-ffbbff89a9d7n%40googlegroups.com.

[Chris Smith]

>  There is a distribute() context manager

I had forgotten about that, thanks for the reminder!

/c

[Aaron Meurer]

On Wed, Feb 21, 2024 at 4:09 PM Chris Smith <smi...@gmail.com> wrote:
>
> > There is a distribute() context manager
>
> I had forgotten about that, thanks for the reminder!

We probably shouldn't advertise it too widely. Like I said, it's not
as bad as the evaluate() context manager, but it has some of the same
fundamental issues.

The main point of it is that there is a global flag in the core to
turn the distribution off, which should make it easier to remove it if
anyone ever wants to put in that work. For instance we could make a
decorator for the tests that makes sure a specific test passes with
automatic distribution turned off (similar to this one
https://github.com/sympy/sympy/blob/ae13ee38f54aa9c8944ef7d103dda778d2a39dbd/sympy/testing/pytest.py#L291).
That way we can start fixing the code incrementally, instead of taking
an "all or nothing" approach, which has failed in the past.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/7a4673b7-4c4e-4101-a4e5-9056847d000en%40googlegroups.com.

[Chris Smith]

In order to solve the issue incrementally, it seems like we would need to identify every line where multiplication is used (on arguments that could possibly be Number and Add) and have them call `Mul` with a `distribute=True` option. 

How to identify where the multiplication lines are located? Set a booby trap in the `flatten` routine that requires the `distribute` flag to be set. Repeatedly run the tests to identify locations and add the `distribute=False`. At that point all tests are unchanged and passing. 

Now, one by one, change a distribute flag to True, run tests, fix tests or code to handle the non-distribution since either could cause a failure: the former because something previously distributed would no longer be so, and the latter because expectations for the algorithm were not met and the algorithm (and maybe the test) need modification.

/c

[Aaron Meurer]

I don't think we need to try to check the code for everywhere a
multiplication happens. We can rely on the tests sufficiently covering
the code.

So what you would do is set the global distribute flag to False, run
the tests, and see what fails. Then as you fix certain tests that are
broken, add a @both_distribute decorator to the test to make sure it
always passes with distribution both off and on.

The downside of this incremental approach is that functions would need
to work both with and without automatic distribution. In many cases,
though, this is something that should happen anyway, as even currently
automatic distribution is not guaranteed. For example, it doesn't
happen with 2*x*(x + y). And it's not uncommon to have un-distributed
expressions because functions like factor() create them.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/cd7ba971-b424-4c23-b61a-a976aebd3020n%40googlegroups.com.

[Chris Smith]

It seems that the tests could be marked as passing for both forms to see where tests fail because a routine is broken. Broken routines could be fixed (as necessary) so they are not assuming distribution. Would the decorator need to be taken off routines once everything was passing?

Except for a single Add from which only a Rational can be subtracted, I suspect that detecting a passing form for a given result would be more tricky than one might expect.

/c

[Aaron Meurer]

On Mon, Feb 26, 2024 at 1:54 PM Chris Smith <smi...@gmail.com> wrote:
>
> It seems that the tests could be marked as passing for both forms to see where tests fail because a routine is broken. Broken routines could be fixed (as necessary) so they are not assuming distribution. Would the decorator need to be taken off routines once everything was passing?

Ideally once we got everything working we would flip the flag default,
and then we could delete the decorator.

The one issue is that it could be more complicated if we want to do
some sort of deprecation, though I'm not sure if we can. If there's a
lot of library code that breaks when you disable automatic
distribution then there will be a lot of user code that breaks as
well.

>
> Except for a single Add from which only a Rational can be subtracted, I suspect that detecting a passing form for a given result would be more tricky than one might expect.

I might not be following exactly what you are saying here, but
wouldn't something like

assert f() == 2*(x + y)

work regardless of whether distribution is enabled or not? It will
either happen to both sides of the equation or to neither side.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/4ef9a536-10c9-42b4-84e0-9b20dd609334n%40googlegroups.com.

[Chris Smith]

>  I might not be following

Say a current test is `assert ans == 2*x + 2*y`. That isn't going to pass when distribution is off. I was imagining that there was some way you could write a decorator that would be able to convert that to `assert f(ans) == f(2*x+2*y)` where `f` is something like `factor_terms`. I'm not sure there is a way to write such a decorator, however.

What we can do, incrementally, is: pick a test function; decorate it with a function that runs the tests in a `distribute(False)` context; fix failing tests. Then repeat that process until 

a) all functions are thusly decorated and 

b) all routines that are identified as failing (because they made an assumption about distribution) are fixed.

Make distribution default to False.

Is that what you had in mind?

/c

[Aaron Meurer]

On Wed, Feb 28, 2024 at 2:23 PM Chris Smith <smi...@gmail.com> wrote:
>
> > I might not be following
>
> Say a current test is `assert ans == 2*x + 2*y`. That isn't going to pass when distribution is off. I was imagining that there was some way you could write a decorator that would be able to convert that to `assert f(ans) == f(2*x+2*y)` where `f` is something like `factor_terms`. I'm not sure there is a way to write such a decorator, however.

But if you instead write 'assert ans == 2*(x + y)' it should work,
because the right-hand side will either remain unevaluated or
distribute, according to whatever the distribute flag is set to.

>
> What we can do, incrementally, is: pick a test function; decorate it with a function that runs the tests in a `distribute(False)` context; fix failing tests. Then repeat that process until
>
> a) all functions are thusly decorated and
> b) all routines that are identified as failing (because they made an assumption about distribution) are fixed.
>
> Make distribution default to False.
>
> Is that what you had in mind?

Yes, except the decorator has to run the test twice, once with
distribute(False) and once with distribute(True) (because the current
behavior also has to continue to work and be tested). This is how the
_both_exp_pow decorator works
https://github.com/sympy/sympy/blob/ab2fb691a90457b65bbf2a7c091c8265be9cee09/sympy/testing/pytest.py#L291.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/2a90c6c6-7d23-468a-a659-f196945c11e4n%40googlegroups.com.