lambdify list of arguments converted silently, why not throw a warning?

[Roberto]

I have seen that lambdify wants a list for the arguments to be treated as symbols. This list can be also give as a python set, e.g. {x,y,z }, which is exactly what would be returned by a .free_symbol property. If lambdify is feed a set like that of the output of .free_symbol it may change their order in the conversion from set to list. This is done silently and may cause major disfunction in the use of the lamdified function because you think x is x, but is y instead and so on. 

I would like to ask developers to check for the type of the lambdify list of arguments and throw a warning to flag that a set is being converted and that the ordering is not guaranteed.

Given that .free_symbols returns a set, not a list, this is very common pitfall in my opinion and must be prevented. 

[Davide Sandona']

You can wrap lambdify into a new function:

def get_lambda(expr, modules="numpy", **kwargs):
    from sympy.utilities.iterables import ordered
    signature = list(ordered(expr.free_symbols))
    return signature, lambdify(signature, expr, modules=modules, **kwargs)

With it, the argument are sorted alphabetically. You can easily see the list of arguments by printing signature.

Davide.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/c1517b79-490b-4a1b-b526-3e3842b55fbco%40googlegroups.com.

[Aaron Meurer]

Can you open an issue in the issue tracker for this? I agree that sets
should not be allowed.

Aaron Meurer

[roberto franceschini]

Hello, I have opened an issue for the input of lambdify. If you want to disallow sets as inputs I think it is fair. However, It would be nice to have that free_symbol can be given directly as input for lambdify, is that possible? should I use something else than free_symbols for this purpose?

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6KuqnjJHTObyDxK0tvaF_Frxzd-MfyQSY94v-oTUWyNAg%40mail.gmail.com.

[Oscar Benjamin]

Hi Roberto,

I already answered this on SO:
https://stackoverflow.com/questions/62798213/keep-the-order-of-parameters-fixed-in-sympy-lambdify-from-free-symbols/62800716?noredirect=1#comment111075099_62800716

If you are using symarray to generate the symbols then you can get the
list of symbols from the symarray directly and then pass those to
lambdify.

I can't think of any situation where it would be a good idea to use
free_symbols to get the arguments to lambdify.

Oscar

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAJxM9JqSoqQQkYH9UUXGyXNqRtY9VHJxKByc8Fv8%2BPaqFudP6w%40mail.gmail.com.

[roberto franceschini]

The situation is the one in which I get a sympy expression out of a function that ends in `return my_polynomial_inRn` and now I want to pass this to another function (e.g. to make plots or whatever) without having to know the names of the symbols involved, which is exactly what free_symbols does for me in this instance.

Do you see another way around this?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxSsngVE_X-aBRYWER5sESZW-e-EnjwWcfGAhM%3D%2Beq97zQ%40mail.gmail.com.

[Oscar Benjamin]

Hi Roberto,

If you have something like
expr = some_function(args)
where some_function creates new symbols (not found in args) then
ideally some_function should also return what the newly created
symbols are or some other object that can tell you them:
expr, syms = some_function(args)
I think that attempting to guess what symbols should be used in any
context by inspecting free symbols is not a good approach.

Oscar

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAJxM9Jpza%3DgRk9%3DZ2_thBS4xN_dkjCaB5PTMcz1hj8wHG_R90g%40mail.gmail.com.

[Aaron Meurer]

If you use the free symbols as arguments, the question is, what order
should they be in? As you noted, they could go in any order.

It is also not hard to do this manually. For example,
lambdify(sorted(expr.free_symbols, key=str), expr) will sort the
symbols in alphabetical order.

Another option would be to require the symbols to be passed as keyword
arguments. As long as your symbol names are valid Python variable
names, lambdify will use them for the function arguments, so you can
pass them as keyword arguments. For example:

>>> expr = x + 2*y
>>> f = lambdify(list(expr.free_symbols), expr) # list(free_symbols) could give any order
>>> f(x=1, y=2)
5

If you have a bunch of symbols, you can put them in a dictionary and
use f(**args).

Aaron Meurer

On Fri, Jul 10, 2020 at 10:47 AM Oscar Benjamin

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxQCPtBmmcjoVKv0jqYFqezsUQ5zWox_Q16x72ygng37Xg%40mail.gmail.com.

[roberto franceschini]

Not sure at all why you say so.

 Adding extra output to the function is excess baggage in my opinion. Indeed free_symbols is a great idea from the sympy developers. I understand why technically one wants to give it as a set, because after all the names are truly meant to be unique. What we need here is just a more transparent conversion to lost or a block for when set are fed to lambdify. 

I am  not very much expert with pull requests but I am working on one. I just need to figure how to import my forked version of dumpy instead of the one from pip. After that I can test my proposed change of lambdify. 

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxQCPtBmmcjoVKv0jqYFqezsUQ5zWox_Q16x72ygng37Xg%40mail.gmail.com.

[roberto franceschini]

Sure, the output of free_symbols can be easily massaged with sorted. This is how I am working now. However the issue remains that a set is silently converted into a list with no guarantee of preserving the order. Working on a pull request...

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6KepRmoA-JT5KzTB9EFx3dhnkPLyZoTqJrhsi5w%2BtWT9g%40mail.gmail.com.

[Roberto]

I am starting to wonder why python is so loose on this. I say "python" because the issue appears for enumerate as well. How can I enumerate a set? Still I am allowed to enumerate a set without receiving any warning. Look at this for instance

set(["a","b"])==set(["b","a"])

list(enumerate(set(["a","b"])))

list(enumerate(set(["b","a"])))

import sympy as sp

_a, _b = sp.symbols('a b')

set([_a,_b])==set([_b,_a])

list(enumerate(set([_a,_b])))

list(enumerate(set([_b,_a])))

Funny enough, moving this code from one notebook to another may change the order in which the sets are ordered. In all cases they are ordered in the same way within the same notebook, but you and I running the same code may get different outcomes!

I do not know if this is special with sets of sympy symbols, but I would have said that deep down enumerate should throw a warning when you try to enumerate a set.

On Thursday, July 9, 2020 at 9:55:15 PM UTC+2, Aaron Meurer wrote:

Can you open an issue in the issue tracker for this? I agree that sets
should not be allowed.

Aaron Meurer

On Thu, Jul 9, 2020 at 3:08 AM Roberto <francesch...@gmail.com> wrote:
>
> I have seen that lambdify wants a list for the arguments to be treated as symbols. This list can be also give as a python set, e.g. {x,y,z }, which is exactly what would be returned by a .free_symbol property. If lambdify is feed a set like that of the output of .free_symbol it may change their order in the conversion from set to list. This is done silently and may cause major disfunction in the use of the lamdified function because you think x is x, but is y instead and so on.
> I would like to ask developers to check for the type of the lambdify list of arguments and throw a warning to flag that a set is being converted and that the ordering is not guaranteed.
> Given that .free_symbols returns a set, not a list, this is very common pitfall in my opinion and must be prevented.
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sy...@googlegroups.com.

[Jared Claypoole]

Hi Roberto,

It looks like you're expecting python sets to do something they're not intended to do.  From the docs, "A set object is an unordered collection".  Of course when you iterate over all the elements they have to come out in some order, but that's an implementation choice.  Elements are not even guaranteed to be returned in the same order each time!  (Though probably they will be.)

I can see how one might expect enumerate to only make sense for containers which have some guaranteed order, but that's not the way it's been designed to work.  It simply iterates over any iterable and gives you an index counting how many elements have been seen so far*.

In general if you want a container with guaranteed ordering, you probably want to use a list or a tuple, or even a dict (which has had ordering guarantees since python 3.7).  You might also be interested in the answers to this stackoverflow question on implementing an OrderedSet in python.

* Actually one is able to change the starting value of this index from 0 to something else, but that's somewhat beside the point.

[roberto franceschini]

Thanks, I think this is why lambdify should not accept anything that hasn't a fixed ordering - this part of lambdify was coded too lighthearted in my opinion. So I am working on a pull request but I would like to have some discussion on what to do to make lambdify better in this sense.

At this time lambdify can give unpredictable outcome, because it accepts set and uses enumerate. That's a bug to correct in my opinion. Rightnow I have a work in progress for a pull request. I have added simple code to warn about the use of set as input, but I would like to give a more general solution to this problem. This, for instance, should warn about other types of inputs that have not fixed order. 

I am tempted to modify by adding specific type checks to warn when used with set, dict, but let list, OrderedDicts, sequence go through as usual. In all cases I'd let the function do the same as it has been doing so far.

Any comment?

To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/ff8185d8-e34b-4353-82c1-0bb9fa3dd610o%40googlegroups.com.

[roberto franceschini]

FYI

https://github.com/sympy/sympy/pull/19792