ChatGPT and SymPy

[Francesco Bonazzi]

https://en.wikipedia.org/wiki/ChatGPT

Some tested examples attached as pictures to this post. Quite impressive...

[Francesco Bonazzi]

[Francesco Bonazzi]

Not everything is perfect... ChatGPT misses the convert_to( ... ) function in sympy.physics.units, furthermore, the given code does not work:

[Francesco Bonazzi]

[gu...@uwosh.edu]

Just for some additional perspective. I have also tried this on some general chemistry word problems. In general, I see it getting basic one logical step processes correct (e.g. a single step dilution or grams -> moles). Things with multiple steps or requiring understanding the physical situation it does poorly on. That said, I think it does better than some of my weakest students. It does not seem to be able to use significant figures in computations (also a problem for my weaker students).

It seems to be improving rapidly. If it can get to reliably differentiating between correct (workable) solutions and erroneous ones, it will be more useful to most people (including my students) than searches of the internet or a cheating sight such as Chegg.

My two cents worth of opinion.

Jonathan

[Aaron Meurer]

It's actually much more sophisticated than that. I asked it to "Write code using SymPy to find the extreme values of the function x^3 + 5x^2 + 3x - 9" and it generated correct code (along with a correct explanation):

Here's the code it generated:

import sympy as sym

x = sym.symbols('x')

f = x**3 + 5*x**2 + 3*x - 9
df = sym.diff(f, x)
x_values = sym.solve(df)

print(x_values)

The URL where you can try it is at https://chat.openai.com/chat. You have to create an account first.

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a6045d80-8cdc-4df9-8cda-589c8d8c822en%40googlegroups.com.

[Aaron Meurer]

I've seen many examples on Twitter and elsewhere of chatgpt doing terribly on actual mathematics. I instead asked it to "Find the extreme values of the function x^3 + 5x^2 + 3x - 9". It got the first part with the derivative and the quadratic equation correct, but then did faulty arithmetic when plugging the values back into the expression (the correct values are 0 and -256/27). It makes a sign error and also gets the addition wrong even with the wrong sign.

My hope is that tools like SymPy can be used as oracles for tools like GPT to help them verify their mathematics. Obviously in this case you don't need SymPy to just verify arithmetic, but I've also seen it do just as poorly on symbolic algebra problems. 

I've also pasted the full log below:

To find the extreme values of a function, we need to find the critical points of the function, which are the points at which the derivative of the function is equal to 0 or does not exist.

The derivative of the function x^3 + 5x^2 + 3x - 9 is 3x^2 + 10x + 3. To find the critical points, we need to solve the equation 3x^2 + 10x + 3 = 0.

To solve this equation, we can use the quadratic formula:

x = (-b ± √(b^2 - 4ac)) / 2a

where a, b, and c are the coefficients of the quadratic equation ax^2 + bx + c = 0. In our case, a = 3, b = 10, and c = 3, so the formula becomes:

x = (-10 ± √(10^2 - 4 * 3 * 3)) / 2 * 3

= (-10 ± √(100 - 36)) / 6

= (-10 ± √(64)) / 6

= (-10 ± 8) / 6

= (-18, -2) / 6

= (-3, -1/3)

These are the critical points of the function x^3 + 5x^2 + 3x - 9.

To find the extreme values of the function, we need to evaluate the function at these critical points and at any points where the derivative does not exist. In this case, the derivative exists for all values of x, so we only need to evaluate the function at the critical points.

If we plug in the critical points into the function, we get:

x^3 + 5x^2 + 3x - 9 = (-3)^3 + 5(-3)^2 + 3(-3) - 9 = -27 + 45 + 9 - 9 = 8

and

x^3 + 5x^2 + 3x - 9 = (-1/3)^3 + 5(-1/3)^2 + 3(-1/3) - 9 = -1/27 + 5/9 + 1 - 9 = 5/9 - 1/27 - 8/9 = -64/27

The extreme values of the function x^3 + 5x^2 + 3x - 9 are 8 and -64/27.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/6af62b19-1fb0-4681-9fd2-5e5fccfcb46fn%40googlegroups.com.

[S.Y. Lee]

> My hope is that tools like SymPy can be used as oracles for tools like GPT to help them verify their mathematics.

In the most general context, "correct mathematics" can also be considered some "grammar".
So there should be some grammar between Type-0 grammar to Type-1 grammar in Chomsky hierarchy. 
In this context, a parser, or a parser with sympy oracle is the solution for such problem,
such that any other ideas to solve such problem can be isomorphic to.

However, building up such parser is off-direction for the researches of deep learning itself,
because it would need a lot of efforts by experts, to interpret the sentence generated by GPT, 
and design a phrase structure grammar for it.

I also thought about an idea that they can just tag arithmetics using SKI combinator calculus.
In this way, there is no wrong arithmetics for every random sequence of alphabets.

However, I'm not sure that if this idea is already refuted by such contemporary researchers
because it should be pretty much well-known.

[Aaron Meurer]

The trend with LLMs is much less structured. It doesn't use any formalism. It just guesses the next character of the input based on training on billions of examples.

That's why I think that tools like SymPy that are more structured can be useful. GPT can already write SymPy code pretty well, much better than it can do the actual mathematics. It may be as simple as automatically appending "and write SymPy code to verify this" to the end of a prompt whenever it involves mathematics. This sort of approach has already been proven to be able to solve university math problems (see https://www.pnas.org/doi/pdf/10.1073/pnas.2123433119, where they literally just take the input problem and prepend "use sympy" and the neural network model does the rest).

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/74847ca3-124b-414d-aa36-01eb91096310n%40googlegroups.com.

[Chris Smith]

In reviewing a PR related to units, I found ChatGPT to get correct the idea that a foot is bigger than an inch, but it said that a volt is bigger than a statvolt (see quoted GPT response [here](https://github.com/sympy/sympy/pull/24325#issuecomment-1354343306)).

/c

[S.Y. Lee]

Wolfram had recently announced the collaboration of chatGPT and wolfram alpha
ChatGPT Gets Its “Wolfram Superpowers”!—Stephen Wolfram Writings

They start to use chatgpt to generate the Wolfram code.
And it is likely to help the issues with correctness about math or science facts
because once it translates to the wolfram functions, and the wolfram function runs without error, the answer is correct.

However, the argument I'd give is that making a combination of code with 
simplify, solve, integrate are not still informative because even though they do things logically correct,
they have problem that they can't inform people how to solve the problems in details.

So I'm thinking about an idea whether language models should really
generate a code that assembles the rules used in simplify, solve.
And then it can be more human readable (or can get more human readable logic from it)
if it is like compose(solve_trig, simplify_cos, ...)
(in some pseudo sympy code)

[Jonathan Gutow]

Having played with these newer Chatbots to see what they can do for word problems of the simpler types encountered in general chemistry, I think this idea has potential. However, what I encountered was that the Chatbots could not do a good job coming up with the proper way to combine things in problems that required logical connection of multiple steps, except in the cases where the Chatbot has been trained on nearly identical exercises. One logical step they did pretty well on.

Jonathan

Dr. Jonathan Gutow

Halsey Science Room 412
Chemistry Department
UW Oshkosh
web: https://cms.gutow.uwosh.edu/Gutow
e-mail: gu...@uwosh.edu
Ph: 920-424-1326

On Mar 23, 2023, at 1:24 PM, S.Y. Lee <syle...@gmail.com> wrote:

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/1fbd88f8-3513-4e02-a576-266352f3531fn%40googlegroups.com.

[Aaron Meurer]

On Thu, Mar 23, 2023 at 12:24 PM S.Y. Lee <syle...@gmail.com> wrote:

Wolfram had recently announced the collaboration of chatGPT and wolfram alpha
ChatGPT Gets Its “Wolfram Superpowers”!—Stephen Wolfram Writings

I wouldn't call this a "collaboration". OpenAI is adding a plugin system to ChatGPT and Wolfram is one of the first plugins https://openai.com/blog/chatgpt-plugins.

If you scroll down on that page, there is also a tool that lets it execute Python code. I don't know if it has access to SymPy, but it likely does, since it seems to have access to other popular libraries like pandas and matplotlib. If anyone has access to ChatGPT Plus, could you check? 

They start to use chatgpt to generate the Wolfram code.
And it is likely to help the issues with correctness about math or science facts
because once it translates to the wolfram functions, and the wolfram function runs without error, the answer is correct.

However, the argument I'd give is that making a combination of code with 
simplify, solve, integrate are not still informative because even though they do things logically correct,
they have problem that they can't inform people how to solve the problems in details.

So I'm thinking about an idea whether language models should really
generate a code that assembles the rules used in simplify, solve.
And then it can be more human readable (or can get more human readable logic from it)
if it is like compose(solve_trig, simplify_cos, ...)
(in some pseudo sympy code)

A language model can do pretty much anything, so long as it's seen enough examples of it before.  You can often just give it examples of what you want in the prompt and it will figure it out.

ChatGPT (and other LLMs) are basically just huge pattern matching machines. That's all they are, which is why they break down whenever they have to do actual logic or reason about something they haven't seen before. So I suspect they could be used quite successfully for symbolic pattern matching, especially when that pattern matching is "fuzzy", like trying to find the best heuristic function to apply to an expression.

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/1fbd88f8-3513-4e02-a576-266352f3531fn%40googlegroups.com.

[Aaron Meurer]

Actually I didn't notice it but there's literally an example of ChatGPT generating SymPy code and executing on https://openai.com/blog/chatgpt-plugins (scroll down to where it says "code interpreter"). It's their main example of calling out to Python. 

Aaron Meurer

[David Bailey]

Dear Group,

I tried to sum a finite set of elements, each consisting of a call to an
undefined function:

import sympy
from sympy import *
k=symbols("k")
f=Function("f")
print("f(k)=",f(k))
print("sum=",sum(f(k),(0,k,10)))

This produced the following output:

f(k)= f(k)

Traceback (most recent call last):
  File "C:\SymPyProject\test.py", line 6, in <module>
    print("sum=",sum(f(k),(0,k,10)))

TypeError: 'f' object is not iterable

I would have expected to get:

f(0)+f(1)+f(2)+f(3)+f(4)+f(5)+f(6)+f(7)+f(8)+f(9)+f(10)

I'm using version 1.12rc1, but I'm 98% certain I got the same error with
1.11

Can anyone help?

David

[Chris Smith]

David, please start a new thread (and try IndexedBase instead of Function and use [] instead of ()).

WRT CHATGPT: talk with it for a while trying to get it to compute the norm of (1/2, 1, 1). When you tell it not to skip steps it can get it right, but it is hard to get it to give you the correct symbolic and decimal answer at the same time. The logic of the first part of the answer doesn't carry over into the second half. And if you get it to succeed, then ask it for the norm of (1/3, 1, 1).

/c

[David Bailey]

On 24/03/2023 02:25, Chris Smith wrote:
> David, please start a new thread (and try IndexedBase instead of
> Function and use [] instead of ()).
>

Thanks, I did consider that, but I thought that undefined functions can
simply participate in algebraic expressions in exactly the same way as
they would in manual algebra.

Is this a bug or a feature of SymPy?

David

[Chris Smith]

>>> sum(f(i) for i in range(3)) == Sum(f(i), (i,0,2)).doit() == f(0) + f(1) + f(2)

True

[David Bailey]

Thanks Chris,

What I had meant to write was Sum(f(k), (k, 0, 10)).doit() ! My main issue was to use sum instead of Sum!

David

[Aaron Meurer]

If you don't have access to the ChatGPT beta, someone made a Streamlit app that does the same thing. You can give it a math problem and it uses ChatGPT to generate and run SymPy code to solve it https://mathgpt.streamlit.app/

Aaron Meurer

[Elimboto Yohana]

ChatGPT is quite impressive! For those who do not have access to ChatGPT and/or Python, can use it with SageCell at https://tssfl.com/viewtopic.php?t=6651

SageCell supports countless Python libraries including SymPy.

I tested ChatGPT with a number of examples, it tries solve even complex maths problems, e.g., :

Example 1:

Find a 4x4 matrix which is a Jordan canonical form of the matrix A = matrix([[5, -1, 0, 0],[9, -1, 0, 0],[0, 0, 7, -2], [0, 0, 12, -3]]). 

assistant: 

To find the Jordan canonical form of a matrix, we need to find the eigenvalues and eigenvectors, and then use them to construct the Jordan blocks. 

First, we find the eigenvalues of A by solving for the determinant of (A - lambda*I), where I is the identity matrix:

|5 - lambda   -1           0           0          |
|9            -1 - lambda   0           0          |
|0             0           7 - lambda  -2          |
|0             0           12           -3 - lambda|

= (5 - lambda)(-1 - lambda)(7 - lambda)(-3 - lambda) - 9*(-1)*(7 - lambda)*(12)
= lambda^4 - 8*lambda^3 + 11*lambda^2 + 38*lambda - 360
= (lambda - 10)(lambda - 5)(lambda + 3)(lambda - 4)

Therefore, the eigenvalues are: lambda1 = 10, lambda2 = 5, lambda3 = -3, lambda4 = 4.

Next, we find the eigenvectors corresponding to each eigenvalue. For lambda1 = 10, we solve (A - 10*I)x = 0:

| -5   -1   0    0 |
|  9   -11  0    0 |
|  0   0   -3    -2 |
|  0   0   12   -13 |

By performing elementary row operations to reduce this matrix, we obtain:

| 1   0   0    1 |
|  0   1   0   3/2 |
|  0   0   1    2  |
|  0   0   0    0  |

So the eigenvector corresponding to lambda1 is x1 = [1, -3/2, -2, 1]. 

Similarly, for lambda2 = 5, we solve (A - 5*I)x = 0:

| 0   -1   0    0 |
| 9   -6  0    0 |
| 0   0   2   -2 |
| 0   0   12   -8 |

By performing elementary row operations to reduce this matrix, we obtain:

| 1  -3  0      0 |
|  0   0  1     -1 |
|  0   0  0      0 |
|  0   0  0      0 |

So the eigenvector corresponding to lambda2 is x2 = [3, 1, 0, 0]. 

For lambda3 = -3, we solve (A + 3*I)x = 0:

| 8   -1   0    0 |
|  9   -2   0    0 |
| 0   0   4   -2 |
| 0   0   12   -6 |

By performing elementary row operations to reduce this matrix, we obtain:

| 1  -1/2  0     0 |
|  0   0      1   -1/2 |
|  0   0      0     0    |
|  0   0      0     0    |

So the eigenvector corresponding to lambda3 is x3 = [1/2, 1, 1/2, 0].

Finally, for lambda4 = 4, we solve (A - 4*I)x = 0:

| 1   -1   0    0 |
| 9   -5   0    0 |
| 0   0   3   -2 |
| 0   0   12   -7 |

By performing elementary row operations to reduce this matrix, we obtain:

| 1  -1  0    0 |
|  0   0  1   -2/3 |
|  0   0  0    0    |
|  0   0  0    0    |

So the eigenvector corresponding to lambda4 is x4 = [1, 1/3, 0, 0].

We can now construct the Jordan canonical form of the matrix A by arranging these eigenvectors in blocks, as follows:

J = [10  1   0   0 ]
        [0   10  0   0 ]
        [0   0   -3  1 ]
        [0   0   0   4 ]

This is the desired 4x4 Jordan canonical form of A.

Example 2:

Find an invertible 3x3 real matrix P which puts the matrix A = matrix([[3,4,0],[-1, -3, -2],[1, 2, 1]]) into rational form. 

assistant: 

To put a matrix in rational form, we want to reduce it to row echelon form and then perform row operations to get all leading coefficients to be 1 and all other entries in the same columns to be 0. Here are the steps to do it for matrix A:

Step 1: Choose the first element in the first row to be the pivot and eliminate all entries in the column below it. To do this, subtract the first row multiplied by (-1/3) from the second row, and add the first row to the third row.

[3  4  0]
[0  1  2]
[4  6  1]

Step 2: Choose the second element in the second row to be the pivot and eliminate all entries in the column below it. To do this, subtract the second row multiplied by 2 from the third row.

[3  4  0]
[0  1  2]
[0 -2 -7]

Step 3: Choose the third element in the third row to be the pivot and eliminate all entries in the column above it. To do this, add the second row multiplied by 2 to the third row.

[3  4  0]
[0  1  2]
[0  0 -3]

Now we have the row echelon form of matrix A. To get it in rational form, we can perform further row operations to get all leading coefficients to be 1. We can achieve this by multiplying the third row by -1/3.

[3  4  0]
[0  1  2]
[0  0  1]

Now we have the rational form of matrix A. To find the invertible matrix P that puts A into this form, we need to keep track of the row operations we performed. 

First, we subtracted the first row multiplied by (-1/3) from the second row. This is equivalent to multiplying by the matrix:

[1  0  0]
[-1/3  1  0]
[0  0  1]

Next, we added the first row to the third row. This is equivalent to multiplying by the matrix:

[1  0  0]
[0  1  0]
[1  0  1]

Then, we subtracted the second row multiplied by 2 from the third row. This is equivalent to multiplying by the matrix:

[1  0   0]
[0  1   0]
[0 -2   1]

Finally, we multiplied the third row by -1/3. This is equivalent to multiplying by the matrix:

[1  0   0]
[0  1   0]
[0  0  -1/3]

To find P, we multiply these matrices in order:

P = [1  0   0] @ [-1/3  1  0] @ [1  0  0] @ [0  1  0] @ [0 -2  1] @ [1  0  0] @ [0  1  0] @ [0  0  -1/3]

Simplifying this expression gives:

P = matrix([[-1/3, -8/3, -2/3],[1/3, 5/3, 2/3],[-4/3, -2/3, 1/3]])

We can check that matrix A can be transformed to its rational form by multiplying it by P and verifying that the result is indeed the rational form:

A_rational = P.inv() @ A @ P

Output:
A_rational = matrix([[1, 0, 0],[0, 1, 0],[0, 0, 1]])

Elimboto

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6L3qy-h94G8OrTqtjrqTu9woZQpXZxaZx7mjcD1L7q32g%40mail.gmail.com.

[Chris Smith]

Use with caution -- the set-up can be right and the answer wrong as in:

What is the norm of vector (1/2, 1, 1)?

The norm of a vector is defined as the length or magnitude of the vector. The norm of a vector v = (v1, v2, ..., vn) is given by the formula:

||v|| = sqrt(v1^2 + v2^2 + ... + vn^2)

So, for the vector (1/2, 1, 1), the norm is:

|| (1/2, 1, 1) || = sqrt( (1/2)^2 + 1^2 + 1^2 ) = sqrt( 1/4 + 1 + 1 ) = sqrt( 6.25 ) = 2.5

Therefore, the norm of the vector (1/2, 1, 1) is 2.5.

The answer: `assert sqrt(sum([i**2 for i in (1/2,1,1)])) == 1.5`

[Aaron Meurer]

It's the same as the example I showed above. It basically can't do math (or really, it can't even do arithmetic). It seems to have an especially hard time with fractions. 

I asked it to generate code using Python to compute the norm or [1/2, 1, 1]. It was able to generate accurate code using pure Python, NumPy, and SymPy to compute the norm. In each case it also confidently claimed the output would be something wrong like 1.5811388300841898 or sqrt(3)/2, but if you actually copy and run the code it gives the correct result. My only gripe with the code it generated is it unnecessarily used CoordSys3D when it could just use Matrix.norm, although that's just random and if you ask it the same thing it might not do that. 

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/9fd4fc7a-32d4-4ab3-b561-f12cbb6ab331n%40googlegroups.com.

[S.Y. Lee]

The only way you always guarantee get right answer is to provide the ChatGPT two correct knowledge, but ask it to pick the better one.

For solving the math problems, I think that these sort of encoding may be helpful.

[Aaron Meurer]

Interesting update on this. I asked ChatGPT with code interpreter the exact same query. It automatically generated SymPy code and ran it to get the correct answer https://chat.openai.com/share/5be9ab93-135f-4a47-a36a-a09979e93954

Aaron Meurer

[Francesco Bonazzi]

Nice. Are there any chances that ChatGPT may add visibility to SymPy? I feel like a lot of people who could be potential users of SymPy don't know of SymPy's existence.

[Aaron Meurer]

I'm hopeful that this will. It seems to automatically go for SymPy whenever faced with a math problem, or at least one that uses symbolic math (it also uses NumPy when it can). The only downside is that the code interpreter hides the code by default, so people won't be able to see that it is using SymPy unless they click the button to show the code that ChatGPT generated. It might end up that a lot of people end up using SymPy with it but most of them never actually realize it. 

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/9d39ff2f-4e99-44da-ab47-0d6100928393n%40googlegroups.com.

[Elimboto Yohana]

It is time to integrate ChatGPT to SymPy!

Elimboto

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6KDhy%3Di3yjOR%3DHHN6vuEMXS5PP2FGEifyu3M4e-y2_2QQ%40mail.gmail.com.