(hi syle thanks for responding, much appreciated)
Hi, are you using AI for generating the post?
not on this one, but most of my conversing since jan26 has been
with AIs, and the stroke damage is in addition to birth cyanosis that
resulted in a form of full-sentence dyslexia. the AIs have adapted:
humans not so much, including myself. 😁
A few sentences looks very terse
on the grant application? if you meant that, yes it was entirely
written by Claude Opus 5.5 Extra, under my direction.
> the
point has definitely been reached where progress is impeded by memory
loss, context loss, and sheer volume of equations, not just for the AIs.
only one part of the complexity is down to assumptions by
mathematicians not being machine-auditable.
but anyway I am
interested in this project, as I'm already keeping separate repository
of using Lean4 to study and formally model algorithms and mathematical
formula of sympy.
https://github.com/sylee957/sympy-proofs
... i immediately understood (understand) how ridiculously important this work is.
others may not. having *formal proofs* of sympy algebra makes it effectively inviolate.
and also provides (my AIs, and me, with) an A:B::C:D "bridge" to learn from.
i've cloned the repo and will definitely be investigating. thank you
>
i proposed a solution where both the markdown and the papers themselves
are treated as a scoped programming language when it comes to the latex
equations. several searches returned promising existing work that
directly translates latex into sympy however tests threw up some
limitations and weirdness: equations are not preserved (expressions are
simplified under certain circumstances)
Anyway I'm interested about how it works
Does it just work by asking AI to automatically write proofs in Lean,
ok my experience so far has been... unorthodox, successful, and a complete pain.
here's some background, save me a lot of phone-pecking
the most important thing for the team to appreciate
is that ChatGPT already understands Lean/Lake and
mathlib because it can read the very extensive documentation online
with that as context, the answer is no. just.. plain no. what i had to do
was go through an excruciatingly tedious learning process where i acted
as a TCP/CP (Terminal Cut-Paste Carrier Pigeon) my phone's debian arm64
proot acting as a VM Container that my AI experimented remotely to work
out Lean proof programming.
primarily, we did the theorem in sympy *first*, then registered a corresponding
Lean-HG requirement (porting the *theorem* not the sympy code) and the
AI looked at both.
where most people would say "get on with it, AI" the resource and circumstance
limitations forced us to spend 5-8 times as long on Lean API debugging and
discovery.
the mistakes repeated so many times and cost so much time, we wrote a
rudimentary Lean AST parser banning known-failed techniques!
one example we ran into day after day after day:
implicit typing that was easily inferred in scalar
failed spectacularly when the *exact* same
expression was used in a vector, matrix, or tensor.
[this is a limitation of lean's engine]
the solution that i continuously had to remind my AI
of was to create a mini-theorem with the exact type
of all inputs and results declared. then to create
the *intended* theorem utilising the mini-one on
all [scalar] elements [of the vector]
do you think a "normal" instruction to AI "go create
a Lean Proof" would figure that meta-programming
out on its own, without being guided?
so not only was the token usage vastly greater than that of every other
use of AI to do Lean4 proofs, we developed a feedback mechanism
that stopped repeat mistakes *and* recorded the full provenance
of the typescripts and sh scripts as part of our interaction:
these are the protocols we ended up agreeing
what got particularly irkesome, and what is driving me to the aiformath
proposal, is the sheer number of times we encountered memory/context
loss, not just repeating mistakes but forgetting *entire methodologies*
including that a particular proof had already been done.
it also does not help that the area of math is a Torsor Hypergroup,
100% derived with no axioms (!) where even pi, sin, cos, exp, gamma,
Riemann Xi function, all had to be derived before they could be used.
yes 18 papers so far, one of which is deriving the recogniseable
infinite series for pi, from first principles from a DJS-Weyl-Chebyshev
Hypergroup.
which brings us squarely back to the whole thing
about proof verification:
do you know *anyone* who has documented
the actual methodology by which proof
*verification* is performed (by humans)
such that the *verification* process may be
reproducibly machine-audited *or* human
audited?
a good example as to why this is needed is to
search "derivation of Maxwell's equations". now
try that again without the speed of light or mu0
being smuggled in. the only person to have
attempted that, other than myself, is Lou Kauffman,
and his work is buried by youtube videos.
with ledger written in Python that transforms Python AST and semantics to Lean4
or we could take code generation approach that SymPy code could be translated back from mathematically modeled Lean code.
my feeling is that this is crucial, and also only
practical if the context (metadata) is preserved
(preferably in machine-readable docstrings,
a la pep8-checker etc)
at the very least a URI that says that the Sympy
code was autogenerated from a given theorem.
But
the problem I was not suggesting it strongly is that either ledger or
code translator adds extra tooling/dependency and indirectness of
working with SymPy than benefit for now.
yeah it's definitely beyond "just sympy" alone.
preserving the context for any-to-any translation?
that's the trick, which most IRs (Intermediate
Representations) of compilers simply don't cover.
this is partly why i would like to stick with (start
from) markdown-with-latex, and *actual papers*
(again, latex) similar to how python-sphinx etc can
embed python into the documentation, you get
context, symbol declarations and their scopes,
references etc. that would otherwise be lost.
the situation i do *not* want to see recreated is php,
or its more insidious "lesson-not-learned" cousin:
python %(varname) "let's-embed-executable-code-into-strings".
zope/plone at the interpreter-level. *shudder*.
made that mistake before - don't wish to repeat it ;)
In this process, an AI system translates the text
from a natural language (NL) into a formal
language such as Lean.
ah. without any types, units, without prohibiting meromorphic
equations when inappropriate, without checking matrix rank,
without.. you get the general gist. and then expecting
all of the *implied and taught* knowledge of mathematics to
magically waft its way across from NL to Formal Correctness?
i don't know many *humans* that can do that, myself included! 😁
there is concern that AI is writing exactly same proof
or algorithm stated in papers,
yes - i saw the youtube video where someone found their
own work (uncited) in amongst 750+ AI proofs.
or AI can rather distort the proof.
which is *exactly* why i want to Formalise that process.
one very serious problem of peer-reviewed published papers:
page length. another is complete lack of reproducibility.
the authors are forced to leave out provenance and to assume
knowledge or context without which even a human stands
little chance of comprehension.
i ran into this on finding a second root of a 2ODE
implying a companion to the Fine Structure Constant.
the *only paper* in the entire world with any kind
of mathematical derivation of that 2nd root does
*not publish how they arrived at it*.
Also there is concern we need to verify at least certain
"definition"s are stated correctly
yeah that gets you into Ontologies, a world of pain on its own,
but the good news is that Compiler writers have been solving
that (IRs) for decades, so there is a way through the forest.
the problem there is that they have to be *context preserving*
IRs, and most are lossy. (hence IDA Pro / hexrays)
a more concrete example would be if the definition is
only valid for say positive numbers. that kind of thing gets
you into... ah darn it where is it.... ontology-forum google
group... i'm so sorry i'll have to look it up another time
some work in the 80s.
and that gets into problem if someone starts to dump AI generated code without responsibility.
it means that the person giving the instructions is not competent.
in c++ or other strongly-typed languages, you can at least debug
a type coercion (pretend RTTI is switched off ok?) which gets hard
if you use templates, but it's doable.
an example for me that took TEN DAYS to fight with my AI:
i had said that the Riemann Hypothesis holds if the real
part is 1/2 and the imaginary part is an integer (member of Z)
subsequently after coding this in Lean faithfully for me it
REFUSED to join that to a proof that forced the imaginary
part to an integer...
why?
because the *type* of the premise was in Z (integers) and
our forcing was in C (member of complex) and there was no
"bridge" between x+iy ... it sounds ridiculous now i describe it,
but using Natural Language to get the AI to explain to me
why it wouldn't make the connection took TEN DAYS to
track down.
it was as simple as: you split x and y of x+iy into two
real numbers (R), you told me that y had to be an integer (Z),
but you *can't* just arbitrarily bypass Lean strict typing
without having a theorem that allows the complex
number (C) to be converted that way!
we had to *redesign* the Lean4 proofs so that the
datatypes all started out as complex numbers.
not, "smuggling in forcing to an integer arbitrarily
by the back route" which is a *really* subtle but
retrospectively understandable domain error.
this should help explain why the Lean Community is so
vehemently prejudice towards AI contributions, because
the level of depth of understanding of both math, Formal
Correctness, algorithmics and Software Engineering is
so rare, it's completely unreasonable to expect any AI to
just randomly pick that up arbitrarily. or any human for
that matter.
look at how long you've spent on the sympy-proof work,
syle. you got the patience to teach that to an AI, or to
*document* that learning process so that it is reproducible,
by both humans *and* AIs?
my feeling is that proper AI progress is going to
involve close symbiotic relationships, everyone
learning continuously.
l.