CoRE insights is here!

[Aaditeshwar Seth]

Long blog: https://wpstaging.core-stack.org/from-landscape-data-to-explainable-diagnosis-neurosymbolic-evidence-cards-on-the-core-stack/

I'm pretty excited and couldn't wait for Kapil to push this to the main website, so sharing the staging link itself to this inside group :D

And apologies to Sameer for stealing the name, it was hard to resist! 

You've heard me talk for a long time about social-ecological patterns, that can we express stresses in a landscape as straightforward logical expressions, like if the aquifer is of this type AND rainfall is like this AND average water balance is so and so THEN it implies xyz. 

Well, a first cut is ready. I've been at it for a few weeks, and our students have been at it for almost a year. But the breakthrough happened only now. 

Here's where you try it out: https://act4d.iitd.ac.in/core-insights/

Demo video: https://www.youtube.com/watch?v=M-wfSXfElpA

Basically, claude is great. You give it a dump of all the CoRE stack variables in a landscape (100+ easily so far, check the variable catalog in the link above) and it would give back an analysis. Mind blowing but also very hard to follow the reasoning, learn from it, and make it transferrable. And it made us look like nobodies - we did all this hard work in getting the data together, and a super intelligent being came and ate away the top of the value chain. 

So we reversed the whole stuff. We extracted these logical expressions by asking claude to write out its reasoning. A lot of hard work after that to normalize the expressions into python evaluable expressions and fix the ton of hallucination in variable naming that claude did and even out the policy of when to declare a stress based on which expressions evaluates true, and yet not disturb claude's prose because that was indeed the intelligent outcome of its knowledge and reasoning. 

And where we are at now is that we have a purely deterministic rules engine which is easy for us humans to understand, tweak, change, and it gives an output in milliseconds, as compared to sending stuff to claude or locally running LLMs on ollama which take easily around 60sec to provide a response. And the accuracy is comparable to claude. 

Read the blog on the journey. We didn't plan it like this but a couple of other good things it led to: 

- The system acknowledges that the CoRE stack data built from external remote sensing and census variables is not complete, and asks for data from the ground to build a final assessment in case its not very sure. This is where local knowledge of the community comes in and the pathway is integrated as a first class citizen in the system. 

- There is an option to pull in an opinion from LLMs, currently wired to go to my local ollama-running workstation, which we term as keeping LLMs in the loop. Humans drive the show. 

- The frontend is currently built in this dashboard like view but we will open up the engine through APIs that can be plugged into chatbots so that community members can landscape stewards can use it to diagnose what's wrong, get a recommendation of solutions, connect with a wider working group in their area, and fire off to collectivize and fix stuff. This is Rainmatter's place-based thesis. 

- And the whole framework is now clearly laid out in front of us: Data -> Insights -> Collectivize -> Action, currently being done for micro-watersheds, but then wireable to be done for waterbodies, farms, forest stands... all entities modeled in the CoRE stack data structure. 

Check it out, especially the triaging app which lets you play around with expressions and get them to do the right thing, and the query evaluation where you can see the system's responses and also compare with claude's mind blowing responses. 

Adi

--

Aaditeshwar Seth
Microsoft Chair Professor, Computer Science and Engineering, IIT Delhi

Co-founder, Gram Vaani; Co-founder, CoRE Stack

Research | CoRE Stack | Gram Vaani | ACT4D 

Technology and (Dis)Empowerment: A Call to Technologists

Latest@GV: Facebook | Twitter | LinkedIn

[Aaditeshwar Seth]

Hi all, I'll talk about the project in this weeks dev call on Friday at 3pm. Do join, it'll be great to get your feedback, and also think about agentic methods to do this more easily next time and reduce the manual effort. 

Zoom Link: https://us02web.zoom.us/j/85657290380?pwd=0RSbT2h27FIBbfYBbIT5UPTgPjzjMV.1
Meeting ID: 856 5729 0380
Passcode: 317250

Note for the dev community: https://groups.google.com/g/core-stack-nrm/ is a different googlegroup than the dev group. Here there is more posting of environment news and papers, so you can join this too if you're interested beyond coding. 

Adi