Relevant to Jas's comments on my idea for AI agent safety

[Alan Karp]

from the link he sent out by mistake, https://github.com/nearai/ironclaw.

• Semantic interposition. Instead of giving the agent raw system access, all interactions go through MCP servers (filesystem, git, etc.). Every tool call passes through a policy engine that can allow, deny, or escalate to the user for approval

--------------
Alan Karp

[Mark S. Miller]

> Relevant to Jas's comments on my idea for AI agent safety

Sorry I missed it. Where do I find your idea and Jasvir's comments?

--
You received this message because you are subscribed to the Google Groups "friam" group.
To unsubscribe from this group and stop receiving emails from it, send an email to friam+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/friam/CANpA1Z3mEBKJa%2B%2ByB38fVL7Ex2dn%2BXjcNMzbLcVW_9UW-CSZzw%40mail.gmail.com.

[Alan Karp]

Well, then, you'll just have to get your priorities straight and show up, won't you?

My basic idea that I've talked about in earlier friams is pretty much what's in that bullet point. 

--------------
Alan Karp

To view this discussion visit https://groups.google.com/d/msgid/friam/CAK-_AD7arwFdk6_1bNTd76SKpT4KsrQ3dywAN-ZGjm_YXHB1JQ%40mail.gmail.com.

[Ben Laurie]

OK, and how does that policy engine work?

To view this discussion visit https://groups.google.com/d/msgid/friam/CANpA1Z3Gr_fedEuUeXvY9bCfkHx2mVi5%2BXarG5y3eOgisGf4ug%40mail.gmail.com.

[Alan Karp]

In my proposal, it does 2 things.  The proxy mediates all communication to and from the LLM agent, and it holds any secrets that the agent needs to authenticate, delegate, invoke, or revoke.  The proxy will only sign requests that satisfy your policy, sort of like a request side PDP.  It knows what data you want kept private and won't distribute it if the agent asks it to.  In a more advanced version, it manages the agent's persistent memory.

--------------
Alan Karp

To view this discussion visit https://groups.google.com/d/msgid/friam/CABrd9SS5ZZ1tPv4%3Db8n%2B%3DapTpaNGGpywzSnhK%3DFN%3DLXd_CmTMw%40mail.gmail.com.

[๏̯͡๏ Jasvir Nagra]

The challenge I was having iirc during the call was - "if I am deducing if a message is acceptable or not" just by looking at the message, we are already lost.  I felt that a separate policy that steps in to decide after the fact violated the grant vs use principle on when you do the check that always felt fundamental to me on what made ocap systems ocap systems.

-- 

Jasvir Nagra

To view this discussion visit https://groups.google.com/d/msgid/friam/CANpA1Z3z--YnCi1_tvm_eDaTfXKf4zhPoBS_P%2BzhxMpooiZ53A%40mail.gmail.com.