This looks like a security nightmare in case someone decides to publish this interface publicly. Prompt injection to exfiltrate sensitive Information being on the top of the list.
You're right. For now, it's only local. For a public deployment, the idea is to have sandboxes and verification steps. That won't completely eliminate the risk of prompt injection, but so far no solution has managed to completely resolve this problem.