You will probably like varlock - it helps get your keys out of plaintext, while giving your agents a schema and additional tools so it can interact with env vars safely. The next step is injecting your keys via proxy, but just varlock is a huge improvement as a first step. Generally provides a ton of quality of live improvements as well, whether working solo or on a team.
if a agent has the keys in the same process, it can easily extract them