Aim Security post-call guardrails support #8356
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
hxtomer
force-pushed
the
feature/aim-post-call
branch
from
3e82516 to
35a7bc3
Compare
|
Hi @ishaan-jaff, just wanted to check in if you've had a chance to look at this yet. Thanks! |
krrishdholakia
changed the base branch from
main
to
litellm_dev_03_12_2025_contributor_prs_p2
|
Hey @hxtomer your branch has merge conflicts with the dev branch - can you please resolve and let me know? Happy to help merge this in |
hxtomer
force-pushed
the
feature/aim-post-call
branch
from
35a7bc3 to
6ebf11e
Compare
|
Hey @krrishdholakia, thanks for putting the effort on this. I fixed the conflicts and avoided the perf issue by whether each guardrail actually needs to register |
| @@ -21,6 +21,7 @@ Documentation = "https://docs.litellm.ai" | |||
| [tool.poetry.dependencies] | |||
| python = ">=3.8.1,<4.0, !=3.9.7" | |||
| httpx = ">=0.23.0" | |||
| websockets = "^13.1.0" | |||
There was a problem hiding this comment.
hey @hxtomer this is not needed. your file is a proxy guardrail. This would make it a dep on the basic pip package.
add it under proxy= on line 56
krrishdholakia
merged commit a2414d0
into
BerriAI:litellm_dev_03_12_2025_contributor_prs_p2
Title
This PR adds support for post-call Aim Security guardrails. Aim guards can analyze LLM output in streaming mode using websockets, in order to ensure a smooth user experience while securing the streamed content with increasing context rather than processing it chunk by chunk.
Type
🆕 New Feature
Changes
I've added a new CustomGuardrail hook that allows Guardrails to consume the response stream as an iterator instead of chunk by chunk. An example usage is in
aim.py.[REQUIRED] Testing - Attach a screenshot of any new tests passing locally
If UI changes, send a screenshot/GIF of working UI fixes
Tested on python 3.9 - 3.12 (3.8 seems to have some dependency issues at the moment).
Thanks in advance for your review :)
Tomer