Fix triton streaming completions bug

[minwhoo]

Title

Reimplements triton streaming handling code that got lost during refactoring.

• For streaming, the api_base url should be appended with "_stream" suffix.
• get_model_response_iterator method returns TritonResponseIterator

Relevant issues

Fixes #8362

Type

🐛 Bug Fix

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 13, 2025 7:42am

[ishaan-jaff]

pls add a test in llm translation

[ishaan-jaff]

bump on this @minwhoo

[minwhoo]

Added streaming test

[krrishdholakia]

Thanks @minwhoo can you share a screenshot of it passing.

Should be good to merge once that's up! 👍

Appreciate your work on this

[minwhoo]

@krrishdholakia Sure thing. Managed to catch a mistake I made. Would this screenshot suffice?

[ishaan-jaff]

LGTM

[ishaan-jaff]

Hi @minwhoo would you be open to working with us on LiteLLM as a founding engineer ? (I'm impressed by your work on this)

If this sounds interesting here's my cal, please book some time: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[minwhoo]

@ishaan-jaff Hey, thanks for the offer—really appreciate it! But at the moment I ’m unable to take on any new commitments. Really great product though. Best of luck on your startup!