You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As of now, Haystack's evaluators which extend LLMEvaluator only support OpenAI. I would like for support through llama.cpp to be added for local/offline/"free" evaluation.
Describe the solution you'd like
After implementing LlamaCppChatGenerator (deepset-ai/haystack-core-integrations#723), it is possible to constrain the output to json. As such, we can split up the instructions of the evaluator into ChatMessages (system for instructions/examples, user for input tuples) and then have the result be in the json format expected.
I have a WIP implementation, and I would like feedback on how to handle the integration of llama.cpp into the existing evaluators.
As of now, I am manually calling the Chat Generator in llm_evaluator to ensure it would be possible. I am hard coding the model and generation kwargs.
My two different ideas to resolve this are as follows:
Allow for passing in of a generator through evaluators and handle logic differently depending on whether api = "openai" or "llama.cpp".
Allow for passing in model_kwargs and generation_kwargs, discard model kwargs when api = "openai", create LlamaCppChatGenerator with kwargs in llm_evaluator using both kwargs when api = "llama.cpp"
Another idea but I don't think it would be ideal is to just have separate evaluator components for llama.cpp.
If I can have some feedback, I can submit a PR with revised changes in a couple of days.
Describe alternatives you've considered
N/A Additional context
N/A
The text was updated successfully, but these errors were encountered:
Thanks for your help with this! Instead of passing generator instances, we can just expand the current approach in the following manner:
Expose an api_params: Optional[Dict[str, Any]] = None init parameter in LlmEvaluator.
Add llama_cpp as a new API and use a LlamaCppChatGenerator as a backend when it's chosen.
Allow users to pass the required API-specific parameters via api_params and, internally, conditionally handle the contents of that dictionary based on the chosen API.
Is your feature request related to a problem? Please describe.
As of now, Haystack's evaluators which extend LLMEvaluator only support OpenAI. I would like for support through llama.cpp to be added for local/offline/"free" evaluation.
Describe the solution you'd like
After implementing LlamaCppChatGenerator (deepset-ai/haystack-core-integrations#723), it is possible to constrain the output to json. As such, we can split up the instructions of the evaluator into ChatMessages (system for instructions/examples, user for input tuples) and then have the result be in the json format expected.
I have a WIP implementation, and I would like feedback on how to handle the integration of llama.cpp into the existing evaluators.
As of now, I am manually calling the Chat Generator in llm_evaluator to ensure it would be possible. I am hard coding the model and generation kwargs.
My two different ideas to resolve this are as follows:
Another idea but I don't think it would be ideal is to just have separate evaluator components for llama.cpp.
If I can have some feedback, I can submit a PR with revised changes in a couple of days.
Describe alternatives you've considered
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered: