You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Previously, when some function running after this setup, the function_setup utility converts both success and failure callbacks to the actual callback object MlflowLogger.
# Execute some function
litellm.completion(model=...)
print(litellm.success_callback) # Output: [<litellm.integrations.mlflow.MlflowLogger object at 0x11e422700>]
print(litellm.failure_callback) # Output: ['mlflow', <litellm.integrations.mlflow.MlflowLogger at 0x3256ad310>]
However, in the latest version (1.59.7), only success callbacks are converted to the callback objects.
# Execute some function
litellm.completion(model=...)
print(litellm.success_callback) # Output: [<litellm.integrations.mlflow.MlflowLogger object at 0x11e422700>]
print(litellm.failure_callback) # Output: ["mlflow"]
Before this PR, the function_setup function does this conversion for both handlers. The PR moves this logic out to the _add_custom_logger_callback_to_specific_event function, but it is only invoked for success handler (event) now. As a result, the conversion does not happen for failure handlers.
This makes failure handlers not executed. The event handler in litellm_logging.py only trigger callback when it is CustomLogger instance, not a string. Therefore, the failure callbacks stored as a string will be ignored.
Relevant log output
Are you a ML Ops Team?
Yes
What LiteLLM version are you on ?
v1.59.7
Twitter / LinkedIn details
No response
The text was updated successfully, but these errors were encountered:
Previously, when some function running after this setup, the function_setup utility converts both success and failure callbacks to the actual callback object MlflowLogger.
Oh i see, since we stopped 'auto' adding any customlogger success callback as a failure event as well, it exposed the issue and we didn't have good testing to catch that.
Thank you for your investigation on this @B-Step62
What happened?
Problem
In the latest version, failure callbacks added as strings are ignored.
Reproducing code
pip install mlflow
(Potential) Root Cause
#7905 changed the handling for custom callbacks when it is added as a string.
Previously, when some function running after this setup, the
function_setup
utility converts both success and failure callbacks to the actual callback objectMlflowLogger
.However, in the latest version (1.59.7), only success callbacks are converted to the callback objects.
Before this PR, the
function_setup
function does this conversion for both handlers. The PR moves this logic out to the_add_custom_logger_callback_to_specific_event
function, but it is only invoked for success handler (event) now. As a result, the conversion does not happen for failure handlers.This makes failure handlers not executed. The event handler in litellm_logging.py only trigger callback when it is
CustomLogger
instance, not a string. Therefore, the failure callbacks stored as a string will be ignored.Relevant log output
Are you a ML Ops Team?
Yes
What LiteLLM version are you on ?
v1.59.7
Twitter / LinkedIn details
No response
The text was updated successfully, but these errors were encountered: