-
Notifications
You must be signed in to change notification settings - Fork 69
(EAI-653): Run LLM-as-a-judge evals on all rated messages #594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ssageToConversation.ts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
re: the re-run evals, doesn't seem like a big deal ultimately. That said, should we store info about evals in the conversations
collection? e.g. a flag (e.g. Message.hasEval
) or pointers to the evals in braintrust (e.g. Message.evals = ["<link to braintrust>"]
?
thats an interesting idea but i'd rather not so that the tracing/online evals can be fully non-mutative, just exists on top of the existing behavior |
Jira: https://jira.mongodb.org/browse/EAI-653
Changes
Notes