Skip to content

(EAI-653): Run LLM-as-a-judge evals on all rated messages #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jan 23, 2025
Merged

Conversation

mongodben
Copy link
Collaborator

@mongodben mongodben commented Jan 15, 2025

Jira: https://jira.mongodb.org/browse/EAI-653

Changes

  • Run LLM-as-a-judge evals on all rated messages
  • Standardize updateTracing func for all routes

Notes

  • Note: there'll be a bit of re-running LLM-as-a-judge evals on rated messages where a message is already LLM as a judge evaluated. Given that approx 2% of messages are rated, and we're running evals on on 10% of all messages. This means that evals will be rerun on .2% of all messages (.1 * .02).

Verified

This commit was signed with the committer’s verified signature.
ziegenberg Daniel Ziegenberg

Verified

This commit was signed with the committer’s verified signature.
oliverklee Oliver Klee
…ssageToConversation.ts

Verified

This commit was signed with the committer’s verified signature.
oliverklee Oliver Klee

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@mongodben mongodben changed the base branch from main to EAI-625 January 16, 2025 21:37
Base automatically changed from EAI-625 to main January 21, 2025 16:26
@mongodben mongodben marked this pull request as ready for review January 21, 2025 17:36
Copy link
Collaborator

@nlarew nlarew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

re: the re-run evals, doesn't seem like a big deal ultimately. That said, should we store info about evals in the conversations collection? e.g. a flag (e.g. Message.hasEval) or pointers to the evals in braintrust (e.g. Message.evals = ["<link to braintrust>"]?

@mongodben
Copy link
Collaborator Author

re: the re-run evals, doesn't seem like a big deal ultimately. That said, should we store info about evals in the conversations collection? e.g. a flag (e.g. Message.hasEval) or pointers to the evals in braintrust (e.g. Message.evals = ["<link to braintrust>"]?

thats an interesting idea but i'd rather not so that the tracing/online evals can be fully non-mutative, just exists on top of the existing behavior

@mongodben mongodben merged commit 0bf7a5c into main Jan 23, 2025
1 check passed
@mongodben mongodben deleted the EAI-653 branch January 23, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants