Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MEAI.Evaluation libraries #5873

Merged
merged 41 commits into from
Feb 13, 2025
Merged

Conversation

shyamnamboodiripad
Copy link
Contributor

@shyamnamboodiripad shyamnamboodiripad commented Feb 11, 2025

Microsoft Reviewers: Open in CodeFlow

@shyamnamboodiripad
Copy link
Contributor Author

shyamnamboodiripad commented Feb 11, 2025

@RussKie RussKie added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Feb 11, 2025
@dotnet-comment-bot
Copy link
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.AI.Evaluation.Reporting Line 88 73.56 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Branch 88 64.8 🔻
Microsoft.Extensions.AI.Evaluation.Quality Line 88 6.44 🔻
Microsoft.Extensions.AI.Evaluation.Quality Branch 88 13.85 🔻
Microsoft.Extensions.AI.Evaluation Line 88 61.54 🔻
Microsoft.Extensions.AI.Evaluation Branch 88 64.29 🔻
Microsoft.Extensions.AI.OpenAI Line 77 71.71 🔻
Microsoft.Extensions.AI.OpenAI Branch 77 53.28 🔻
Microsoft.Extensions.AI.Ollama Line 80 78.2 🔻
Microsoft.Gen.MetadataExtractor Line 98 57.35 🔻
Microsoft.Gen.MetadataExtractor Branch 98 62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.Abstractions 83 85
Microsoft.Extensions.AI.AzureAIInference 91 92
Microsoft.Extensions.AI 88 89
Microsoft.Extensions.Caching.Hybrid 86 87

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=947607&view=codecoverage-tab

@stephentoub
Copy link
Member

I rebased this on main after merging API changes, and added a commit here that adapts to those changes (just a few tweaks needed).

@dotnet-comment-bot
Copy link
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.AI.Ollama Line 80 78.2 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Line 88 72.06 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Branch 88 64.8 🔻
Microsoft.Extensions.AI.Evaluation.Quality Line 88 7.57 🔻
Microsoft.Extensions.AI.Evaluation.Quality Branch 88 16.42 🔻
Microsoft.Extensions.AI.Evaluation Line 88 58.67 🔻
Microsoft.Extensions.AI.Evaluation Branch 88 56.67 🔻
Microsoft.Extensions.Caching.Hybrid Line 86 82.92 🔻
Microsoft.Extensions.AI.OpenAI Line 77 68.75 🔻
Microsoft.Extensions.AI.OpenAI Branch 77 50.41 🔻
Microsoft.Gen.MetadataExtractor Line 98 57.35 🔻
Microsoft.Gen.MetadataExtractor Branch 98 62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.AzureAIInference 91 92
Microsoft.Extensions.AI.Abstractions 83 85
Microsoft.Extensions.AI 88 89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=950078&view=codecoverage-tab

Copy link
Member

@peterwald peterwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are items we need to follow up on, but nothing that should block at this point in my opinion.

@shyamnamboodiripad shyamnamboodiripad enabled auto-merge (squash) February 12, 2025 23:01
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 12, 2025
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 12, 2025
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 12, 2025
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 12, 2025
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 12, 2025
@dotnet dotnet deleted a comment from dotnet-comment-bot Feb 13, 2025
@shyamnamboodiripad shyamnamboodiripad merged commit 89b0d7e into dotnet:main Feb 13, 2025
6 checks passed
Comment on lines +29 to +30
// To avoid infinite loops, ignore exceptions that were already seen.
if (seen.Add(current))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you get an infinite loop anyway? This method only loops for AggregateException.InnerExceptions, and I don't see how that could ever return a collection that includes the same AggregateException:

  • AggregateException.InnerExceptions returns ReadOnlyCollection<Exception> so one cannot use that to modify the InnerExceptions collection of an existing AggregateException.
  • AggregateException.InnerExceptions is not virtual so a class derived from AggregateException cannot override InnerExceptions to return something circular.
  • The AggregateException constructor copies the list of inner exceptions so AggregateException.InnerExceptions won't be affected if the caller later modifies the original list.
  • C# does not allow the following so you cannot pass the AggregateException reference to its own constructor.
    using System;
    
    public class MaliciousException : AggregateException {
        public MaliciousException()
            : base(innerExceptions: [this]) // error CS0027: Keyword 'this' is not available in the current context
        {
        }
    }

Exponential complexity would be possible though, if there are m layers of n AggregateException instances and each exception in layer k has all exceptions in layer k + 1 as its inner exceptions. So the HashSet will in theory protect against that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-ai-eval Microsoft.Extensions.AI.Evaluation and related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants