Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Kernel function can only be called with primitive parameters. Class parameters don't work. #6310

Closed
q-stefanmuscalu opened this issue May 17, 2024 · 4 comments · Fixed by #6335
Assignees
Labels
python Pull requests for the Python Semantic Kernel

Comments

@q-stefanmuscalu
Copy link

q-stefanmuscalu commented May 17, 2024

Describe the bug
I am writing a plugin in native Python. The kernel functions only work if the parameters are primitive types, such as str, float, bool. The agent fails to call functions that take classes as parameters.

To Reproduce

Example class parameters

This example is where the kernel function takes a class parameter:
https://gist.github.com/q-stefanmuscalu/15c804fee87f5cbd5f7024307c4a9ff6

class HolidayRequest(KernelBaseModel):
    start_date: str = Field(
        ...,
        description='The date when the holiday period starts in ISO 8601 format',
        examples=['2024-01-23', '2020-06-15'])
    end_date: str = Field(
        ...,
        description='The date when the holiday period ends in ISO-8601 format',
        examples=['2024-01-23', '2020-06-15'])

class HolidayPlugin:

    @kernel_function(name="BookHoliday", description="Book a holiday")
    def book_holiday(
        self,
        request: Annotated[HolidayRequest, "A request to book holiday."]
    ) -> Annotated[bool, "The result is the boolean value True if successful, False if unsuccessful."]:
        return True

The full error trace is here. The agent calls the book_holiday tool with a string, rather than an instance of HolidayRequest, although the HolidayRequest annotates every field and extends KernelBaseModel

 pydantic_core._pydantic_core.ValidationError: 1 validation error for HolidayRequest
   Input should be a valid dictionary or instance of HolidayRequest [type=model_type, input_value='Book holiday from 13th to 20th of August 2023.', input_type=str]
     For further information visit https://errors.pydantic.dev/2.7/v/model_type

Example primitive parameters

This example is where the kernel function takes primitive parameters and the agent successfully calls it:
https://gist.github.com/q-stefanmuscalu/8164996f3bfabe9bb53d979bb826bd9a

class HolidayPlugin:

    @kernel_function(name="BookHoliday", description="Book a holiday")
    def book_holiday(
        self,
        start_date: Annotated[str, "The date when the holiday period starts in ISO 8601 format. Example: '2024-01-23', '2020-06-15'"],
        end_date: Annotated[str, "The date when the holiday period ends in ISO-8601 format. Example: '2024-01-23', '2020-06-15'" ]
    ) -> Annotated[bool, "The result is the boolean value True if successful, False if unsuccessful."]:
        return True

The output is:

Q: Book my holiday for 13th - 20th of August 2023.
A: Your holiday has been booked for the 13th to the 20th of August 2023.

Printing the arguments when the tool is called does prove the agent provided the parameters in the right format 2023-08-13, 2023-08-20

Expected behavior
I expect the Agent to make a tool call to book_holiday where it constructs an instance of HolidayRequest and then replies with Your holiday has been booked...

Screenshots
N/A

Platform

  • OS: Mac
  • IDE: VS Code
  • Language: Python
  • Source: semantic kernel version 0.9.8b1

Additional context
I found a single example in the unit tests where a kernel function takes class parameters. This is why I expect kernel functions to handle class parameters.

@markwallace-microsoft markwallace-microsoft added python Pull requests for the Python Semantic Kernel triage labels May 17, 2024
@github-actions github-actions bot changed the title Kernel function can only be called with primitive parameters. Class parameters don't work. Python: Kernel function can only be called with primitive parameters. Class parameters don't work. May 17, 2024
@q-stefanmuscalu
Copy link
Author

Here are the plans generated by the agent in both cases:

To book your holiday for the 13th to the 20th of August 2023, follow these steps:

1. Use the `holiday-BookHoliday` function to book the holiday. You will need to provide a request in the form of a string that specifies your desired holiday dates. For example, the request could be "Book holiday from 13th to 20th of August 2023."

2. Once the holiday is booked, use the `UserInteraction-SendFinalAnswer` function to confirm the booking to you. The answer could be something like "Your holiday for the 13th to the 20th of August 2023 has been successfully booked."

This plan utilizes the available functions to achieve your goal of booking a holiday for the specified dates.

and

To book your holiday for the 13th to the 20th of August 2023, follow these steps:

1. Use the `holiday-BookHoliday` function to book the holiday. You will need to provide the `start_date` and `end_date` parameters.
   - `start_date`: "2023-08-13"
   - `end_date`: "2023-08-20"

2. Once the holiday is booked, confirm the booking details and dates.

3. Use the `UserInteraction-SendFinalAnswer` function to communicate the successful booking of your holiday for the specified dates back to you. The answer should include the confirmation of the holiday booking from the 13th to the 20th of August 2023.

This plan ensures that your holiday is booked for the desired dates and that you are informed of the successful booking.

@q-stefanmuscalu
Copy link
Author

I looked at the tool schema sent to the LLM and it's wrong:

{
    "type": "function",
    "function": {
        "name": "holiday-BookHoliday",
        "description": "\n        You need to find out the `start_date` and `end_date` of the holiday and then call \n        the `book_holiday` tool with a single argument called `request`. \n        The `request` is an instance of the Python class `HolidayRequest`. \n        Calling `book_holiday` with another type will fail.\n        ",
        "parameters": {
            "type": "object",
            "properties": {
                "request": {
                    "description": "A request to book a holiday",
                    "type": "string"
                }
            },
            "required": [
                "request"
            ]
        }
    }
}

The part that says "type": "string" should be "type": "object"

@moonbox3
Copy link
Contributor

Thanks for filing @q-stefanmuscalu. We have some work in the next few days to improve the schema handling for complex types. Tagging @eavanvalkenburg for visibility.

@moonbox3 moonbox3 self-assigned this May 17, 2024
@moonbox3 moonbox3 removed the triage label May 17, 2024
@q-stefanmuscalu
Copy link
Author

Sounds good! Thank you @moonbox3 !

github-merge-queue bot pushed a commit that referenced this issue May 20, 2024
… Memory Connectors. (#6335)

### Motivation and Context

The Python code base could handle some primitive types for the schema
for tool call objects and kernel parameter metadata. However, it
couldn't properly handle the more complex JSON schemas for tool call
objects.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR introduces:
- JSON schema handling for KernelParameterMetadata and tool call
objects.
- Updates to the tool call utils method to properly recurse on the
KernelParameterMetadata's stucture.
- Adds unit tests for this code.
- Add experimental_class/experimental_functions to various parts of the
code base like Memory Connectors and OpenAPI plugin.
- Fixes #6310 
- Fixes #6280

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Pull requests for the Python Semantic Kernel
Projects
Status: Sprint: Done
3 participants