OpenAI flavor #8155

harupy · 2023-04-03T09:56:26Z

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

(Please fill in changes proposed in this fix)

How is this patch tested?

Existing unit/integration tests
New unit/integration tests
Manual tests (describe details, including test results, below)
- with Azure OpenAI (using a real API key)
- with OpenAI (using a real API key)

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly in the documentation preview.

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

mlflow-automation · 2023-04-03T09:56:43Z

Documentation preview for 357f841 will be available here when this CircleCI job completes successfully.

More info

Ignore this comment if this PR does not change the documentation.
It takes a few minutes for the preview to be available.
The preview is updated when a new commit is pushed to this PR.
This comment was created by https://github.com/mlflow/mlflow/actions/runs/4716412992.

harupy · 2023-04-06T09:18:36Z

mlflow/openai/__init__.py

+    :param path: Local filesystem path to the MLflow Model with the ``openai`` flavor.
+    """
+    wrapper_cls = _TestOpenAIWrapper if _MLFLOW_OPENAI_TESTING.get() else _OpenAIWrapper
+    return wrapper_cls(_load_model(path))


Could not find a good way to mock requests in the UDF

Makes sense :)

mlflow/openai/__init__.py

jinzhang21

Verified basic functionality works as intended. This is awesome, @harupy ! https://e2-dogfood.staging.cloud.databricks.com/?o=6051921418418893#mlflow/experiments/469643393101378/runs/3991bf0fa56f4cf0ba8864bfc3a60a47

BenWilson2

LGTM! Great work @harupy

mlflow/openai/__init__.py

Signed-off-by: harupy <hkawamura0130@gmail.com>

mlflow/openai/api_request_parallel_processor.py

harupy · 2023-04-15T02:56:15Z

mlflow/openai/__init__.py

+        _save_example(mlflow_model, input_example, path)
+    if metadata is not None:
+        mlflow_model.metadata = metadata
+    model_data_subpath = "model.json"


Suggested change

model_data_subpath = "model.json"

model_data_subpath = "model.yaml"

@jinzhang21 maybe we should use yaml here as well?

Right, but @sunishsheth2009 mentioned it doesn't work with YAML because Langchain doesn't serialize / format it properly. Does it work with OpenAI? I'd prefer to use YAML for consistency here.

Langchain doesn't serialize / format it properly

@sunishsheth2009 Can you elaborate on this?

Yes!! So in Langchain, I think its a bug with Langchain, but this is how it stores the yaml file

_type: !!python/object/apply:langchain.agents.agent_types.AgentType - zero-shot-react-description allowed_tools: - Search - Calculator llm_chain: _type: llm_chain llm: _type: openai best_of: 1 frequency_penalty: 0 logit_bias: {} max_tokens: 256 model_name: text-davinci-003 n: 1 presence_penalty: 0 request_timeout: null temperature: 0.0 top_p: 1 memory: null output_key: text prompt: _type: prompt input_variables: - input - agent_scratchpad output_parser: null partial_variables: {} template: 'Answer the following questions as best you can. You have access to the following tools:

but json is stored like this:

{ "llm_chain": { "memory": null, "verbose": false, "prompt": { "input_variables": [ "input", "agent_scratchpad" ], "output_parser": null, "partial_variables": {}, "template": "Answer the following questions as best you can. You have access to the following tools:\n\nSearch: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [Search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: {input}\nThought:{agent_scratchpad}", "template_format": "f-string", "validate_template": true, "_type": "prompt" }, "llm": { "model_name": "text-davinci-003", "temperature": 0, "max_tokens": 256, "top_p": 1, "frequency_penalty": 0, "presence_penalty": 0, "n": 1, "best_of": 1, "request_timeout": null, "logit_bias": {}, "_type": "openai" }, "output_key": "text", "_type": "llm_chain" }, "allowed_tools": [ "Search", "Calculator" ], "_type": "zero-shot-react-description" }

See that the type is incorrect in yaml but it is correct in json.
Maybe we can create a bug report with Langchain if we decide to go yaml with Langchain Agent. 🤔

Thanks!

_type: !!python/object/apply:langchain.agents.agent_types.AgentType

does seem incorrect.

Filed langchain-ai/langchain#2998

Signed-off-by: harupy <hkawamura0130@gmail.com>

harupy changed the title ~~OpenAI flavor~~ [DO NOT REVIEW YET] OpenAI flavor Apr 3, 2023

harupy force-pushed the openai-flavor branch from 1eceea0 to 0a2afd6 Compare April 6, 2023 05:44

github-actions bot added area/models MLmodel format, model serialization/deserialization, flavors rn/feature Mention under Features in Changelogs. labels Apr 6, 2023

harupy commented Apr 6, 2023

View reviewed changes

harupy force-pushed the openai-flavor branch from e09c17d to 41825c7 Compare April 6, 2023 16:27

harupy changed the title ~~[DO NOT REVIEW YET] OpenAI flavor~~ OpenAI flavor Apr 6, 2023

harupy commented Apr 7, 2023

View reviewed changes

mlflow/openai/__init__.py Show resolved Hide resolved

harupy commented Apr 7, 2023

View reviewed changes

mlflow/openai/__init__.py Outdated Show resolved Hide resolved

harupy commented Apr 7, 2023

View reviewed changes

mlflow/openai/__init__.py Show resolved Hide resolved

harupy force-pushed the openai-flavor branch 2 times, most recently from 8e00ffd to 70af162 Compare April 8, 2023 00:17

harupy marked this pull request as ready for review April 10, 2023 03:38

jinzhang21 approved these changes Apr 11, 2023

View reviewed changes

BenWilson2 approved these changes Apr 11, 2023

View reviewed changes

harupy commented Apr 13, 2023

View reviewed changes

mlflow/openai/__init__.py Outdated Show resolved Hide resolved

harupy force-pushed the openai-flavor branch 2 times, most recently from 6934854 to bee4e70 Compare April 13, 2023 09:33

harupy added 5 commits April 14, 2023 08:45

Openai flavor

2a871bc

Signed-off-by: harupy <hkawamura0130@gmail.com>

Fix

33d9154

Signed-off-by: harupy <hkawamura0130@gmail.com>

Fix

f8a1977

Signed-off-by: harupy <hkawamura0130@gmail.com>

Improve tests

ee707e7

Signed-off-by: harupy <hkawamura0130@gmail.com>

Fix message

eacc2ee

Signed-off-by: harupy <hkawamura0130@gmail.com>

harupy force-pushed the openai-flavor branch from 4a1a961 to eacc2ee Compare April 13, 2023 23:45

harupy added 3 commits April 14, 2023 09:19

openai.yaml

64fc1be

Signed-off-by: harupy <hkawamura0130@gmail.com>

use subprocess

42eb168

Signed-off-by: harupy <hkawamura0130@gmail.com>

Update

f65f8cd

Signed-off-by: harupy <hkawamura0130@gmail.com>

harupy commented Apr 15, 2023

View reviewed changes

mlflow/openai/api_request_parallel_processor.py Outdated Show resolved Hide resolved

harupy commented Apr 15, 2023

View reviewed changes

harupy added 4 commits April 15, 2023 14:56

Use threadpool

9bb8a6f

Signed-off-by: harupy <hkawamura0130@gmail.com>

Use yaml

8380a66

Signed-off-by: harupy <hkawamura0130@gmail.com>

Rename

25f6083

Signed-off-by: harupy <hkawamura0130@gmail.com>

Fix mock

357f841

Signed-off-by: harupy <hkawamura0130@gmail.com>

harupy merged commit 6a33d1d into mlflow:master Apr 17, 2023
25 of 26 checks passed

harupy mentioned this pull request Apr 17, 2023

Add openai flavor in model.rst #8242

Merged

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenAI flavor #8155

OpenAI flavor #8155

harupy commented Apr 3, 2023 •

edited

mlflow-automation commented Apr 3, 2023 •

edited

harupy Apr 6, 2023

BenWilson2 Apr 11, 2023

jinzhang21 left a comment

BenWilson2 left a comment

harupy Apr 15, 2023

jinzhang21 Apr 15, 2023 •

edited

harupy Apr 15, 2023

sunishsheth2009 Apr 15, 2023

harupy Apr 17, 2023

harupy Apr 17, 2023

	model_data_subpath = "model.json"
	model_data_subpath = "model.yaml"

OpenAI flavor #8155

OpenAI flavor #8155

Conversation

harupy commented Apr 3, 2023 • edited

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

mlflow-automation commented Apr 3, 2023 • edited

harupy Apr 6, 2023

Choose a reason for hiding this comment

BenWilson2 Apr 11, 2023

Choose a reason for hiding this comment

jinzhang21 left a comment

Choose a reason for hiding this comment

BenWilson2 left a comment

Choose a reason for hiding this comment

harupy Apr 15, 2023

Choose a reason for hiding this comment

jinzhang21 Apr 15, 2023 • edited

Choose a reason for hiding this comment

harupy Apr 15, 2023

Choose a reason for hiding this comment

sunishsheth2009 Apr 15, 2023

Choose a reason for hiding this comment

harupy Apr 17, 2023

Choose a reason for hiding this comment

harupy Apr 17, 2023

Choose a reason for hiding this comment

harupy commented Apr 3, 2023 •

edited

mlflow-automation commented Apr 3, 2023 •

edited

jinzhang21 Apr 15, 2023 •

edited