Add support for transformers as a named flavor #8086

BenWilson2 · 2023-03-23T19:23:32Z

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Initial PR for transformers flavor, providing support for:

save_model
load_model
log_model

How is this patch tested?

Existing unit/integration tests
New unit/integration tests
Manual tests (describe details, including test results, below)

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly in the documentation preview.

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

Add support for HuggingFace transformers as a named flavor.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

mlflow-automation · 2023-03-23T19:23:50Z

Documentation preview for 3b851a7 will be available here when this CircleCI job completes successfully.

More info

Ignore this comment if this PR does not change the documentation.
It takes a few minutes for the preview to be available.
The preview is updated when a new commit is pushed to this PR.
This comment was created by https://github.com/mlflow/mlflow/actions/runs/4565232106.

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

mlflow/transformers.py

tests/diviner/test_diviner_model_export.py

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

mlflow/transformers.py

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

harupy · 2023-03-29T04:34:25Z

mlflow/transformers.py

+    """
+    Convert a submitted component-based model in a dictionary to a namedtuple
+    """
+    ComponentModel = namedtuple("ComponentModel", model, rename=True)


I would pull this out from this function so that we can use isinstance(..., ComponentModel)

After looking at that type validator and how it's not really possible to do the dynamic renaming in a clean way, I deleted these functions and implemented a subclass of NamedTuple. It's a bit more complex, but it gives us the ability to do explicit type validation to ensure that only properly qualified objects are able to be processed in the module.

harupy · 2023-03-29T04:42:54Z

mlflow/transformers.py

+    elif _isinstance_named_tuple(model):
+        try:
+            return get_task(model.model.name_or_path)
+        except RuntimeError as e:


Suggested change

except RuntimeError as e:

except Exception as e:

Catching Exception might be safer in case huggingface suddenly changes RunTimeError to a different error type.

excellent point. I also updated the handling in the pip requirements function (since the root of the issue is identical (but called from a different transformers function that is internal to the loading of a model instance).

harupy · 2023-03-29T09:14:06Z

mlflow/utils/docstring_utils.py

+    def get_version_ranges(module_key):
+        versions = _ML_PACKAGE_VERSIONS[module_key]["models"]
+        min_version = versions["minimum"]
+        max_version = versions["maximum"]
+        return min_version, max_version


Can we move this function in the top level?

Excellent idea. renamed and added usage notes

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

dbczumar · 2023-03-29T21:29:49Z

mlflow/ml-package-versions.yml

+      pip install git+https://github.com/huggingface/transformers
+  models:
+    minimum: "4.25.1"
+    maximum: "4.27.3"


Suggested change

maximum: "4.27.3"

maximum: "4.27.4"

ah, good catch.

dbczumar · 2023-03-29T21:34:38Z

mlflow/transformers.py

+          from transformers import pipeline
+
+          pipeline = "csarron/mobilebert-uncased-squad-v2"
+
+          with mlflow.start_run():
+              mlflow.transformers.save_model(
+                  transformers_model=pipeline,
+                  path="path/to/save/model",
+              )


This seems to fail:

Traceback (most recent call last): File "save.py", line 6, in <module> mlflow.transformers.save_model( File "/Users/corey.zumar/mlflow/mlflow/utils/docstring_utils.py", line 235, in version_func return func(*args, **kwargs) File "/Users/corey.zumar/mlflow/mlflow/transformers.py", line 275, in save_model _validate_transformers_model_dict(transformers_model) File "/Users/corey.zumar/mlflow/mlflow/transformers.py", line 141, in _validate_transformers_model_dict model = transformers_model.model AttributeError: 'str' object has no attribute 'model'

(Transformers version 4.27.3)

I.... don't even know... what I was thinking with that.... UGH. Fix incoming.

dbczumar · 2023-03-29T21:35:07Z

mlflow/transformers.py

+          from transformers import pipeline
+
+          pipeline = "csarron/mobilebert-uncased-squad-v2"
+
+          with mlflow.start_run():
+              mlflow.transformers.log_model(
+                  transformers_model=pipeline,
+                  artifact_path="my_pipeline",
+              )


This seems to fail (https://github.com/mlflow/mlflow/pull/8086/files#r1152507646)

dbczumar · 2023-03-29T21:36:16Z

mlflow/transformers.py

+                "the model is not a language-based model and requires a complex input type "
+                "that is currently not supported."
+            )
+        _logger.warning(f"This model is unable to be used for pyfunc prediction due to {reason}")


Can we clarify that the pyfunc flavor won't be added to the model?

Added a sentence explaining this

dbczumar · 2023-03-29T21:38:53Z

mlflow/utils/docstring_utils.py

+        required_pkg_versions = f"``{min_ver}`` -  ``{max_ver}``"
+
+        notice = (
+            f"The '{integration_name}' package is known to be compatible with the following "


Suggested change

f"The '{integration_name}' package is known to be compatible with the following "

f"The '{integration_name}' MLflow Models integration is known to be compatible with the following "

dbczumar · 2023-03-29T21:39:17Z

mlflow/transformers.py

+
+
+@experimental
+@docstring_version_compatibility_warning(integration_name=FLAVOR_NAME)


Can we also apply this decorator to load_model()?

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

dbczumar · 2023-03-30T04:57:47Z

mlflow/transformers.py

+                "that is currently not supported."
+            )
+        _logger.warning(
+            f"This model is unable to be used for pyfunc prediction due to {reason} "


Nittiest of nits (reads better):

Suggested change

f"This model is unable to be used for pyfunc prediction due to {reason} "

f"This model is unable to be used for pyfunc prediction because {reason} "

dbczumar

LGTM! Tested manually and it works very well for language & image use cases. Thanks @BenWilson2 !

harupy · 2023-03-30T08:43:24Z

mlflow/transformers.py

+    base_reqs = ["transformers"]
+    try:
+        base_reqs.extend(_model_packages(model))


Suggested change

base_reqs = ["transformers"]

try:

base_reqs.extend(_model_packages(model))

try:

base_reqs = ["transformers", *_model_packages(model)]

A minor simplification :)

harupy

LGTM once https://github.com/mlflow/mlflow/pull/8086/files#r1150353739 is addressed!

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

tafaust · 2023-05-09T11:29:36Z

Quick question: we're currently trying to log a pytorch lightning module with transformers in it (logging transformer model and processor via callback).

We are currently stuck loading these models with our lightning module wrapper - are there any best practices from the mlflow team?

I cannot find anything in the docs.

BenWilson2 · 2023-05-09T11:51:52Z

@tahesse what is the nature of the issue that you're experiencing? Would you like to file an issue so that we can have a place to discuss and you can share example code?

tafaust · 2023-05-09T12:20:34Z

@BenWilson2 I think it is pretty much #4245 except that our pretrained model is wrapped with PytorchLightning module like in https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Donut/CORD/Fine_tune_Donut_on_a_custom_dataset_(CORD)_with_PyTorch_Lightning.ipynb

Do you think it makes sense to open a new issue for it? I feel like there is just a best practice section missing in the docs for it. The mlflow.pytorch.autolog() was of no help because the {Train,Val}Dataloader seem to be pickled with the Lightning module and the they wrap the huggingface datasets and the caching makes it impossible to load it on a different filesystem.

BenWilson2 · 2023-05-09T16:06:30Z

Do you have an example that works properly that you'd be willing to contribute and are interested in updating documentation guidelines for this use case? If so, please feel free to file a PR and we can discuss there :)

tafaust · 2023-05-11T13:48:18Z

@BenWilson2 Unfortunately, I am not able to open source my companies code without permission.

BenWilson2 added 2 commits March 23, 2023 15:05

Add support for transformers

1da16ad

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

fix pydoc rendering

ad79779

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

github-actions bot added area/models MLmodel format, model serialization/deserialization, flavors rn/feature Mention under Features in Changelogs. labels Mar 23, 2023

BenWilson2 requested review from harupy, dbczumar and WeichenXu123 March 23, 2023 19:24

fix the cross version parser

4b20526

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 commented Mar 23, 2023

View reviewed changes

mlflow/transformers.py Show resolved Hide resolved

BenWilson2 commented Mar 23, 2023

View reviewed changes

tests/diviner/test_diviner_model_export.py Show resolved Hide resolved

BenWilson2 added the only-latest If applied, only test the latest version of each group in cross-version tests. label Mar 23, 2023

BenWilson2 added 5 commits March 23, 2023 15:36

remove pointless separation of tokenizer fetch

d70ba6b

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

remove invalid attr

a9b4bcc

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

update lazy import test

0c76cbe

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

add test libs

841de13

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

fix linting issues

85ae0ad

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 force-pushed the transformers-flavor branch from 7743e93 to 85ae0ad Compare March 23, 2023 22:08

exclude non LLMs from pyfunc flavor

cb2a8c9

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Show resolved Hide resolved

dbczumar reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Show resolved Hide resolved

harupy reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

harupy reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Show resolved Hide resolved

harupy reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

harupy reviewed Mar 24, 2023

View reviewed changes

mlflow/transformers.py Outdated Show resolved Hide resolved

PR feedback

a976c11

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

harupy reviewed Mar 29, 2023

View reviewed changes

BenWilson2 added 5 commits March 29, 2023 13:13

Building a proper dict -> typed NamedTuple conversion

ff4075c

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

remove commented out code

b8a3699

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

make the wrapper class private to remove from docs display

ef055a3

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

refactoring

195c0c1

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

fixes for regressions in 4.26.x builds of transformers

1af5728

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

dbczumar reviewed Mar 29, 2023

View reviewed changes

PR feedback

6203547

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

dbczumar reviewed Mar 30, 2023

View reviewed changes

dbczumar approved these changes Mar 30, 2023

View reviewed changes

harupy reviewed Mar 30, 2023

View reviewed changes

harupy approved these changes Mar 30, 2023

View reviewed changes

BenWilson2 added 2 commits March 30, 2023 09:49

apply missed fix

10ba0f8

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

remove unused import

3b851a7

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 enabled auto-merge (squash) March 30, 2023 15:37

BenWilson2 merged commit 2a7393b into mlflow:master Mar 30, 2023

BenWilson2 deleted the transformers-flavor branch May 9, 2023 16:06

s-udhaya mentioned this pull request May 20, 2023

[WIP] Add transformers flavor #7473

Closed

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for transformers as a named flavor #8086

Add support for transformers as a named flavor #8086

BenWilson2 commented Mar 23, 2023

mlflow-automation commented Mar 23, 2023 •

edited

harupy Mar 29, 2023

BenWilson2 Mar 29, 2023

harupy Mar 29, 2023 •

edited

BenWilson2 Mar 29, 2023

harupy Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 29, 2023

dbczumar Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 29, 2023

dbczumar Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 29, 2023

BenWilson2 Mar 29, 2023

dbczumar Mar 30, 2023 •

edited

dbczumar left a comment

harupy Mar 30, 2023

harupy left a comment

tafaust commented May 9, 2023

BenWilson2 commented May 9, 2023

tafaust commented May 9, 2023 •

edited

BenWilson2 commented May 9, 2023

tafaust commented May 11, 2023

	f"The '{integration_name}' package is known to be compatible with the following "
	f"The '{integration_name}' MLflow Models integration is known to be compatible with the following "



		@experimental
		@docstring_version_compatibility_warning(integration_name=FLAVOR_NAME)

	f"This model is unable to be used for pyfunc prediction due to {reason} "
	f"This model is unable to be used for pyfunc prediction because {reason} "

Add support for transformers as a named flavor #8086

Add support for transformers as a named flavor #8086

Conversation

BenWilson2 commented Mar 23, 2023

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

mlflow-automation commented Mar 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harupy Mar 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbczumar Mar 30, 2023 • edited

Choose a reason for hiding this comment

dbczumar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harupy left a comment

Choose a reason for hiding this comment

tafaust commented May 9, 2023

BenWilson2 commented May 9, 2023

tafaust commented May 9, 2023 • edited

BenWilson2 commented May 9, 2023

tafaust commented May 11, 2023

mlflow-automation commented Mar 23, 2023 •

edited

harupy Mar 29, 2023 •

edited

dbczumar Mar 30, 2023 •

edited

tafaust commented May 9, 2023 •

edited