Add swiftformer #22686

shehanmunasinghe · 2023-04-10T10:14:17Z

What does this PR do?

Adds 'SwiftFormer' into huggingface/transformers

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. Add SwiftFormer #22685
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @NielsRogge @alaradirik

using add-new-model-like

D-Roberts · 2023-04-10T22:31:25Z

Are there plans for tensorflow version? I am interested in having the tf model available for use in a downstream task in my work/research

amyeroberts · 2023-04-13T16:04:27Z

Hi @shehanmunasinghe, thanks for opening this PR!

Could you make sure to fill out all the necessary documentation for the model in the README and swiftformer.mdx file?

Quick question about the modeling code - it seems that all of the model components are copied from ViT i.e. their architecture and forward pass are exactly the same - is this correct?

@D-Roberts I don't know of anyone working on the TensorFlow version of this, looking through the open PRs or issues. @shehanmunasinghe - do you know of anyone who is working on a TF port?

shehanmunasinghe · 2023-04-14T05:01:53Z

Hi @amyeroberts, Thanks for your response.

Please note that this is a Work In Progress (WIP) pull request. The changes to the modeling code and the documentation will be reflected once I push them.

Hi @D-Roberts, currently I'm not aware of anyone working on the TensorFlow version of this.

amyeroberts · 2023-04-14T10:23:21Z

@shehanmunasinghe OK - sounds good :) Let us know when the PR is ready to review. In the meantime, please don't hesitate if there are any questions.

@D-Roberts - would you be interested in porting this model once the pytorch version is merged in?

D-Roberts · 2023-04-14T14:29:43Z

@amyeroberts I am still working on porting the Efficientformer; I am interested in having both in tf to train in some downstream tasks / research... I would like to do the port for swiftformer too but can't commit to it right now due to time constraints (I do this in my spare time).. I'll revisit after I am done with the efficientformer and after the swiftformer torch pr here is merged too.

amyeroberts · 2023-04-14T14:54:54Z

@D-Roberts Of course, no worries, and thank you for your work on adding EfficientFormer :) I've opened an issue - #22771 - to add the TF version of this model where future discussions on how, who and when can be organised.

HuggingFaceDocBuilderDev · 2023-04-14T17:37:46Z

The documentation is not available anymore as the PR was closed or merged.

This reverts commit c2e05e2.

shehanmunasinghe · 2023-04-15T07:24:30Z

Hi @amyeroberts , this is now ready for your review.

shehanmunasinghe · 2023-04-27T19:10:47Z

Hi @amyeroberts , I have resolved the issues that were raised during the code review. Please take a look.

amyeroberts · 2023-04-28T13:50:31Z

@shehanmunasinghe Great! I'm away for a few days, but will re-review when I'm back at my computer at the start of next week.

amyeroberts

Thanks for the continued work on this model and updating this PR! Overall looking good and nice and clean 👌 🚀

Most of the comments are in the modeling file and relate to class definitions. Overall, structure and layout is good, there's just a few places to reowrk and should be good to merge soon :)

All model specific layers should be implements as a nn.Module with the model name as a prefix
All layers should have as few keyword arguments as possible and use the config object to instantiate itself. There were quite a few keyword arguments which are essentially hardcoded. When these are removed, a lot of the model logic should simplify.

src/transformers/models/swiftformer/configuration_swiftformer.py

src/transformers/models/swiftformer/modeling_swiftformer.py

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

shehanmunasinghe · 2023-05-06T07:46:11Z

Hi @amyeroberts, thanks for your time and effort in reviewing this. This is my first pull request on this repo and I'm glad to hear your constructive comments. I have applied the suggestions you made and updated the code again.

amyeroberts

Very nice! 🔥

Just a few nits on docstrings and configuration attribute names. Once resolved I think we're good to merge!

src/transformers/models/swiftformer/configuration_swiftformer.py

src/transformers/models/swiftformer/modeling_swiftformer.py

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

shehanmunasinghe · 2023-05-10T05:49:01Z

Hi @amyeroberts , I have fixed those issues and pushed the updated code.

However, as indicated here one test is failing, though this has nothing to do with tests/models/whisper/test_modeling_whisper.py.

FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_pt_tf_model_equivalence - AssertionError: 1.04904175e-05 not less than or equal to 1e-05 : outputs.encoder_hidden_states_0: Difference between PyTorch and TF is 1.049041748046875e-05 (>= 1e-05).

shehanmunasinghe · 2023-05-11T05:53:15Z

Hi @amyeroberts , I have fixed those issues and pushed the updated code.

However, as indicated here one test is failing, though this has nothing to do with tests/models/whisper/test_modeling_whisper.py.

FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_pt_tf_model_equivalence - AssertionError: 1.04904175e-05 not less than or equal to 1e-05 : outputs.encoder_hidden_states_0: Difference between PyTorch and TF is 1.049041748046875e-05 (>= 1e-05).

All checks are passing now.

amyeroberts

Really nice PR 🔥 Thanks for adding this model and for iterating!

I noticed just one config attribute which should be updated before merging - apologies for not catching before, after that we can merge. Congrats on getting your first model contribution added 🤗

src/transformers/models/swiftformer/configuration_swiftformer.py

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

shehanmunasinghe · 2023-05-12T05:35:38Z

Hi @amyeroberts , thanks for approving this PR. I have updated everything and now I think it can be merged.

* Commit the automatically generated code using add-new-model-like * Update description at swiftformer.mdx file * remove autogenerated code for MaskedImageModeling * update weight conversion scripts * Update modeling_swiftformer.py * update configuration_swiftformer.py * Update test_modeling_swiftformer.py * update modeling code - remove einops dependency * Update _toctree.yml * update modeling code - remove copied from comments * update docs * Revert "update docs" This reverts commit c2e05e2. * update docs * remove unused reference SwiftFormerImageProcessor * update dependency_versions_table.py * update swiftformer.mdx * update swiftformer.mdx * change model output type - no attentions * update model org name * Fix typo * fix copies * Update tests/models/swiftformer/test_modeling_swiftformer.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/auto/image_processing_auto.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/auto/feature_extraction_auto.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/swiftformer.mdx Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/swiftformer/configuration_swiftformer.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update modeling_swiftformer.py fix-copies * make style, make quality, fix-copies * Apply suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * make style Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Add suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Add suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * make fix-copies * Update modeling_swiftformer.py * Update modeling_swiftformer.py * Add suggestions from code review Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

shehanmunasinghe added 2 commits April 4, 2023 17:57

Commit the automatically generated code

8dd1ca2

using add-new-model-like

Merge remote-tracking branch 'upstream/main' into add_swiftformer

d37e355

Update description at swiftformer.mdx file

1ab0476

amyeroberts mentioned this pull request Apr 14, 2023

TF Swiftformer #22771

Closed

2 tasks

shehanmunasinghe added 8 commits April 14, 2023 20:25

remove autogenerated code for MaskedImageModeling

1de0db5

update weight conversion scripts

13169a9

Update modeling_swiftformer.py

4363d6a

update configuration_swiftformer.py

c1dbc0d

Update test_modeling_swiftformer.py

3d6c400

Merge remote-tracking branch 'upstream/main' into add_swiftformer

4947175

update modeling code - remove einops dependency

a6832b2

Update _toctree.yml

bc1da70

shehanmunasinghe added 9 commits April 14, 2023 23:13

update modeling code - remove copied from comments

6bfaec6

update docs

c2e05e2

Revert "update docs"

5717bdc

This reverts commit c2e05e2.

update docs

a1354c0

remove unused reference SwiftFormerImageProcessor

fa09eca

Merge remote-tracking branch 'upstream/main' into add_swiftformer

5d5e580

update dependency_versions_table.py

cffe50e

update swiftformer.mdx

8560039

update swiftformer.mdx

3fedb84

shehanmunasinghe marked this pull request as ready for review April 15, 2023 06:41

make style, make quality, fix-copies

0140ceb

shehanmunasinghe requested a review from amyeroberts April 28, 2023 05:09

shehanmunasinghe changed the title ~~[WIP] Add swiftformer~~ Add swiftformer Apr 28, 2023

amyeroberts reviewed May 5, 2023

View reviewed changes

shehanmunasinghe and others added 4 commits May 5, 2023 22:17

Apply suggestions from code review

4a63ce4

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Apply suggestions from code review

df62654

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

make style

f04a9cc

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Merge remote-tracking branch 'upstream/main' into add_swiftformer

e8e5791

amyeroberts reviewed May 9, 2023

View reviewed changes

shehanmunasinghe and others added 6 commits May 10, 2023 07:20

Add suggestions from code review

5b5b1d2

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Add suggestions from code review

465dc1a

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Merge remote-tracking branch 'upstream/main' into add_swiftformer

5fca4cb

make fix-copies

b538c9d

Update modeling_swiftformer.py

ce74bf5

Update modeling_swiftformer.py

85bdaaf

Merge remote-tracking branch 'upstream/main' into add_swiftformer

ea387e0

shehanmunasinghe requested a review from amyeroberts May 11, 2023 05:53

amyeroberts approved these changes May 11, 2023

View reviewed changes

src/transformers/models/swiftformer/configuration_swiftformer.py Outdated Show resolved Hide resolved

shehanmunasinghe and others added 2 commits May 12, 2023 08:20

Add suggestions from code review

a0cb449

Co-Authored-By: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Merge remote-tracking branch 'upstream/main' into add_swiftformer

09db33b

amyeroberts merged commit c045249 into huggingface:main May 12, 2023
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add swiftformer #22686

Add swiftformer #22686

shehanmunasinghe commented Apr 10, 2023 •

edited

D-Roberts commented Apr 10, 2023 •

edited

amyeroberts commented Apr 13, 2023

shehanmunasinghe commented Apr 14, 2023

amyeroberts commented Apr 14, 2023

D-Roberts commented Apr 14, 2023 •

edited

amyeroberts commented Apr 14, 2023

HuggingFaceDocBuilderDev commented Apr 14, 2023 •

edited

shehanmunasinghe commented Apr 15, 2023

shehanmunasinghe commented Apr 27, 2023

amyeroberts commented Apr 28, 2023

amyeroberts left a comment

shehanmunasinghe commented May 6, 2023

amyeroberts left a comment

shehanmunasinghe commented May 10, 2023

shehanmunasinghe commented May 11, 2023

amyeroberts left a comment

shehanmunasinghe commented May 12, 2023

Add swiftformer #22686

Add swiftformer #22686

Conversation

shehanmunasinghe commented Apr 10, 2023 • edited

What does this PR do?

Before submitting

Who can review?

D-Roberts commented Apr 10, 2023 • edited

amyeroberts commented Apr 13, 2023

shehanmunasinghe commented Apr 14, 2023

amyeroberts commented Apr 14, 2023

D-Roberts commented Apr 14, 2023 • edited

amyeroberts commented Apr 14, 2023

HuggingFaceDocBuilderDev commented Apr 14, 2023 • edited

shehanmunasinghe commented Apr 15, 2023

shehanmunasinghe commented Apr 27, 2023

amyeroberts commented Apr 28, 2023

amyeroberts left a comment

Choose a reason for hiding this comment

shehanmunasinghe commented May 6, 2023

amyeroberts left a comment

Choose a reason for hiding this comment

shehanmunasinghe commented May 10, 2023

shehanmunasinghe commented May 11, 2023

amyeroberts left a comment

Choose a reason for hiding this comment

shehanmunasinghe commented May 12, 2023

shehanmunasinghe commented Apr 10, 2023 •

edited

D-Roberts commented Apr 10, 2023 •

edited

D-Roberts commented Apr 14, 2023 •

edited

HuggingFaceDocBuilderDev commented Apr 14, 2023 •

edited