added deit onnx config

[0xrushi]

What does this PR do?

Added DeiT OnnxConfig to make this model available for conversion

@chainyo

#16308

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[chainyo]

I saw that DeiT also have a DeiTForMaskedImageModeling implementation, is it a feature also available in onnx ?

[0xrushi]

BeiT also has BeiTForMaskedImageModeling, but there isn't anything for that in features.py

[lewtun]

You can add something like

"masked-im": AutoModelForMaskedImageModeling

to the _TASKS_TO_AUTOMODELS attribute of the FeaturesManager. I don't think we currently support this feature for TensorFlow, so no need to include this in __TASKS_TO_TF_AUTOMODELS.

Feel free to then include this feature as part of vit, deit, and beit

[chainyo]

What does this PR do?

Added DeiT OnnxConfig to make this model available for conversion

@chainyo

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Also pinging @LysandreJik, @lewtun and @NielsRogge because he did the implementation of DeiT.

Btw Did you try to convert one DeiT model with your add ?
If so you could add the converted model to the ONNXConfig for all organization, it would be awesome!

[lewtun]

Thanks for adding DEIT to the ONNX exporter @rushic24 ! Overall this looks great and I think it would be neat if we can also include the masked image modelling feature :)

[lewtun]

You can add something like

"masked-im": AutoModelForMaskedImageModeling

to the _TASKS_TO_AUTOMODELS attribute of the FeaturesManager. I don't think we currently support this feature for TensorFlow, so no need to include this in __TASKS_TO_TF_AUTOMODELS.

Feel free to then include this feature as part of vit, deit, and beit

[0xrushi]

Thanks @lewtun , I just added those.

I'm trying to add a README to ONNXConfigForAll,

How does onnx work in ViT?
I tried the code below, but its not working

from transformers import ViTFeatureExtractor, ViTModel
import torch
from datasets import load_dataset
from onnxruntime import InferenceSession


dataset = load_dataset("huggingface/cats-image")
image = dataset["test"]["image"][0]

feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTModel.from_pretrained("google/vit-base-patch16-224-in21k")

inputs = feature_extractor(image, return_tensors="pt")

session = InferenceSession("onnx/model.onnx")

# ONNX Runtime expects NumPy arrays as input
outputs = session.run(output_names=["last_hidden_state"], input_feed=list(inputs))

[chainyo]

How does onnx work in ViT?
I tried the code below, but its not working

Could you check on netron.app what are the inputs of the onnx converted model and then check if the input_feed is correct. Btw it should be a Dict not a List.

[0xrushi]

How does onnx work in ViT?
I tried the code below, but its not working

Could you check on netron.app what are the inputs of the onnx converted model and then check if the input_feed is correct. Btw it should be a Dict not a List.

And inputs itself is a dictionary

The input as seen from netron is pixel_values, I did try dict(inputs), inputs, list(inputs)

[0xrushi]

I just figured it out. You can find it here https://huggingface.co/OWG/DeiT :-)

[chainyo]

I just figured it out. You can find it here https://huggingface.co/OWG/DeiT :-)

It seems really good! 🤗

[lewtun]

Thanks for iterating on this @rushic24 and great job on pushing a model to the Hub 😍 !

This LGTM so gently pinging @sgugger for final approval

[lewtun]

For reviewers: we picked masked-im as the name to match the pre-existing masked-lm used for language modeling

[sgugger]

Thanks for the PR. It all looks good but there is a datasets submodule added in this PR that should be removed before it can be merged.

[lewtun]

Thanks for the fix!