[CLIP] allow loading projection layer in vision and text model #18962
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
patil-suraj
changed the title
| @dataclass | ||
| class CLIPVisionModelOutput(ModelOutput): | ||
| """ | ||
| Base class for model's outputs that also contains a pooling of the last hidden states. |
There was a problem hiding this comment.
| Base class for model's outputs that also contains a pooling of the last hidden states. | |
| Base class for vision model's outputs that also contains a pooling of the last hidden states. |
There was a problem hiding this comment.
Also didn't we create this class for the projection output? The pooling output is also part of the previous output "BaseModelOutputWithPooling" -> should we maybe focus on the image_embeds here instead of the last_hidden_state?
| @dataclass | ||
| class CLIPTextModelOutput(ModelOutput): | ||
| """ | ||
| Base class for model's outputs that also contains a pooling of the last hidden states. |
There was a problem hiding this comment.
| Base class for model's outputs that also contains a pooling of the last hidden states. | |
| Base class for text model's outputs that also contains a pooling of the last hidden states. |
|
Just to understand better - what is a checkpoint that uses such a projection layer? Is that really part of CLIP or rather of the model built on top of CLIP? Also if one uses the Wondering if we should instead create a new head here instead of forcing it into the same class? Or is this an architecture that the official CLIP is using often? |
The projection layers are already part of the CLIP model. Those are used to convert the final hidden states of vision and text model into clip embedding space. transformers/src/transformers/models/clip/modeling_clip.py Lines 880 to 881 in a261147 The reason we added So if a user needs either the text embeds or vision embeds they need to load the whole clip model, or write a custom wrapper module to include the projection layer (which is what we did for safety checker in diffusers https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_clip.py#L880)
Good point! Then maybe we could add |
I'd prefer that solution too! |
This PR allows optionally loading projection layers in
CLIPVisionModelandCLIPTextModel.There are some tasks in
diffusersthat rely either on text or image projections but not both. This will allow using only the related modality model instead of loading the full model or having to write wrappers.