Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make CLIPFeatureExtractor accept batch of images as torch.Tensor. #14650

Open
fcakyon opened this issue Dec 6, 2021 · 6 comments
Open

Make CLIPFeatureExtractor accept batch of images as torch.Tensor. #14650

fcakyon opened this issue Dec 6, 2021 · 6 comments

Comments

@fcakyon
Copy link
Contributor

@fcakyon fcakyon commented Dec 6, 2021

🚀 Feature request

Make CLIPFeatureExtractor (or any FeatureExtractor in general) accept batch of images as torch.Tensor.

Motivation

Currently batch of images as torch.Tensor are not treated as a batch, it has to be a List[torch.Tensor] but it is not the case when using native Pytorch DataLoader. Can we update this line so that it accepts batches as torch.Tensor. Maybe we can check if the tensor has 4 dimension then assume it is a batch?

@patil-suraj @TobiasNorlund

@aphedges
Copy link
Contributor

@aphedges aphedges commented Dec 6, 2021

@fcakyon, why was I mentioned here? The only time I edited this file was to add a missing space in an error message.

@fcakyon
Copy link
Contributor Author

@fcakyon fcakyon commented Dec 7, 2021

My apologies @aphedges. Saw your name on PR.

@patil-suraj
Copy link
Member

@patil-suraj patil-suraj commented Dec 7, 2021

Thanks for the issue!

Since a batch of images is a very common input I think we should support this. Pinging @NielsRogge, @sgugger

@fcakyon Also note that when resizing, the feature extractors convert the tensors to PIL images, so the most efficient way to use these is to pass a list of PIL images.

@fcakyon
Copy link
Contributor Author

@fcakyon fcakyon commented Dec 7, 2021

@patil-suraj I was using as:

PIL Images > Pytorch DataLoader > batch as torch.Tensor > CLIPFeatureExtractor > CLIPModel

But in this workflow CLIPFeatureExtractor treats batch as torch.Tensor as a single image tensor and as a result CLIPModel gives error.

Is the intended usage different?

@patil-suraj
Copy link
Member

@patil-suraj patil-suraj commented Dec 14, 2021

We could support batch as tensors but as explained in the above comment it won't be efficient. So I would recommend passing a list of PIL images.

@jaketae
Copy link
Contributor

@jaketae jaketae commented Dec 29, 2021

@patil-suraj Do you still see supporting batched tensors as necessary/beneficial? It seems pretty straightforward (e.g. checking ndim), but I thought I'd ask for confirmation before jumping into a draft PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants