Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add zero-shot object detection pipeline #18445

Open
NielsRogge opened this issue Aug 3, 2022 · 6 comments
Open

Add zero-shot object detection pipeline #18445

NielsRogge opened this issue Aug 3, 2022 · 6 comments
Labels
Good First Issue

Comments

@NielsRogge
Copy link
Contributor

@NielsRogge NielsRogge commented Aug 3, 2022 •
edited

Feature request

We currently have OWL-ViT in the library, which is capable of performing zero-shot object detection.

It would be great to have a pipeline for this task, with the following API:

from transformers import pipeline

pipe = pipeline("zero-shot-object-detection")
pipe("cats.png", ["cat", "remote"])

This pipeline could default to the https://huggingface.co/google/owlvit-base-patch32 checkpoint. Also check out the demo notebook that showcases the model.

This can be implemented similar to other pipelines (we already have one for zero-shot image classificaiton with CLIP, so it would be very similar to that one). For an example PR that added a pipeline, see #11598.

Motivation

Pipelines are great for abstracting away all the complexity for quick inference with a model.

Your contribution

I can assist with this, together with @Narsil.
@LysandreJik
Copy link
Member

@LysandreJik LysandreJik commented Aug 3, 2022

As seen with @alaradirik this morning, this could also be leveraging the custom pipeline feature that was implemented last week, especially if this pipeline works with a very limited number of artchitectures.

cf #18079

@NielsRogge
Copy link
Contributor Author

@NielsRogge NielsRogge commented Aug 3, 2022

cc @alaradirik

@MocktaiLEngineer
Copy link
Contributor

@MocktaiLEngineer MocktaiLEngineer commented Aug 3, 2022

Can I take this up and work on it?

@alaradirik
Copy link
Member

@alaradirik alaradirik commented Aug 4, 2022

Hi @MocktaiLEngineer! Of course, you can also @NielsRogge, @Narsil or me if you need any help or have any questions.

@LysandreJik
Copy link
Member

@LysandreJik LysandreJik commented Aug 5, 2022

cc @sgugger as we chatted about it as well

@sgugger
Copy link
Member

@sgugger sgugger commented Aug 5, 2022

I was going to do a custom pipeline on this today actually, as the dev advocates want more examples of it :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Good First Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@alaradirik @LysandreJik @sgugger @NielsRogge @MocktaiLEngineer