Open class detection models can identify objects from both known and novel/unseen classes, adapting to new object categories dynamically, while closed-set detection models are limited to a fixed set of predefined classes they were trained on and cannot recognize objects outside this set.
Released in 2024 by Ultralytics, YOLOv11 is a relatively recent update to the famous YOLO object detection architecture. These models are trained and evaluated on a widely-used COCO2017 (‘Common Objects in Context’) dataset which contains annotations for training object detection. Train2017 contains 118K images, Val2017 contains 5K images which are used for evaluation. COCO2017 comprises 80 object categories in total.
COCO dataset and YOLO series traditionally represent the closed-set object detection setup when the class set (80 object classes in total) is defined before any detection model is created. Any such model is then trained to detect objects of the pre-defined class set and is unable to output predictions for any class outside the set without a special adjustment.
Paligemma-2 is a family of vision-language models released by Google in 2024. These models can be evaluated on more than 30 tasks, including image captioning and visual question answering (VQA). Object detection is also among the suite of tasks that Paligemma-2 is designed to perform.
Paligemma-2’s flexibility stems from the fact that a vision-language model may be flexibly prompted by means of natural language. Some of Paligemma-2 family models were trained specifically to solve the open class detection task, allowing user-driven detection via natural language prompts.
This work focuses on:
A prompt that enables the object detection mode must be of a specific form:
Model outputs a set of bounding boxes in the format:
We first need to reproduce the COCO2017 benchmark results for the well-known YOLOv11 family. Once a reliable and reproducible benchmark result is achieved, it can be used to evaluate any other model including one from the Paligemma-2 family.