YOLO v11x significantly outperforms all PaliGemma-2 variants on COCO val2017 (36,781 objects, 80 classes), achieving mAP@0.5:0.95 of 0.536 compared to the best PaliGemma variant (10B-finetuned) at 0.296.
This represents an 81% performance gap, demonstrating that specialized object detectors remain substantially superior to vision-language models (VLMs) for this task.
PaliGemma-2 exhibits severe limitations on small objects:
AP_small: 0.370AP_small: 0.097 (10B-finetuned)PaliGemma-2 demonstrates architectural constraints in object discovery:
mAR@100: 0.685 (finds 68.5% of all objects)mAR@100: 0.399 (finds 39.9% of all objects)Notably, PaliGemma mAR@10 (0.396) and mAR@100 (0.399) show minimal improvement (+0.3pp), indicating the model cannot generate more valid detections even when allowed higher detection limits.
This contrasts sharply with YOLO’s +4.4pp improvement, demonstrating fundamental architectural limitations rather than calibration issues.
All models were evaluated using identical low-threshold settings (conf_threshold=0.001, iou_threshold=0.7) to measure theoretical capability rather than production performance.
This approach reveals:
The apparent paradox - YOLO having lower per-class precision (0.03-0.65) but higher AP (0.536) - is explained by metric definitions:
conf>=0.001, measuring TP/(TP+FP) at one thresholdYOLO’s aggressive detection strategy (recall=0.93) generates many predictions, lowering precision at minimal thresholds but providing excellent precision-recall trade-off across the full confidence range.
PaliGemma’s conservative strategy (recall=0.54) shows higher precision at low thresholds but cannot find enough objects to compete in AP.
Higher resolution does not improve performance beyond 448px, suggesting computational or attention mechanism constraints.
Increasing classes_per_call beyond 10 typically degrades performance, indicating the model struggles with multi-class simultaneous detection.
PaliGemma-2 28B with 4 classes-per-call achieved only 0.009 mAP, the worst result despite being the largest model tested. This catastrophic underperformance may be related to an observation from the original paper, which notes that “a possible factor related to the relatively worse transferability of PaliGemma 2 28B is that the underlying Gemma 2 27B model is trained from scratch, as opposed to the 2B and 9B models, which are distilled.”
This training methodology difference could explain why the 28B variant fails to leverage its larger capacity effectively for object detection fine-tuning, demonstrating that distilled models may exhibit superior transfer learning properties compared to models trained from scratch.
Models perform well on:
Critical failures on:
These patterns indicate PaliGemma-2 relies on distinctive visual features and struggles with scale, multiplicity, and occlusion.
VLMs are not production-ready for object detection: The 2-3x performance gap between YOLO and PaliGemma-2 on COCO demonstrates specialized detectors remain essential for serious detection tasks.
Architectural constraints dominate: Low recall even at minimal confidence thresholds (conf=0.001) indicates fundamental model limitations, not calibration issues. Small object detection remains a critical weakness.
Optimal configuration for PaliGemma-2 inference: 10B parameter model, 448px resolution, 1 class per call, fine-tuned for closed-set setup.
Research directions: small object detection, multiple object handling, and scaling efficiency (28B underperforms 10B) require architectural innovations rather than parameter scaling.
This analysis demonstrates that while vision-language models show promise for multimodal tasks, they currently lack the architectural specialization required for competitive object detection performance on standard benchmarks like COCO.