Improving Zero-Shot Models with Label Distribution Priors
We propose a new approach for zero-shot labeling of large image datasets, CLIPPR (CLIP with Priors), which adapts zero-shot models for regression and classification on unlabelled datasets. Our method does not use any annotated images. Instead, we assume a prior over the label distribution in the dataset. We then train an adapter network on top of CLIP under two competing objectives: i) minimal change of predictions from the original CLIP model ii) minimal distance between predicted and prior distribution of labels. Our method is effective and presents a significant improvement over the original model.