InstructCV
InstructCV is an open-source PyTorch implementation presented at ICLR 2024 that transforms text-to-image diffusion models into versatile vision generalists through instruction tuning. Developed by researchers including Yulu Gan and Ahmed Alaa, this project establishes a unified language interface for computer vision, allowing diverse tasks to be executed via natural language prompts rather than specialized model architectures. The core innovation involves casting traditional computer vision tasks such as segmentation, object detection, depth estimation, and classification as text-to-image generation problems. By training a diffusion model on a multi-modal dataset constructed with paraphrased task instructions, InstructCV steers the model from pure image generation to instruction-guided visual analysis. The software supports PyTorch 1.5 and above and includes a pretrained checkpoint capable of handling multiple benchmarks like NYUv2, ADE-20K, and COCO. It leverages the InstructPix2Pix architecture and provides