CM3leon is a state-of-the-art generative model that enables both text-to-image and image-to-text generation. It is a multimodal model that combines the functionality of autoregressive models with low training costs and inference efficiency. The model is trained using a recipe adapted from text-only language models, including retrieval-augmented pre-training and multitask supervised fine-tuning stages.CM3leon achieves state-of-the-art performance in text-to-image generation, even with five times less compute than previous transformer-based methods. It is capable of generating sequences of text and images conditioned on arbitrary sequences of other image and text content, expanding the functionality of previous models that were limited to either text-to-image or image-to-text generation.The model has been multitask instruction-tuned for both image and text generation, resulting in significant improvements in tasks such as image caption generation, visual question answering, text-based editing, and conditional image generation. CM3leon outperforms Google’s text-to-image model and achieves an impressive Fréchet Inception Distance (FID) score of 4.88 on the widely used image generation benchmark, establishing a new state of the art.CM3leon’s capabilities shine in complex object generation and text-guided image editing tasks. It excels in generating coherent imagery that follows input prompts, even when dealing with constraints and compositional structures. Moreover, the model performs well in tasks such as text-guided image editing, text-to-image generation with compositional prompts, and answering questions about images.Despite being trained on a relatively small dataset, CM3leon’s zero-shot performance compares favorably against larger models trained on more extensive datasets. It demonstrates the potential of retrieval augmentation and the impact of scaling strategies on autoregressive model performance. CM3leon’s versatility and excellent performance make it a valuable tool for various vision-language tasks.
