OpenAI published research on 17 June 2020 describing Image GPT, a model that applies the GPT architecture to image generation and classification tasks [source]. The work demonstrated that unsupervised pre-training on image data, using the same transformer architecture developed for language, could produce coherent image completions and achieve competitive performance on classification benchmarks.

The model was trained on sequences of pixels rather than text tokens. Images were resized to low resolution—32×32 or 64×64 pixels—and flattened into one-dimensional sequences. The transformer then predicted pixels autoregressively, similar to how GPT-2 predicts the next word in a sentence.

Image GPT achieved 96.3% accuracy on CIFAR-10 classification when fine-tuned with labels, placing it among the top unsupervised methods at the time. On image completion tasks, the model generated plausible continuations of partial images, though outputs remained constrained by the low resolution required for computational tractability.

OpenAI noted that the approach required substantial compute resources. Training the largest model, iGPT-XL with 6.8 billion parameters, involved processing 1.4 million images over extended periods. The team acknowledged that pixel-level autoregression was less efficient than methods operating on compressed representations or latent codes.

The research explored whether insights from language model scaling would transfer to vision. While Image GPT demonstrated that transformers could learn useful image representations without labels, the computational cost and resolution limits highlighted challenges distinct from text modelling. OpenAI released model weights and code for three model sizes to support further research.

The work preceded later developments in vision transformers and diffusion models, which adopted alternative approaches to image generation and understanding.