Poster
in
Workshop: Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)
A Comprehensive Benchmark of Human-Like Relational Reasoning for Text-to-Image Foundation Models
Colin Conwell · Tomer Ullman
Keywords: [ relational reasoning ] [ Text-to-image foundation models ] [ performance benchmarks ] [ psychology ]
Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only 22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.