Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. However, applying this reasoning to multimodal domains, where understanding text and images together is essential, remains a substantial challenge. To address this, we introduce WinoVis, a novel dataset specifically designed to probe text-to-image models on pronoun disambiguation within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel evaluation framework that isolates the models' ability in pronoun disambiguation from other visual processing challenges. Evaluation of successive model versions reveals that, despite incremental advancements, Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally surpassing random guessing. Further error analysis identifies important areas for future research aimed at advancing text-to-image models in their ability to interpret and interact with the complex visual world.

Source: Official arXiv:2405.16277 abstract; public preprint version; ACL Anthology record cross-checked.

Problem and motivation

Text-to-image models can generate visually plausible images without resolving which entity an ambiguous pronoun refers to. Existing WSC-style reasoning evaluations are mainly textual and do not isolate multimodal pronoun disambiguation from image-generation artifacts.

Method and contribution

WINOVIS contains 500 WSC-adapted scenarios generated through GPT-4 prompting and manually filtered for textual ambiguity, illogical content, visual indistinctiveness, and redundancy. The evaluation uses Stable Diffusion generations, DAAM cross-attention heatmaps, caption filtering, 90th-percentile heatmap thresholding, and IoU-based overlap/decision rules. A single pronoun-to-entity association is accepted when the pronoun heatmap crosses IoU 0.4 with one or the stronger of the two referents.

Findings and evidence

The paper evaluates Stable Diffusion versions labeled 1.0/1.5/2.0 and XL in Tables 2-3; the setup prose says 1.1/1.5/2.0/XL. SD 2.0 reports 56.7% precision, 24.2% recall, 34.1% F1, and 36.1% certainty, with 55 correct, 42 incorrect, and 172 neither outcomes in the table. SDXL produces mostly unusable heatmap decisions. Error analysis shows much weaker handling of visually distinct entities than disparate entities.

Limitations and future directions

Limitations: Entity separation, semantic entanglement, captioned images, DAAM availability only for open Stable Diffusion models, bias, and limited scenario diversity affect validity. The metrics assess heatmap/association behavior rather than human-level or task-functional multimodal reasoning. GPU/CPU, OS, runtime, and replication details are unknown.

Future work: Improve entity separation and entanglement filtering, study bias, expand context and entity diversity, and develop interpretability methods for more diffusion models.

Sources and identifiers

• Published version published_public_page
• Public preprint · PDF public_full_text
• arXiv version arXiv 2405.16277

When to cite this paper

Cite this paper when your work uses or compares wINOVIS as a 500-scenario benchmark for pronoun disambiguation in text-to-image generation.

• WINOVIS as a 500-scenario benchmark for pronoun disambiguation in text-to-image generation.
• a DAAM heatmap plus IoU evaluation pipeline that separates entity overlap, caption artifacts, and pronoun-association decisions.
• the SD 2.0 result when documenting that visually convincing diffusion outputs can still show weak multimodal pronoun resolution.
• the distinct-versus-disparate entity error analysis when motivating harder visual common-sense benchmarks.

Citation

@inproceedings{ezzatiJivan2024picturingambiguity,
  author = {Brendan Park and Madeline Janecek and Naser Ezzati-Jivan and Yifeng Li and Ali Emami},
  title = {Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge},
  year = {2024},
  booktitle = {ACL 2024 Main Conference},
  pages = {355-374},
  publisher = {Association for Computational Linguistics},
  doi = {10.18653/v1/2024.acl-long.22},
  url = {https://aclanthology.org/2024.acl-long.22/}
}

Park, B., Janecek, M., Ezzati-Jivan, N., Li, Y., & Emami, A. (2024). Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge. In ACL 2024 Main Conference (pp. 355-374). https://doi.org/10.18653/v1/2024.acl-long.22

B. Park, M. Janecek, N. Ezzati-Jivan, Y. Li, and A. Emami, "Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge," in ACL 2024 Main Conference, pp. 355-374, 2024, doi: 10.18653/v1/2024.acl-long.22