CLARITY Framework Overview with Rule and LLM Components
Figure 1: Overview of the CLARITY framework . The symbols and denote rule-based and LLM-based components, respectively.
Paper context
Paper title: CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems Abstract: NL2SQL systems deployed in industry settings often encounter ambiguous or unanswerable queries, particularly in interactive scenarios with incomplete user clarification. Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution, overlooking realistic failure modes. We introduce Clarity, a framework for automatically generating an NL2SQL benchmark with multi-faceted ambiguities and diverse user behaviors across both single- and multi-turn settings. Using a constraint-driven pipeline, Clarity transforms executable SQL into ambiguous queries, augmented with grounded conversational continuations and schema-level metadata. Empirical evaluation on Spider and BIRD shows that leading NL2SQL systems, including those based on strong LLMs, suffer significant performance degradation under multi-faceted ambiguity. While these systems often detect ambiguity, they struggle to accurately localize and resolve the underlying schema-level sources. Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2SQL systems. Passages referencing this figure: not assess whether systems correctly localize the schema elements responsible for uncertainty. Consequently, high ambiguity-detection accuracy may mask failures in schema-level identification and resolution. We argue that fine-grained, schema-grounded evaluation enables more diagnostic assessment of NL2SQL robustness, particularly under multi-faceted ambiguity and realistic interaction settings. Figure 1: Overview of the CLARITY framework . The symbols and denote rule-based and LLM-based components, respectively. Schema Entity Use Case AMBROSIA AmbiQT NoisySP SQUAB PRACTIQ MMSQL BIRD-INTERACT CLARITY (OURS) Type of Data - Single turn Single turn Single turn Single turn Multi turn Multi turn Multi turn Single and multi turn Column Unifacet Amb ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Unifacet Unans ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
