SERA: Evaluating LLM-based semantic ranking over structured tables

LLM-based data systems are moving beyond text-to-SQL toward semantic query processing, where natural language requests are executed through operators that combine relational processing with LLM-driven inference. A central operator in this setting is SemanticRank, or semantic top-k: given a user intent and a set of relational records, the system orders rows according to a user-defined criterion. Current systems implement this operator as a single black-box LLM prompt. This abstraction hides a key execution problem: the reliability and cost of semantic ranking depend on the criterion being evaluated, the evidence available in the table, and the physical strategy used to compare candidate records. We study semantic ranking as a logical database operator and introduce a multi-domain benchmark for identifying the factors that govern reliable execution over tabular data. Our framework evaluates five dimensions: criterion complexity and specification, computable versus semantic execution, evidence availability, and ranking strategy. We compare proprietary, open-source, and reasoning-oriented LLMs across datasets, specification regimes, evidence conditions, and execution strategies using rank-aware metrics. Our analysis shows that SemanticRank cannot be implemented reliably as a single prompting primitive. Ranking quality is shaped by criterion structure, and formula-based specifications help reasoning-oriented execution. Real-world names can introduce prior-knowledge bias that anonymisation reduces. Full execution is competitive on small candidate sets, while partitioning changes the cost-quality trade-off without consistently improving accuracy. These findings identify key factors for reliable SemanticRank execution.