Description: Evaluates an LLM's ability to disambiguate user requests for generating SQL queries based on the given business rules and database schema. A question can either be answered using the schema, a combination of the schema and the question, or requires additional information to be answered.

Number of Samples: 245

Language: Portuguese

Provider: iFood

Evaluation Method: Accuracy score to evaluate the LLM's ability to accurately classify user requests into one of the three categories: schema-based, schema-and-question-based, or additional-information-required. Ground truth was curated using domain expert labels.

Data Collection Period: April 2024 - May 2024