What does the AI need to know about your database to get the SQL right?
· Toni Martir, arquitecto de software
Someone asks the AI assistant for the October sales-by-customer report. The AI writes the SQL statement (the technical instruction that pulls the data out of the database), the report runs first time without a single error, and the total does not match the one from accounts: the statement added up amounts including VAT and counted the cancelled invoices.
The AI did not misread anything: it read the only thing it had, the database catalogue, which is the list of its tables and columns with the type of each column. That says nothing about what each piece of data means. To get it right, the AI needs, above all, three things spelled out in words: what each column is, how the tables join and what each code stands for.
The October sales report: what the catalogue says and what the AI is missing
An example with three columns from an invoicing database. If invoice 125 exists in series A and in series R, joining the lines by number alone adds each invoice's lines to the other one. All three statements run without an error: the figure comes out wrong with no warning.
What the AI receives
To write the statement, the AI starts from the schema: the description of the database it receives with each question. Oracle spells out what its assistant, Select AI, sends to the AI: the name of each table, its columns and any comments written for them. Without comments, only the names are left, and in the databases behind BIRD, a public benchmark for these assistants, table and column names «are frequently abbreviated» and hard to understand.
A few lines of explanation, twenty more right answers
BIRD also measured how much the missing part weighs. It brings together 95
large databases from 37 professional fields, and each question comes with a
hint written by experts: for instance, that in its basketball database a centre
is stored as pos = C. In the tests its authors published in 2023, GPT-4 returned the right
result for 34.88 out of every 100 questions without the hints, and for 54.89
with them. Same model, twenty more correct answers in every hundred.
What to write down for it
Databricks compares its assistant to a new analyst joining the company: to understand what the data represents, it relies on the descriptions of the tables and columns. Oracle lists what is worth describing, among other things: what each column is for, what it means to the business, what units it is in and which values it accepts, and which keys join one table to another.
Those descriptions can live in the database itself, as comments on each table and column: Oracle adds them to what the AI receives once an option is switched on. They are written once and arrive with every question. Describing the tables in the question's area is enough, because the whole database will not fit in what the AI reads at once, as I explained when looking at RAG.
Describing the schema cuts down the errors, but the figure still has to be checked; I explained how to do that without being a programmer in how you know whether the AI figure is right.
So what does it need to know?
What the names leave out: what each column is, how the tables join and what each code means. Once that is written in words, the AI receives it with the schema and gets more answers right.
Since version 4.0.22, the Report Manager designer has local schemas for its AI copilot: a file on your computer, read from the catalogue of each connection, where you describe the tables, columns, relationships and allowed values. If the database belongs to Odoo, SAP Business One, SAP R/3, Business Central or Dynamics 365 Finance and Operations, the schema comes ready-made from the schema library, which recognises the system from the database's tables. In the SQL editor of each of the report's datasets you choose the local schema or one from the Reportman AI cloud.
Sources
- Oracle, DBMS_CLOUD_AI Package,
section «Profile Attributes», attribute
object_list: «For translation natural language to SQL, the object name, object owner, object columns and comments are sent to the AI provider using HTTPS requests.» - Jinyang Li et al., Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD, arXiv, version 3 of November 2023): in the abstract, «95 databases with a total size of 33.4 GB, spanning 37 professional domains»; section 3.3, «database table and column names are frequently abbreviated, which are difficult to understand» and «"center" can be represented by "pos = C" in the database professional_basketball»; Table 2, «Testing Data» column, GPT-4: 34.88 «w/o knowledge» and 54.89 «w/ knowledge» (+20.01).
- Databricks, Curate an effective Genie Agent: «Think of Genie as a new data analyst joining your company. Like any new team member, Genie needs clear context to be effective. It relies on quality table and column descriptions to understand what the data represents».
- Oracle, Select AI Concepts, section «Metadata Enrichment»: «Candidate information to include: Table and column descriptions: purpose, business definitions, units, and allowed value ranges Keys and relationships: primary/foreign keys, join paths», among others.
- Oracle, Examples of Using Select AI, section «Example: Improve SQL Query Generation»: «If you have table and column comments in your database tables, enable "comments":"true" parameter in DBMS_CLOUD_AI.CREATE_PROFILE function to retrieve table level and column level comments. The comments are added to the metadata of the LLM for a better SQL generation.»
- Report Manager, what is new in version 4.0.22: «Local schemas for the AI copilot — describe the tables, columns, relations and allowed values of a database in a file on your computer, read from the catalog of the connection», «choose them, with the cloud schemas, in the SQL editor of the datasets» and «Import from the schema library: ready-made schemas of ERP systems (Odoo, SAP Business One, SAP R/3, Business Central and Dynamics 365 Finance and Operations), with the system detected from the tables of your database».
Toni Martir, arquitecto de software · · sources checked on