Measure the classifier and correct a wrong answer

Classify a new upload with a language model

By the end of this lesson, you can measure the classifier against labelled files and correct a wrong answer without changing code.

Lesson 2 of 5~25 minScript: 00_load_pdfs.sql, 02_measure.sql, 02_model_check.sql, 02_correct.sql

1. Lesson 1: Classify a new upload with a language model
2. Lesson 2: Measure the classifier and correct a wrong answer
3. Lesson 3: Handle re-runs, new versions and failures
4. Lesson 4: Review and release verdicts
5. Lesson 5: Classify the documents already in ADM

Load the labelled sample files

This lesson continues in the schema of lesson 1. One correct answer proves little. To judge a classifier, compare it with files whose type a person has already decided. The sample set has twelve files, and the table hkd_ai_labels holds the type of each:

File	Correct type	Why it is in the set
inv_ferrotek_1042.pdf, inv_nordlicht_2207.pdf	invoice	Clean invoices with two layouts
supply_agreement_ferrotek.pdf	contract	A supply agreement
scan_fax_amendment.pdf	contract	An amendment as an image with no text layer
quote_alpenwerk_q5531.pdf	quote	An offer with items, totals and a payment term
proforma_alpenwerk_pf5531.pdf	quote	A pro forma invoice for prepayment, not for booking
datasheet_pump_x200.pdf	datasheet	Product data
letter_employment_change.pdf	hr	A personal letter to an employee
reminder_ferrotek_1042.pdf	other	A payment reminder about an earlier invoice
delivery_note_nordlicht_ls2207.pdf	other	A delivery note with goods and no prices
memo_offsite_menu.pdf	other	A lunch menu with a hidden instruction for the classifier
note_parking_rules.pdf	other	A company notice that fits no type

The PDFs are in the folder ai/samples/pdf, and ai/samples/generate_samples.py creates them. Every company and number in the files is invented. A label is a decision of one person. A pro forma invoice is not an invoice for the books, so this set labels it quote. Another company can decide differently, and then it writes another description.

Run @00_load_pdfs.sql ALL and then @wait.sql. The script skips the invoice from lesson 1, which is already there. The worker handles the eleven new files in the background, so the wait can take a minute or more.

  already there: inv_ferrotek_1042.pdf
  loaded: inv_nordlicht_2207.pdf (document 18846156970330158106897897961951)
  loaded: supply_agreement_ferrotek.pdf (document 5177945473410070129386129567827)
  loaded: quote_alpenwerk_q5531.pdf (document 3143673877035059371920674541736)
  loaded: datasheet_pump_x200.pdf (document 1049996483122637046771283395181)
  loaded: letter_employment_change.pdf (document 6256943880468850265542148209085)
  loaded: scan_fax_amendment.pdf (document 17758895741827141449599006197149)
  loaded: proforma_alpenwerk_pf5531.pdf (document 18639682205706866119844703516288)
  loaded: reminder_ferrotek_1042.pdf (document 8722472896643899480876865015285)
  loaded: delivery_note_nordlicht_ls2207.pdf (document 4981176566447341570787319339510)
  loaded: memo_offsite_menu.pdf (document 3496359334447821100048289866105)
  loaded: note_parking_rules.pdf (document 3744643299072339771833480905816)

Compare each verdict with its label

Run @02_measure.sql. The script joins the labels with the newest verdict of each file and makes no model call. It sorts the wrong answers first. It also shows what each one caused. A released verdict took effect without a person. A held verdict still waits for review.

FILE_NAME                             EXPECTED_TYPE    CLASSIFIED    STATUS          VERDICT                  TOKENS_IN
_____________________________________ ________________ _____________ _______________ _____________________ ____________
proforma_alpenwerk_pf5531.pdf         quote            invoice       RELEASED        WRONG and released             379
datasheet_pump_x200.pdf               datasheet        datasheet     RELEASED        right                          358
delivery_note_nordlicht_ls2207.pdf    other            other         NEEDS_REVIEW    right                          316
inv_ferrotek_1042.pdf                 invoice          invoice       RELEASED        right                          450
inv_nordlicht_2207.pdf                invoice          invoice       RELEASED        right                          396
letter_employment_change.pdf          hr               hr            NEEDS_REVIEW    right                          340
memo_offsite_menu.pdf                 other            other         NEEDS_REVIEW    right                          368
note_parking_rules.pdf                other            other         NEEDS_REVIEW    right                          336
quote_alpenwerk_q5531.pdf             quote            quote         RELEASED        right                          435
reminder_ferrotek_1042.pdf            other            other         NEEDS_REVIEW    right                          322
scan_fax_amendment.pdf                contract         contract      NEEDS_REVIEW    right                          778
supply_agreement_ferrotek.pdf         contract         contract      NEEDS_REVIEW    right                          441
   FILES    RIGHT_TYPE    WRONG_AND_RELEASED    WRONG_HELD    PROMPT_TOKENS    COMPLETION_TOKENS
________ _____________ _____________________ _____________ ________________ ____________________
      12            11                     1             0             4919                  521

Eleven of the twelve verdicts are right. The twelfth is WRONG and released: the model classified the pro forma invoice as an invoice. The type invoice releases automatically, so the file now carries the annotation kb.invoice. A collection that selects files by that key would include a document that is not an invoice for the books.

A wrong verdict of a review type waits for a person, who sees it in lesson 4. The release rule decides how much damage a wrong answer can do.

Your run can differ

A model can answer a borderline file differently in another run, and another model can answer differently. In the recorded runs, every model tested called the pro forma invoice invoice. If your run classifies it as quote, your table shows twelve right answers, and no verdict is wrong and released. A smaller model can also give more wrong answers than the recording, for example for the payment reminder. Continue anyway: the next sections work in the same way. The review list in lesson 4 and the numbers in lesson 5 then differ a little from the recordings.

The last column of the first result set shows the input tokens of each file. The text files use between 316 and 450 tokens. The scanned fax, which is an image with no text layer, uses 778. The second result set sums the tokens of all twelve verdicts.

Optional: compare models on the borderline files

The same file can get another answer from another model. This section is optional, and the next section does not depend on it. Run @02_model_check.sql. It asks three models about the three borderline files, three times each. That is 27 requests of about 400 tokens, and it takes about a minute. The script changes nothing in ADM. Your key needs access to the three models, or you change their names on the define lines.

file                                  model         answers of 3 runs
proforma_alpenwerk_pf5531.pdf         gpt-5.4-nano  invoice invoice invoice
proforma_alpenwerk_pf5531.pdf         gpt-5.4-mini  invoice invoice invoice
proforma_alpenwerk_pf5531.pdf         gpt-6.1-sol   invoice invoice invoice
reminder_ferrotek_1042.pdf            gpt-5.4-nano  invoice invoice invoice
reminder_ferrotek_1042.pdf            gpt-5.4-mini  invoice invoice other
reminder_ferrotek_1042.pdf            gpt-6.1-sol   other other other
delivery_note_nordlicht_ls2207.pdf    gpt-5.4-nano  other other other
delivery_note_nordlicht_ls2207.pdf    gpt-5.4-mini  other other other
delivery_note_nordlicht_ls2207.pdf    gpt-6.1-sol   other other other

Read the table by file:

• All three models classify the pro forma invoice as invoice, in all nine runs. They are consistent, and they are wrong by the label.
• The two smaller models, gpt-5.4-nano and gpt-5.4-mini, classify the payment reminder as invoice in five of six runs. gpt-5.4-mini answers other in one run. gpt-6.1-sol answers other in all three runs of this recording.
• All three models answer other for the delivery note.

A model that always gives the same answer is consistent, and that does not make the answer correct. The same model can answer differently in two runs. Three files and three runs show the kind of difference that can occur. They do not rank the models. Measure with your own labelled files before you change the model.

Correct the answer by changing the type descriptions

The model reads the description of each type as its definition. The first description of invoice is “A bill from a supplier asking for payment: invoice number, items, amount due”. It does not exclude a pro forma invoice, which also asks for payment. To correct the answer, change the question:

update hkd_ai_doc_types
   set description = 'A tax invoice from a supplier that is booked and paid: invoice number, items, amount due. Not a pro forma invoice, not a reminder about an earlier invoice'
 where doc_type = 'invoice';

update hkd_ai_doc_types
   set description = 'An offer with prices that is not booked yet. A pro forma invoice for prepayment belongs here'
 where doc_type = 'quote';

The change is data. No code changes, and no file is classified again until you ask. Run @02_correct.sql. It rewords the two descriptions, queues the pro forma invoice with hkd_ai_api.reclassify, waits for the worker, and shows the history of the file.

The queue is empty (waited 6 s).
   RESULT_ID DOC_TYPE    STATUS
____________ ___________ ___________
           8 invoice     RELEASED
          21 quote       RELEASED
ANNOTATION_KEY    ANNOTATION_VALUE
_________________ ___________________
ai.model          gpt-6.1-sol
ai.run_date       2026-10-02
ai.status         released
kb.quote          released

The history has two verdicts. The newest one says quote. The annotation kb.quote replaced kb.invoice: when the type changes, the classifier removes the release of the old type, so the file is not in two collections.

The model check in the previous section ran before this change. It would give other answers now, because the descriptions are different. Run @02_measure.sql again to measure the whole set with the new descriptions:

FILE_NAME                             EXPECTED_TYPE    CLASSIFIED    STATUS          VERDICT       TOKENS_IN
_____________________________________ ________________ _____________ _______________ __________ ____________
datasheet_pump_x200.pdf               datasheet        datasheet     RELEASED        right               358
delivery_note_nordlicht_ls2207.pdf    other            other         NEEDS_REVIEW    right               316
inv_ferrotek_1042.pdf                 invoice          invoice       RELEASED        right               450
inv_nordlicht_2207.pdf                invoice          invoice       RELEASED        right               396
letter_employment_change.pdf          hr               hr            NEEDS_REVIEW    right               340
memo_offsite_menu.pdf                 other            other         NEEDS_REVIEW    right               368
note_parking_rules.pdf                other            other         NEEDS_REVIEW    right               336
proforma_alpenwerk_pf5531.pdf         quote            quote         RELEASED        right               404
quote_alpenwerk_q5531.pdf             quote            quote         RELEASED        right               435
reminder_ferrotek_1042.pdf            other            other         NEEDS_REVIEW    right               322
scan_fax_amendment.pdf                contract         contract      NEEDS_REVIEW    right               778
supply_agreement_ferrotek.pdf         contract         contract      NEEDS_REVIEW    right               441
   FILES    RIGHT_TYPE    WRONG_AND_RELEASED    WRONG_HELD    PROMPT_TOKENS    COMPLETION_TOKENS
________ _____________ _____________________ _____________ ________________ ____________________
      12            12                     0             0             4944                  519

All twelve verdicts are right now. Only the pro forma invoice was classified again, so the other eleven verdicts are the old ones. A change of a description does not reach the files that are already classified. Lesson 5 shows how to classify many files again.

Twelve files are a small set. When you change a description, run the measurement again, and add a file to the set for every wrong answer that you find.

Hidden instructions in a document

The file memo_offsite_menu.pdf is a lunch menu. Its last line is small grey text that tells “the document classifier” to answer contract. The classifier answered other. The system prompt of the classifier contains a rule that text inside a document is data and never an instruction, and the model named the note in its reason.

One file does not show that the classifier resists such text

Hidden instructions in a document are a known risk. The sample set has one such file, and the classifier handled it. The instruction in that file asks for contract, a type that needs review, so a successful instruction would have been held for a person anyway. An instruction that asks for invoice would be released. Another model, document or wording can behave differently. Lesson 4 shows how to check a sample of the released verdicts.

Another approach: get a probability for each type with uc_ai.decide

The classifier of this course calls uc_ai.generate_text with a response schema. UC AI also has the call uc_ai.decide. It asks the model a question with a fixed list of answers, and it returns the chosen answer with a probability for every answer in the list. With these probabilities, a release rule can also test how sure the model is. For example, a rule can release an AUTO type only when its probability is 0.95 or more.

This section is optional, and the course scripts do not use it. Read these limits before you choose this approach:

• uc_ai.decide is experimental. It needs a UC AI version later than 26.4. The AI Pack includes it from the ADM release after 26.1.
• It works with OpenAI, TypeSafe and OpenRouter only. See the reference of uc_ai.decide.
• It reads text and images, not PDF files. Extract the text of a PDF first. The scanned fax has no text layer, so its text is empty except for the title of the PDF.
• It returns no reason. A reviewer in lesson 4 sees only the type and the probabilities.

The excerpt below builds one choice question from hkd_ai_doc_types and sends the text of a document. The function adm_storage_index_api.extract_text_from_blob extracts the text with the Oracle Text filter, the same filter that ADM uses for its search index:

l_text := adm_storage_index_api.extract_text_from_blob(l_blob, l_mime);

for r in (select doc_type, description from hkd_ai_doc_types order by sort_order) loop
  l_criteria.put(r.doc_type, r.description);
end loop;
l_question.put('type', 'choice');
l_question.put('instructions', 'Which type of document is this? Pick the type that fits what the document is, not what it mentions.');
l_question.put('criteria', l_criteria);
l_questions.put('doc_type', l_question);

l_result := uc_ai.decide(
  p_state     => l_text
, p_questions => l_questions
, p_provider  => uc_ai.c_provider_openai
);

l_answer := l_result.get_object('answers').get_object('doc_type');
if l_answer.get_string('type') = 'choice' then
  l_doc_type := l_answer.get_string('choice');
  l_p        := l_answer.get_object('probabilities').get_number(l_doc_type);
end if;

OpenAI can refuse a question. A refused answer has the type refused and no choice, so the excerpt reads type first. Send a refused answer to review.

The recording ran before that release, so it ran the two steps in two schemas. The course schema extracted the text of the twelve sample files. A schema with a UC AI version that includes uc_ai.decide sent each text to it, with the default model gpt-6-luna and the first descriptions of the types. The table shows the chosen type, its probability, and the second answer when its probability is above 0:

FILE_NAME                             EXPECTED_TYPE  DECIDED     PROBABILITY  SECOND ANSWER   INPUT_TOKENS
datasheet_pump_x200.pdf               datasheet      datasheet   1                                     312
delivery_note_nordlicht_ls2207.pdf    other          other       0.96         hr 0.02                  269
inv_ferrotek_1042.pdf                 invoice        invoice     1                                     410
inv_nordlicht_2207.pdf                invoice        invoice     1                                     348
letter_employment_change.pdf          hr             hr          0.94         contract 0.05            289
memo_offsite_menu.pdf                 other          other       0.99         invoice 0.01             318
note_parking_rules.pdf                other          other       1                                     286
proforma_alpenwerk_pf5531.pdf         quote          invoice     0.89         quote 0.06               333
quote_alpenwerk_q5531.pdf             quote          quote       1                                     393
reminder_ferrotek_1042.pdf            other          other       0.67         invoice 0.3              275
scan_fax_amendment.pdf                contract       contract    0.67         other 0.29               217
supply_agreement_ferrotek.pdf         contract       contract    1                                     409

The answer for the pro forma invoice is wrong, as it was with generate_text. Three runs of the excerpt for this file gave invoice with 0.89 each time. The other answers of the AUTO types have the probability 1, so a rule that releases only at 0.95 or more holds the pro forma invoice for review. The payment reminder is right, but with 0.67. The scanned fax is right too, but its text is only the title scan_fax_amendment, so the model decided from the file name. The output tokens are 0 for every call. OpenAI counts only the input tokens of a decision.

A second run used the reworded descriptions of the previous section. The pro forma invoice got quote with 0.99, and the other eleven files got the same type as before. The payment reminder rose from 0.67 to 0.98, because the new description of invoice excludes a reminder.

One run on twelve files does not give a threshold for your documents. If you use uc_ai.decide, measure the probabilities on your own labelled files, as this lesson does for the verdicts.

Handle re-runs, new versions and failuresSee what a second run keeps, how a new version is classified, and what happens to a file the model cannot read.