What is a schema?
A schema is the set of rules you follow when you label training pages. A Smart Extract model learns those rules from your examples and applies them to pages it has never seen. You never write the rules down for the model. You show them.
One page, many schemas
Smart Extract lets you decide what a model outputs: which text to transcribe and in what order, how to group it into regions, which entities to tag, which styles to mark, and how to classify the page. Every one of those decisions is a rule. The full set of rules is your schema.
This guide uses a single running example, a page from the 1876 probate calendar for England and Wales. Each section shows the same page labelled under a different set of rules, so you can see exactly how a change in the schema changes the output.
- 01TranscriptionCharacters, line breaks, what to transcribe, reading order
- 02CoordinatesOptional. Learned only if you label them
- 03TablesGrids with merged cells and header rows
- 04RegionsSemantic or visual grouping, and nesting
- 05Named entitiesLabels on spans of text, which can overlap
- 06Text stylesBold, italic and other visual marks
- 07Page classificationKey-value pairs for the whole page
- 08Structured dataFrom markup to a data model
One rule above all: be consistent
Whenever the model produces output, it is choosing between alternatives: this character or that one, a line break or none, region A or region B, a person tag or no tag. To choose, it needs something on the page that tips the decision. That something can be visual, such as bold type, a position on the page, or a symbol. It can be linguistic, such as the words themselves or what comes before them. Two things that look the same and read the same cannot be labelled differently. The model would have nothing to go on.
The other half is consistency. The model learns the pattern between pages and their labels, including every inconsistency in the labels. If the same kind of thing is labelled two ways in your training data, the model learns that it is sometimes one and sometimes the other.
Two conditions bound what can be learned
The information must be recoverable from the page – either visible to a trained eye or inferable from linguistic and structural context. A category that requires outside knowledge cannot be learned.
It must be annotated consistently: the model learns the pattern between pages and their annotations, including any inconsistency in them.
Rule of thumb: if a person looking at the page cold could not make the classification, the model cannot learn it.
Transcription conventions
Some of the rules in a schema are ones you already apply, perhaps without noticing.
Characters
When you see a glyph in the image and type a character in the editor, you are applying a rule. The capital E below is normally transcribed as “E”. You could transcribe it as “e” instead, and the model would learn to output “e” whenever it sees similar shapes. The same goes for diacritics, abbreviations, or a symbol like the one on the right: write “M”, “male”, or the Unicode character ♂. Any of these can be learned, as long as you always do the same thing.
“E” or “e”
“M”, “male” or ♂Line breaks
Even keeping or dropping line breaks is a rule. Transcribe each visual line as its own line, and the model outputs lines. Transcribe an entry as one block, and the model outputs blocks.
What to transcribe
The most natural schema transcribes every piece of text on the page. But that is also a choice. Leave out the header on every training page and the model learns never to output headers. Transcribe only the name and the value under “Effects”, and the model learns to output just those two fields from each entry and nothing else. Transcribe only the handwritten additions, and it learns to skip the print.
Reading order
The order in which you transcribe is the order the model learns. On this two-column page you could read entry by entry, taking the name on the left and then its paragraph on the right, or read the whole left column first and then the right.
Coordinates are optional
Earlier generations of models needed a baseline for every line before they could read it. Smart Extract models do not. You can transcribe without any markup on the image, which makes ground truth much faster to produce. But coordinates are a schema rule like any other: if none of your training pages have them, the model outputs none. If they do, it learns to output them for the same elements.
You can also mix: coordinates for all lines, or only for lines in the left column. Whether the model outputs coordinates, and whether they are accurate, has no effect on the quality of the transcription itself. They are an additional output, not a prerequisite.
Atlas was trained without coordinates, so it outputs none. A model fine-tuned on pages that carry coordinates will produce them.
Tables
Tables are labelled in the editor as grids of rows and cells, including merged cells and header rows. A Smart Extract model is not limited to one table per page, and other text can sit around a table. Text that visually spills past a cell boundary is still placed in the correct cell, because the model reads the meaning of the text alongside its position.
Regions
Regions are the first schema element that adds information rather than just capturing text. A region groups lines. What the grouping means is up to you. It can be semantic: all lines that belong to one person go into a region called testator. It can be visual: everything in the left column goes into left_column, everything on the right into right_column. Or both at once.
The third version uses something new with Smart Extract: regions can nest. A testator region contains a left-column and a right-column region. On a newspaper page, paragraphs could nest inside articles. Nested regions can be thought of as a tree, and structured that way if you want to export into a database. Regions never overlap, though. Each line belongs to exactly one region at each level.
The more complex the rule, the more examples the model needs. A rule like “a separate region for handwritten marginalia, but only in the body, not in the header” is learnable. It just needs enough pages that show it.
Region coordinates
Whether a region carries coordinates is independent of whether it appears in the output, and independent of line coordinates. You can draw boxes at the column level and none at the testator level, or only at the testator level, or none at all.
Named entities
Named entities are labels on spans of text inside a line. You define them yourself in the collection’s tag settings, and you can call them whatever you like. The model learns to apply a label whenever the words, their context, or their position on the page match what it saw in training.
Whatever the labels, the model needs some way to tell them apart. That can be visual: bold type instead of plain, a position in the top left rather than the top right, text that sits below an image. Or it can be semantic: the word itself and how it is spelt, or the context around it, for instance that it comes after the phrase “who died”. If any of these aspects differ, the model should be able to distinguish one entity from another.
Where a label’s boundary is a matter of judgement, the schema decides. Is “France” an organisation? Is “the government of France”? Whatever you decide, decide it the same way every time. The vaguer or more varied a category, the more examples it needs.
Entities can overlap
Unlike regions, entities can overlap. Here address runs from “late of” to the occupation, and county marks a single word inside it. Each word can belong to any number of entities. Entities can cross line breaks but not region boundaries.
Text styles
Text styles record how text looks: bold, italic, underlined, struck through, subscript, superscript and a few more. Like everything else, they are schematic. You can mark the heavy type on this page as bold and leave italics unmarked. You can also use a style for something other than appearance, for instance marking every date as italic. The model learns whatever pattern you apply.
Page classification
The last kind of label applies to the whole page. In the page metadata editor you add key–value pairs such as Language: English or Type: Printed. A page can carry several values for the same key. The model learns to predict the pairs for new pages.
What can be classified comes back to the one rule. A model could probably learn to sort undated memoranda into 18th and 19th century, because paper, layout and hands change over time. It could not sort them into 1761 and 1762, because nothing on the page distinguishes the two. Unless, that is, each page states its year somewhere in the text. Then the model has the information it needs and the class becomes learnable.
From markup to structured data
Combine these elements and a schema stops being markup and becomes a data model. Regions and entities can be used together to define fields in a relational structure. In practice the fields are often set by the regions, with entities marking values inside them. Classifications add attributes at page level. A model trained this way can produce a full transcript with the structure attached, or just the structured fields on their own.
Export today goes through the standard Transkribus formats. A custom export shape still needs a short script. An interface for defining the export structure directly is in development.
Existing transcripts are already a schema
Every transcript produced in Transkribus so far follows an implicit schema: coordinates for every line and region, usually no entities. You can train a Smart Extract model on that data as it is, and it will reproduce that schema on new pages. Nothing needs relabelling. To use the capabilities above, add the richer labels to the same data.
Try it on a page
Atlas is the pre-trained Smart Extract model. Upload a page and see the structured output it was trained to produce.
The probate calendar page contains public sector information licensed under the Open Government Licence v3.0.