A first draft, not a finished score: testing whether frontier models can read music accurately

01Background

Over the past fifteen years, I’ve devoted evenings to arranging orchestral scores for viola ensembles.² The creative part is satisfying: choosing pieces that suit the register, revoicing dense chords, discovering counter-melodies, and deciding which viola carries the main line. The part that fatigues me is transcribing: manually entering hundreds of bars of notes, accidentals, rests, and dynamics into notation software before starting the arrangement.

Every arranger dreams of an app that perfectly converts sheet music into notes you can manipulate. The technology that supports this is called optical music recognition (OMR). You take a photo or PDF of sheet music, export it to a clean MusicXML file, and open it in a music notation editor, where the creative process starts.

Optical character recognition (OCR), a related technology, is already highly successful at digitizing printed text (one vendor advertises accuracy above 99%¹). Earlier this week, while attending a Carnegie Hall concert, I took a photo of the program and asked Claude to calculate the concert runtime based on the listed works. It correctly identified every piece and estimated the runtime ranges for them. However, music isn’t text. While text flows linearly from left to right, sheet music is closer to a 2D graph, where a symbol’s position affects its meaning. To transcribe a score, an OMR system must:

• Read up and down as well as across. Multiple staves sound simultaneously, and polyphonic voices must align vertically in time.
• Treat spatial position as semantic pitch. A notehead represents a different pitch depending on the line or space it occupies on or around a five-line staff.
• Parse composite visual glyphs. Pitch and duration aren’t separate tokens. They’re expressed together through noteheads, stems, flags, beams, and augmentation dots.
• Track state across bars. The clef at the start of the system, the key signature, and earlier accidentals in the measure dictate the pitch of every subsequent note.
• Solve duration arithmetic. Notes and rests in every bar must sum up exactly to the time signature, accounting for tuplets, ties, and partial pickup bars.
• Attach floating expressions. Dynamics, slurs, hairpins, and articulations sit above, below, or across the notes they modify.

There are a lot of constraints, but I’m optimistic that we can have great OMR too.

02Motivation

A few years ago, seeking to shortcut the process of transcription, I ran a few music scores through Audiveris,³ an open-source OMR app that outputs MusicXML. After wrestling with its output, I realized fixing its mistakes took significantly longer than entering the parts by hand.

Audiveris reads a page like an assembly line. It runs about twenty steps in sequence. First, it finds the staff lines to understand the visual scale. Next, it identifies beams, noteheads, and stems. The rest of the symbols are identified by a classifier neural network. At the end, it reconstructs the rhythm. Each step builds on the ones before it.

This assembly line can suffer from error compounding, in which a single stem or dot missed in an early step can cascade into a measure that doesn’t have enough beats at the end. This causes spurious rhythms to appear or, in many cases, crashes the app. It’s frustrating to use because it’s slow and there’s no guarantee you’d be able to save its output.

03Metric

Recently, a friend and fellow violist introduced me to some exciting news. OpenAI’s Astra⁴ release landing page advertised dramatic OMR accuracy improvements over its prior models. The announcement cited a score called OMR-NED, which is short for OMR normalized edit distance.⁵

OMR-NED is an error metric based on two sums:

1. The edits. How many notation primitives you must insert (I) or delete (D) to transform the transcription into the reference score. An incorrect symbol counts as two edits: one to delete it and one to insert the correct one.
2. The size of both scores. The total symbol count of the transcription (N_pred) and the reference score (N_ref) combined. A “symbol” here is an atomic visual piece of notation: a notehead, a stem, an accidental, a beam, a dot, or a tie.

Divide the edits (1) by the combined size (2) and you get a normalized distance from zero to one. Zero means an exact match; one means the two scores have nothing in common. OpenAI advertised “accuracy” as one minus that number:

Accuracy = 1 − OMR-NED = 1 − (I + D) / (N_pred + N_ref)

Since it counts visual notation symbols rather than raw pitches alone, slurs, dynamics, and beams carry weight alongside notes. A lower OMR-NED indicates a more accurate transcription.

04Method

When I saw OpenAI’s claim of Astra’s vastly improved OMR accuracy from 0.19 to 0.84 (an OMR-NED reduction from 0.81 to 0.16) over GPT-5.6 Sol, I needed to see the results for myself. I assembled a test set of scanned string quartet pages and ran them through four frontier models from OpenAI and Anthropic, alongside the latest version of Audiveris. OpenScore,⁶ an open project that transcribes public domain scores, provided human-engraved reference transcriptions for each page as an answer key. I evaluated every output with musicdiff¹¹ and colored every incorrect note red.

• The pages: Eight pages from five string quartets by Schubert, Haydn, Elgar, Dvořák, and Beethoven. The images are authentic scans of historical printed editions from IMSLP,⁷ downloaded from a dataset compiled for OMR benchmarking⁸ that pairs each scan with OpenScore’s transcription. The models could only access the raw scan and never the answer key. I selected the first six pages at random before running the tests, and then added two famous openings so errors could be quicker to identify by ear.
• The models: GPT-6 Astra and GPT-6.1 Sol from OpenAI (run in Codex), and Claude Opus 5.5 and Claude Fable 5.1 from Anthropic (run in Claude Code). Every run was an isolated session with the same prompt and no access to the web or answer keys. One difference that slipped past my prompt: Codex allowed the OpenAI models to execute shell commands, which they used to crop and zoom into sections of the scan. Claude Code was configured with read-only image access.
• The settings: Each model processed each page at medium and high reasoning effort, using two sources: the full page alone, and the full page accompanied by separate cropped images of each musical system, which AI suggested might improve the accuracy. That produced 128 total runs, one attempt per condition.
• The intermediate format: Rather than asking models to output thousands of lines of deeply nested, verbose MusicXML that can exhaust context and output tokens, the prompt requested ABC notation,⁹ a compact, human-readable text shorthand for scores. Every ABC output was converted to MusicXML using LEGATO’s standard converter.¹⁰
• The baseline: Audiveris 5.11.0 processed each of the eight pages once. Since the scans are relatively low resolution (about a third of the 20-pixel staff height Audiveris recommends), I enlarged them 3× first.
• Cost: I logged token usage and runtime across all runs to evaluate the trade-offs.

05Results: accuracy

The chart below shows each system’s OMR-NED across all eight pages, evaluated on full-page scans at high effort. Lower is better.

For detailed results of every high-effort transcription, complete with error markings and synthesized audio playback, see OMR test: full results.

The standout performance came from GPT-6.1 Sol on page 1, the opening of Schubert’s Death and the Maiden quartet. Almost the entire page transcribed cleanly, scoring 0.032, or 0.968 accuracy, which is the best single score in the entire experiment.

At first glance, the engraved render looked virtually identical to the reference. However, in the musicdiff output, a red note surfaced in bar 3 of the cello part. Schubert steps the bass line down from E to D. Sol transcribed an E again, repeating bar 2’s pitch. A few bars later (bars 7 to 9), the cello switches into tenor clef, and Sol transcribed four consecutive notes exactly one staff position too high.

I couldn’t understand why it made these mistakes. It had gotten almost everything else correct, and these notes weren’t any harder to see in the source.

I re-ran page 1 in a completely fresh session. The exact same repeated E returned in bar 3. Another cello mistake in bar 18 repeated identically, though other errors shifted and the overall page score degraded to 0.063.

Since its reasoning isn’t available in the logs, I prompted Sol in a fresh session to brainstorm why it might have made the bar 3 error. While I can’t trust that it matches the transcription session’s reasoning, it sounded like a mistake a human could make:

“The image shows bar 2’s upper E on the second ledger line above the bass staff, while bar 3’s D is in the space above the first ledger line, one staff position lower. I likely carried over bar 2’s pitch because both bars have the same low A followed by an accented upper half note, overlooking that small vertical difference.”

An inattentive or inexperienced human sight-reader might make a similar mistake of assuming the notes repeated, especially without listening to the other instruments for clues about the musical direction.

Image quality didn’t explain the variance in accuracy either. The two Haydn pages are sharp, high-contrast, modern prints, yet both landed in the bottom half of Sol’s results. The faded, low-contrast Elgar page landed in the top half.

Use the player below to hear how errors sound across different pages. Select a piece at the top, choose a model and effort level, and listen to the original, the transcription, or isolate the wrong notes.

06Results: listening test

Transcriptions with high error rates sound like an agonizing first rehearsal (an experience I’m very familiar with). If you know the piece intimately, incorrect pitches or rhythms stick out immediately. However, on the top-performing runs, mistakes can camouflage themselves in the harmony.

Each round below plays the AI transcription alongside the answer key. See if you can identify the original.

07Results: token cost

For every model tested, high effort improved accuracy over medium effort, but used many times more tokens. Sol consumed 19 times more tokens at high effort (surging from 0.50M to 9.65M tokens across the benchmark) to improve its OMR-NED substantially, from 0.549 to 0.171. Astra burned 9 times more tokens (0.22M to 1.96M) to move from 0.418 to 0.295. The Claude models roughly doubled their token usage, achieving smaller accuracy bumps.

Page by page, more tokens didn’t reliably track better scores.

In Codex, Sol took advantage of its shell access to crop and zoom into sections of the scan on six of the eight pages. Those six pages accounted for 99% of Sol’s total token consumption.

However, the data showed that cropping and zooming did not guarantee better scores. The two pages where Sol never zoomed used under 40,000 tokens each and achieved OMR-NED scores of 0.152 and 0.212. Meanwhile, page 7 (Schubert) burned 2.7M tokens cropping and re-cropping the image, only to end up as Sol’s worst result of the test at 0.415.

The runs that were provided system crops followed a similar path. While feeding separate line images alongside the full page helped the Claude models (aggregate OMR-NED improved on Fable from 0.659 to 0.526 and Opus from 0.541 to 0.496), it hurt Sol’s and Astra’s performance (Sol’s aggregate OMR-NED worsened from 0.171 to 0.321). My untested hypothesis is that the crops drew Sol’s and Astra’s attention away from the full page, which offers context like recurring motifs.

For a long transcription, spending the tokens might still be cheaper than hours of human entry. But for AI developers, this highlights the subtle hazards of unconstrained agentic vision: tool use like cropping and zooming can burn tokens and simultaneously fragment the global context needed to reason about the task.

08Results: error patterns

While a larger sample size of transcriptions might tell a different story, a few types of failure surfaced during this test:

1. The answer key differs from the scan. In the Elgar Violin II part, the third and fourth notes of a measure are on the staff’s bottom line (E), which Sol, Astra, and Audiveris transcribed as E. However, OpenScore’s answer key specifies G for both notes. OpenScore either used a different edition or made a typo (I plan to investigate and follow up). Despite transcribing what the scan shows, the models receive a lower accuracy score when graded against different notes.

on the page: E, E

OpenScore answer key: G, G

2. Alto clef is misread. In what felt like a cruel viola joke, Anthropic’s Opus and Fable models struggled most with the viola part. Viola music is primarily written in alto clef (where middle C sits on the middle line), but both Claude models repeatedly transcribed the notes as if they were written in treble clef. This resulted in pitches consistently transposed by a seventh. This clef blindness caused 142 wrong notes in Opus and 111 in Fable, while Sol and Astra recognized the correct clef. This might surface a pretraining imbalance in which alto clef is underrepresented.

alto clef Opus, Fable: A4
on the page: B♭3

3. Errors hide in intermediate notation serialization. In ABC notation, you can specify relative octaves by using commas, case, and apostrophes: C,, C, c, c' is the sequence of octaves from low to high register, where middle C is C. On Schubert’s page 7, Sol omitted the trailing commas for the cello’s lower octave. That syntactic slip caused 120 of the 170 pitch errors in that part. For AI practitioners, this demonstrates how intermediate serializations can cause single-point-of-failure bottlenecks: the model’s visual perception of the note’s attributes was accurate, but omitting a single character in the ABC token stream created drift in the MusicXML export.

Sol wrote: G4
on the page: G3

4. Ledger lines degrade visual perception. Notes on two or more ledger lines have higher error rates than ones within the five-line staff. Astra’s error rate increased from 16% to 27%, Opus’ from 51% to 61%, and Fable’s from 60% to 78%. Sol went the other way, with fewer wrong pitches on two or more ledger lines (6%) than within the staff (13%). In the Elgar example below, a D♭6 sits on two ledger lines above the treble staff, tightly squeezed between a flat accidental, a phrasing slur, and the expression ten. Audiveris and every model misread it.

on the page: D♭6

Scan quality failed to predict pitch accuracy elsewhere. On the clean, high-contrast Dvořák page 4:

• Sol and Astra achieved near perfection, missing only 1% and 3% of pitches.
• Opus misread 68% of the pitches, its worst page by that measure.

Astra missed 39% of pitches on pages 1 and 5, compared to 10% or fewer across the other six pages.

09Conclusions

OMR hasn’t caught up to OCR, and it isn’t ready to fully automate score transcription. Today’s frontier models are getting more accurate and could make automated transcription practical soon. GPT-6.1 Sol, OpenAI’s most recent frontier model at the time, output the most accurate transcriptions in the high-effort category.

In this experiment, Audiveris failed to produce an exportable file on three of eight pages, and the files it did produce were tangled in broken measure durations. By contrast, every frontier model at high effort produced a valid, openable MusicXML file for every single page. Instead of entering notes or attempting to untangle spurious tuplets by hand, I received an editable rough draft in minutes.

However, I couldn’t predict where the mistakes would be.

Treat a model’s transcription as a first draft and check every bar for mistakes before using it.

Pitch accuracy wasn’t associated with scan quality, and other than ledger lines, I couldn’t identify any visual patterns linked to error rates.

For now, I must continue to proofread my arrangements, but for the first time since I started, generating a first draft with frontier models and OMR might save me time.

10Next experiments

The limited sample size (eight pages with one run per setting) leaves many open questions. Here are a few items that I’d like to try next:

• Run every page more than once. The page 1 Sol rerun repeated two cello mistakes and changed the rest. More runs might help me discover more error patterns.
• Give every model the same tools. Codex let Sol and Astra crop and enlarge the scan, and Sol spent most of its tokens on the pages where it did. Allowing the Claude models to use the same tools would make the comparison fairer.
• Test rendered pages as well as scans. Rendering OpenScore’s answer key files removes noise characteristics like paper texture, bleed-through, and skew. Comparing them with scans would separate errors caused by the noise from errors in reading the music.
• Have a model check its own work. The model could render its transcription, compare the result with the scan, and fix what differs before it answers.
• Check the harmony. Music21¹³ can label valid chords in each bar and flag notes that don’t fit for verification.
• Test open OMR models. MuSViT¹² and LEGATO¹⁰ were trained on sheet music and might make a cheaper first pass than a frontier model.
• Try xhigh and max effort. Every model scored better at high effort than at medium, so the higher levels would show whether that trend continues.
• Use OMR-NED as a training signal. OMR-NED can be computed automatically for the OpenScore scores, so it could serve as the reward for fine-tuning an open model with reinforcement learning.
• Proofread against the scan. Audiveris and some commercial OMR apps can lay the scan over the transcription. Checking by hand that way might be the fastest path to an accurate transcription.