How a solo iOS app follows an actor through a scene

Actors rehearse with a reader: someone who speaks the other parts so you can practise yours. Readers are hard to find at 11pm before an audition. Scriptmate is that reader. You import a PDF of your sides, tap your character, and the app plays every other line, listens while you speak, and delivers the next cue when you finish.

I designed, built and shipped it alone, on iOS 16 and up. It went live in January 2026. What follows is the part that was actually hard.

1. Knowing when the line is over

Speech recognition is built to transcribe. Scriptmate needs something narrower and stranger: given a line the actor is supposed to say, decide the instant they have said enough of it. Actors paraphrase, restart, swallow the last word and skip half a sentence. Waiting for an exact transcript means the cue never fires.

Two engines behind one protocol

The app talks to a VoiceRecognizer protocol with two implementations. On iOS 16 to 25 it uses SFSpeechRecognizer, forced on-device, with partial results on and the current line's words passed as contextual strings so the recognizer is biased toward the script.

request.shouldReportPartialResults = true
request.requiresOnDeviceRecognition = true
request.taskHint = .dictation
request.addsPunctuation = false

let lineWords = text.components(separatedBy: .whitespaces).filter { !$0.isEmpty }
var hints = Array(Set(lineWords + contextualStrings))
if hints.count > 100 { hints = Array(hints.prefix(100)) }
request.contextualStrings = hints

On iOS 26 it switches to the new SpeechAnalyzer and SpeechTranscriber. That path keeps one transcriber open for the whole rehearsal and hot-swaps vocabulary per line with setContext, which removes the 200 to 400 ms cold start per line, sidesteps the roughly one minute task timeout of the old API, and gives per-word confidence and audio time ranges. Audio is converted to 16 kHz mono int16 with AVAudioConverter before it goes in.

The matcher

Transcribed tokens are aligned greedily and in order against the normalised expected line. A match may skip up to two expected words (lookahead 2), and a resume pass restarts from the furthest matched position if recognition restarts mid-line. Word equality is a chain, cheapest first:

• Exact match after normalisation: lowercase, diacritics stripped, apostrophes collapsed, digits spelled out per language with NumberFormatter.
• A curated homophone table per language: to, two, too in English, silent-h pairs in Spanish, ce, se in French, das, dass in German.
• Words of three characters or fewer must match exactly, and a negation guard stops shouldn't matching should.
• Levenshtein similarity of at least 0.80 for short words and 0.70 for longer ones.
• Double Metaphone for English, so their and there collide the way they do in the recognizer.

Korean is tokenised per Hangul syllable, because the recognizer's spacing rarely agrees with the script's. Coverage is absolute rather than a ratio: a cue of seven words or fewer may leave one word unmatched, a longer cue two.

The state machine

Each line runs through a small actor with four states: idle, listening, advancing, stalled. A final result advances. A partial result advances if the tail confidence, the mean over the last three segments, is at least 0.4; otherwise the app needs two consecutive matching partials. A 700 ms refractory window carries across lines so one loud word can't fire twice, and eight seconds of silence moves the line to stalled and logs it.

static let minTailConfidence: Float = 0.4
static let refractoryMs: Double = 700
static let requiredConsecutiveMatches = 2
static let silenceTimeoutSeconds: Double = 8.0

The bug that taught me to timestamp audio

The nastiest failure came from SpeechAnalyzer finalising the previous utterance one or two seconds into the next line. Three incidental words from the old line, it, the, of, would walk the cursor to seven of nine and fire the cue before the actor had said anything. Two fixes. First, a minimum share of genuinely matched words (0.6), so position alone can't advance. Second, an AudioClock that stamps every buffer, so a result is only considered if it ends after the line started, and finals are filtered to words whose time range begins after the cue start:

guard result.range.end > cueStartTime else { return } // stale

if awaitingPreviousFinal {
    if result.isFinal {
        awaitingPreviousFinal = false
        let filtered = Self.wordsSpoken(startingAfter: cueStartTime, in: result.text)
        ...
    } else if result.range.end > cueStartTime + Self.awaitFinalLimit { // 3 s safety valve
        awaitingPreviousFinal = false
    } else {
        return // hold volatile results until the previous utterance finalises
    }
}

There is an honest limit here. Two lines that share four of five words, "Hey, I need some help here" and "Wait, I need some help!", cannot be told apart by the matcher. A test documents that, and the fix lives upstream in the timestamps rather than in cleverer string matching.

2. Keeping the reader out of the recording

Self tape mode records video of the actor while the app speaks the other parts. If the reader's voice reaches the microphone, the take is ruined. The session is configured for playAndRecord in videoRecording mode with A2DP allowed, so the reader plays through the actor's AirPods or wired headphones while the camera mic captures only the room.

var options: AVAudioSession.CategoryOptions = [.allowBluetoothA2DP, .mixWithOthers]
if isBluetoothHFPInput { options.insert(.allowBluetoothHFP) }
try audioSession.setCategory(.playAndRecord, mode: .videoRecording, options: options)

if selectedOutput.portType == .builtInSpeaker {
    try audioSession.overrideOutputAudioPort(.speaker)
} else {
    try audioSession.overrideOutputAudioPort(.none) // headphones or Bluetooth: default routing
}

Without headphones the default output is the earpiece, which is far quieter than the speaker, and the loud speaker option is labelled in the UI as the one that may bleed into the mic. The rehearsal screen has its own single session owner that prefers Bluetooth, then a headset mic, then USB audio, and publishes connect and disconnect events for the banner in the main view.

One detail that cost a day: AVAudioSession.interruptionNotification does not fire for the app's own text to speech. So the view model explicitly pauses the recognizer before the reader plays, rather than relying on the system to tell it.

The reader itself wraps AVSpeechSynthesizer with a 0.5 second watchdog that recreates a stuck synthesizer, and a plausibility check on didFinish: a finish that arrives faster than 20 ms per character is treated as bogus and retried once. Voices resolve through six tiers, from a character's assigned Personal Voice on iOS 17, through premium and enhanced system voices, down to the language default, with the novelty voices blocklisted. Since September a composite player prefers a take the user recorded themselves and falls back to synthesis.

3. Turning a messy PDF into a scene

Sides arrive as PDFs, photos of pages, and occasionally screenshots of a text message. The client decides how to send them. Photos get on-device orientation scoring and Vision OCR. Small PDFs go up as base64 for the model to read visually. Larger PDFs are split per page with PDFKit; if a document yields fewer than 500 characters it is treated as scanned and OCR is merged in.

The backend is a single Cloudflare Worker calling Claude Haiku 4.5 at temperature 0 with forced tool use, so the tool's JSON schema is the output contract: a title, characters in order of first appearance, and dialogue turns. Stage directions and parentheticals are excluded by the prompt, shared lines are emitted once with the speakers joined. Long scripts are chunked at 12,000 characters with a one-page overlap and parsed eight chunks at a time; pages are fenced with markers and the model tags every turn with its page, which is how overlap is removed on reassembly.

Everything around the model is about not losing a user's upload. Requests carry an idempotent attempt ID so a retried import replays the finished reply instead of paying twice. Failures and successes are captured to R2 with the Cloudflare ray ID, and the app builds a prefilled support email with the file attached when an import fails. Usage is metered per user in KV, billed only on success. Haiku input costs one dollar per million tokens and output five, so the emitted schema drops anything the client can derive, which alone saved about ten percent of output tokens.

The latency work was the least glamorous and the most valuable. Output is bound at roughly 45 to 50 tokens per second, so a 200 KB King Lear with 1,048 turns and 26 roles simply cannot be fast. But it can stop failing: after one focused session it went from a 502 to 184 seconds, a 108 KB musical from 304 to 138 seconds, and a 39 KB play from a CPU kill to 81 seconds. The three root causes were the ten millisecond CPU limit of the free Workers plan, a content filter refusing one chunk of Lear, and the model's default temperature. Four of the first fourteen importing users had re-uploaded identical files with no logged error, which is what sent me looking.

4. How it was built

SwiftUI with MVVM and a composer per screen: Foo.swift, Foo+ViewModel.swift, Foo+Composer.swift, where compose() wires production dependencies and preview() wires fakes. Every external dependency is a protocol with a fake beside it, which is why 642 Swift Testing tests run without a network, a microphone or a store.

The process leaned on Claude Code as a reviewer and a second pair of hands. The repository's CLAUDE.md reads as an operating manual: how to build, how to test, never hand edit the project file, always run the suite after changing logic, and a house style rule that user copy never uses em dashes. Architecture notes for the data layer, view hierarchy and test strategy live next to the code, written for the agent as much as for me, and every change ships through the same review and the same 642 tests regardless of who typed it.

Monetisation is StoreKit 2 wrapped by RevenueCat, with a fallback to raw StoreKit if RevenueCat fails: monthly at $9.99, annual at $69.99, seven day trials, and a paywall whose headline is chosen from a three question onboarding survey. Review requests are earned rather than scheduled: an import chain or a completed rehearsal, a ten minute suppression after any error, at most three asks spaced 21 and 30 days apart, and a sentiment gate so unhappy users reach support instead of the App Store.