ASS support for webvtt/srt

[tresby]

What should the engine do?

Been playing around with subtitles on my Rivulet build.

Trying to get some things like positioning, colour, font face etc working but these stylings seemed to be stripped.

Is there any chance to add the option to preserve this data from SRT and VTT subs abut like richText with dvb_teletext?

Also been playing around with some server side stuff with my dvb_teletext streams and thought I would mention I managed to get position working with dvb_teletext with -txt_page ass. Is that something that is easy to add to the dvb_teletext part?

Thanks as always

Motivating media or use case

VOD playback for text based subs with styling.

Live TV playback with vtt styling via a custom sub renderer.

Area

Codec / container support

Host app / integration context

Rivulet

Would you be willing to open a PR?

Maybe, with guidance

[superuser404notfound]

Thanks for this, and for the -txt_page ass note in particular, which turned out to be the thread that unpicks the rest.

Short version: the styling you are asking for already reaches the engine on every one of these formats, and it gets thrown away in one place on our side. So this is smaller than it looks, and it is one piece of work rather than three.

Everything already arrives as ASS

libavcodec converts text subtitle formats into an ASS event line before we ever see them, so AVSubtitleRect.ass carries the markup already:

• SRT goes through ff_htmlmarkup_to_ass (libavcodec/htmlsubtitles.c): <b>/<i>/<u>/<s> become {\b1}/{\i1}/{\u1}/{\s1}, <font color=...> becomes {\c&HBBGGRR&}, <font size=...> becomes {\fs<n>}, <font face=...> becomes {\fn<name>}, <br> becomes \N.
• WebVTT (libavcodec/webvttdec.c) maps <i>, <b> and <u> to the same {\i1}/{\b1}/{\u1}.
• dvb_teletext already runs with txt_format=ass here, which is exactly why its colours survive today.

What discards it is our own parser. On the plain-text path SubtitleRectText.cleanASSBody strips every {...} block with a regex, and on the rich-text path coloredRuns(fromASSEventLine:) handles \c / \1c and says so in its own doc comment: "All other override tags are ignored".

So there is no per-format parser to write. One ASS override-tag parser, and SRT, WebVTT, teletext and ASS all light up at once.

The model has to grow first

The thing you pointed at cannot carry what you want yet. SubtitleTextRun is currently colour and nothing else:

public struct SubtitleTextRun: Sendable, Equatable {
    public let text: String
    public let color: SubtitleColor?
}

The shape to take it to:

public struct SubtitleTextRun: Sendable, Equatable {
    public let text: String
    public let color: SubtitleColor?
    public let isBold: Bool
    public let isItalic: Bool
    public let isUnderlined: Bool
    public let isStruckThrough: Bool
    public let fontName: String?
    public let fontSize: Int?
}

New fields defaulted in the initializer so existing call sites keep compiling, plus a cue-level placement type on SubtitleCue for {\anN} and {\pos(x,y)}. Positions in a libavcodec-produced ASS line are in the default ASS play resolution (384x288, ASS_DEFAULT_PLAYRESX/Y in libavcodec/ass.h), so they want normalizing to [0, 1] before they reach a host, the way SubtitleImage.position already is.

SubtitleRectText.teletextBody(fromASSEventLine:) already has the right result shape: rich text when a run carries an attribute, flattened plain text otherwise. Generalized to the full attribute set, unstyled cues stay .text and nothing regresses for hosts handling only that case.

One part will not work, and it is worth knowing up front

WebVTT cue settings (line, position, align, size, vertical) are not converted by FFmpeg at all. webvttdec.c says so at the top of the file:

 * @todo need to support extended markups and cue settings

Only the inline tags are mapped. So VTT bold, italic and underline will work through the above, and VTT positioning will not, until webvttdec learns cue settings upstream. That is a genuine FFmpeg patch rather than something that can be done on this side.

SRT positioning has a similar caveat: it arrives only when the demuxer supplies AV_PKT_DATA_SUBTITLE_POSITION side data, which most .srt files do not carry. A leading {\anN} written into the text does survive ff_htmlmarkup_to_ass, so that much works.

If you want to take it

The whole change is: extend SubtitleTextRun and add the placement type, generalize coloredRuns into a full override-tag parser in SubtitleRectText.swift, then route the text codecs to it in SubtitleDecoder (sidecar) and EmbeddedSubtitleDecoder (in-container). Tests go in Tests/AetherEngineTests/ against raw event lines, so the parser needs no fixture media.

Happy to review as it goes, and equally happy to build it if you would rather just have it. The teletext positioning you asked about falls out of the same parser, since those lines carry the same tags.

[superuser404notfound]

Built and shipped in 5.26.0 (#239), so no need to pick this up yourself.

It went in as described above: one ASS override parser rather than one per format, so SRT, WebVTT, teletext and ASS all light up together.

What a host gets now:

public struct SubtitleTextRun: Sendable, Equatable {
    public let text: String
    public let color: SubtitleColor?
    public let isBold: Bool
    public let isItalic: Bool
    public let isUnderlined: Bool
    public let isStruckThrough: Bool
    public let fontName: String?
    public let fontSize: Int?
}

plus SubtitleCue.placement, a SubtitleTextPlacement carrying the ASS numpad alignment and, when the cue asks for one, an anchor normalized to [0, 1] against the video frame, the same convention SubtitleImage.position uses. \pos is normalized against a declared PlayResX/Y when the ASS header provides one and against libavcodec's 384x288 default otherwise, which is what the synthesised lines use.

Two things worth knowing for Rivulet:

• No option gates it. An unstyled cue still arrives as .text with the same string as before, so existing handling keeps working, and SubtitleCue.text still flattens either body. You only need new code for the cues you actually want styled.
• The teletext positioning you asked about works, since those lines carry the same tags and the parser no longer discards them. \an and \pos land on the cue rather than splitting a run.

Unchanged: preserveASSMarkup still hands over raw event lines and the script header, if you would rather drive libass yourself for a given track.

Leaving this open for the WebVTT half. Cue settings (line, position, align, size, vertical) still do not reach the engine, because libavcodec does not convert them, and that is an upstream change rather than something this side can fix. VTT inline bold, italic and underline work today. If VTT positioning is what you actually need for the live path, say so and it can go on the FFmpeg patch queue.

Thanks for the report, and for the -txt_page ass aside in particular. That was the detail that made it obvious the markup was already arriving and only being thrown away here.

[tresby]

Hey Vincent,

Thanks for looking into this!

Tested an SRT with colour and working great!

For Webvtt - I was trying to get Rivulet to have all subtitles render through the custom renderer, rather than native subs which are less flexible - so would be good to be able to pass the webvtt subs through aether and displayed in the appropriate placement etc - but all good if its not something you want to do :)

For dvb_teletext, I was having some trouble getting positioning to work the way it was with my hls webvtt ffmpeg command in my live tv proxy.

Although I can't really explain it, I asked claude to explain it, I also made a fix for some missed 'tabbed' spaces that weren't being detected.

Hopefully the following is helpful!

Teletext: derive placement from the grid row, and widen the blank-row predicate

Two related things on the 5.26.0 #233 placement work, both specific to libzvbi teletext. Line numbers are Sources/AetherEngine/Subtitles/SubtitleRectText.swift at 5.26.0.

1. Teletext encodes vertical position as a row index, not as \an

#233 derives placement from \an / \pos only:

// line 168
let placement = (alignment == nil && position == nil)
    ? nil : SubtitleTextPlacement(alignment: alignment, position: position)

libzvbi doesn't generally emit \an. It sends a full 24-row page and encodes vertical position as which row the text starts on — expressed as leading \Ns. Some pages do carry an explicit alignment ({\an2}(COW MOOS) is a real example from AU FTA), but most don't, so placement comes back nil for the majority of broadcast captions and the positioning information is lost.

That information is worth keeping: broadcasters move captions to the top of frame specifically to avoid covering on-screen graphics, so it's editorial intent rather than incidental layout.

The engine already recognises this — edgeTrimmed's own comment (line 229) calls it "a row-positioning newline" — but treats it purely as a #107 rendering artefact and strips it.

Where it fits

Everything needed is already in scope at line 168, and edgeTrimmed doesn't discard it until line 170:

var runs = pieces.map { $0.style.run($0.text) }
let placement = ...                        // ← line 168: leading newlines still present
guard let trimmed = edgeTrimmed(runs) else { return nil }   // ← line 170: gone

The styled walker also already normalises both escapes that matter (lines 147-148): \N/\n → \n, and \h → space. So by line 168 the run sequence is exactly the newline-delimited grid the derivation needs.

Must be teletext-gated, since styledRuns is shared with SRT / ASS / WebVTT where a leading newline is not row positioning. teletextBody is already the teletext-specific entry point.

The algorithm (from a working implementation)

This has been running in production in a Python HLS proxy (teletext → WebVTT) for a while, so the shape is field-tested:

# split the ASS text into grid rows on \N / \n; blank rows are KEPT as rows,
# because the row ORDINAL is the positioning information
for i, chunk in enumerate(re.split(r'\\[Nn]', part)):
    if i: rows.append([])
    chunk = chunk.replace('\\h', ' ')      # hard-space padding → real spaces
    if chunk: rows[-1].append((colour, chunk))

first_row = None
for idx, runs in enumerate(rows):
    if not ''.join(t for _, t in runs).strip():
        continue                            # blank grid row: skipped for output,
                                            # but idx already counted it
    if first_row is None:
        first_row = idx                     # ← the identification

# only when the page carried no explicit \anN
if an is None and first_row is not None:
    if first_row < 24 // 2:                 # upper half of the 24-row page
        # → top of frame (equivalent to \an8)

So: the ordinal index of the first non-blank row, upper half → top.

Two things learned the hard way, both worth keeping:

Deliberately coarse — top vs bottom only. Mapping each row to its own line:N% looked more faithful but made captions jitter vertically. Teletext bottom-anchors a block, so a two-row caption starts at row 19 and a one-row at row 21; consecutive cues then rendered at alternating heights and the text visibly jumped between them, which reads as a blink. Only "did the broadcaster move this to the top" carries meaning.

\an present must win, including when it means bottom. An early version tested if not settings to decide whether to fall back to the grid row. {\an2} is explicitly bottom, correctly produces no cue settings, and that empty result was then mistaken for "no information" — promoting bottom-anchored captions to the top of frame. The fallback has to be gated on an is None, not on the derived output being empty.

2. The styled path's blank/trim predicates are narrower than the plain path's

Separate, smaller, and possibly the cause of stray blank lines on coloured cues.

The plain path tests blankness with Unicode-aware trimming (lines 21, 66, 302):

s.trimmingCharacters(in: .whitespacesAndNewlines)

The styled path uses an explicit three-character test (edgeTrimmed, lines 240 and 247):

let d = String(first.text.drop(while: { $0 == " " || $0 == "\t" || $0 == "\n" }))
while let c = s.last, c == " " || c == "\t" || c == "\n" { s.removeLast() }

and collapseInteriorBlankLines (line 266) uses #"\n(?:[ \t]*\n)+"#.

CharacterSet.whitespaces covers Unicode general category Zs, which includes U+00A0 NO-BREAK SPACE; [ \t] does not. So a padding row made of anything other than literal space/tab/newline is trimmed on a plain cue and survives on a styled one — which is the "sub line / blank / sub line" artefact, and it would appear only on coloured pages.

The ASS \h escape is already normalised to a plain space at line 148, so this only bites if libzvbi emits the character directly rather than the escape. Worth checking against a real page rather than assumed — but there's precedent for U+00A0 turning up in caption payloads in this codebase: CEA608Decoder.swift:253 explicitly maps \u{00A0} → " ".

Cheapest fix is to make the styled predicates use the same .whitespacesAndNewlines / CharacterSet test the plain path already uses, so the two paths can't disagree.

Why this matters downstream

Blank rows and column padding sit inside the colour runs, e.g.

{\c&HFFFF00&}\h\h\hWe want to figure out\h\h{\c&HFFFFFF&}\h

so a host can't fix either of these itself: by the time cues arrive, the runs are trimmed and the row ordinals are gone. Both have to happen where the ASS line is still intact.

[superuser404notfound]

Thanks for testing, and for the write-up. The teletext half turned out to be right about a case I would not have found on my own, and the WebVTT half turned out to be right about me being wrong, so all three parts landed.

Teletext placement

The premise is not quite what libzvbi does, but the conclusion holds for a case that does happen.

gen_sub_ass already derives the vertical position from the grid row and emits it as \an:

vertical_align = (2 - (av_clip(i + 1, 0, 23) / 8));
av_bprintf(&buf, "{\\an%d}", alignment + vertical_align * 3);

so thirds rather than halves, with the horizontal component coming from its own leading/trailing analysis. The engine has parsed that since 5.26.0 and hands it over as SubtitleCue.placement.

What matters is that the whole block sits behind is_subtitle_page, and that flag comes from the row-0 header (libzvbi-teletextdec.c:617):

// Check for disabled NEWSFLASH flag and enabled SUBTITLE and SUPRESS_HEADER flags
ctx->subtitle_map[pgno] = (!(flags1 & 0x40) && flags1 & 0x80 && flags2 & 0x01);

When a broadcaster does not set all three, or the header has not been seen yet, the else path writes the whole page instead: no \an, no per-row trim, one " \N" per grid row with the empty ones included. That is exactly when the row ordinal becomes the only carrier of the position, which is the case the Python implementation is handling, and the engine's edge trim was removing it before anything could read it.

So it is in, gated on \an being absent and on teletext only, and it applies ffmpeg's own formula to the ordinal rather than a second scale, so the two paths cannot disagree about where a caption belongs. Both of the hard-won details survive: coarse bands only, and an \an that arrived always wins.

One correction to the reasoning, worth recording because it is what makes the gate load-bearing: on a page libzvbi does flag, leading \Ns are not the grid row at all. They are the empty_lines / 2 filler gen_sub_ass writes to nudge the block inside its anchor, so reading them as an ordinal there would produce nonsense. The an is None check is carrying more weight than the row counting.

The whitespace fix

The predicate difference was real and it is fixed, but it cannot be what you are seeing on teletext. IS_TXT_SPACE catches U+00A0 explicitly:

#define IS_TXT_SPACE(ch) ((ch).unicode < 0x0020 || (ch).unicode >= 0xe000 || (ch).unicode == 0x00a0 || ...)

and every space it produces is written as the \h escape, which the parser already normalises to a plain space. So a literal NO-BREAK SPACE cannot reach the engine from libzvbi. The mismatch still needed fixing for SRT, WebVTT and ASS payloads, which can carry one, so the styled path now trims exactly what the plain path trims.

For the blank lines themselves I found a different candidate and fixed that too. The interior blank-line fold worked per run and across one run boundary, which misses the shape the source produces most easily: the padding of an otherwise empty row carries the spacing attribute that changes colour, so the blank row lands in a whitespace-only run of its own and breaks the chain between the two text rows. It now folds the flattened text and re-splits along the run boundaries, so run structure cannot hide a blank row from it.

If blank lines survive that, the fastest way to settle it is the ASS line itself. -loglevel debug makes the teletext decoder print every one it emits as subtext:...:txetbus, and one of those from an affected page tells us more than any amount of reasoning from the rendered output.

WebVTT

The "libavcodec does not convert them" note was a conclusion about the wrong layer, and pushing on it was the right call. The decoder does drop line / position / align. The demuxer keeps them: libavformat/webvttdec.c attaches the verbatim settings string to every packet as AV_PKT_DATA_WEBVTT_SETTINGS, and matroskadec.c propagates the same side data for WebVTT in Matroska. Nothing upstream was needed, only reading one layer up.

They arrive as SubtitleCue.placement now. A percentage line becomes an anchor point plus the alignment row, anchored to the frame edge it is nearer, because the spec's default line alignment pins the box top and line:90% would otherwise hang a two-line cue off the frame. A line number keeps only the half of the frame it names, since line boxes need a rendered line height the engine does not have. align becomes the column. size and vertical have no equivalent in the placement model and are ignored, and a position without a line keeps only the column, because an anchor point needs both axes.

While testing that I found something worse sitting underneath it: the standalone WebVTT demuxer was never compiled into the FFmpeg build, only the decoder. avformat_open_input rejected every .vtt file with AVERROR_INVALIDDATA, so external WebVTT subtitles never loaded at all, on any path. Same shape as a gap I fixed for raw PGS sidecars a while back, and it hid for the same reason: the codec was present, so the problem looked like it had to be somewhere else. That is fixed in the FFmpeg build this release pins.

One boundary worth being explicit about, since it decides whether this covers what Rivulet needs. All of the above is the engine's own demux path. For a remote HLS source playing through the native bypass, the WebVTT rendition belongs to AVPlayer and the engine never demuxes it, so those cues do not reach subtitleCues and a custom renderer cannot draw them. For embedded tracks, including WebVTT in Matroska or MPEG-TS, selectSubtitleTrack(index:) is already the host-overlay path and is independent of the native renditions, so rendering everything yourself works there today. If the HLS rendition case is the one you need for the proxy, open a separate issue and I will look at what it takes.

All of it is in 5.27.0.