String Codecs
July 12, 2025 • 2 min read
While working on the "QR Transfer Protocol" I was running into frequent bugs and annoyances developing compact representations for the QR code data payload.
'QRTP': a
chunk index: a
chunks total: a
'ack' hash: a
chunk data: 8
What I needed was a utility for typed, debuggable, and compact string encodings.
Using JSON.stringify as a baseline, take the following string:
{"foobar":true,"zabzob":false,"hash":"abc123","value":1.56}Compare this to a hand-crafted encoding which takes advantage of the known schema, where redundant keys, symbols, and delimiters are removed:
10abc123|1.56Writing these optimized encodings by hand can be a pain when the layout keeps changing, so I wrote a small codec.ts utility with a tiny DSL which infers types and compacts the data as much as possible, removing redundant delimiters, creating smaller alphabets for enums and so on…
const codec = codec("QRTP<index:num>/<total:num>:<ack:text>");The codec's encode and decode functions convert objects which match the schema to/from compact strings like "QRTP4/5:abc123".
To infer the types from the string took some work. Leading to this gnarly utility type.
type PatternType = 'text' | 'num' | 'bool' | 'list' | 'nums' | 'pairs' | 'numPairs' | 'enum';
type ExtractEnumValues<T extends string> = T extends `enum[${infer Values}]`
? Values extends string
? Values extends ''
? never
: Values extends `${infer First},${infer Rest}`
? First | ExtractEnumValues<`enum[${Rest}]`>
: Values
: never
: never;
type ParseType<T extends string> = T extends `${infer Base}-${string}`
? ParseType<Base>
: T extends 'text'
? string
: T extends 'num'
? number
: T extends 'bool'
? boolean
: T extends 'list'
? string[]
: T extends 'nums'
? number[]
: T extends 'pairs'
? Array<[string, string]>
: T extends 'numPairs'
? Array<[number, number]>
: T extends `enum[${string}]`
? ExtractEnumValues<T>
: string;
type ExtractFields<T extends string> = T extends `${string}<${infer Field}:${infer Type}>${infer Rest}`
? { [K in Field]: ParseType<Type> } & ExtractFields<Rest>
: T extends `${string}<${infer Field}>${infer Rest}`
? { [K in Field]: string } & ExtractFields<Rest>
: {};
type Expand<T> = T extends infer O ? { [K in keyof O]: O[K] } : never;
type CodecData<T extends string> = Expand<
ExtractFields<T> & {
payload?: string;
}
>;And that's it, that's the whole post! Thank you for coming to my TED talk. The codec utility can be found here though I wouldn't recommend it for anything too important at time of writing.