I Cloned My Own Voice for My Website

I hit play on the intro video for this site and heard a stranger speaking my words.

The voice was articulate. Warm. Well-paced. And completely wrong. It was an off-the-shelf synthetic voice, the kind you pick from a dropdown menu, and the moment it started narrating something I’d written, I felt it in my gut — this isn’t me. If the whole point of the site is that I’m openly showing the work as I go, the voice can’t belong to someone else.

So I cloned my own voice. The interesting part isn’t that I did it. It’s everything that went wrong on the way there, and the one boring verification step I should have run first.

Finding the sample

You can’t clone a voice without audio. I had a better source than I expected: years of recorded video updates — me, alone, talking to a camera for ten minutes at a stretch. A solo monologue is the ideal raw material for voice cloning. One speaker, no one talking over them, natural rhythm, and plenty of it to choose from.

The content of those recordings doesn’t matter. This is worth being blunt about: a voice clone copies timbre — the texture and color of a voice — not the words. The reference clip is disposable. The model never repeats anything in it. What gets published is a brand-new script spoken in the cloned voice, so the source recording stays private and the output carries none of its original content. I pulled about twenty seconds of clean speech, normalized the audio levels, and called that the raw material.

The obvious way, and why it broke

Most modern text-to-speech engines clone a voice using one of two paths.

The first path feeds the model the reference audio plus a transcript of what’s being said in it — this is called in-context learning, where the model uses the example to figure out how to continue. The second path extracts something called a speaker embedding (often an “x-vector”), which is a compact digital fingerprint of how a voice sounds — its timbre — with the actual words stripped out entirely.

I registered my voice and used the in-context path, because it’s the default and it generally gives the best quality when the reference clip and the target speech are in the same language. I asked it to speak one sentence.

It produced two and a half minutes of audio.

Not my sentence stretched out. A runaway. The model latched onto the reference and never found the exit. I tried a shorter clip. A cleaner clip. A different segment of the recording. Same result every time: a clean reference, a short prompt, and a minute or more of unintelligible drift. Meanwhile, all eleven voices already installed on the system worked perfectly through the exact same path.

The actual fix

The tell was that the other path — the x-vector one — was stable.

The blend feature, which works purely on those speaker embeddings, produced a clean twelve-second clip of my voice on the first try. No runaway. No drift. Just my voice, saying what I’d asked it to say.

Here’s what was happening: the in-context path was destabilizing on my specific recording when paired with the engine’s fixed reference transcript. The x-vector path doesn’t care about any of that. It pulls the timbre and synthesizes the new text from scratch. So I routed my voice through it.

One configuration flag. No model retraining. I tagged the reference clip as if it were in a different language than the target, which is exactly how the engine decides to use timbre-only mode — it’s the same mechanism it uses for cross-lingual voice cloning. My voice now rides the same rails as a voice cloned from a clip in another language.

Then I did the thing I should have started with. I ran the synthesized output back through speech-to-text. If the transcription comes back matching my exact script, the clone is intelligible. It did. That round-trip — speak, then transcribe, then diff against the input — is the cheapest reality check available for generated audio. It would have caught the runaway in one step instead of the five it took me to debug it blind.

The pipeline, concretely

None of this is exotic. All of it is open source. The shape:

Pull the source with yt-dlp — it handles Loom, YouTube, and most video players. yt-dlp -x --audio-format wav.
Cut a clean ~20s window with ffmpeg — apply a high-pass filter to kill low-end rumble, normalize loudness, and downsample to the model’s expected rate: ffmpeg -ss 30 -t 22 -af "highpass=f=70,loudnorm=I=-18:TP=-1.5,aresample=24000" -ac 1 sample.wav.
Register it with the TTS engine — a self-hosted Qwen3-TTS model behind a small API that speaks the same language as OpenAI’s audio endpoints.

the mechanism: x-vector routing and the verification pipeline give me the detail

Why two clone paths exist. Qwen3-TTS (and most modern zero-shot TTS systems) can condition synthesis two ways. The in-context path concatenates the reference audio + its transcript with your target text and lets the model complete it — high fidelity, but fragile: if the reference transcript drifts from the actual audio, the model latches onto the reference and never terminates. The x-vector path instead extracts a compact speaker embedding (a d-vector / x-vector from a speaker-encoder sub-network), discards the reference audio entirely, and conditions the acoustic model on the embedding alone. It’s timbre-only — no transcript dependency — which is why it’s also used for cross-lingual synthesis.

The routing flag. The server decides which path to use by comparing target_language to ref_language in the voice manifest. When they differ, it assumes cross-lingual, skips the in-context decoder, and uses the embedding path. Set ref_language to anything other than your target language and you’re in stable mode with no other changes:

{ "wav": "voices/sample.wav", "name": "my-voice", "ref_language": "ko" }

The preprocessing that actually matters. Before registration, normalize the reference clip with ffmpeg — the model is sensitive to DC offset and loudness variance:

ffmpeg -ss 30 -t 22 -i source.wav \
  -af "highpass=f=70,loudnorm=I=-18:TP=-1.5,aresample=24000" \
  -ac 1 sample.wav

The only verification step worth running. Pipe synthesis output into whisper and diff against your input script. A runaway produces a multi-minute transcript; a bad clone produces mostly empty output. Both fail immediately:

whisper output.wav --language en --output_format txt
diff <(cat output.txt) <(cat script.txt)

Intelligibility is a measurable property. “Sounds fine” is not.

Synthesize with POST /v1/audio/speech {voice: "jon", input: "..."}.
Verify by transcribing the output with Whisper and diffing against the script. A runaway shows up as a 150-second clip where you asked for 12. A broken clone shows up as near-empty transcription. Both get caught before anything ships.

The whole thing is wrapped in a small script now, so next time it’s one command. The video itself is composed in HyperFrames — HTML and GSAP timeline animations rendered to MP4 — with the narration dropped in as the audio track.

What I’d tell you

Three things, if you’re cloning a voice:

Timbre, not content. The reference clip is disposable and its words never surface. That makes the privacy story simple and the sourcing easy — any clean solo recording works.
Two clone paths, very different failure modes. In-context conditioning is higher fidelity and more fragile. X-vector embedding is timbre-only and rock-stable. If the fancy path runs away, drop to the embedding. You’ll know in seconds, not hours.
Verify generated audio by transcribing it. You can’t eyeball a waveform. Round-trip it through speech-to-text and compare to the script. “Sounds right” is a feeling. Intelligible is a property you can measure.

The voice on this site is mine now. It took one good recording, one stubborn bug, and one boring verification step I should have run first.

Built on: Qwen3-TTS (Alibaba) for the voice model · yt-dlp and ffmpeg for sourcing · Whisper (OpenAI) for the verification round-trip · HyperFrames + GSAP for the video. All of it open or self-hostable — none of this needs a vendor.