A Voice Assistant on My Desk That Talks to My Local LLM (Part 5)

July 26, 2026·9 min read

By the end of Part 4, the site had a mouth from Part 3 (a local Qwen on the desktop under my desk) and a hippocampus (Redis memory that remembers you between visits). It still lived entirely inside a browser tab. This part gives it a body: a small round screen on my actual desk that listens when I tap it, ships my voice off to be transcribed, asks the same local Qwen the website uses, and reads the answer back out loud.

The hardware is a Waveshare ESP32-S3-Touch-AMOLED-1.75. It is a lovely little board with a genuinely thin paper trail once you get past "blink the screen." If you googled that board name and landed here, this is the post I wanted when I started. Fair warning, same as the rest of the series: this is a field report, written while the mistakes are still warm.

The Goal

Three things, because that format keeps working:

1. A physical object on the desk that answers questions using my LLM, running on my hardware, not a cloud assistant renting me my own voice back.
2. Have it respect the same backend flag from Part 3. When I flip the GrowthBook toggle between my house box and my friend's box, the desk gadget follows, no reflash.
3. Do the speech-to-text and text-to-speech without standing up yet another server. The thinking stays local. The transcription and the voice can be somebody else's free compute.

House rules from the rest of the series still hold: nothing on my network exposed without a token, and the part that reasons about my questions runs on my own GPU.

The Board, Honestly

Here is what is actually on it:

• A 466x466 round AMOLED, driven by a CO5300 controller over a 4-lane QSPI bus.
• A CST9217 capacitive touch panel on I2C.
• An ESP32-S3 with 16MB flash and 8MB PSRAM, which matters, because you render a full-frame canvas in PSRAM.
• And the part that cost me a full day: the audio path is two separate chips, and almost every tutorial treats it like one.

That last point is the whole reason this post has a hardware section. The display and touch are well-trodden. The audio is where people (me) lose a day.

The Pin Map

Every pin, so you can skip the datasheet archaeology. These are the values I run in config.h:

Display (CO5300, QSPI)
  SCLK 38   D0 4   D1 5   D2 6   D3 7   CS 12   RST 39
  466 x 466, column offset 6

Touch (CST9217, I2C @ 0x5A)
  SDA 15   SCL 14   INT 11   RST 40

Audio I2S (shared by both codecs)
  MCLK 42   BCLK 9   WS/LRCLK 45   DOUT 8 (to speaker)   DIN 10 (from mic)
  PA enable 46  (NS4150 amplifier)

Codecs on the same I2C bus as touch (SDA 15 / SCL 14)
  ES8311  @ 0x18   output only  (DAC -> NS4150 -> speaker)
  ES7210  @ 0x40   input only   (dual mics -> ADC -> I2S DIN)

The I2C bus is shared three ways: touch, the output codec, and the input codec. The I2S bus is shared two ways: the ESP is the clock master, and both codecs are slaves hanging off the same MCLK, BCLK, and word-select lines.

The Two Chips Nobody Draws

This is the single fact that would have saved me the most time, so it gets its own heading.

The ES8311 is a mono codec, and on this board it is wired for output. It takes I2S data on GPIO8, converts it to analog, and hands it to an NS4150 class-D amplifier that drives the speaker. The ES8311 has a microphone input on paper. On this board it is not the microphone. Do not spend an afternoon configuring its ADC. I did.

The ES7210 is a separate 4-channel audio ADC (Everest Semiconductor datasheet), and it is the microphone. The board's two mics feed the ES7210, which serializes them onto the I2S bus and drives GPIO10, the ESP's I2S data-in pin. It is its own I2C device at address 0x40, with its own register map, entirely independent of the ES8311.

So the mental model is: GPIO8 goes down to the speaker chip, GPIO10 comes up from the microphone chip, and they are different silicon. The ESP32 generates one set of clocks (MCLK 42, BCLK 9, WS 45) and both chips slave to them. Once that clicked, everything else was downhill.

The Architecture

  Tap the orb
     |
  ES7210 mic ADC --I2S DIN(10)--> ESP32-S3
     |                               |
     |                        build a 16 kHz WAV in PSRAM
     |                               |
     |                     HTTPS POST /api/assistant
     v                               v
  ES8311 <--I2S DOUT(8)--    Cloudflare Pages Function
  speaker                          |   Workers AI Whisper  (speech -> text)
     ^                             |   local Qwen via the Part 3 tunnel  (the answer)
     |                             |   Workers AI aura-1   (text -> speech, 16 kHz WAV)
     +--------- audio/wav ---------+

Whisper and aura-1 run on Cloudflare Workers AI, which has a free daily allowance and needs no box of mine. The reasoning step in the middle, the part that actually answers the question, hits the same local Qwen the website uses. The GrowthBook flag from Part 3 decides house or fever, and the device inherits that decision for free because it calls through the same function the browser does.

Doing It Wrong First

My first recordings were pure silence. Not quiet audio. Digital zero, every sample. Whisper, handed four seconds of nothing, does the polite thing and hallucinates: it returned "you", and once, memorably, "have a great time and let me know when you are back."

The debugging went in stages, each one a small archaeology dig:

1. I configured the ES8311's microphone. Wrong chip. See above.
2. I found the ES7210, ported a driver, and still read zeros. The chip answered on I2C at 0x40, so it was alive. It just was not producing audio.
3. I put the raw microphone peak into the device's telemetry, because opening the USB serial port resets the ESP32-S3 and I kept missing the boot logs. Watching the peak sit at exactly 0 over the network told me the ADC was clocked wrong, not wired wrong.

The actual bug was one byte. The ES7210 has a MAINCLK register (0x02) that sets the ADC clock divider, and for 16 kHz audio with the master clock this board runs (16000 times 256, which is 4.096 MHz), that register has to enable the clock doubler and the DLL on top of the divider. The correct value is 0x01 | (1 << 6) | (1 << 7), which is 0xC1. I had written 0x01. With the wrong divider the ADC ran unclocked and produced flat zeros, and, being an ADC, it reported no error about it. Silence is a valid output.

The fix came from the board's own repository. Waveshare ships an esp-idf/05_Spec_Analyzer example that reads the microphone for an FFT display, and it delegates to Espressif's esp-bsp ES7210 driver. That driver has a clock coefficient table, and the row for 4.096 MHz at 16 kHz spells out every divider bit. Porting that table's values verbatim turned the peak from 0 to a live, breathing noise floor on the first flash. If you are fighting this chip, the driver is the source of truth, and the coefficient table is the part to copy exactly.

One more from the same fight: do not hand-gate the clock-off register (0x01) trying to be clever about power. The esp-bsp sequence leaves it alone. So should you.

The Two-Core Rule

The ESP32-S3 has two cores, and on a display device you must respect the division of labor or the screen stutters. I render the full round canvas out of PSRAM at roughly 15 frames a second on core 1. Every blocking network call, and TLS is very blocking, lives on core 0 behind a mutex-guarded snapshot. The first time I put an HTTPS poll inside the render loop, the display froze solid for the length of the request, and it was obvious within one visit that the draw loop and the radio could not share a thread.

The voice recording earns its own task for a related reason. When you tap the orb, the "listening" cue appears instantly on core 1, but if the actual capture is queued behind a stack of network polls it can start seconds late, after you have already finished talking. Giving the recorder a dedicated task drops the tap-to-capture gap to about 60 milliseconds, so the moment the screen says listening, the microphone is genuinely open. Half my early transcription failures were not audio quality at all. They were timing, my voice landing in the silence before the window opened.

Making It Talk Back

Getting audio out was its own small puzzle. Cloudflare's aura-1 text-to-speech returns MP3 by default, and the ESP32 has no cheap MP3 decoder I wanted to carry. The trick is that aura-1 accepts format parameters: ask for encoding: linear16, container: wav, and sample_rate: 16000, and it hands back a plain PCM WAV at exactly the rate the ES8311 already plays. The device streams that WAV straight off the wire into the speaker with no decoding step.

I return it from the Pages Function as a binary audio/wav body with the transcript and answer riding in response headers, rather than base64 inside JSON, which would have tripled the payload for no reason. On the device, the reply gets read into PSRAM and clocked out to the codec.

The last snag was comedy: the first spoken answers sounded like a walkie-talkie. Full-scale synthesized speech was overdriving the tiny speaker into a rasp. Scaling the samples to about 60 percent gave the amplifier headroom and the voice cleaned right up. The synthesized ping and whoosh sounds had always been fine because they were generated quieter, which is exactly the clue I ignored for an hour.

Design Decisions

Why speech-to-text and text-to-speech live on Workers AI, not the house box. The local GPU has one job worth protecting: running the model that answers questions. Whisper and aura on a free tier keep that card free for the model people actually talk to, and the transcription and the voice are commodity work. The reasoning stays home.

Why the device respects the flag instead of pinning a backend. Same answer as Part 3. One toggle should move everything. The desk gadget calling through the same function as the browser means it can never disagree with the website about which LLM is live.

Why a tap and not a wake word. An always-listening microphone on my desk is a thing I do not want, and a tap is a clean act of consent. The orb turns to listening only when I ask it to.

Why the microphone peak lives in telemetry. Because opening the native USB serial port resets this board, and I kept missing the one boot log I needed. A number in the heartbeat I could watch over the network turned a guessing game into a measurement. That single move is what actually cracked the silence bug.

The Gotchas, So You Skip Them

Board-specific, hard-won, in the order they bit me:

• The microphone is the ES7210, a different chip from the ES8311. The speaker codec cannot record on this board. Configure the ADC, not the DAC.
• The MAINCLK register must match your master clock. For 16 kHz at 4.096 MHz MCLK, register 0x02 is 0xC1, not 0x01. Wrong divider means clean, error-free silence.
• Copy the esp-bsp ES7210 coefficient table verbatim. The board's 05_Spec_Analyzer example is the working reference. Do not improvise the clock bits.
• MCLK has to be running before the ES7210 will take its configuration. Bring the I2S clocks up first, then talk to the chip.
• Mute the amplifier (PA on GPIO46) while recording. The idle class-D amp couples hiss straight into the ADC and buries your voice under a wall of noise the size of your signal.
• Average the two microphone channels, do not sum them. Summing clips on loud input, and clipped audio makes Whisper hallucinate confident nonsense.
• DIN is GPIO10, DOUT is GPIO8, and they land on different chips. Cross them in your head and you will chase silence on one end and confusion on the other.
• Keep all networking off the render core. Any blocking call in the draw loop freezes the screen for its full duration.

Conclusion

Part 3 gave the site a mouth, Part 4 gave it a memory, and this part gives it a body sitting on my desk. It listens on a tap, transcribes on somebody else's free GPU, asks the same local Qwen the website asks, and speaks the answer back through a speaker the size of a coin, for a running cost of zero dollars a month plus one ESP32.

The honest status: the voice loop works end to end, the reasoning is genuinely mine, and the hardware surprises were all in the audio path that the docs wave past. If you have this exact board and a weekend, the pin map and the one clock byte above are the two things that would have given me back my lost day. The rest is the fun part.

Tags:

Enjoyed this post? Give it a clap!