How we built an AI sax quartet in 3D sound

How we built an AI sax quartet in 3D sound

A CognitoRiverDelta builder project: Don Wood had the idea and the musical direction, and an AI agent working with him made most of the technical decisions. It went from idea to a finished 3D recording in about an hour and a half, on one Mac, with free tools.

New in iteration 2: play along. Three AI players, and you play the fourth part. A moving box on the score shows you where you are.

Listen first

Use headphones for the 3D versions: the four players stand around you, from your front-left to your front-right.

The dated pairs

Iteration 2: be the fourth player (under 20 minutes)

Measured: from the first mention of iteration 2 (15:38:33) to built, tested and merged (15:45:20) took under 7 minutes, and it was live on the public site at 15:48:04, under 10 minutes from idea to the world. That includes three new ideas added mid-build, and designing how a builder project grows by iterations. We evolve quickly when we're given the right feedback and suggestions for improvement.

Our goals for iteration 2, set on a whim, and that's the truth. Don Wood had these ideas while it was being built, and the agent built each one as it came:

1. Let a person play along: three AI players, and you play the missing part.
2. A moving box on the score, large, thin-lined and rounded, that follows the music over your notes.
3. Five 3D recordings: the full quartet in a ring around you, everyone 5 feet further back, and four "empty chair" versions that seat you where the missing player sits.
4. Slow it down to learn: the music and the score slow down together, with the pitch unchanged.
5. Web only, on any screen: phone, tablet or a big screen, with no app to install and nothing sent anywhere.

The AI agent decided:

• Proportional notation: left to right is time, so the box glides smoothly with the music, a simple score we draw ourselves in the page, with no outside libraries.
• The notes as each sax reads them: saxophones are transposing instruments (soprano and tenor in B-flat, alto and baritone in E-flat), so each player sees their own written notes, with the note names printed under their part.
• The room as geometry: each seat is a point on a circle about 3.9 m from the centre. From an empty chair, facing the centre, you hear the player straight across the ring about 7.8 m in front of you, and the other two 45 degrees to your left and right, 5.5 m away. Sounds from behind are a little duller, as they are for real ears.
• Slowing down without changing the pitch: the browser itself slows the recording and keeps every note in tune, on iPhone and iPad too. The score follows the recording's own clock, so the box slows down with the music. At 50% speed, two real seconds moved the music and the box exactly one second.
• Re-using the final take: the same players and the same timing as take 5, so every minus-one mix lines up with the score exactly.
• Nothing sent anywhere: the page runs in your browser, and it only remembers which saxophone you picked, on your own device.

Our honest note: this is not a world-class recording, and it isn't trying to fool anyone. It shows what young builders can dream up, build and share with the world in under an hour, and then improve in minutes.

Who decided what

We're honest about this, because it's the point: a human sets the goal and keeps the judgment, and AI agents do a lot of the work, including many of the decisions.

Don Wood decided:

• the goal: four AI sax players, one finished audio file, in spatial audio;
• yes to the free tools the agent proposed;
• the musical shape of the final take: build toward a climax, then a quiet close. He listened to the takes and chose take 1's arc over the AI bandleader's flatter plan;
• that this should become a builder example for the Academy.

The AI agent decided:

• The music: a Bach chorale (BWV 66.6). It is public domain, written in exactly four voices (soprano, alto, tenor, bass), and those map neatly onto soprano, alto, tenor and baritone sax.
• What "AI players" means here: an AI model doesn't generate the sound. Instead, a local AI is the bandleader. It reads the piece's six phrases and gives each player directions: loud or soft per phrase, smooth or detached, how loosely to play the beat, and a short note in words. Software saxophones then play each part from that direction. This keeps the music real and the AI's role clear, and every player stays separately controllable.
• Each player as its own performance: the music becomes one expressive part per player, with louder and softer notes, swells on long notes, a breath at each phrase end, and slightly human timing. The free synthesizer plays each part into its own track.
• The 3D mix, written from scratch instead of downloaded: a model of a listener's head, after a well-known 1998 research model. Sound reaches the far ear a fraction of a millisecond later and a little duller (the head is in the way), so your brain hears where each player stands. On top of that come wall and floor reflections and a shared concert-hall echo.
• Always finish: if the AI bandleader is unavailable, a written plan takes over, so the demo never fails.
• Use the project's shared AI queue: take 2 timed out because the Mac's AI was busy, so the agent routed the bandleader through the project's own queue, which waits its turn.
• Check it with numbers, not only ears: the agent measured the loudness of each part of the piece, to confirm take 4 really builds and then drops (about 16 decibels quieter at the end).

What went wrong, and what we changed

• The AI bandleader's first plan was flat. Take 3 had the soprano loud from the second phrase to the end, even though its own note said "peak at phrase five". Don Wood heard it, and chose the shape he wanted. The agent kept the bandleader's ideas and applied take 1's shape on top. Lesson: AI suggests, people decide.
• The agent's own mistake. It had moved the baritone part down an octave, below a real baritone sax's lowest notes. Bach's bass line already fits the baritone as written. The agent found the error while writing this page and fixed it in take 5. Lesson: writing down what you did is a check in itself.
• The honest limits. These are sampled saxophones, not players breathing in a room, and the head model is a good simple model, not a measurement of your own ears. It is a first iteration.

What spatial audio is, in one breath

Stereo puts sounds between your left and right ears. Spatial (3D) audio places them around you: in front, at the sides, even behind. On ordinary headphones this is done binaurally, by copying the tiny time and tone differences between your two ears. That's all this project does, and you can do it too.

Try it yourself, at $0

New in the Academy: Play along, then write your own quartet part, a 4-week project guide for any instrument, or your voice. Learn a part slowly with the three AI players, speed up week by week, then write and play a part of your own.

The Academy's musicians path walks you through it step by step: record several parts (you can be all four players), mix them into one file, use AI as an arranger whose ideas you choose from, lead a band of software players, and place them around the listener. Its last project is "A quartet in 3D sound".

Credits

• Music: Johann Sebastian Bach, chorale BWV 66.6 (public domain).
• Free, open-source tools: music21 (the chorale), FluidSynth (the synthesizer), the MuseScore General sound set (the saxophones, MIT licence), and NumPy and SciPy (the 3D mix). Thank you to their makers.
• Conceived and directed by Don Wood, who leads CognitoRiverDelta and takes responsibility for it, built with an AI agent and the project's own local AI as bandleader. Nothing left this Mac.