AI Music Exploration — Building Sound from Scratch

June 20, 2025

The Spark

Browsing ElevenLabs the other day, I spotted their effects API. Clean interface, solid docs, obvious use case: what if my particle text background could react to AI-generated music?

One prompt later, I had a play button generating random lo-fi house tracks that made my homepage particles dance like an equalizer. The music is basic. The feature was gold.


My Single-Shot Prompt

"help me create a new feature, the feature should include lowfi house music generation on my home page using elevenlabs api, i'll provide the API key. I want to create a play button that generates a new random song that will be played on the home page. The music while playing should change the particles in the interactive EWJ.DEV text to be a music equalizser where the top of the leters would grow and shrink with the music. To start lets expose the generate api that will return a 1 day cached audio file. Meaning that the audio would be generated again every day if someone pressed play. Make sure to ask any clarifiying questions and remember to always ask clarifying questions so that we are always aligned on the out comes of the task"

Result:

Working feature in one conversation. Check it out at ewj.dev.

But why stop there?


Three Hours to a Music Editor

The homepage feature got me thinking: what if I built a proper music layering app? Not for making Grammy-worthy tracks, but for letting non-musicians experiment with AI-generated audio layers.

Three hours later

a full music editor where you can generate, tweak, and download AI content. Multiple layers, real-time mixing, browser-based audio processing.

Warning: The output isn't perfect

The music sounds like a robot having a seizure. That's not the point. The point is abstracting away the deep AI knowledge so builders can focus on the creative workflow.


What I Actually Learned

Audio layering is an art form. Combining multiple generated tracks into something listenable requires understanding frequency ranges, timing, and how sounds interact. AI can generate individual pieces, but composition still needs human taste.

AudioContext is surprisingly powerful. Browser audio APIs handle real-time processing, visualization, and manipulation without external dependencies. You can build legitimate audio tools entirely in JavaScript.

Tone.js saves months of work. Raw AudioContext is like writing assembly. Tone.js gives you synthesizers, effects, and scheduling out of the box. Use it.

Production ready AI apps are possible. Not easy, but possible. You need a range of knowledge in the domain you're building for, plus willpower to push through the rough edges. But a functioning prototype in 3 hours? That's the new normal.

Domain expertise matters less than knowing the right terms. I don't need to understand AudioBuffer parsing or frequency analysis. I just need to know those concepts exist and let AI fill the implementation gaps.


The Bigger Picture

We're entering an era where you can build in domains you don't fully understand. The constraint isn't technical knowledge, it's knowing what questions to ask and recognizing good output from bad.

Six months from now, expect to see more "impossible" solo projects: complex audio tools, 3D modeling apps, data analysis platforms built by people who learned the domain vocabulary yesterday.

The builders who win will be the ones who can spot opportunities, frame problems clearly, and ship fast enough to learn from real users.


Got an Idea?

If you're sitting on a concept but don't know where to start, reach out. I'm always hunting for the next challenge that pushes the boundaries of what one person can ship.

The future belongs to builders who can turn curiosity into working code in an afternoon.

Bottom line: Stop waiting for permission to build in unfamiliar domains. Learn the vocabulary, frame the problem, and let AI handle the rest.

Comments

No comments yet.