A 72-Hour Engineering Sprint for Translating Thought to Speech
Our engineers recently spent 72 hours with a participant in our VOICE study, at his home, for a hackathon. The goal was simple: rapidly prototype as many features as possible to improve the experience of using Neuralink to communicate with thought, and test every one of them with the end user.
Why build next to the user
The input to a speech brain-computer interface (BCI) is a person’s own neural activity, which varies with fatigue, attention, and the way someone imagines speaking. Small details like how long it takes to start a recording, how easy it is to fix a mistake, or whether the phone needs to be unlocked first can be the difference between a tool someone tolerates and a tool someone reaches for every day.
This can be challenging to assess purely from virtually collected data and analytics, and it is not something we can simply try for ourselves. None of the engineers building this software have an implant, so we cannot feel the difference between a good design and a frustrating one; only our participants can. The highest signal often comes from sitting next to a participant, watching where they hesitate, hearing what frustrates them, and shipping a fix shortly afterwards. Working side by side with a participant compressed weeks of feedback loops into hours. Ideas went from whiteboard to prototype to real-world test in a single sitting, and the ones that did not work were discarded just as quickly.
Here is some of what we built.
1. Neural voice activity detection
Speech decoding used to require pressing a button to start and stop recording. We built a system that automatically detects when a user is and is not trying to speak, so the app can start and stop on its own. This is a big step towards making communication completely hands-free for people with speech impairment.
2. Real-time neural visualizations
One of the hard problems in BCI is figuring out which intended or imagined actions produce the strongest neural activity, since that directly affects decoder performance. We built visualizations that give participants real-time feedback on the intensity and distribution of signals detected by the implant. Our participant used them to test different imagined speaking patterns and see which ones recruited the most activity, turning model tuning into something closer to a game.
3. A computer use agent
We prototyped a way for users to trigger a Grok agent by thinking of saying “Hey Neuralink” followed by a task, which the agent then carries out autonomously on the computer. It is an early illustration of how increasing the bandwidth between the brain and computers could help close the gap between thought and action.
4. A mobile app for speech BCI
Over the last few months, we have put significant effort into Bloom, our mobile app designed to make speech BCI more accessible. The hackathon was a chance to watch Bloom in daily use, outside the lab, and fix what got in the way.
5. Keyboard dictation with BCI
Standard keyboards and built-in dictation can be difficult or impossible for people with ALS to use. We built an iOS keyboard integration that routes BCI-based dictation into any app, streamlining text entry and reducing the need to switch between modes.
6. A faster editing flow
Decoders are not perfect, and mistakes can be awkward or costly. Driving time-to-edit toward zero is crucial for dictation. To do this, we iterated on several editing interfaces that integrate our own suggestions into the keyboard and let users quickly correct text in any app. There is still more to optimize, but the result is a dramatic improvement over the first version of Bloom.
7. Lock screen Live Activity
Starting a conversation should not require unlocking a phone and opening an app. We added Bloom as a Live Activity so speech can begin right from the lock screen, with at-a-glance status and quick controls. The goal is to drive everyday communication friction toward zero.
8. Phone call integration
Voice calls are an essential part of daily life. We integrated Bloom with iPhone calling so BCI-generated speech is injected directly into the call’s microphone stream, giving the person on the other end clear, consistent audio. We also tested an optional greeting at the start of calls, such as “This call is from someone using an assistive communication device.”
This one meant a lot to the team. Our participant and his wife started their relationship over phone calls, and watching the two of them joke with each other on the phone again was one of the most memorable moments of the week.
What we took away
The features matter, but the bigger lesson is about how we work. When engineers build next to the people they are building for, priorities become clearer as they observe first-hand where the user experience shines and where it needs to be improved. And the motivation is hard to overstate: there are few things in engineering as rewarding as watching someone use something you built that morning to talk to the people they love.
Our participant, along with his wife and caregiver, were integral collaborators in the process; opening up their home, giving candid feedback, suggesting ideas, and pushing the team to go further.
Join us
The hackathon is over, but we are just getting started. The list of things we want to improve and build keeps growing.
We need exceptional engineers across machine learning, software, and many other disciplines to help shape a technology that will transform human interaction and advance human agency in an era of AI. If this resonates, visit neuralink.com/careers.
Neuralink devices are investigational and have not been approved by the FDA or other regulatory authorities. This post features a voluntary clinical trial participant sharing his personal experience, which may not reflect all participants or future outcomes.