# Writing a Screen Reader in Rust

2024-11-20

I wanted to know how screen readers actually work, so I built a small one for Windows.

## Backstory

In June 2025, a new law, the Barrierefreiheitsstärkungsgesetz (BFSG), comes into effect in Germany and requires a _lot_ of websites, basically anything that sells to consumers, to be accessible. So German website owners are currently scrambling to conform to the new regulations. This is also how I first got exposed to the topic of web accessibility: more and more of our clients at [next.motion](https://next-motion.de) are asking for audits and improvements in this space.

So recently, while working on the accessibility of a website, I got the idea to dig deeper into how screen readers actually work. I knew how to label a button, but I only had a vague idea of what happens between pressing Tab and hearing that label read out loud. What better way to understand something than to build it yourself?

To keep things simple, I decided to only focus on Windows for now, because it's by far the most popular OS among the people I build for. I named the project Aria, after the set of roles and attributes for HTML elements.

## Choosing the right tools

Screen readers need low-level access to OS APIs, both for reading what's on screen and for interacting with applications. Rust is pretty much the only low-level programming language I'm somewhat comfortable with, so it was an easy choice for this project. The borrow checker is the cherry on top, because I don't feel like shooting myself in the foot with memory bugs while I'm also learning a bunch of new APIs.

As for functionality, I figured a basic screen reader should be able to:

- Access information about UI elements on the screen
- Read text aloud (text-to-speech)
- Listen and react to user input
- Play audio files (for sound effects)

There are also a few things I won't implement at this stage, mainly a GUI and support for reading anything but the focused element. A real screen reader also lets you move through content that can't receive focus, like the paragraphs and headings on a website. Browsers do expose all of that through their accessibility APIs (Chromium's [accessibility documentation](https://www.chromium.org/developers/accessibility/) is a good starting point if you're curious), but supporting it would be a whole project of its own.

After searching around GitHub a bit, I found [leexgone/uiautomation-rs](https://github.com/leexgone/uiautomation-rs). It's a Rust wrapper around the Windows UI Automation API, which provides programmatic access to most user interface elements on the desktop, as well as the ability to interact with them. This is exactly what I need for the first point on my list!

For text-to-speech, I knew that Windows has a built-in speech API, I just needed a way to access it from Rust. Luckily, Microsoft maintains the [windows crate](https://github.com/microsoft/windows-rs), which has bindings for pretty much the whole Windows API, speech synthesis included. That covers point two.

For points three and four, I searched for utility crates on [crates.io](https://crates.io) and found [rodio](https://crates.io/crates/rodio) for audio playback and [mki](https://crates.io/crates/mki) for keyboard input.

## Reading UI elements

UI Automation makes it easy to hook into Windows interface events. For example, to get notified whenever the focused element changes, I created a `FocusChangedEventHandler` struct and registered it as a focus changed handler:

```rust
struct FocusChangedEventHandler {
    previous_element: Mutex<Option<UIElement>>,
}

let automation = UIAutomation::new()?;
let handler = UIFocusChangedEventHandler::from(FocusChangedEventHandler {
    previous_element: Mutex::new(None),
});

automation.add_focus_changed_event_handler(None, &handler)?;
```

From then on, Windows calls the handler's `handle` method every time focus moves. It sometimes gets called several times in quick succession for the same element, though, which meant a single press of Tab could announce the same button over and over. To prevent that, the handler remembers the previous element and compares runtime IDs, which UI Automation uses to tell elements apart. If the new element has the same ID as the previous one, there's nothing new to say:

```rust
impl CustomFocusChangedEventHandler for FocusChangedEventHandler {
    fn handle(&self, sender: &UIElement) -> uiautomation::Result<()> {
        let mut previous = self.previous_element.lock().unwrap();

        if let Some(element) = previous.as_ref() {
            if element.get_runtime_id()? == sender.get_runtime_id()? {
                return Ok(());
            }
        }

        *previous = Some(sender.clone());
        drop(previous);

        let name = sender.get_name()?.trim().to_string();
        let control_type = sender.get_control_type()?;

        log::info!("Focus changed to: {name} ({control_type:?})");
        Ok(())
    }
}
```

<SameRuntimeIdFigure />

This shortened version stops at logging. In Aria, the next step is building the actual announcement from the element's name and its localized control type, so a button named Save becomes "Save, button." The control type is also what decides whether to play a sound before speaking, but more on that later.

## Reading text

The Windows text-to-speech API is pretty straightforward. You create a [`SpeechSynthesizer`](https://learn.microsoft.com/en-us/uwp/api/windows.media.speechsynthesis.speechsynthesizer.synthesizetexttostreamasync?view=winrt-26100), call `SynthesizeTextToStreamAsync` with the text you want to read, and hand the resulting stream to a `MediaPlayer`:

```rust
let synthesizer = SpeechSynthesizer::new()?;

let stream = synthesizer
    .SynthesizeTextToStreamAsync(&HSTRING::from("Hello, World!"))?
    .get()?;

let source = MediaSource::CreateFromStream(
    &stream,
    &stream.ContentType()?,
)?;

let player = MediaPlayer::new()?;
player.SetSource(&source)?;
player.Play()?;
```

The `.get()` call waits until synthesis is done, which is fine for a "Hello, World!" but not for a screen reader that has to keep listening to your keyboard while it talks. So in Aria, speech runs asynchronously, which comes back to haunt me a little at the end of this post.

## Playing sounds

This one is also quite simple. For my implementation, only three sounds are needed: a startup sound, a shutdown sound, and a sound for when you focus an input field. The input sound is an earcon, a short sound that tells you something about the interface, and it plays right before the field's name is read:

```rust
if control_type == ControlType::Edit || control_type == ControlType::ComboBox {
    play_sound(INPUT_FOCUSSED_SOUND);
}
```

I skipped any fancy asset management and just included the sound files as byte arrays, so they're baked into the executable:

```rust
pub const STARTUP_SOUND: &[u8] = include_bytes!("../assets/sounds/startup.mp3");
pub const SHUTDOWN_SOUND: &[u8] = include_bytes!("../assets/sounds/shutdown.mp3");
pub const INPUT_FOCUSSED_SOUND: &[u8] =
    include_bytes!("../assets/sounds/input-focussed.mp3");
```

## Listening to keyboard input

Similar to how you can listen to focus changes, you can listen to keyboard input using the `mki` crate. Escape stops whatever Aria is currently saying, and every other key goes to `on_keypress`, which reads keys back to you while you type in an input field:

```rust
mki::bind_any_key(Action::handle_kb(|key| {
    use Keyboard::*;

    match key {
        Escape => TTS::stop(true).unwrap(),
        _ => on_keypress(format!("{:?}", key)),
    }
}));
```

`TTS` here is Aria's own little wrapper around synthesis and playback. Both the keyboard hook and the focus handler talk to the same one, so either of them can control what's currently being said.

## Putting it all together

With all the pieces working on their own, the tricky part was deciding what should happen when input arrives while Aria is still talking. Say focus is on a text field and Aria is in the middle of "Name, edit" when you tab over to Save. The old announcement should stop right away and "Save, button" should start. Pressing Escape should stop speech without starting anything new. Simple rules, but focus events, key presses and audio playback all come from different places, and they all have to agree on what's currently being said.

<SpeechInterruptionFigure />

In the end, I moved the speech handling out of the event handlers into one place that owns the current announcement, and added some nice-to-have features on top, like a simple command-line interface and a configuration file for customizing the screen reader's behavior.

Aria only follows focus, which is a tiny part of what people actually need from a screen reader. But building it made missing labels a lot easier for me to notice, because when a button has no useful name, there's very little a screen reader can say about it.

A build, as well as the source code (MIT), is available on [GitHub](https://github.com/Coyenn/Aria). You can grab the latest release [here](https://github.com/Coyenn/Aria/releases/latest).