Central bank announcements usually carry little surprises in terms of policy, but we listen to them to update our internal outlooks on what we think future decisions are going to be. As humans listening to other humans, we not only pay attention to the content of their words (ie a “hawkish” or a “dovish” speech) but to *how* those words are delivered. Like in any conversation, we subconsciously interpret vocal cues—tone, pacing, nervousness, stuttering, fidgeting and other emotional markers.
But we tend to be extremely biased when evaluating these things, easily swayed by charisma, reputation or appearance.
What if you could train a model to quantify emotional expressions of central bankers in announcements and see how these variables affect markets? Can subtle vocal shifts signal underlying sentiment and influence asset prices?
What I built
I worked from sixteen FOMC press conferences between May 2019 and May 2025 from the Federal Reserve's YouTube channel. I run automated speaker diarization (the process of differentiating each speaker in the video), it produces a timeline of the conference marked up by speaker.
With the Chair's speech isolated, the pipeline cut each conference into clips of roughly ten to thirty-five seconds, splitting on pauses that seemed natural from the transcript (so a clip never chops a sentence in half). Each clip has metadata like exact start and end time, lined up to the time zone.
Older emotion recognition models handled vision and audio separately, one model looks at the face, another listens to the voice, and the two are stitched together afterwards. That works badly in this case.
I used R1-Omni, a multi-modal emotion recognition model built on top of DeepSeek-R1. It watches the video and listens to the audio jointly rather than separately. Because it is LLM-based, it isn't restricted to a fixed list of emotions, it simply describes what it sees in open language. R1-Omni also explains itself with a thinking trace: for every clip it produces a written rationale. And it's open-weights, so I can run inference myself.
Running inference on this was not trivial. I had to rent an AWS Nvidia GPU instance powerful enough and pass each clip sequentially, with video and intermediate stages in cloud storage.
Lining it up against markets
The last step consists of joining the clips to market data. I got one-second tick data on S&P 500 futures and two-year Treasury futures and tried to match each clip's real time to the market.
That produced the dataset of several thousand clips, each with an emotion label, the model's reasoning, a timestamp, and the subsequent price path of different horizons. From there the statistical work is fairly conventional: compare what happens after emotionally non-neutral clips against neutral ones, control for what prices were already doing and for the actual policy surprise, and repeat across a range of time horizons.
The truth is I don't think this research can identify an alpha in any way (yet), but the idea was to prove if open source LLM-esque models can be used for adjacent fields that are not purely language-based, and to prove if the harness and orchestration are workable. Also, in some way to highlight that as LLMs advance they might be able to conquer fields previously thought as "human-only" like understanding emotional responses, "reading" the mood of the markets, etc.