Crude Signal, reading 200,000 WSJ articles for oil
Research that turned 200,000 Wall Street Journal articles into daily news themes, to test whether the news says something about oil prices that the prices do not.
Before large language models made text cheap to read, the question was already there. Does what the newspaper talks about tell you something about oil that the price does not? This was my first time turning messy text into a signal, and the habit it taught me, making the middle step readable by a person, shows up in everything I have built since.
Finance research had already shown that news attention carries information. Bybee, Kelly, Manela and Xiu's paper Business News and Business Cycles, published in the Journal of Finance in 2024, used a topic model over about 800,000 WSJ articles to show that attention to news themes tracks the economy and helps forecast returns. For oil specifically, Brandt and Gao found in 2019 that macro news helps predict oil returns over longer horizons, while geopolitical news moves prices without predicting them. The team's question was whether attention to news topics adds forecasting power for crude oil on top of models that only use prices.
I built the text side. I put together and cleaned more than 200,000 WSJ articles, which meant removing duplicate wire stories, splitting the text into words and stripping out common words and boilerplate. Then I fit an LDA topic model, which gives every article a mix of themes, and turned those mixes into a daily series of how much coverage each theme got.
Then I read the topics. For each one I looked at its top words and its most typical articles, named the ones that made sense, merged near-duplicates and dropped the noise. Geopolitical conflict came through as one of the clearest drivers. Reading them by hand is what made the series usable for economists, because every theme that went into a forecast had a name a person agreed with.
In the paper I contributed to, adding the news features cut the out-of-sample forecast error by 18% compared with the same model without news, and raised the backtested Sharpe ratio of the resulting strategy from 0.07 to 0.45. These are the paper's model results on historical data, not a live trading record, and the credit for the model belongs to the team. My part was the text pipeline and the topic review that fed it.
Research for the University of International Business and Economics. I am happy to go through the methods and results on a call.
LDA was the right tool in 2023 because it needs no labels and its output is easy to read. Nobody chose the topics in advance, and each one is a list of words an economist can argue with. Its weakness is short text, and work by Bai and co-authors in 2022 showed that headlines are too short for standard topic models.
Today I would keep the same shape and swap the parts. I would use embeddings to group short headlines, an LLM to suggest topic names and flag groups that do not make sense, and a person to approve the names before any series goes into a forecast. The idea I carried into every later project is that the middle step should be readable by a person, because that is what lets a team trust a model's output and correct it.