One of the problems with language models is that there is a cut-off date in the training data, that is, the model does not know current events that go beyond that date. Rapier In certain sectors it can be a serious problemis precisely the objective of Talkie-1930, a language model trained solely on texts from before 1930. If you’ve ever wondered what it would be like to talk to someone from the past, there’s an AI for that.
A vintage language model. This is how These LLMs have been baptized who are trained with historical content. Talkie-1930 is a model with 13 billion parameters that does not have access to modern information nor can it consult the Internet, but has only been trained with books, newspapers and other texts from before 1930.
To explore the model, the researchers had Claude converse with the model, evaluating his responses. The model showed great knowledge of the world, with many historical details of the time, and a great ability to imitate the style of Victorian authors such as Dickens, although somewhat limited in more satirical formats.
More than a cultural experiment. Talkie is the closest thing to talking to someone educated in the early 20th century. This turns the model into a window that allows us to explore the mentality and culture of a past time and learn how society, politics or daily life were described back then. But beyond curiosity, Talkie-1930 also functions as a “control subject” to better study the functioning of AI and achieve important advances.
Predicting the future. By being “frozen” in 1930, Talkie makes it possible to better measure how far a model can extrapolate and predict the future from historical patterns alone, without cheating with later data. To test this anticipatory capacity, the researchers showed up to 5,000 descriptions of subsequent historical events, taken from the “On this day” section of the New York Times, and measured the model’s degree of surprise.
The result was that the model showed more surprise in the decades after the data cutoff, especially in the 1950s and 1960s, but then its degree of surprise stabilized. According to the researchers, this suggests that predictive performance improves as the time horizon becomes longer, but they point out that it will be necessary to train older models to be able to measure it well.
Invention. Demis Hassabis, CEO of Google DeepMind, raised a very interesting question at a conference recently: if an AI with a limit of knowledge until 1911 could reach the theory of relativity that Einstein discovered in 1915. In this sense, models like Talkie-1930 are a very interesting tool to observe its ability to generate new ideas that can lead to discoveries.
No pollution. Is one of the problems that the models have trained with large corpuses of current data, in which the evaluation data itself usually also sneaks in and ends up causing their capabilities to be overestimated. With vintage models there is no contamination and that allows you to carry out very specific experiments, such as seeing if you are able to learn to program without having any prior computer knowledge. Talkie-1930 is open source and is available on Github.
Image | Xataka
GIPHY App Key not set. Please check settings