Katharina Zweig: "All of my books can be summarized by the following demand: Software should only be used where it can technically deliver the desired results. I'm not even sure if that's an ethical demand. I think it's more of a common sense demand. Technology should only be used where it can work. Many AI models are loaded with the expectation of being objective or optimal. But they are not."
"If we look at the newer large language models, such as ChatGPT, they basically have the same weaknesses as their predecessors. They sometimes invent things, they hallucinate: they fill in gaps instead of simply saying 'I don't know'.”
"The software has no concept of what is true or false. It calculates probabilities and recognizes patterns in texts. But it doesn't understand the world, it can't think or comprehend contexts. And that's because of the technology behind it."
"Here's an example: If I stopped in the middle of ….. – you would have thought: Oh, something is missing. Many would infer that I would have said 'this sentence' next, some would have guessed I would have said 'talking to you' next. That's exactly what language models do: they guess which word is most likely to come next. They have learned this from millions of texts. The machine is trained by giving the machine only the beginning of a text and letting it guess what would have come next. Whether its guess was right or not then influences how its internal settings are adjusted. In essence, a large language models consists of thousands of equations with between millions and up to trillions internal settings. These are not classic algorithms, i.e., procedures that are mathematically proven to lead to the best possible results. This kind of software is called a heuristic – a code that often leads to good results, but without any guarantee. So, for any prompt the machine computes a probability table for all words in its vocabulary and then it chooses one of it as the next word. After that, it repeats the process using the prompt and the new word, chooses a new word, repeats the process for the prompt and the newly generated words, and so on. A stochastic process."
"There is one important variable: the so-called temperature. This is a parameter that can be set for creating texts – not when training the model, but for generating the responses. If the temperature is 0, the machine always takes the most probable word. This results in texts that are a bit dull and quickly become repetitive. At 2, it gives the same probability to all words and just takes any of it – it can be Japanese, then Arabic, then English again, German, completely jumbled. In language models, the value is usually set to 0.7 to 0.9, and in ChatGPT it is sometimes set to 1. This allows for enough creativity to make it pleasant to read. But it is not reliable."
"Language models process purely linguistic forms. In other words, texts without any real connection to reality. But it is precisely this connection to the real world that would be crucial if we want such systems to really do something for us – for example, acting independently on the internet or preparing decisions. Large language models are intended to increasingly perform such tasks for all situations in life as digital assistants. But it is not a good idea, for example, to use a language model as an independent assistant for travel planning. Last year, I wanted to book a trip to Budapest. The train was canceled at short notice, and the railway company offered me a connection with four changes, each with only a few minutes' transfer time. As someone with common sense, I know that this won't work, so I'd rather take the car. The machine might have simply adopted the suggestion from the railway company's website. It doesn't think for itself the way we do."
"I use AI as a creative sparring partner, for example, to help me with coding or translation, and it often works great. But we should only use it where we can understand and verify the answers ourselves."
"The machine produces texts that only appear to make sense. My RPTU colleague and linguistics professor Jan Georg Schneider uses the term 'intelligible textures' to describe this. He emphasizes that the human reader interprets the texts and fills them with intelligence. On the machine's side, there is no understanding, only pieces of words strung together. Knowing this, I can decide what I want to use it for and what I don't."
"Algorithmic systems have been helping to make decisions about people for several years now. For example: Who gets a loan? How does an insurance company assess the risk of damage? In the US, they even assess the risk of recidivism for someone standing trial. These are highly sensitive decisions that change people's lives."
"I am currently thinking deeply about the way we talk about AI. Why does it seem so natural for humans to say that a machine thinks or understands us or summarizes a text? Do we really mean the same process as if we were observing humans doing something? If we want to explore the social impact on us as individuals and on society, AI research is not just a topic for computer science; we need interdisciplinary exchange. I work closely with colleagues from psychology, philosophy, linguistics, and political science. This allows us to develop a common language to describe what AI systems do. What does 'understanding' mean for a machine? What is “decision-making” in the algorithmic sense – and what is it in the human sense?"
"One project in our Algorithm Accountability Lab at RPTU is TrADeMaS, which stands for Transparency of Algorithmic Decision Making Systems, supported by the federal ministry BMFTR. We are investigating whether machine-made decisions can be explained in a mathematically robust and psychologically effective manner. Such systems use machine learning to evaluate patterns from past data and, on this basis, make decisions about the future – for example, whether someone is creditworthy or eligible for a job. The GDPR and the planned EU AI Regulation ensure that those affected must be able to understand why a decision has been made. To this end, there are so-called explainable AI approaches that are designed to make AI explainable. We are examining whether these approaches really create transparency."
"We look at the issue from two angles: mathematically – how robust are these approaches? Can they detect errors or manipulation – or can they be tricked into saying “everything is fine” even though there is discrimination, for example? And secondly, psychologically: how well do people understand the explanations they receive from a machine? Just because we as computer scientists believe that something is a good explanation does not mean that it is understandable to users. And we see that much of what looks like an explanation is not."
"In an earlier project, we looked at algorithmic decision-making systems in the criminal justice system. These are already being used in some countries to help judges assess whether someone is likely to reoffend. Here, too, we worked in an interdisciplinary manner – with researchers from the fields of computer science, psychology, law, and political science – and asked: What are the limits of such systems? Can responsibility be handed over to a machine when it comes to freedom or imprisonment? We believe that criminal law is far too complex to leave decisions to machines. Legal subtleties, moral considerations, and social contexts play a role in court proceedings – things that algorithms cannot grasp today."
"My mission is to ensure that as many people as possible understand how AI works and what software can and cannot do. So that everyone can decide for themselves: Do I want to use this or not? For me, that's the only way to keep pace with developments."
Zweig: Weiß die KI, dass sie nichts weiß?, Heyne Verlag München, 2025, ISBN: 978-3-453-21907-6, Leseprobe (German only)
Montag, Lachmann & Zweig: Addictive features of social media/messenger platforms and freemium games against the background of psychological and economic theories. International journal of environmental research and public health 16 (14), 2612, 2019
Deutsches Museum, Katharina Zweig gives a lecture in the series Wissenschaft für jedermann, November 2025
Katharina Zweig was a guest on the Scobel Podcast, Dezember 2025
