The technology behind AI for clinicians

A closer look at the technology beneath clinical AI, for an audience of practising doctors. Dr Bakshi sets out what intelligence means when you come to it from neuroscience rather than engineering, walks through where each branch of the field now stands, and closes on the three capabilities a clinician can already put to work. What follows is an edited transcript.

Having done this review, I find it interesting that many people have a very narrow, or even flawed, view of what artificial intelligence is. Being a neurosurgeon, and probably the only neurosurgeon and neuroscientist who is also running an AI company, I will take the liberty of saying this: if you take a neuroscience point of view, intelligence is much more than an ability to learn.

When people talk about AI, they talk about learning. Machine learning, deep learning, all of that. But intelligence is the ability to perceive, learn, reason, adapt and interact. At the end of the day the job of intelligence is to manage the complexity of life, and to survive. That is what intelligence is about, and artificial intelligence aims to replicate it. AI equals the ability to learn is not complete. It needs to be much more than that.

There is work happening in at least eight different domains.

Vision. The computer vision field is at least a decade ahead of language. AlexNet, a convolutional neural network, came out around 2012, and by 2015 these systems started becoming better than humans. Today computer vision models are far better than humans at a lot of classification tasks. So it is not surprising that radiology and pathology systems built on computer vision have developed successfully. A lot of them are in production, and a lot of them are affecting clinical work today.

Language. The next big advance, over the last four years or so, is the arrival of large language models, which can understand, interpret and generate human language. Go back ten or fifteen years and everybody used to laugh that AI would be very mathematical, very analytical, and would never understand emotions. Well, guess what. Today AI can write better poetry. It can compose better music. It can read emotion in video and in written text. We have all been surprised by how quickly these capabilities escalated.

Data analysis. You will know from your own experience that humans struggle with data in more than two dimensions. We draw an x-y graph, we study correlations, we look at a linear trend over time. Add a third dimension and the mind starts to boggle. If you struggle really hard you can hold three. Do you know what computer systems can do? They can handle a million dimensions effortlessly.

What does a million dimensions mean? Take one patient. Make age one dimension, gender another, the first symptom a third, the second symptom a fourth. You can build a million dimensions and then compare patients across all of them. Humans cannot. That is a fundamental capability these large systems have and we do not, and it is what lets us use them to do things a human being would not manage given a million years.

Sound. Then there is auditory AI, the ability to recognise, interpret and generate speech and sound. There is a tool called Suno. You type in that you want a song a bit like one band, with lyrics like a particular rapper and music like a particular composer, and it gives you something fairly close to all three, from text alone.

Cognition. This is the ability to replicate human thought processes: reasoning, learning, problem solving. There is something called chain-of-thought processing happening now. OpenAI has just released a model called o1, and it is coming out with an IQ of almost 120. You all know the average human IQ is 100. These systems are getting closer and closer to replicating the cognitive capability of most people.

Embodiment. The domains above sit on a server. They have no physical manifestation. But when you talk about robotics, or geospatial AI, or AlphaFold predicting the three-dimensional structure of a protein from nothing more than its amino acid sequence, that becomes embodied AI, with its own capabilities.

All of that is theory. Fine. What does it mean for us as clinicians?

Three things are here and now.

The first is retrieval-augmented generation. You give the system a body of knowledge – Harrison's textbook of internal medicine, Robbins for pathology, twenty-five PDF documents – and say, this is the knowledge you will use to answer questions. There is deep mathematics underneath (embeddings, and the machinery around them) that we will not go into, but the effect is that the knowledge arrives at the point where you need it. If you are a clinician making a decision, it brings that knowledge to the decision. If you are a patient asking a question, it brings it to you.

The second is speech to text and text to speech. You can talk to an agent, the agent understands you, and the agent carries out what you asked. That opens access to far more people than typing into a computer, or even into WhatsApp, ever will.

The third is language itself. Multilingual is a solved problem. Modern systems understand Tamil, Telugu, Persian, Russian – name a language, type in it, and the system will answer you in it.