Theoretical physicist Matt Hodgson from the University of York talks to Hamish Johnston about the role that large language models, or chatbots, can play in physics
What are large language models or chatbots?
A large language model (LLM) is a type of artificial intelligence. The goal of the model is to generate human-like text in response to prompts from a user. This could be answering questions, writing e-mails and even writing poems and stories. One LLM that has been making headlines recently is ChatGPT. It is highly advanced as it is trained on an enormous amount of data, with the current version using Internet data up to 2021.
How did you become interested in chatbots?
I have been using neural networks – a type of artificial intelligence – to look at complex mathematical relationships. Similarly, chatbots learn language-based relationships and I became recently aware of the potential of chatbots, and specifically ChatGPT, thanks to a collaborator who specializes in AI. I then began experimenting with ChatGPT and discovered that it could be useful in many of my daily tasks.
Any examples?
I use them to write e-mails or summarize a long e-mail if I don’t have the time to read it. If I’m rushing off to give a lecture, and I get an important but long e-mail, I could ask the chatbot to summarize it in just a couple of bullet points. If I write an abstract for a conference and I realize it is actually limited to 500 characters, not 1000, and if I have left it to the last minute, I can give it to a chatbot and say “make this into 500 characters”. Of course, I always reread the output just to make sure the chatbot hasn’t introduced anything I don’t like.
Can chatbots be used for maths?
Chatbots are language-based models, so you can’t expect it to solve advanced mathematical expressions. However, it has been trained on examples where people have performed mathematical derivations, so although not what it is designed to do, it can do basic maths. I would always favour tools designed for mathematics, like Wolfram Alpha, which has a proven track record.
What about computer code?
Yes. I found it particularly useful for writing computer code. You could ask ChatGPT to look over your code and identify mistakes. It is almost like having a private tutor who can spot your mistakes and tell you how to improve your code.
Do you have any examples?
I was writing a lecture on solving Schrödinger’s equation and I gave the mathematical form of my solution to ChatGPT and asked it to write a Python code that generates a GIF. It was able to write a code in a few seconds. It is not that I couldn’t have written the code myself, but it is time-consuming. What would have taken me 30 minutes before, now takes less than five minutes, and saving that time has given me the opportunity to improve the quality of my lectures and introduce more interesting examples to my students.
Are chatbots widely used in academia?
They are not used as much as they should be. I feel that we have been caught off guard a little bit by the explosion of ChatGPT. We’re now playing catch-up, part of which is trying to protect the academic integrity of our courses. In physics we don’t have the same problem that disciplines with more essay writing have. Yet we already deal with advanced tools capable of assisting the work a physicist does on a daily basis. For example, Wolfram Alpha has been around for a long time, and for a physics student that is a more effective cheating tool because it can perform quite advanced mathematical derivations. But we shouldn’t become too complacent and underestimate what chatbots can do.
Is it good for students to use them?
It is really important that we stress to our students that they have to practise fundamental skills before they can use something like ChatGPT. Understanding the underlying concepts and principles is necessary to use these tools effectively and interpret their results. This is as true for written work as it is for mathematics. That includes writing papers and performing a literature review. If you get a chatbot to do it for you, you never learn what a good literature review is. We also need to think hard about how we are going to change the way we assess things like lab reports. Do we now start looking at closed-book lab reports or do we assess higher-level skills so we’re looking for the deep analysis that a chatbot cannot do?
What role can chatbots play in scientific publishing?
I think there shouldn’t be a lot of concern around this, but again, we shouldn’t become too complacent. Every tool is designed to make work more productive, and we shouldn’t rely entirely on the tool to do the job. The tool is there for people to use and to create something beyond what they could do without it. However, it could be that because these chatbots are so accessible, journals might see an enormous increase in bogus papers that are written entirely by chatbots and that could be a real problem. You then run this risk that the chatbot writes the paper and the chatbot reads the paper, and humans are bypassed entirely. Like with anything, if it’s abused to that extent, then it is a problem.
Do you think it’s a good idea that some science publishers have introduced policies requiring authors to document how they’ve used the chatbots?
We never declare a computer did the calculations, so why should it matter if you have a tool that helps you with writing? If the peer-review process works correctly, it should be able to identify, say, a literature review that’s been entirely written by AI. If a conscientious scientist declares in their paper that they have used a chatbot to help improve their introduction, I don’t really know what the reader is supposed to think when reading the author’s declaration other than, well good, I’m glad it’s improved the accessibility of the paper.
Could chatbots be a good leveller for people who don’t have English as their first language?
Absolutely. That is definitely a positive and I would hope that it improves accessibility for scientists who are in a country where they can’t get a good English education. It could also help native speakers who are just poor communicators. After all, it’s always a shame for someone to have a brilliant idea, but then it fails to be communicated properly. I think it’s recognized in academic writing, certainly in physics, that the accessibility of papers is poor compared to the quality of the research. The chatbot revolution: how physicists are using large language models in academia
Could chatbots also help interdisciplinary research?
Yes. If I speak to a chemist about my work, because there is some overlap in our interests, but I use words that they don’t understand, it can be a real slog for them to try and understand what I’m doing. So I could say to the chatbot “I have written this abstract for a physics conference, can you rewrite it in language that a chemist could easily understand”. I would be always cautious, however, because it might change the meaning of what I’ve written.
Could there be diversity issues with chatbots?
In principle, yes, but we have to be careful about this. It depends on the data from which the model is trained and whether that dataset has implicit biases. But that is not to say it doesn’t have potential uses. For example, if you were sloppy and didn’t write using inclusive language, you could just give it to the chatbot and say “make this inclusive”. I would always remain wary and avoid relying entirely on the chatbot to do this for you. We all have unconscious biases and not addressing them in favour of relying on AI to address them for you is not a good idea because the chatbot won’t always be there to assist.
Are you positive about the future of chatbots?
Yes. I think the biggest advantage is improving the clarity of academic writing. You could write lab instructions and give it to a chatbot and ask whether the instructions are clear or whether they need improving. It’s kind of like running it past 100 or 1000 people before giving it to your students. Hopefully, by doing this it increases the accessibility of the instructions and the students can focus on the physics. It’s also important to remember that we are only in the early days of these chatbots and more sophisticated chatbots will certainly come.
