What is your experience of using AI for research in condensed matter theory?

 I have been dabbling a little with using AI (at a very basic level) to help me with some research problems. For example, in a recent preprint, I did the maths in the Appendix with help from AI.

I am curious to hear from the experience of others of using it for research in condensed matter theory and theoretical chemistry. 

Peter Woit made a similar request to the high-energy theory community. The responses are helpful and worth reading. An earlier post about AI in math drew the following concrete suggestions from "Greg"

"In math, one thing I found useful to get my bearings was to go through some of my published papers and ask it to try and improve the main result.

Another is to do an adversarial audit of a paper you are very familiar with (yours if you have the stomach for it, someone else’s if not) and then see which of the complaints are pedantic and which are actual.

A third is to give you a history and references/citations for an argument that it provides.

A fourth is to suggest different ways to simplify an argument, even if they are only half-formed in your mind or if you are having trouble making them precise. (Try to be as precise as possible, but don’t use the same filter that you would use with a colleague. Sometimes vague is the best you can do.)

If you are thinking of proposing a question or conjecture in a grant application or preprint, ask it to stress test the conjecture. Don’t lean too much in one direction when prompting it and asking repeated questions (unless there is a particular thread you are trying to untangle, the use your nose, of course), but try to have it explore the positive direction in one query and the negative direction in a follow-up."

[A year ago, I posted about a benchmarking paper for condensed matter theory. I have not seen any updates on that work. For example, have the latest AI models improved significantly on these benchmarks?]

About a month ago, my office mate, Henry Nourse, gave a nice talk, "Is my skillset safe in an AI future?" at the UQ Condensed Matter group meeting. Henry gave a very concrete example of how he used it for a problem involving Raman scattering from quantum spin chains.

Henry stimulated me to see whether AI could help with a problem I was struggling with. It was basically a mathematics problem involving a Fourier transform. There were things that I thought were probably true, but after days of messing with algebra, my aging brain could not prove them. Google Gemini gave the answer in less than a minute. On the one hand, that is impressive. On the other hand, the absolute key was knowing precisely what question to ask it and knowing whether the anwer made sense. Furthermore, it took hours for me to check the result and to refine it, sometimes through further prompts. Besides getting the result, a nice benefit is that AI can quickly generate a LaTeX file that describes the mathematical argument and provides relevant references. Nevertheless, there are still hours more work of checking and polishing the text.

[Aside. The proof made use of various Bessel function identities that I had forgotten. I had used them extensively in a paper about thirty years ago.]

This success motivated me to use Google Gemini to develop, start to solve, and analyse a "simple" classical two-dimensional model of springs interacting with Ising spins. It is simple enough that I think most of the analysis can be done analytically. I have been working on related models on and off for the last couple of years. However, again due to my aging brain, I often get stuck on algebra or the subtleties of the theory of elasticity. AI does the algebra quickly. Furthermore, all the equations are formatted in LaTex. Great!

Again, it bears repeating that a thorough knowledge and intuition about the underlying physics is probably essential for knowing what prompts to make, and whether the answers make sense and are useful. The fact that I had struggled with related models offline beforehand really guided my interaction with AI.

Again, everything still needs to be checked. Sometimes AI makes suggestions that I think may be "hallucinations" [e.g. suggesting connections to the Svetitsky-Yaffe conjecture for lattice gauge theory] that may have arisen from earlier questions about unrelated topics. When I ask similar questions on a later day, it sometimes gives a different answer. For example, once it neglected to include the effect of one term in the Hamiltonian.

Initially, I struggled to keep records and manage the workflow. Eventually, I found it helpful to have a single LaTex file which is like a rough draft of a paper, into which I insert results. I found my engagements with AI both interesting, stimulating, and stressful. I now limit my sessions to short periods of time and then polish and edit the LaTex document in between sessions. This helps me consider what the next steps are. I am glad I am retired and can do all this at a leisurely pace.

What I am doing is incredibly basic compared to what may be possible. I am just using the free version of Google Gemini.

My limited experience raises all sorts of questions about the potential and pitfalls of graduate students using AI in research. Will it enhance or hinder their education? Potentially both?

There are some great quotes from Terence Tao in a recent New York Times article

“The effort needed to solve problems [without AI] is often very instructive. It teaches you something. It’s like going to the gym and having a goal to lift a weight a hundred times. Now, A.I. can solve questions without really getting any value out of them. It’s like having machines that can lift weights for you at the gym.”

“It feels to me like a really clever student who has memorized everything for the test but doesn’t have a deep understanding of the concept,..”

“This has some value,... But once someone shows a specific path to the waterfall, people just take that path. They don’t spend as much time looking for other paths.”

I also recommend Tao's recent lecture at the International Congress of Mathematicians.

For now, I would love to hear from others, particularly in condensed matter theory or theoretical chemistry.

What do you use AI for? What have you learned from the experience? How do you go about it? For example, how do you decide what prompts to use? How do you manage workflows? How do you document your work? What is it good at? What is it bad at? What potential benefits do you see? What pitfalls?