We gave ChatGPT a college-level microbiology quiz. It blew the quiz away.

You wouldn’t know it from interacting with Siri or those technical-support, call-center robots, but artificial intelligence has made some incredible advances in a short amount of time. Earlier this year, the tech world was abuzz with various generative AI programs that could, on command, create entirely new, never-before-existing images or works of art.

Today, the tech world is abuzz again over ChatGPT, a chat bot that is said to be the most advanced ever made. Just how advanced is ChatGPT? It can create poems, songs, and even computer code. On Twitter, Thomas Ptacek showed that it could convincingly construct a passage of text on how to remove a peanut butter sandwich from a VCR, in the voice of the King James Bible.

In my opinion, that is truly remarkable. But as a scientist, I want to know more about ChatGPT’s abilities. Can it answer hard questions about difficult topics? Can it synthesize information? Can it analyze an experiment?

To find out more, I subjected ChatGPT to a microbiology quiz. The questions I asked would be appropriate for a final exam for an undergraduate biology major who just completed an introductory course in microbiology. Note that ChatGPT tends to drone on a bit, so I truncated many of its answers for the sake of brevity.

Q1. In two sentences or less, explain the difference between Gram positive and Gram negative bacteria.

A1. Gram positive bacteria have a thick peptidoglycan layer in their cell walls, while Gram negative bacteria have a thin peptidoglycan layer. This difference in the cell wall structure of the bacteria leads to different staining properties when they are exposed to crystal violet and iodine, the two dyes used in the Gram staining procedure.

Grade: Mostly correct. The ChatGPT failed to mention that Gram positive bacteria have a single membrane, while Gram negative bacteria have two membranes. (This is probably the more important distinction between the bacteria.) It also is incorrect about the dye; the correct dyes are crystal violet and safranin. (Iodine is used to “fix” the crystal violet dye.)

Q2. In three words or less, transformation, transduction, and conjugation are all examples of what?

A2. Genetic exchange in bacteria.

Grade: Correct (if we forgive the word “in”). I am impressed by this, as it shows that ChatGPT is capable of identifying the most important common feature of these microbiological processes. Transformation occurs when a bacterium incorporates DNA from its environment; transduction occurs when a virus transfers genes from one bacterium to another; and conjugation occurs when one bacterium injects DNA directly into another bacterium. All of them are forms of genetic transfer.

Q3. Complete the simile. Prions are to protein as viroids are to what?

A3. Prions are to protein as viroids are to RNA. [Truncated]

Grade: Correct. I am very impressed by this. It understands similes. (Seriously?!) Prions are infectious proteins (one form of which causes mad cow disease), and viroids are infectious RNA molecules that cause disease in plants.

Q4. An emergency room patient presents with a terrible headache and stiff neck. The doctor orders a spinal tap to collect cerebrospinal fluid. A Gram stain of the CSF reveals the presence of Gram negative diplococci. What is the diagnosis?

A4. Based on the information you provided, the Gram stain of the cerebrospinal fluid (CSF) shows the presence of Gram-negative diplococci, which are bacteria that are typically oval-shaped and occur in pairs. This finding is consistent with the diagnosis of meningitis. [Truncated]

Grade: Correct.

Q5. A microbiologist does a plate count to determine the concentration of bacteria in a test tube. He performs six ten-fold dilutions and puts 1 mL of the final dilution on a Petri plate. 205 colonies were observed on the Petri plate. What was the original concentration of bacteria in the test tube?

[A5 is long but worth reading in its entirety. A screenshot is below.]

Grade: 100% correct. (My jaw is on the floor.)

Q6. In five words or less, what is the point of Koch’s postulates?

A6. Establish causality between microbe and disease.

Grade: Nailed it (again, forgiving the word “and”).

Q7. A mutation occurs in the promoter region of a gene so that sigma factor can no longer recognize or bind to it. In one sentence, what will happen to expression of the gene?

A7. Expression of the gene will be decreased because sigma factor cannot recognize or bind to the promoter region.

Grade: Correct.

Q8. A scientist wants to clone a gene into a plasmid. How should he decide which restriction enzyme to use?

A8. There are several factors to consider when choosing a restriction enzyme for cloning a gene into a plasmid. The first and most important factor is the compatibility of the restriction enzyme with the plasmid vector. The restriction enzyme must produce compatible sticky ends that can be ligated to the sticky ends of the plasmid vector, allowing the gene to be inserted into the plasmid. The scientist should also consider the size of the gene and the restriction enzyme’s recognition site, as well as the presence of any potential restriction sites within the gene itself. [Truncated]

Grade: Correct.

Q9. A natural killer cell encounters a somatic cell that lacks MHC I molecules on its surface. In one sentence, what is the likely outcome?

A9. If a natural killer cell encounters a somatic cell that lacks MHC I molecules on its surface, the natural killer cell is likely to kill the somatic cell.

Grade: Correct. (A natural killer cell — also called an NK cell — is sort of like the “secret police” of the immune system. They monitor body cells, asking to see “identification” (MHC I molecules). If a cell cannot produce that identification, it is killed on the spot.)

Q10. A scientist wants to test the efficacy of a new antibiotic. He infects ten rats with Salmonella and administers the antibiotic to all ten rats. All of them recover. What is the flaw in this experiment?

A10. One potential flaw in this experiment is that the scientist only used ten rats. This is a relatively small sample size, and it is possible that the results of the experiment are not representative of the population as a whole. It is generally recommended to use a larger sample size in experiments in order to increase the reliability and validity of the results. Additionally, the experiment should include a control group of rats that do not receive the antibiotic in order to compare the effects of the antibiotic with the effects of no treatment.

Grade: Correct. (Though, I don’t think the comment about the rats being insufficiently representative of the population is relevant here — this isn’t an epidemiology study.)

ChatGPT blows away the microbiology quiz

I went into this project as a total skeptic. I was completely certain that I could ask ChatGPT questions that would be far too difficult — not because the content is hard, but because my questions required that the AI demonstrate fairly specialized knowledge, an ability to synthesize that knowledge, an ability to be concise in its responses, and an ability to construct a mathematical solution to a microbiology word problem. With the exception of Q1, ChatGPT passed with flying colors. If I were grading the quiz, I would give ChatGPT a 95% — which is far better than what most human students likely would get.