Can AI Understand Literature? (Claude 3.5, GPT 4o, Llama 3.1 405B evaluated)

Introduction

Introduction

In this article, we'll delve into the fascinating intersection of modern AI, specifically Large Language Models (LLMs), and literature. As a computer programmer with a deep interest in AI, and a literature enthusiast who runs a YouTube channel focused on discussing and interviewing about literature, this intersection lies at the nexus of my twin passions. We'll examine how various LLMs—GPT-40, Mini GPT-40, Llama 3.1 405B, and Claude 3.5—interpret and understand literature.

Benchmarking LLMs

Overview

Benchmarking LLMs is a critical process. Given the non-deterministic nature of these models, benchmarks must be scientific and verifiable yet complex enough to elude basic lookup through search engines.

Set-Up

For this study, we’ll be comparing five different models: GPT-40, Mini GPT-40, Llama 3.1 405B, and Claude 3.5. Our focus will be how these models can enhance and intersect with the reading and studying of literature.

Biblical References in Moby Dick

For our first test, I asked each model to identify five biblical references in Herman Melville's "Moby Dick."

• Meta 405B: B grade, satisfactory but not exceptional.
• GPT Mini 40: Identified key references but lacked depth—B grade.
• GPT 40: Good but a step below expectation—B grade.
• Claude 3.5: Provided comprehensive answers, especially noting Rachel weeping for her children—A grade.

The Whiteness of the Whale in Moby Dick

I then asked about the significance of the whiteness of the whale.

• Meta 405B: Detailed and thorough—A grade.
• GPT Mini 40: Clearly articulated but lacked certain nuances—C grade.
• GPT 40: Included significant points but not exhaustive—B grade.
• Claude 3.5: Most comprehensive and accurate—A grade.

The Significance of the Town-Ho Story in Moby Dick

Next, we discussed the purpose of the Town-Ho story, an interlude in "Moby Dick."

• GPT Mini 40: Long-winded but hit the core points—B grade.
• GPT 40: A better balance of depth and brevity—A grade.
• Meta 405B: Unique insights not considered by GPT models—A grade.
• Claude 3.5: Reasonable but lacked a certain critical depth—B grade.

Samuel Johnson on American Independence

We asked why Samuel Johnson opposed America's independence.

• GPT Mini 40: Basic, missed critical moral components—B grade.
• GPT 40: Included moral arguments about slavery—A grade.
• Meta 405B: Lacked moral depth—B grade.
• Claude 3.5: Comprehensive and morally nuanced—A grade.

Ezekiel Chapter 16

We explored the unique and significant aspects of Ezekiel Chapter 16.

• GPT Mini 40: Missed the depth of visceral imagery—C grade.
• GPT 40: Better but still surface-level—B grade.
• Meta 405B: Comprehensive yet concise—B grade.
• Claude 3.5: Included imagery and allegorical analysis—A grade.

Foucault's Power Theories

We compared how different models discussed power in capitalist vs. pre-capitalist societies according to Michel Foucault.

• GPT Mini 40: Consistent but somewhat redundant—B grade.
• GPT 40: Detailed and insightful—A grade.
• Meta 405B: Introduced additional Foucault concepts—A grade.
• Claude 3.5: Familiar and strong across multiple concepts—A grade.

Nietzsche's Eternal Return

We queried whether Nietzsche believed in the concept of the eternal return.

• GPT Mini 40: Aligns with scholarly consensus—did not literally believe—C grade.
• GPT 40: More nuanced, admitted the possibility—B grade.
• Meta 405B: Complex and noted metaphysical implications—B grade.
• Claude 3.5: Kept open the possibility of literal belief—A grade.

Carl Naussgard's My Struggle

For Carl Naussgard's "My Struggle," we asked for unique and compelling aspects.

• GPT Mini 40: Raw honesty and stream of consciousness noted—B grade.
• GPT 40: Added the blending of autobiographical fiction—A grade.
• Meta 405B: Complex and deep—A grade.
• Claude 3.5: Comprehensive and philosophical depth—A grade.

War Poetry in Wilfred Owen's Style

Lastly, we asked each model to produce a war poem in Wilfred Owen's style.

• GPT Mini 40: First poem was surprisingly aligned—A grade.
• GPT 40: Good but not nearly as impactful—B grade.
• Meta 405B: Mostly simplistic and lacked depth—C grade.
• Claude 3.5: Captured some essence but not all—B grade.

Conclusion

Collectively, the GPT-40 mini model surprisingly performed well, particularly in poetic exercises. Claude 3.5 and meta 405B also demonstrated strong comprehension and synthesis capabilities.

Keywords

• Literature
• AI
• Large Language Models
• Biblical References
• Moby Dick
• Samuel Johnson
• Ezekiel Chapter 16
• Michel Foucault
• Nietzsche
• Carl Naussgard
• Wilfred Owen

FAQ

Q: Why is Michel Foucault's theory of power important in this study?

A: Foucault's theories help us understand how AI interprets complex social and philosophical concepts, reflecting their capabilities to handle abstract academic material.

Q: Did any model excel in providing poetic imitations?

A: Surprisingly, GPT-40 mini produced the most compelling Wilfred Owen-style war poem.

Q: How did the models perform in identifying biblical references in literature?

A: Claude 3.5 provided the most comprehensive and accurate biblical references in "Moby Dick."

Q: Were there any significant differences in grade distribution among models?

A: Generally, larger models like Claude 3.5 and GPT-40 performed better with a greater frequency of 'A' grades compared to smaller models.

Q: Can AI be effectively used to understand and interpret literature?

A: Yes, this study shows that AI can provide valuable insights and even offer reasonable poetic imitations, although they still occasionally fall short in depth and nuance.