Beyond the Hype: A Realistic Look at Large Language Models • Jodie Burchell • GOTO 2024

Beyond the Hype: A Realistic Look at Large Language Models • Jodie Burchell • GOTO 2024

Introduction

Thank you for the absolutely lovely introduction. A version of this talk was given in Porto last year and has since changed quite a bit. My name is Jodie Burchell, and I am currently working as a developer advocate at JetBrains, all nicely branded today. I've been a data scientist for almost 10 years and have spent a significant portion of my career in Natural Language Processing (NLP). One amusing anecdote: I worked with early large language models like BERT, and even years later, the CEO of my former company seems to not realize that they have been working with AI for the past seven years. Prior to my career in data science, I completed a PhD in psychology, and I've been watching the current space with both interest and concern.

AI Hype Cycle

For the past two years, we've been in a full-on AI hype cycle. Claims have ranged from models like Lambda showing sentience to speculation that huge swathes of the white-collar job market will be replaced by generative models, even reaching doomsday AI apocalypse predictions. This overwhelming storm of opinions makes it challenging to discern the real capabilities and limitations of these models.

Context and History of Large Language Models

Many people think models like ChatGPT came out of nowhere, but they are part of a long history of research in NLP. The early models aimed to automate tasks with huge amounts of text, like classification and summarization, rather than generating text. Large language models belong to a type of model called neural networks, originally proposed to mimic the human brain. Key technical advancements like CUDA, Common Crawl datasets, and Long Short-Term Memory networks (LSTMs) have paved the way for these models.

LSTMs couldn't scale up due to sequential processing limitations. Transformer models solved this and allowed models to grow significantly larger. Generative Pre-trained Transformers (GPTs) emerged from this research, with GPT models stacking decoder units to increase their size and accuracy.

Perception and Claims

Despite impressive capabilities, LLMs face exaggerated claims, including achieving Artificial General Intelligence (AGI). This perception harks back to sensationalized moments in history, like IBM's Deep Blue beating Garry Kasparov in chess. Skill-based assessments of intelligence in LLMs can be misleading, confusing the output with the mechanism.

Generalization and Intelligence

Real intelligence involves solving unfamiliar tasks, not just performing well on specific tasks seen during training. Researcher François Chollet defines generalization levels from narrow to universal. Narrow systems include simple calculators, while extremity means human-level intelligence. LLMs mostly show narrow to local generalization, falling short of AGI.

Practical Applications

LLMs are useful for natural language tasks such as translation, text classification, summarization, and question answering (QA). QA can be enhanced through techniques like fine-tuning and Retrieval-Augmented Generation (RAG), where additional context helps the model answer questions more accurately.

Building a RAG Pipeline

In a demonstration, a Python project using LangChain, an open-source package, works with a large PDF document. The project shows LLM's capacity to answer questions about PyCharm by chunking the document into embeddings stored in a vector database. Using retrieval systems and powerful models like ChatGPT-3.5, the application can accurately answer queries.

Challenges in Deployment

Deploying RAG and LLM applications isn't straightforward. Several hyperparameters significantly impact performance, such as chunk size, embedding models, and vector database retrieval strategies. Effective deployment also requires choosing the right LLM specialized for the task, fine-tuning, and proper performance measurement.

Conclusion

LLMs are powerful but limited and currently far from achieving AGI. Their true strength lies in solving natural language tasks, provided they are carefully tuned and deployed with performance in mind.

Keywords

• NLP
• Large Language Models
• GPT
• Neural Networks
• Transformer Models
• Fine-Tuning
• Retrieval-Augmented Generation (RAG)
• Artificial General Intelligence (AGI)
• LangChain
• Question Answering (QA)
• Performance Measurement
• Deployment Challenges

FAQ

Q: What is a large language model
A: Large language models are types of neural networks designed to process and generate human-like text based on vast amounts of language data.

Q: How do LLMs work
A: They use transformer models to avoid sequential processing, allowing them to capture relationships between words and generate text predictions.

Q: Are LLMs close to achieving Artificial General Intelligence (AGI)
A: No, they mostly show narrow to local generalization and are far from achieving human-like broad or extreme generalization.

Q: What are the practical applications of LLMs
A: LLMs are effective in natural language tasks like translation, text classification, summarization, and question answering.

Q: What is Retrieval-Augmented Generation (RAG)
A: RAG involves pulling in relevant external information to help LLMs answer questions more accurately, combining document retrieval with language model predictions.

Q: What are the challenges in deploying LLMs
A: Challenges include tuning hyperparameters, choosing the right model specialized for the task, and accurate performance measurement to avoid issues like hallucinations.

Q: Can LLMs solve problems they haven't seen before
A: LLMs struggle with generalization and often can't solve unfamiliar problems accurately, indicating they are not truly intelligent in the AGI sense.