LlamaIndex Webinar: Improving RAG with Advanced Parsing + Metadata Extraction

Introduction

Hey everyone, Jerry here from LlamaIndex. Today we're excited to host another edition of the LlamaIndex webinar series. The topic of today's webinar is best practice data preparation for RAG (Retrieval-Augmented Generation).

Introduction

I'm super interested in this topic because we've been discussing the importance of the data processing layer in building any question-answering or knowledge assistance system. This includes parsing, transformations like chunking, metadata extraction, and indexing.

Today, we are co-hosting a workshop with the co-founders of DZ with Rhys Leonard on improving RAG systems. We will show you step-by-step how parsing and especially the metadata extraction piece can directly lead to performance improvements. This will be a practical and technical workshop.

Workshop Overview

Introduction of Hosts

• Rhys Leonard: One of the founders and CEO of DZ, specializing in metadata creation and standardization.
• Jerry: From LlamaIndex, focusing on the data processing layer including parsing and indexing.

Agenda

1. Brief overview of both LlamaParse (for parsing) and DZ (for metadata creation).
2. Role of metadata in RAG and ways high-quality tags can be used to improve retrieval.
3. Experimental setup and results regarding parsing and metadata impact on performance.

Common Problems in Rag

Four core problems in RAG systems, especially with complex documents like research papers, include:

1. LLMs selecting the wrong document or chunk
2. Inability to pull from tables, charts, and complex formats
3. Latency and cost issues as data volume scales
4. Scalability issues

Key Areas: Parsing and Metadata

1. Parsing: High-quality parsing can extract information from tables, charts, and images while maintaining links between document sections.
2. Metadata: Improves retrieval by embedding relevant chunks or documents and allows quick document filtering.

Workflow

1. Large volume of research papers.
2. Parsing using LlamaParse.
3. Page-based chunking as a baseline.
4. Metadata creation using DZ.
5. Embedding metadata into an index using Quadrant as the vector DB.
6. Two approaches utilizing metadata:
   1. Filtering the embedding space.
   2. Using a router query engine.

LlamaParse Overview

LlamaParse is part of the LlamaIndex Cloud offering and helps in parsing complex documents like PDFs, PowerPoints, and HTML. It's excellent at extracting images, charts, tables, and can be customized through input prompts.

DZ Overview

DZ focuses on creating high-quality, standardized hierarchical metadata. This includes chunk-level and document-level metadata. The metadata aids in quickly filtering documents and enhancing retrieval accuracy.

The Role of Metadata in RAG

Use Cases

1. Pre-filtering data ahead of LLM usage.
2. Creating intelligent document clusters.
3. Improving routing accuracy using metadata for quick decision-making.
4. Using descriptive metadata to answer frequently asked questions.

Filtering Embeddings

Metadata allows embedding space to be effectively segmented, making retrieval efficient. The vector autoindex retriever can filter embedding spaces based on metadata attributes.

Router Query Engine

Using metadata allows the Router Query Engine to evaluate document-level metadata for better decision-making, enabling faster and more accurate retrieval.

Experimental Setup and Results

Experiments

• Large volume of research papers with 100 documents and 100 questions.
• Comparison between simple text splitting (using PyPDF) and high-quality parsing (using LlamaParse).
• Addition of metadata for enhanced retrieval.

Results

1. Parsing Improvements:

   • LlamaParse outperformed PyPDF by correctly maintaining linkages between document sections.

2. Metadata Impact:

   • Significant improvement in retrieval accuracy and scalability.
   • Domain-specific metadata proved most impactful.

Key Takeaways

• Metadata significantly enhances retrieval accuracy and scalability.
• High-quality parsing is critical for RAG systems dealing with complex documents.
• Using both parsing and metadata together yields the best results.

Q&A Session

Questions covered topics such as:

• Metadata's role in summarizing content.
• Supporting custom models for extraction.
• Dynamic metadata generation during user interactions.

Conclusion

The presentation highlighted that metadata is underexplored but critical for improving RAG systems. Metadata allows better filtering and routing, leading to higher accuracy and scalability.

For more detailed information, a waiting list for DZ's API is available for those interested in metadata extraction and labeling services.

---

Keywords

• LlamaParse
• DZ Metadata
• RAG Systems
• Retrieval-Augmented Generation
• Parsing
• Metadata Extraction
• Data Preparation
• Vector DB

---

FAQ

Q: What is the purpose of LlamaParse? A: LlamaParse is designed to parse complex documents to make them suitable for LLMs, extracting images, charts, and metadata efficiently.

Q: How does metadata improve RAG systems? A: Metadata helps by quickly filtering documents, improving the accuracy and scalability of retrieval, especially as data volumes increase.

Q: What types of documents were used in the experimental setup? A: The experiments used scientific research papers, which are complex in format.

Q: Can I use custom models for metadata extraction? A: Yes, DZ supports custom models, allowing users to connect their preferred models for metadata extraction.

Q: How is metadata generated and validated in DZ? A: Metadata can be Auto-suggested by the system or manually defined. It is validated through human-in-the-loop workflows to ensure accuracy.

Q: Does using metadata help with document summarization? A: While the primary goal in the experiment was retrieval accuracy, metadata can be useful for summarization if high-level themes are identified and tagged.