LLM Deployment and Challenges in the Space With Pedro Torruella (OctoAI)

Introduction

Hi everyone, I'm Annie and we're at the AI Engineer Worlds Fair. Today, I'm here with Pedro. Hi Pedro!

Pedro: Hi, Annie, nice to meet you.

Annie: Nice to meet you too! Could you introduce yourself?

Pedro: Sure. My name is Pedro. I am a DevRel Engineer at OctoAI, and I've been with them for over a year and a half. Super excited to be here.

Annie: That's awesome! Could you tell me more about the product and the problems you guys are solving?

Pedro: Absolutely. We are a GeniSys platform. What does that mean? We package models, both open source and closed source, and optimize them to squeeze out the most performance. Then, we add necessary features so you can take these models into a production environment. High-level, we often say that we give you the OpenAI API experience on open-source models to ensure smooth migration from other models. That's just one of the many things we can do for you.

Annie: Interesting. Any other problems you're solving?

Pedro: Yes, we actually have two product verticals. We have media generation (Media Gen) for all your diffusion models and a couple of image processing endpoints. And we also have our LLM (Large Language Model) solution. We offer an embedding endpoint and more. These are available on our SaaS API for public deployment. Depending on your needs, there's a progression. For instance, if you have a lot of traffic, we can handle that and deploy it for you. If you have your own GPUs or AWS/GPC credits, we provide containers called OctoStack that you can deploy on your own terms.

Annie: What are some of the challenges you believe will remain in deploying models within the next two to three years?

Pedro: One challenge is the size of the models. Although there are talks about perhaps not needing big models, research shows that they will get bigger, requiring more hardware and optimizations. That’s where we come in. We make it possible to put these big models into existing hardware. That is a significant challenge, but we’re working towards solving it.

Annie: Anything specific you see changing in the future?

Pedro: Actually, yes. We were discussing this over dinner yesterday. The adoption of agent-based programmatic workflows is something I think will permeate the rest of the software ecosystem. The agentic workflow is changing how we do software, and I’m very excited about that.

Annie: That sounds fascinating. Anything else you'd like to share with our viewers?

Pedro: Sure. We have a couple of talks lined up. We’ll be at the AWS booth this afternoon with a fine-tuning workshop. Tomorrow, I’ll be discussing LLM optimizations and the tricks we use to serve LLMs. We are here to help since it’s still early days in the AI world. We appreciate it if you come to OctoAI, talk to me, and let’s work together to make you successful.

Keywords

• OctoAI
• GeniSys platform
• Model Optimization
• SaaS API
• Media Generation
• LLM
• Deployment Challenges
• Agentic Workflow
• LLM Optimizations

FAQ

Q: What does OctoAI offer?

A: OctoAI packages and optimizes models, both open-source and closed-source, and provides necessary features for production environments. The aim is to offer an OpenAI API-like experience with open-source models.

Q: What are the product verticals of OctoAI?

A: OctoAI has two product verticals: Media Generation, which includes diffusion models and image processing endpoints, and LLM solutions, which include embedding endpoints.

Q: What are the challenges in deploying models in the next few years?

A: One of the main challenges is the size of the models, which are expected to grow, requiring more hardware and optimizations.

Q: How does OctoAI address deployment challenges?

A: OctoAI helps by enabling large models to run on existing hardware, providing both public deployments and customizable container-based solutions for client-specific needs.

Q: What future changes are foreseen in the software ecosystem?

A: The adoption of agent-based programmatic workflows is expected to change how software development is done, making it more dynamic and efficient.