LLMOps: Everything You Need to Know

building with ai by

Autoblocks

LLMOps: Everything You Need to Know

Table of Contents What is LLMOps?

3

Introduction

3

The Need for New Tooling

4

Why is LLMOps Tooling Important?

5

LLMOps vs. MLOps

5

Distinct Underlying “Jobs to Be Done”

7

Distinct Product Builders

9

Distinct Product Workflows

11

How Autoblocks Helps You Build Better AI Products

Autoblocks

13

1

Autoblocks

2

Rather than working with ML teams across silos, product teams can be much more proactive in ..

What is LLMOps?

Evaluating where they can leverage AI to enhance user experience, an

Introduction

Executing the actual implementation.

2023 was a breakout year for AI adoption. Although some of this tailwind is driven by FOMO (my competitors are thinking about AI, so I should, too), most of it is driven by a genuine paradigm shift in what it means to “ship an AI product.”

Finding the Right Use Cases for GenAI

Two specific breakthroughs catalyzed this shift.

Breakthrough #1: The barrier to entry for shipping AI products has been slashed.

The Need for New Tooling Due to these trends, there’s rightfully a lot of exuberance and urgency among product teams to deliver exceptional AI products.

Under the old paradigm of traditional machine learning (ML), integrating AI in your products meant:

One of the problems that product teams quickly run into, however, is that existing tooling isn’t well-suited to accommodate the unique infrastructure requirements of building with GenAI.

Boatloads of data, Dedicated ML staff, Massive compute resources, and

This creates a burning need for a new class of tooling that helps product teams build with GenAI. The important thing to note is that this new class of tooling is focused on threading the needle of being

A very specific use case.

Today? A product engineer can use foundation models — through a cloud API provider or open-source library — to deploy the v1 of an AI product in just a few hours.

Familiar enough to fit into product teams’ workflows, an Bespoke enough to actually be useful for managing GenAI-powered products.

Breakthrough #2: Product managers and engineers can now own the end-to-end AI product development process.

This tooling has been described as a lot of things. Generally, the AI community seems to be triangulating around “LLMOps.”

Some argue that since the tooling can be generalized to other types of foundation models, a better term for the category is “FMOps,” or foundation model operations. This quickly turns into a matter of semantics and personal preference.

In addition to the lower barrier to entry, the primary stakeholder within a company responsible for building these features has migrated from ML teams to product teams.

This is an exciting development because product teams are closest to user experience. They are the best stakeholders to identify where AI can fit into their product.

Autoblocks

3

Autoblocks

4

Why is LLMOps Tooling Important?

LLMOps vs. MLOps Now, let’s address a question we often hear from product leaders: Can’t our company’s MLOps platform handle this?

The short answer is no.

GenAI and ML are both “AI,” but they’re more different than similar as far as product leaders are concerned.

Three main reasons differentiate LLMOps from MLOps, and we elaborate on each of these in the sections below They enable distinct underlying “jobs to be done, They have distinct builders, an They require a distinct product workflow.

MLOps platforms like Weights & Biases and Arize AI will have you believe that LLMOps is “MLOps for LLMs.” However, we believe that’s a narrow outlook and should be taken with a grain of salt.

There’s a reason that MLOps platforms evangelize this view, and ML practitioners readily subscribe to it. Humans are, by nature, pretty uncomfortable with change. It ends up being more palatable to think of the GenAI paradigm shift as “an evolution of traditional ML,” versus viewing it as an entirely different beast.

Autoblocks

5

Autoblocks

6

Distinct Underlying “Jobs to be Done”

The tooling for GenAI and traditional ML is so fundamentally different because the underlying “jobs to be done” that GenAI and ML enable are fundamentally different.

Machine learning offers some level of personalization while maintaining relatively high predictability.

ML models are intensively trained on a lot of data to make decisions (usually predictive or classification) with a high degree of confidence.

For example, based on her prior activity, Netflix’s recommendation engine decides which show to display on Olivia’s home screen.

GenAI, on the other hand, provides a high degree of personalization but with lower predictability.

GenAI foundation models are random (or stochastic) by design. Their unpredictability and adaptiveness are where their magic lies.

Going back to the Netflix example, instead of solely relying on an ML model to guess what Olivia wants to watch, a GenAI product could allow her to tell Netflix what she’s in the mood for — “I’m in the mood for a light rom-com, and my favorite rom-com is The Switch” — and surface appropriate movie recommendations.

Because of these different underlying “jobs to be done,” it’s fair to view GenAI and ML as two distinct tools in a product team’s toolkit. Each one requires specific infrastructure to be effectively leveraged.

Autoblocks

7

Autoblocks

8

Distinct Product Builders

Not only are GenAI and ML distinct tools in a product team’s toolkit, but the people who are best equipped to leverage these tools are different.

As mentioned, product teams’ dependency on ML teams to ship AI products has largely vanished. Product teams can now own the end-to-end process.

MLOps platforms are facing a classic innovator’s dilemma. Their core audience is ML practitioners. But ... they see the immense potential that GenAI has unlocked.

Hence, they’re trying to simultaneously participate in this latest wave of AI without alienating their “in-seat” user base.

Meanwhile, LLMOps platforms like Autoblocks are building specifically for product teams, from the ground up.

Instead of forcing product teams to learn about F-scores and loss factors, LLMOps platforms conform infrastructure to the needs of product builders (rather than the other way around).

MLOps tools make the lives of machine learning practitioners and data scientists easier.

LLMOps tools make the lives of product teams and developers easier.

Autoblocks

9

Autoblocks

10

Distinct Product Workflow

Despite how turnkey LLMs are, using them in a commercial product presents unique challenges to the product development lifecycle.

A good way to mentally model these challenges is by segmenting the AI product development cycle into two categories: testing and evaluation.

Testing: How do I set up my product?

There are an infinite number of ways you can configure your LLM app. You can change the model, the model parameters, the model weights, the prompts you give the model, the data you enrich the response with, etc.

LLMs are dynamic black boxes, so the best way to evaluate them is by ..

Testing them thoroughly, and Keeping a close pulse on how users engage with your app post-production.

LLMOps solutions can help you test, understand, and evaluate user interactions at scale.

Examples of LLMOps Solution Observability & analytic Evaluatio Traces & run trees Unlike a traditional software product, GenAI products often demand an MVP approach, where you ship a product and then iterate on it post-production.

LLMOps solutions look to enable that refinement process by giving product teams the tools they need to iterate effectively.

LLMOps solutions can help you play with different levers and determine the best configuration.

Examples of LLMOps Solution Testing suit Prompt playgroun Prompt managemen Model management

Autoblocks

Evaluation: What do users think of my product?

How to Build World-Class GenAI Products

11

Autoblocks

12

How Autoblocks Helps You Build Better AI Products

Insight #3: Every product team has its own way of building. ome people like to use application frameworks like LangChain, while others want to build their own LLM chains. Some like proxy solutions, while others don’t want a third party between their GenAI product and API model provider. Some want to manage prompts in code, while others prefer to manage them in a UI.

S

Autoblocks is an end-to-end GenAI product platform that allows product teams to quickly and collaboratively iterate on their GenAI products.

After talking to hundreds of companies building with GenAI, Autoblocks was born out of three unique insights:

Autoblocks is highly adaptive and conforms to your preferences. We obsess over removing friction from the iteration process, instead of adding to it.

Autoblocks 2.0: The GenAI Product Platform

Insight #1: Consolidating tooling increases product velocity.

Using disparate tools for different parts of the AI product development cycle seems fine in theory. However, this piecemeal approach leads to a major undesirable consequence: it slows down the iteration process!

A unified platform like Autoblocks helps you avoid the dread of clicking into different tools as you try to understand, improve, and differentiate your AI products.

Ready to leverage the best LLMOps solution?

Insight #2: Iterating on GenAI products is a highly collaborative process.

et Started

G

Building world-class GenAI products requires several stakeholders, including product managers, developers, domain experts, and business leaders.

The Autoblocks platform enables collaboration among all these stakeholders, leading to better product improvements, without compromising velocity.

Autoblocks

13

Autoblocks

14