Scaling Analysis Without Scaling the Team

At Dagster, we’ve always believed in building the tools we need ourselves. That’s how Dagster began. We saw that existing data platforms weren’t powerful enough to support a company without a dedicated team of experts.

Internally, we’ve always run a lean data team. In fact, our internal Dagster platform, managing over a thousand assets and several thousand materializations each day, is maintained by a single dedicated data engineer (you can see our internal Dagster platform).

More recently, I joined as Dagster’s first dedicated data analyst. The team was thrilled to have someone who could dig deeper into our data platform. But we quickly realized that Dagster alone didn’t give me the same ability to scale my analytical workflows.

The Problem

As I settled into the role, it quickly became clear just how much of a backlog was waiting. Our data platform had scaled impressively in terms of infrastructure, but that didn’t necessarily make human analysis any easier.

At the same time, something remarkable was happening in AI. Large language models weren’t just getting smarter; their ability to process massive amounts of context meant they could finally hold the entire shape of a data platform in their “mind” at once.

For the first time, it seemed possible to build an assistant that truly understood our data environment, rather than one that simply guessed at SQL snippets.

So we started experimenting. What if analysis could scale the same way Dagster scales pipelines? What if I could create a context-aware copilot for data—one that everyone at the company could use? That prototype became Compass.

How We Use Compass

As Compass’s first user, I’ve seen firsthand how the process evolved. In the beginning, most of its answers needed to be double-checked manually. That experience convinced us to focus more on building around a centralized context store.

Instead of having Compass infer the important parts of our data stack, I could now curate everything myself, the tables that should be used and the business rules that govern them.

This kind of institutional knowledge is critical, yet it rarely fits neatly into traditional database systems. It’s where analysts bring the most value. With Compass, though, I was able to design workflows that felt like an extension of my own intuition.

As Compass matured, it began supporting more of my workflows, and the number of use cases it could handle continued to grow:

Conversations with the data

The most common scenario I see now is someone doing exploratory analysis on their own. This might start with a single question or evolve into a long back-and-forth as users test and refine a hypothesis.

It’s not uncommon to browse the Compass channel and find conversation threads that are thirty messages deep, a conversation that might once have taken up my entire day.

Multi-User Conversations

One major advantage of chat-based data tools like Compass, compared to traditional dashboards, is how naturally they support collaborative, data-driven discussions. A user can start a conversation with Compass and, once they uncover something interesting, bring in other team members.

That might mean pulling me in to double-check a SQL query, or inviting a sales manager to talk through a trend.

Having these conversations within the same tool everyone already uses makes collaboration far smoother than asking people to switch between platforms or deal with extra logins.

Expertise and Context

Every Compass conversation is backed by the centralized context store, which means every question is grounded in the company’s institutional knowledge. Over time, that knowledge base reinforces itself.

For example, a user might add their own insight such as noting that “there should be a custom Salesforce field that maps to customer service tier.” Compass routes that contribution back to me so I can review and approve it before it becomes part of the shared context store.

Similarly, if Compass encounters a question it can’t answer, I’m automatically notified. That ensures I’m involved when necessary but not bogged down by questions the platform can already handle on its own.

Prototyping and Onboarding

Once Compass became the de facto expert on our company’s data, nearly every data-related project started with it. Many Dagster team members now use Compass to write and refine queries before using them elsewhere. Even new dashboards and features typically begin life as Compass-powered prototypes.

Scheduling and Integration

Because Compass is built on top of Slack’s API, it can take advantage of everything Slack offers for normal conversations. You can schedule reports, request recurring analyses, or even tie insights directly to upcoming meetings and events—all without leaving the chat.

Lessons Learned

Our goal in designing a data copilot was the same as our goal with Dagster itself. We’ve always understood the importance of data practitioners and we never wanted to remove the need for an analyst. Instead, we wanted to scale their workflows.

If anything, I now feel more connected to the data than I did before Compass. Back then, I was constantly jumping between requests, trying to keep up. Now, everything is centralized. I have standardized workflows that ensure people get accurate answers and I can spend more time on meaningful analysis.

The more we’ve built Compass, the more we’ve relied on it. Along the way, a few lessons have stood out:

• Scale without scaling the team. Compass allows me to focus on higher-order work instead of drowning in one-off requests.
• Conversations beat dashboards. The fastest path to insight often isn’t another chart, it’s a question and a thoughtful back-and-forth.
• Organizational memory matters. Compass remembers the reasoning behind analyses, so we’re never starting from scratch.
• Humans and AI are stronger together. Compass provides technical leverage, and I bring nuance and understanding. Neither is enough on its own but together, they’re transformative.

Why it Matters

Traditionally, growing data needs have meant growing the data team. Compass broke that equation. It lets us scale our ability to answer questions and generate insights without increasing headcount at the same pace.

Compass began as our internal data copilot, built to solve our own bottlenecks. Today, it’s becoming much more than that. As we bring other organizations online with Compass, we’re seeing how the right combination of people, platforms, and AI can transform data teams.