#011 – Kubernetes for Humans Podcast with Charity Majors (Honeycomb.io)

Itiel Shwartz: Hello everyone, and welcome back to another episode or special episode of the Kubernetes for Humans podcast. Today with me on the show, live from KubeCon, we have Charity Majors. Charity, would you like to introduce yourself?

Charity Majors: My name is Charity, and I am the co-founder and CTO of Honeycomb.io. I’m here at KubeCon.

Itiel Shwartz: How long have you been coming to KubeCon?

Charity Majors: This is my second time.

Itiel Shwartz: What happened at the previous KubeCon? Was it relevant?

Charity Majors: Honestly, I mostly go to conferences when they invite me to talk. 

Itiel Shwartz: We just had a panel together, right?

Charity Majors: Yeah, you can probably catch it on YouTube where the CNCF posts the panel discussions.

Itiel Shwartz: Maybe tell us a bit about your history. I know you’re a well-known figure in the industry, but for anyone just joining us, could you share a bit about your background and what led you to software and eventually founding your own company in a very crowded and challenging space like observability?

Charity Majors: Sure. I don’t have a CS degree or any formal credentials. I was a music major dropout. 

Itiel Shwartz: What did you play?

Charity Majors: Classical piano. But I’ve been running computer systems since college. My first job was when I was 17, and I’ve been an Ops engineer for many years. Most recently, I was at Parse, a mobile backend as a service. We got acquired by Facebook, and the process of trying to make Parse reliable was what led Christine Yen, my co-founder, and me to start Honeycomb. We were building a next-generation platform before the tools really existed, and it was impossible to live without those tools anymore.

Itiel Shwartz: Parse was huge back then, right? For those who don’t know, Parse was like Heroku for mobile apps, where we would run your app, and you’d just have an API SDK, and we’d make it work. There were over a million mobile apps running on Parse by the time I left.

Charity Majors: Yes, it was a great product. I still hold a grudge against Zuckerberg because Facebook killed it. People wanted to buy it off them, but they didn’t think it was worth the lawyer fees or something. It made me really angry.

Itiel Shwartz: It was a great product. I used Parse during my degree for a project, and it was so neat.

Charity Majors: Yes, it really was.

Itiel Shwartz: Let’s talk about what you’re currently doing. 

Charity Majors: Honeycomb was founded almost eight years ago, which is insane. Back then, everything was about monitoring. Nobody was using the term observability, and we were the ones who kind of originated it. We were trying to articulate how what we were doing was different. We knew it wasn’t a monitoring product; it wasn’t based on metrics or just logs. It was about making a product that developers could use to interrogate their code in real-time in production. Observability, as a term, comes from control theory, which is about understanding what’s happening inside your system by observing the outputs. If you accept that definition, certain things follow: you need high cardinality support, high dimensionality support, traceability, and an exploratory, open-ended approach. Metrics, on the other hand, are just numbers with some tags; you’ve discarded all the context. With Honeycomb, you can slice and dice the data in real-time, identifying exactly what’s causing issues like a spike in errors, down to specifics like Android devices using a particular build in a specific region. Debugging often comes down to understanding what’s different between the things you care about and the things you don’t.

Itiel Shwartz: That makes a lot of sense, especially considering the noise we deal with in modern observability. There are so many metrics, and most of them aren’t relevant until they are.

Charity Majors: Exactly. We started, and for the first three or four years, our definition of observability gained steam. Then everyone jumped on the bandwagon, saying they did observability too. Now, everyone doing any kind of telemetry claims to do observability, which dilutes the meaning of the word. But it’s better than nobody caring about it at all.

Itiel Shwartz: I can relate. We also have a company in this space, but we’re not a classical observability tool. We’re not APM. How do you differentiate yourself in such a crowded market where everyone uses the same terminology?

Charity Majors: The best way to differentiate is by getting developers to try it. Once they do, they realize it’s different. Many people get value from what I’d call “observability 1.0” with metrics and logs, and they’re better off than before. But they could be an order of magnitude better if they collected data in a way that provides a single source of truth, from which you can derive metrics, logs, and traces. You can’t go in the other direction. Honestly, we get most of our customers from engineers who have used Honeycomb, change jobs, and then bring us along because they know there’s nothing else like it.

Itiel Shwartz: Developers can be hard to sell to because they don’t always see the need for a new tool, even if it’s an order of magnitude better.

Charity Majors: Exactly. No one wants to learn a new tool unless it’s significantly better than what they had before.

Itiel Shwartz: Your pitch is very rational, more so than many other tools. On a different note, I read one of your posts about never building your own database. Did you write your own database?

Charity Majors: Yes, we did. And it was kind of a tongue-in-cheek statement. I’ve spent my entire career telling people not to write a database unless they absolutely have to. We didn’t set out to write a database, but we did. If Snowflake had existed at the time, we might have built on top of it, but that would have locked us into certain choices that I think would have resulted in an inferior product.

Itiel Shwartz: So, are you happy with your own database?

Charity Majors: We’re constantly miserable with it, but from a good place. It was a tragic necessity.

Itiel Shwartz: Writing a database seems like opening a whole new company. Even dedicated database companies often struggle.

Charity Majors: Yes, but it needed to be done.

Itiel Shwartz: Let’s talk about the future. You mentioned that in three years, things are only going to get worse with more metrics, more complexity, more services, and more systems. Beyond observability, what do you think will happen in the industry?

Charity Majors: This is a great time to be alive in computing because we’re moving farther up the stack, allowing engineers to do more powerful things. But the real question is how engineers who grow up in a post-ChatGPT world will develop compared to those of us who came before it. Writing software has become easier, but running and owning software is still hard. My main concern is how anyone will become a good senior engineer without putting in the time to learn the hard way. But answers will emerge.

Itiel Shwartz: I also wonder about the impact of AI on software development. ChatGPT is great for people who already know how to code, but it’s concerning for those who are just starting.

Charity Majors: Yes, and I hope that developing with LLMs will push developers into a modern era of observability and testing in production because you can’t rely solely on unit tests when using LLMs.

Itiel Shwartz: That’s an optimistic view. Any final thoughts or things you’d like to promote?

Charity Majors: We’re here at KubeCon with a booth. If anyone wants a demo, feel free to reach out.

Itiel Shwartz: It was a pleasure having you on the show.

Charity Majors: Thanks for having me.

[Music]

Charity Majors is an ops engineer and accidental startup founder at honeycomb.io. Before this, she worked at Parse, Facebook, and Linden Lab on infrastructure and developer tools, and always seemed to wind up running the databases. She is the co-author of O’Reilly’s Database Reliability Engineering, and loves free speech, free software, and single malt scotch. 

Itiel Shwartz is CTO and co-founder of Komodor, a company building the next-gen Kubernetes management platform for Engineers.

Worked at eBay, Forter, and Rookout as the first developer.

Backend & Infra developer turned ‘DevOps’, an avid public speaker who loves talking about infrastructure, Kubernetes, Python observability, and the evolution of R&D culture.  He is also the host of the Kubernetes for Humans Podcast. 

Please note: This transcript was generated using automatic transcription software. While we strive for accuracy, there may be slight discrepancies between the text and the audio. For the most precise understanding, we recommend listening to the podcast episode