karloscodes

Logs are enough

· 3 min read

AI-aided. I wrote the first version. AI helped me polish the writing.

In 2021, Coinbase paid Datadog around $65 million to watch its infrastructure. The number surfaced on Datadog’s own earnings call, and the industry’s reaction was telling. Not “how absurd,” but “how do we get a customer like that.”

That’s the observability business. It sells three pillars, metrics, traces, and logs, each with its own pricing meter. And somewhere along the way, the three pillars stopped being a product tier and became a definition of professionalism. Teams of four now feel negligent without distributed tracing.

Look at what each pillar is for. Traces follow a request across dozens of services, because no human can. Metrics compress logs that have become too voluminous to read. Dashboards give forty engineers one shared picture of a system none of them fully knows. All three exist for bigness, tools for understanding systems that outgrew understanding.

Your system didn’t. In a monolith on one box, the trace is a stack trace with a request ID on it. The latency percentiles are sitting in your request logs’ duration field, one query away. Error-rate alerts derive from the same lines. The shared mental model the dashboard sells? You have it, in your head, because the system fits there. You’d be buying X-ray glasses for a body you can see.

The title oversells by two small items, so let’s be exact. Dead servers don’t log, so you need one external ping that notices you’re gone. And a full disk kills logging first, so you need one disk alert, plus logrotate, which was capping disks long before anyone sold observability. Keep a week or two of history and compress the rest away. Almost every question you’ll ask your logs is about the last few days, and logs you no longer have can’t leak.

Logs, a ping, and a disk alert. What you’re skipping is the metric pipeline, the trace collector, and the per-host pricing, and none of them was protecting you.

I built LogNorth on this. It stores log events, groups the errors, and emails you when a new error shows up or an old one comes back. It keeps 90 days by default. It answers on the OpenTelemetry endpoints for traces and metrics, because the SDKs expect them, and throws away what arrives there. The logs are the product.

Set up the ping and the disk alert first. They are the only two things in the catalog your box can’t tell you about itself, because when either one fires, the box has already stopped telling you anything.

New posts in your inbox. Unsubscribe anytime.

Or grab the RSS feed.