karloscodes

Logs are enough

Wed, 08 Jul 2026 00:00:00 GMT

In 2021, Coinbase paid Datadog around $65 million. Not for infrastructure. For watching it. The number surfaced on Datadog's own earnings call, and the industry's reaction was telling: not "how absurd," but "how do we get a customer like that."

That's the observability business. It sells three pillars, metrics, traces, and logs, each with its own pricing meter. And somewhere along the way, the three pillars stopped being a product tier and became a definition of professionalism. Teams of four now feel negligent without distributed tracing.

Look at what each pillar is actually for. Traces follow a request across dozens of services, because no human can. Metrics compress logs that have become too voluminous to read. Dashboards give forty engineers one shared picture of a system none of them fully knows. All three exist for bigness: tools for understanding systems that outgrew understanding.

Your system didn't. In a monolith on one box, the trace is a stack trace with a request ID on it. The latency percentiles are sitting in your request logs' duration field, one query away. Error-rate alerts derive from the same lines. The shared mental model the dashboard sells? You have it, in your head, because the system fits there. You'd be buying X-ray glasses for a body you can see.

The title oversells by two small items, so let's be exact. Dead servers don't log: you need one external ping that notices you're gone. And a full disk kills logging first: you need one disk alert, plus logrotate, which has been capping disks since before observability was a word. Keep a week or two of history and compress the rest away. Almost every question you'll ask your logs is about the last few days, and logs you no longer have are logs nobody can leak.

Logs, a ping, and a disk alert: the complete tower at your scale, and notice there's still not a pillar in it. What you're deleting is the industry, the metric pipelines and trace collectors and per-host pricing, not the last two percent of safety.

You don't need three pillars to hold up one box.

Five nines for customers who sleep

Wed, 01 Jul 2026 00:00:00 GMT

99.9% uptime allows 43 minutes of downtime a month. 99.99% allows four. The entire argument between those two numbers, the architecture reviews, the multi-region diagrams, the on-call rotations, is a fight over 39 minutes.

Now price the 39 minutes. The industry's own rule of thumb says each extra nine costs roughly ten times more: redundant everything, failover that needs rehearsing, a second region you hope stays in sync, a pager that owns somebody's weekend. The most expensive minutes in engineering, bought monthly, forever.

Then ask who would have noticed them. If you sell B2B, your customers use the product during their working hours: call it 50 of the week's 168. They are asleep, commuting, or living their lives for the other 70%. Downtime at 3am on a Sunday costs you nothing but the uptime badge. Most of the 39 minutes you're insuring against would land in hours when nobody's there.

And there's a ceiling no SLA budget can break: availability multiplies. Your app times your cloud provider times your DNS times your payment processor. You cannot out-nine your dependencies. When a big cloud region has a bad afternoon, half the internet goes down together, and customers blame the internet, not you. Inside the herd, outages are forgiven. The nine you bought above your dependencies was never reachable.

What reliability looks like at your scale is different and mostly free. An external ping that tells you you're down before a customer does. A backup you've actually restored, with a stopwatch running. A status page that tells the truth, and the email that says what broke and when it will be fixed. Customers don't remember the 20 minutes you were down. They remember whether you told them.

Some businesses earn the nines: payments, hospitals, anything with real traffic at 3am. The signal is in your access logs, not in your ambitions. Open them and look at what happens at night.

Buy the nines your customers are awake for.

Names yes, geometry no

Wed, 24 Jun 2026 00:00:00 GMT

You learned hexagonal architecture at a job with hundreds of engineers, and it made sense there: layers as treaties between teams, interfaces as borders nobody crosses without paperwork. Then you started something small. A side project, a tiny company. And you brought the hexagon with you, because that's what professional code looks like.

The skills transferred. The context didn't.

Keep the half that costs zero lines of code: the names. Domain-Driven Design got this part right, and it's the thin part of a thick book. Call things what the business calls them. If the accountant says invoice, the class is Invoice, not BillingDocumentEntity. Draw boundaries as vocabulary: this module talks about shipments, that one about payments, and they don't borrow each other's words. That's thinking, not architecture, and it's worth the same at any headcount. The rest of the DDD catalog, the aggregates and repositories and factories, is geometry by another name. Leave it in the book.

Because the geometry's promise is seductive. Someday you'll swap Postgres. Someday the big refactor will glide along those segregated interfaces. Someday never comes. At the old job, that was fine: the premium was split across a hundred engineers, and insurance you never claim is what bigness costs. At your scale, the whole premium lands on five of you, or just on you. The DTOs, the mappers, the interfaces, the wiring: a handful of hands maintains all of it, waiting for a someday that was never yours to begin with. Enterprise insurance on a small house.

The hexagon was a treaty between a hundred hands in one codebase. There's no one to sign treaties with anymore. Code that fits in one head needs no protection from other heads. Smallness already does the hexagon's job. Small requires less, and the requiring less is the whole opportunity.

And staying small is a decision with champions. 37signals has run Basecamp and HEY for millions of users with a few dozen programmers, on one monolith, by choice, for two decades. They didn't fail to grow. They refused the bigness so they'd never need the treaties.

The hexagon was sold to big enterprises, in books written for engineering at scale. Nobody wrote the book for your scale. So here's the short version: names yes, geometry no. Small isn't a phase to apologize for. It's a competitive advantage, and skipping the hexagon is how you cash it.

Keep the language. Lose the geometry.

Aggregate at write time

Wed, 17 Jun 2026 00:00:00 GMT

ClickHouse is short for Clickstream Data Warehouse. It was built at Yandex to power Metrica, their web analytics platform, which grew from 200 million events a day to more than 25 billion by 2016. They open sourced it, it's a marvel, and it has become the reflex: need analytics, need reports, store the raw stream and query it later.

Here's the detail buried in their own history. Metrica ran for years on pre-built aggregates. Yandex moved to querying raw events because, at their scale, the combinations of reports people wanted exploded past what rollups could hold. ClickHouse exists because Yandex outgrew aggregation. The threshold is measured in billions of events a day.

At that scale, the warehouse makes sense. At your scale, most reports your product produces never need one. The move that replaces it: aggregate at write time. Keep one row per metric per hour, and when an event arrives, bump its counter instead of storing the event.

INSERT INTO hourly_views (hour, views)
VALUES (date_trunc('hour', now()), 1)
ON CONFLICT (hour) DO UPDATE SET views = views + 1;

That's the whole trick. The report isn't computed when someone opens it. It was built all along, one increment per event, by the same write that recorded it. The raw event is gone, and the answer is already sitting on disk.

That skips the part the pitch leans on hardest. The warehouse doesn't just store your data, it promises a generic engine: pipe every raw event in, often through something like Kafka, and query it any way you want later. That power is the real product, and it's the deeper trap. Generic querying is for when you don't know what you'll ask. You do. The reports your product has to produce, a usage summary for you or a stats page for your customers, were designed up front and rarely change. You're not running a research lab, you're rendering the same reports on a schedule. Renting a generic engine to answer fixed questions is paying twice, the pipeline that feeds it and the engine that serves it, for a number a pre-computed total hands you with one SELECT.

So compute the answer once, because you can. You know your questions.

Now count the rows. Aggregated by day, a year of any report is 365 rows. By hour, 8,760. Track ten reports at hourly grain and a full year is under 90,000 tiny rows, fewer than one busy afternoon of raw events.

Look at the size of it. A healthy small product does 10,000 tracked events a day. Stored raw, that's 3.65 million rows a year, most of a gigabyte, growing forever, re-scanned every time a report runs. Aggregated hourly, it's 8,760 rows. About a megabyte. A thousand times less storage. Four hundred times fewer rows. The same reports, instant. The work happens once, when the event lands, and never again.

I run my analytics this way: hourly rollups in SQLite. Six months of production traffic came to 14 megabytes. Every chart is a read of pre-computed rows, instant on a $5 VPS.

There's a second prize. Raw events stored forever are pure liability: a bill with a breach radius. Aggregates are small by construction and anonymous by arithmetic. The privacy policy writes itself.

You are not Yandex. Be glad. They need the generic engine: their questions are unknowable and their data has nine zeros. Yours fit on a phone. So skip the machine they're stuck running and put the saved time and money into the product.

That's the decision, and it isn't close. Aggregated reports are a competitive advantage, and you get it for being small and refusing to go generic. Simple wins.

Store answers, not questions.

The database is your queue

Thu, 11 Jun 2026 00:00:00 GMT

Kafka was built at LinkedIn to move activity streams. By the time they wrote it up, it was carrying 7 trillion messages a day across 4,000 brokers. Serious engineering for a serious problem. Their problem.

Here's the reflex it created. You need background jobs: send the welcome email, retry the webhook, build the nightly report. The tutorial says install Redis, add a queueing library, and if you're "serious," RabbitMQ or Kafka. Three systems running before your first job does.

It's a trap at your scale. And it's not your fault: the books were written for Google scale, and nobody told you they weren't talking to you.

Now count your jobs. Not the jobs you might have. The ones you have. A normal product runs a few thousand a day, with rush hours that might touch a few per second. You are about to deploy LinkedIn's artillery against a queue that fits on a napkin.

The database you already run handles this without noticing. A jobs table and SELECT ... FOR UPDATE SKIP LOCKED is the whole trick, and every serious framework has it packaged. Rails 8 went further: the default queue is now the database. Not for toy apps. HEY, 37signals' email service, pushes about 20 million jobs a day through it. That's 230 jobs a second, on the technology the tutorials told you couldn't handle your fifty.

Do the math. Four thousand jobs a day is one job every twenty seconds. The database queue, proven at 230 jobs a second, clears your whole day in 17 seconds. LinkedIn's Kafka would clear your year in two hundredths of a second. One of these is a reference class. The other is a costume.

And the database queue gives you something the broker never will: one transaction. Save the order and enqueue its email in the same commit, and there is no crack between systems for a job to fall into. With a separate broker there is always a crack, and the industry's fix for it is the outbox pattern, which is, read it slowly, a jobs table in your database. The cure for the second system is the first one.

Your jobs become rows. You inspect the queue with a SELECT. You count failures with a WHERE. Your backup backs it up. Nothing new to monitor. One less system that can break.

And half your queue doesn't even need to be a queue. The welcome email can leave a minute late. The nightly report was never urgent. This is where cron stops being the boring old tool and becomes the discovery of your architecture: a fifty-year-old scheduler, zero dependencies, that has outlived every framework it ever served. One crontab line sweeping the table every minute is real-time enough for most of what a business does, and it breaks less often than a worker fleet. Batch was never the compromise. It was the right size all along.

You're small. At your scale you get to do less. Take that: it's a competitive advantage. Simple wins, and when you grow, you adapt.

The queue you're shopping for is already running. It's called your database.

You're allowed to just use it.

Do you know what one server can do?

Wed, 10 Jun 2026 12:00:00 GMT

In 2024, Gunnar Morling ran a challenge: aggregate one billion temperature readings from a text file, as fast as you can. The winning solution took 1.5 seconds. Not on a cluster. On 8 cores of a single rented server.

One billion rows. 1.5 seconds. One machine.

I keep meeting engineers who can sketch Google's architecture from memory but can't tell me what one modern server does. We memorized their numbers and never measured our own. So: do you know what one server can do?

Here's the box. A Hetzner AX42 costs about €50 a month. Eight cores, 64 GB of ECC RAM, NVMe storage. The challenge machine was bigger, but results were capped to eight of its cores, five years older than these. The whole machine costs less per month than one hour of the engineer architecting around it.

64 GB of RAM means the entire customer database of almost every B2B company on earth fits in memory, with room left for ten copies.

Same story on disk. A million-row table, the kind that gets called big in planning meetings, is about 100 MB with its indexes. Twenty-five phone photos. The billion-row file from Morling's challenge was around 13 GB. The box holds forty of them on half a terabyte of NVMe. You are not running out of disk. You have never been close.

NVMe does random reads on the order of a million IOPS. The spinning disk that your architecture patterns were designed around did about two hundred. Two hundred. Every storage pattern you learned was a workaround for that number.

Throughput is arithmetic. A sustained 1,000 requests per second is 86 million requests a day. A plain Go service does tens of thousands of requests per second on eight cores; Rails, nobody's idea of fast, does thousands. SQLite takes tens of thousands of inserts per second batched in WAL mode, on the database engine everyone tells you is a toy.

You will probably never see 86 million requests in a day. Stack Overflow served one of the busiest sites on the internet from about nine web servers. That was a decade ago, on weaker hardware than the €50 box.

Now put a normal business next to those numbers. A B2B product with a thousand daily active users peaks at a few dozen requests per second. Its database, after years of trading, holds a few gigabytes. Its background work is thousands of jobs a day, not millions. A normal business fits in a corner of this machine, and the machine is bored.

So why do we shard databases that fit in RAM and run queue clusters for jobs one core could clear before lunch?

Because we're running someone else's playbook. The books and talks that taught us architecture were written by Googlers and Amazoners, solving Google and Amazon problems. Sharding made sense: their data didn't fit on one machine. Yours does. The advice isn't wrong. It's wrong at your scale.

That's the whole ask: measure. Your numbers, not theirs. Don't engineer at their scale. Engineer at yours.

Before you add an Elasticsearch cluster

Sun, 31 May 2026 12:00:00 GMT

I've reached for Elasticsearch more than once without really thinking about it. You need search, ES is what you reach for. It's the reflex.

Here's what the reflex skips, and it's the thing that matters most when you're small. Elasticsearch isn't a feature you add. It's a system you run. Another database sitting next to the one you already have, kept in sync, one more thing to monitor, back up, and get paged for. At 3am, alone, because you are the whole ops team.

That cost has nothing to do with speed. For a solo founder or a small team, the question was never "is Elasticsearch faster." It's "do I want to operate a search cluster to find out." Usually the honest answer is no.

Because Postgres already searches, and for the size most of us actually run, it isn't a compromise.

ALTER TABLE docs ADD COLUMN search tsvector
  GENERATED ALWAYS AS (
    to_tsvector('english', coalesce(title,'') || ' ' || coalesce(body,''))
  ) STORED;

CREATE INDEX docs_search_idx ON docs USING GIN (search);

SELECT id, ts_rank(search, q) AS rank
FROM docs, websearch_to_tsquery('english', $1) q
WHERE search @@ q
ORDER BY rank DESC LIMIT 20;

No new service. No sync. The index updates in the same transaction as the write, so stale search results aren't a bug you can have. It's a column and an index on the database you already run.

Does skipping the cluster cost you speed? I measured it, and put the whole benchmark online. A million real product reviews, Postgres and Elasticsearch side by side. This isn't a lab-grade benchmark and doesn't try to be. It's numbers from a Mac Mini M4 running Docker, meant to show roughly how search slows down as the data grows, not to give you exact figures:

rows	typical	worst case
10,000	0.1 ms	1 ms
100,000	0.8 ms	12 ms
1,000,000	15 ms	143 ms

Read the top two rows, because that's the size most apps actually are. At a hundred thousand documents Postgres answers a typical query in under a millisecond. No human can tell that apart from a dedicated cluster. You would run a second system to save time nobody can notice.

The bottom row is real, though. At a million rows the worst queries cross 100 ms. But what makes a query slow isn't the size of the table, it's how many rows it matches. Your everyday searches stay instant even at a million rows. Only broad common-term queries over a large dataset cost you.

In practice that ties back to dataset size, and other benchmarks put the working range clearly. Xata recommends starting with Postgres-only search for tables under 100,000 rows. Neon, testing at ten million, found Postgres's built-in search had gotten too slow by then, while a stronger search index kept up. So the range is roughly this: the built-in search is comfortable into the hundreds of thousands, and once you're into the millions it's worth adding something stronger.

You have two options inside Postgres, and you reach for them in order. First, the built-in full-text search, the code above. Nothing to install, it ships with every Postgres, and for a small-to-medium dataset it's all you need. The costs are mild: it adds some storage, and its ranking is basic, so it won't give you finely tuned relevance. Second, pg_search, an extension that adds the same kind of ranking Elasticsearch uses, directly to Postgres. In Neon's test it matched or beat Elasticsearch and handled a single database up to hundreds of millions of rows. The catch is it's a third-party extension, so you need a Postgres you can install it on.

So here's my opinion, plainly. Solo and small-to-mid teams over-engineer search constantly, and it's almost never about load. It's fear, or habit, or wanting the resume line. We reach for the cluster before we have the problem the cluster solves.

Match the tool to your scale. Start with the search that's built into every Postgres. Into the millions, pg_search gives you stronger ranking without leaving the database. Elasticsearch comes last, for when you outgrow a single machine: billions of rows, search load you need off your main database, or scale you have to spread across several servers. Those are real reasons, and you'll know when they're yours.

Until then you're not running Google. You're running a search box over a dataset that fits in your database. Add one index. Don't run big-company infrastructure until you have big-company problems.

Debug production with AI agents

Thu, 05 Mar 2026 12:00:00 GMT

Something breaks. You open the logs. You scroll. You search. You find the error. You context switch to the code. You trace it. You fix it. You deploy. You check if it worked.

That's an hour. Maybe two. More if you're unlucky.

Now imagine: you tell an AI to check your logs. It finds the error, knows your codebase, writes the fix. You review it or not. Deploy, verify, done.

Fifteen minutes. Maybe less - the AI can do it all.

Same outcome. A fraction of the time and effort.

The tight loop

Error in logs → AI reads it → AI writes fix → AI deploys it → AI verifies → Done

The AI has context. It knows your codebase. It sees what broke and why. It can do it all.

Why this matters if you're solo

Some bugs are quick. Typo, missing null check, obvious fix.

Others are nasty. You're tracing through five files, reading stack traces, trying to reproduce it locally, losing an afternoon.

The AI doesn't get tired. It reads the logs, traces the code, and finds the issue. The nasty bugs that steal your afternoon? They become 15-minute problems.

What you actually need

For this to work, the AI needs access to your errors and logs. Not raw log files scattered across servers. Structured, searchable, linked, grouped. Something with an API the agent can query.

A tool that groups errors, links related logs, tracks occurrences, shows what happened before and after - that context helps you debug faster. Turns out it helps the AI too. The same structure that makes logs useful for humans makes them useful for agents.

The rest is straightforward:

An LLM with access to your codebase
A deployment pipeline the agent can trigger
A way to verify the fix worked

No fancy APM. No distributed tracing. Just your logs, an AI, and a deploy script.

What this enables

No scrolling. No context switching. No losing your afternoon.

This is what AI unlocks for solos with simple stacks. Your small codebase isn't a limitation - it's an advantage. The AI can understand it, navigate it, fix it.

One person. One codebase. One tight loop. LogNorth.

Small Data

Sun, 15 Feb 2026 12:00:00 GMT

I opened Fusionaly's database yesterday. Six months of production web analytics. Thousands of pageviews, sessions, referrers, all the dashboards.

14 megabytes.

That's absurdly small. I could email the whole thing.

Turns out that's all I need.

I'm building two more products the same way.

Formlander handles form submissions. Three months in production, same architecture. LogNorth captures errors and logs. Two weeks old, same pattern.

SQLite for all of them. One file per product. No clusters. No shards.

The pattern isn't the file size. The pattern is what I didn't build.

No event streaming — SQLite handles my write volume fine. No separate analytics database — I aggregate on write, queries stay fast. No replication — daily VPS snapshots are enough for a business that won't die if it's down for an hour.

The industry would call this unscalable. They're confusing my scale with Google's. Google processes billions of events per second. I process thousands per day. That's their rounding error. To me, it's a business.

The math is almost embarrassing.

A busy site does 50,000 pageviews per day. Ten years of that is 180 million events — somewhere between 50 and 80 gigabytes. With aggregation, retention, and compaction, smaller still.

Check your production database. How big is it actually? You might be surprised.

"But what if I outgrow it?"

You'll know. The database will tell you. The queries will slow down. The file will get big.

And when that day comes, you'll have the revenue to solve it. That's a good problem. Most businesses never get there.

What you shouldn't do is build for imaginary scale today. I wasted years doing that.

Your data is small.

So why are you building like it isn't?

That's not a limitation — it's your leverage. Small enough to self-host. Small enough to own completely.

Embrace it.

SQLite for Web Analytics: How I Made It Work

Fri, 23 Jan 2026 12:00:00 GMT

I like building things to see if they work. Sometimes that means ignoring the standard advice.

Web analytics usually means ClickHouse, or Kafka, or at least Postgres with TimescaleDB. Big tools for big data. But most sites don't have big data — they have thousands of visits, maybe tens of thousands. Single-tenant analytics isn't a scale problem.

So I wondered: could a plain relational database handle this? And if so, how simple could we make it?

Aggregation changes everything

Nobody needs millisecond-level granularity in their dashboard. Hourly buckets are fine. And once you aggregate by hour, the numbers get small fast: 24 rows per day per dimension. A year of data fits in megabytes, not gigabytes.

At 100K visits/day, you're looking at roughly 26 GB/year. Scale to 500K and it's 130 GB. Even a million daily visits only hits 260 GB/year. This isn't big data. This is just... data.

So yes, any relational database handles this easily. But that got me thinking: if the data is this small, do we even need a database server?

What if we used SQLite?

SQLite has one limitation that scares people off: concurrent writes. Only one writer at a time. Under heavy load, writes queue up and block each other.

But analytics doesn't need instant consistency. Nobody's refreshing dashboards 50 times per second. What matters is never losing an event. So instead of fighting SQLite's limitation, I leaned into it — with coordination across layers.

The browser buffers

The SDK batches events every 200ms and retries three times with exponential backoff. If the network drops, it caches up to 100 events in localStorage. When you close the tab, sendBeacon() fires off whatever's left.

The SDK doesn't fire and forget. It listens for the server's response.

The server pushes back

Sometimes the server says "slow down." When it's overwhelmed, it returns 503 with a Retry-After header. The SDK respects that — waits, then tries again. Events don't drop, they just arrive later.

This handshake keeps the pace manageable. The browser and server agree on how fast events flow.

The database absorbs writes gradually

Events land in a staging table — dead simple schema, optimized for fast INSERTs. If SQLite is busy, the write retries up to 10 times with backoff. A background job drains the staging table in batches and rolls everything into hourly aggregates.

The dashboard reads from those aggregates. Under high load it might lag a few minutes — but you're not a stock trader. Who cares?

One process. One file. No Redis queue, no Kafka buffer, no worker cluster. Coordination lives in code, not infrastructure.

This isn't for everyone

Could SQLite keep up at a million visits/day? Maybe. But if you're at that scale, you're not who I built this for. You have a team. You can afford managed Postgres. Go use Plausible or roll your own.

This is for the rest of us. Most sites see 10K-50K visits/day — that's the reality for most projects, most businesses, most side hustles.

It scales further than you'd think

It's not 2010 anymore. SQLite on NVMe delivers sub-millisecond writes. The bottleneck everyone worried about a decade ago just isn't there.

Start with a 512MB server on Hetzner. When you outgrow it, bump to 1GB. Then 2GB. Then 32GB. Vertical scaling is underrated — no network hops, no distributed coordination, no consistency headaches between nodes.

Backups? Litestream streams every change to S3 in near-realtime. Point-in-time recovery, disaster recovery — all handled. Or just snapshot the file and ship it somewhere safe.

Radical simplicity, pragmatic results. One process. One file. If you ever outgrow it, you'll know exactly when.

Want to try it? Fusionaly is open source on GitHub.

Stop Self-Hosting Interplanetary Rockets

Sun, 07 Sep 2025 11:52:28 GMT

Most self-hosted apps weren’t really built for self-hosting.
They’re just the SaaS product, open-sourced and thrown over the wall.

Which means you inherit everything that made sense for SaaS at scale:

Multi-tenant databases built for millions of users
Layers of services (Redis, Kafka, Elastic, ClickHouse, background workers, queues…)
Ops complexity designed for an SRE team, not one person

It’s like buying an interplanetary rocket when all you needed was a car to get across town.

Sure, it’s impressive. But you don’t need that kind of firepower.
You don’t need to launch payloads to Mars.
You just want to run your company’s data safely, simply, and on your own terms.

The Overkill Problem

When you run these “self-hosted” SaaS clones, you’re taking on:

Infrastructure designed for global traffic you’ll never see
Databases meant for multi-tenant complexity you’ll never need
A maintenance burden you’ll never want

Self-hosting business tools should give you more control, not more headaches.
But too often, it ends up feeling like running AWS from your living room.

What Self-Hosting Should Mean

Self-hosting should be about:

Control – you own your data
Simplicity – you can run it on one server
Sustainability – you can maintain it yourself

You don’t need an interplanetary rocket.
You need a reliable car: easy to drive, easy to keep, gets the job done.

A Different Approach

That's why I built Fusionaly this way:

One server
One payment, own it forever
Bring your own AI key (no hidden usage costs)
Evaluation made simple: one-command devbox, no hoops

Designed for self-hosting from the start, not retrofitted from SaaS.

Closing

Self-hosting shouldn’t feel like operating NASA mission control.
It should feel like running your own tool, on your own terms.

Stop self-hosting rockets.
Start self-hosting tools you can actually run.

Learn more about Fusionaly

Life is a counterbalance act

Sun, 17 Aug 2025 10:38:08 GMT

People say life is about balance. I used to say it too.

Turns out, it’s not.

Perfect balance is boring. It’s static. Nothing happens there.

The good stuff happens at the edges. When you work hard. When you play hard. When you lean all the way in. That’s where growth happens. That’s where the magic is.

But stay there too long and you burn out. The edge takes everything out of you.

That’s why life isn’t balance, it’s counterbalance. You push one way, then pull back. You swing between intensity and rest, between yes and no.

It’s not about sitting in the middle. It’s about moving back and forth with intention. That’s the rhythm of a full life.

True wisdom is knowing when to push and when to pull back. When to burn bright and when to cool down.

Life isn’t balance. Life is counterbalance.

Don't forget to sharpen your axe

Sat, 17 May 2025 08:27:45 GMT

Every evening, after my day job, I carve out a few hours for my side projects as a part-time solopreneur, and my brain goes wild. A new feature to make my app pop. A slick refactor to clean up that messy code. A whole new tool that could be the one. It’s like a hundred doors open at once, each whispering, “Pick me.” I love it, the rush of ideas, the itch to tinker. But here’s the truth: it’s a double edged sword.

For years, I let that flood carry me. I’d code a bit here, sketch a new app there, chase every shiny thought. It felt creative, alive. But nothing shipped. I was running in circles, burning out, and hating myself for it. My side hustle, my escape, became a trap. Too many ideas, not enough focus.

The hardest lesson I’ve learned is prioritization. Not the sexy kind with color-coded to-do lists, but the brutal kind: picking one thing and ignoring the rest. I started asking, What’s the smallest move with the biggest impact? One feature that solves a real pain. One bug that’s blocking users. One hour of deep work. It’s not fun, it feels like strangling my curiosity. But it works. I’ve shipped projects I’m proud of, even if they don’t pay my bills yet, because I learned to say no to 99% of my brain’s noise.

I’m driven, alive, chasing something real. Still, I miss something when I don’t tinker and explore, a spark, a sense of play that keeps my soul lit. So I found a balance. Most nights, I’m ruthless: one task, one goal, no distractions. But I let myself play. Spend time with a new language, sketch a crazy app, or just break something for fun. It’s not productive, and that’s the point. Focus ships; tinkering keeps me human.

Life’s funny, isn’t it? The thing that makes you effective —prioritization— can dim your spark if you’re not careful. Don’t let it. Make time for something that brings you joy. You’ll ship more and stay alive inside. Just don’t forget to sharpen your axe from time to time.

Thriving in the AI era

Sat, 08 Feb 2025 09:30:26 GMT

My Vision as a Software Engineering in the AI Era

Core Beliefs:

As a techno-optimist, I believe that AI amplifies human potential rather than replacing it
The future belongs to those who learn to leverage AI while maintaining their agency
Engineering excellence will be redefined by judgment, not just coding skills
Our most valuable assets remain uniquely human: critical thinking, principles, and values
Be you, nobody can beat you at that, authenticity becomes an even stronger differentiator in an AI world

The Evolution of Skills:

Traditional role boundaries are dissolving, being "just" a backend engineer is limiting
AI tools give us "superpowers", the ability to confidently work across the entire stack
Deep understanding still matters:
- Fundamentals are non-negotiable
- Learning comes from books, reflection, and writing
- AI is a tool, not a replacement for knowledge
Focus shifts to higher-level capabilities:
- Strategic thinking
- System design
- Decision-making
- Technical judgment
- Clear communication

Industry Shifts:

Teams get smaller but more impactful:
- Fewer engineers, but more versatile
- Full-stack renaissance
- Blending of engineering, product, and design roles
Market dynamics intensify:
- More companies, smaller sizes
- Increased competition
- Network becomes crucial for opportunities

Keys to Success:

Master AI collaboration without becoming dependent
Build deep foundations in computer science and engineering principles
Develop strong problem-solving, decision-making and strategic thinking skills
Excel at communication:
- Articulate complex ideas simply
- Write clear, compelling documentation
- Bridge technical and non-technical perspectives
- Build influence through effective communication
Cultivate your authentic voice and perspective, authenticity is your competitive advantage
Build a robust professional network and brand
Never stop learning, but learn strategically

Writing this has been an exercise in optimistic futurism. As someone who believes in the positive potential of technology and the opportunities AI will create, I'm genuinely excited to see how these thoughts age. Future me, if you're reading this in 2030, I hope you're smiling at how far we've come. Time will tell!

Total Cost of Ownership, a software engineering perspective

Sat, 14 Dec 2024 13:14:08 GMT

Engineering is fundamentally about trade-offs. Every decision carries weight, implications, and most importantly, costs. When talking about costs in software engineering, we're not just dealing with financial expenses, we're considering time investment, cognitive overhead, operational complexity, and team resources. While software engineers excel at discussing technical merits, architectural patterns, and code quality, there's often a blind spot when it comes to understanding and considering the Total Cost of Ownership (TCO) of our solutions.

The developer cost blind spot

Software engineers are trained to value technical excellence: clean code, elegant architectures, and engineering purity. This mindset, while valuable, often creates a blind spot when evaluating solutions. We excel at discussing algorithmic complexity, scalability, and code maintainability, but frequently overlook or underestimate the total cost of ownership of our technical decisions.

This blind spot manifests in how we evaluate solutions. We might build a technically superior approach without considering its maintenance burden, or choose cutting-edge technology without factoring in the cost of keeping the team proficient with it. Cost optimization isn't about choosing the cheapest solution, it's about understanding the full spectrum of costs: development effort, maintenance burden, operational complexity, team expertise, the price of future changes and how they all compound in time.

The hidden costs in our solutions

What makes a solution good from an engineering perspective doesn't always align with what makes it sustainable and efficient in the long run. A beautifully designed microservice architecture might be technically elegant, but the operational complexity, infrastructure demands, and cognitive overhead of maintaining multiple services could turn it into an overwhelming burden for the team.

To showcase this concept more clearly, let's examine a simpler example that many developers face: building a personal blog, comparing building everything yourself versus hosting your posts on a blogging platform.

From a pure TCO perspective, both approaches require an initial investment, though they differ significantly in scope and time to market:

Initial Investment

Custom blog:

Research time: Comparing static site generators, hosting options, and deployment strategies
Learning curve: Understanding the chosen technology stack
Implementation:
- Basic: Theme selection/creation, content structure
- Advanced: SEO optimization, tagging system
- Infrastructure: Domain setup, deployment pipeline
- Features: RSS feeds, contact forms, email subscriptions
Initial deployment and testing

Managed platform:

Research time: Comparing available platforms
Learning curve: Platform-specific features
Setup:
- Basic: Theme selection, content structure
- Advanced: SEO settings, tagging system
- Infrastructure: Domain connection
- Features: Built-in RSS, forms, subscriptions

The cost different is already quite evident, especially in terms of time investment. The time to market is significantly better with the managed platform, it can have you publishing content within hours, while a custom solution might take weeks to reach the same functionality level.

Ongoing costs over time

The true cost difference becomes evident as time passes. A custom blog requires constant attention: security updates, dependency management, framework upgrades, and fixing broken features. Each year brings new challenges: major version migrations, growing technical debt, and possibly even complete rebuilds as technologies become obsolete.

Meanwhile, a managed platform quietly evolves, with improvements and new features appearing automatically. The time saved compounds, hours not spent on maintenance become hours invested in creating value.

The opportunity cost

The most significant hidden cost manifests in the opportunities foregone while managing infrastructure (Opportunity cost). Those hours spent troubleshooting deployment issues could have produced new articles. The mental energy devoted to maintaining dependencies could have gone into engaging with readers or growing your professional network. Time invested in upgrading frameworks could have been spent developing expertise in your core domain or building valuable industry relationships. These missed opportunities accumulate over time, often far exceeding the visible costs of either solution.

The TCO difference becomes substantial when accounting for these opportunity costs. While building a custom blog might serve additional purposes like learning specific technologies or having complete control over the platform, from a pure cost perspective, the managed solution often wins, not just in direct costs, but in freeing up time and mental energy for higher-value activities.

Building a cost-conscious engineering culture

Cost consciousness must become an integral part of the software engineering mindset, as poor cost assessment can lead to significant problems down the line. Solutions may become unsustainable as they scale, technical debt accumulates faster than expected, and valuable resources get drained from maintaining over-engineered systems. Most critically, high operational costs can limit business opportunities and make it difficult to pivot when needed.

When engineers factor in cost considerations from the start, they naturally gravitate toward more pragmatic solutions. This doesn't mean always choosing the cheapest option, rather, it's about understanding the long-term implications of technical decisions. A more expensive initial investment might lead to lower maintenance costs, while a seemingly cheap solution could become a resource drain over time.

Consider introducing a "TCO check" in your architectural decision records or design reviews. For each significant technical decision, include a section that answers:

What's the initial development cost?
What ongoing maintenance will be required?
What risks might we face in six months or a year?
What alternatives did we consider and what were their cost implications?

Make TCO reviews a regular part of your engineering practices and establish a clear ownership of it. Teams should be accountable for their cost decisions, just as they are for their service reliability. This simple practice helps teams develop a habit of considering the complete cost picture and make it part of their engineering workflow.

Conclusion

Cost isn't just a business concern, it's a fundamental engineering constraint that deserves the same attention as performance, scalability, and maintainability. The most elegant solution might not be the right one if its TCO makes it impractical or unsustainable. This might mean choosing a managed service over building a custom solution, using established technologies over cutting-edge ones, or accepting some technical compromises to meet business constraints. Of course, every engineering decision depends on context, sometimes building your own solution is the right strategic choice, but that's a story for another post.

By embracing cost consciousness in software engineering decisions, we build not just better systems, but more viable and sustainable ones. The next time you're evaluating a technical decision, remember to factor in the total cost of ownership, your future self and your company, will thank you.

New and trendy: Convenience at what cost?

Sat, 07 Dec 2024 12:18:59 GMT

The software industry is obsessed with the new and trendy. Every few months, a new tool, framework, or methodology emerges, promising to revolutionize how we build software. While innovation is essential and drives progress, the adoption of new and rapid changing technologies creates substantial hidden costs that we rarely discuss. The key isn't to avoid innovation, but to understand how these choices impact our organizations over time, impacts that often only become apparent years later, when the accumulated weight of our decisions begins to show.

Compounding effect of change rates

What's often overlooked in technology decisions is how the rate of change itself becomes a critical risk factor. Technologies that change rapidly create a compounding burden on organizations. This isn't just about keeping up with the latest versions, it's about the multiplicative effect of change across an entire technology stack.

This isn't an argument against innovation or new technologies, many truly revolutionary tools have transformed our industry for the better. Rather, it's about understanding the full cost of adopting technologies that change rapidly, especially in critical parts of our stack. The faster a technology evolves, the more resources we need to dedicate to keeping up with its ecosystem.

Consider a modern web application. If each layer of the stack, frontend framework, state management, API layer, database, deployment platform, changes significantly every two years, teams aren't dealing with linear change. They're facing an exponential increase in complexity and maintenance burden. A change in one layer, often forces changes in adjacent layers, creating a cascade of updates that can paralyze development teams.

Not to mention that a major version upgrade of any core technology typically implies hundreds of thousands of dollars in immediate refactoring costs or an equivalent amount in technical debt waiting to be paid.

The real cost of rapid changing technologies

The real cost lies in how these changes compound over time, in the opportunity cost of delayed features, in the impediments to move forward or in the cost of not adapting to changes.

The total costs manifest in multiple ways:

Direct costs

Time spent implementing migrations
Time spent creating release plans
Time spent in training and skill development
Time spent in testing and quality assurance

Hidden Costs

Lost productivity during learning curves
Opportunity cost of delayed features
Risk of introducing new bugs
The risk of increased operational complexity
The risks of not adapting to changes

The cost of inaction

While chasing every new trend carries significant costs, once you've committed to a technology, refusing to adapt to its evolution can be even more expensive. Each skipped update makes the next migration more difficult. What might have been a week-long update can become a year-long rewrite if delayed too long, talent pool starts shrinking, developers starts demotivating. Quite a challenge.

The abandonment risk

There's another hidden cost that haunts trendy technology choices: the risk of complete abandonment. When a new technology fails to gain long-term traction, companies who adopted it early face difficult decisions. Popular JavaScript frameworks have disappeared overnight, taking their ecosystems with them. Suddenly, you're not just maintaining your application, you're maintaining your foundations.

The power of Lindy technology

But there's hope. The Lindy Effect suggests that the future life expectancy of non-perishable things like technology is proportional to their current age. In other words, technologies that have been around for a long time are likely to stay around for longer. This isn't just theoretical, it's a powerful framework for evaluating technology choices.

Consider these examples:

SQL databases have outlived countless NoSQL alternatives that promised to replace them. While NoSQL has its place, organizations that chose PostgreSQL over the latest document store trend found themselves with a battle-tested solution that scales predictably and maintains data integrity, critical factors that become apparent only years into a project.

Traditional server-side rendering and HTML continue to power some of the web's most reliable systems, while many Single Page Application frameworks have come and gone. Companies that built complex SPAs often find themselves rewriting their applications every few years, while server-rendered HTML continues to work reliably across all platforms with minimal maintenance.

These technologies haven't merely survived; they've thrived. They've been battle-tested across countless scenarios, their failure modes are well understood, and their documentation is comprehensive. Most importantly, they remain stable and often come with Long Term Support cycles, providing predictable upgrade paths and security updates.

Strategic innovation vs trend-chasing

The key to successful technology adoption lies in distinguishing between strategic innovation and trend-chasing. Before adopting new technology, organizations should ask:

What's the ecosystem's stability and long-term outlook?
What problem does it truly solve? Is there a Lindy alternative?
What role will it play? it worth the risks?
What is its rate of change, can we keep up with it?
Do we have the resources to maintain it as it evolves?
How will this decision age over the years?

Conclusion

In an industry that often equates "new and trendy" with "better," choosing stable technology can feel counterintuitive. However, success is about making thoughtful, pragmatic technology choices that acknowledge the true cost of change. At the end, you are not Google, and you don't have the budget to innovate like it.

Innovation is crucial for progress, but the most successful organizations are those that can distinguish between transformative changes worth the investment and trends that might not justify their long-term costs.

The key is finding the right balance between innovation and stability. Remember: your users don't care about how cutting-edge your technology stack is. They care about reliable, efficient solutions to their problems. In many cases, proven technology is the best way to deliver exactly that.

Dependencies: Convenience at what cost?

Mon, 04 Nov 2024 20:57:29 GMT

Installing dependencies in our projects is easy and convenient. With just a few installation steps, you can solve complex problems that would take time to implement from scratch. But is it always a good investment?

Dependencies hide complexity from us, but that doesn't mean we should ignore what comes in the package. On the contrary, understanding the full scope of each dependency becomes crucial, the devil is in the details.

And, it's not just about the direct dependency you're installing. What about its dependencies? And the dependencies of those dependencies? Who maintains each one of them? There's a dangerous multiplier effect here, hidden in plain sight and often ignored.

The real problems

This web of direct/indirect dependencies creates several critical issues:

Security vulnerabilities: A critical security issue buried six levels deep in a dependency chain can compromise an entire application. The more dependencies a project has, the larger its attack surface becomes.

Maintenance overhead: Package updates are never isolated events. Dependencies conflict with each other regularly. Package A might require version 1.x of Package C, while Package B demands version 2.x, creating complex dependency resolution challenges that waste development and maintenance time.

Build bloat: You are carrying the baggage of each dependency and its dependencies, making the dependency tree larger than expected. Simple projects can end up with hundreds of megabytes of dependencies. Anyone familiar with those projects where node_modules is larger than the actual application code by a factor of 100?

What to do?

Smart dependency management requires a strategic approach:

Question every new dependency. Does the functionality justify the overhead of another package? Could this be implemented in-house with reasonable effort?
Use dependency analysis tools, review the source code and repository health of each new dependency you add to your project, checking:
- How actively is it maintained?
- How many open issues and pull requests?
- What's the test coverage?
- How many contributors?
- How many dependencies?
Maintain regular and automated update schedules while carefully evaluating major version updates.
Implement version locking through common tools like package-lock.json or yarn.lock, etc.
Monitor dependency health continuously:
- Set up automated security scanning
- Watch for deprecated packages
- Document dependencies that require special attention

Sustainability is hard but worth it

Optimizing for the short term is not enough. While it's easy to bring in a dependency to fix an immediate problem, it might not be the best solution for your project's long-term health. This isn't an argument for zero-dependency projects, you shouldn't always reinvent the wheel. The key is thoroughly evaluating the benefits and risks of new dependencies.

Dependency management is about exercising balance –a lost art in software engineering. It's about making informed decisions that consider both immediate needs and long-term sustainability. After all, today's quick fix shouldn't become tomorrow's maintenance nightmare.

Two phased deployments are useful

Wed, 25 Sep 2024 16:35:10 GMT

Nowadays, zero-downtime deployments are the norm in the software industry. Strategies like Rolling updates, Canary releases, and Blue-green deployments have become ubiquitous, being used by literally everyone and every platform out there. These methods allow updating applications with little to no perceived downtime, seamlessly transitioning users to new versions.

However, while these strategies effectively address the downtime part of the equation, they shift the trade-offs elsewhere. All of these deployment approaches require having multiple copies of the application, gradually rolling out changes to a subset of servers or pods while the old version continues to run on others. This process ensures service availability throughout the deployment, but it also means that two versions of the application must coexist temporarily within the same infrastructure, which has several implications.

Common issues

Case study: Deleting a table column with zero downtime using rolling strategy.

This operation is typically done in a single step: the code and migrations work together to delete the column from the database and stop using it in the code.

During the deployment window, which could last from a few seconds to minutes in a rolling deployment, a critical issue emerges. Once the new image is deployed and migrations are applied, the new version of code could be executed on some replicas while the old version runs on others. The old version begins to throw errors if it tries to access the already deleted column, while the new version works correctly.

This dichotomy generates error spikes during deployments and, in general, could lead to data corruption, inconsistencies, and outages. Most alarmingly, these are not the only problems. Performing such an invasive change in one step implies a bigger risk: it eliminates the possibility of a clean rollback, only allowing fast-forward error solving.

Fortunately, there’s a better way:

The Two-Phased Solution

To address these shortcomings and mitigate the associated risks, a two-phased deployment approach becomes crucial. This is not just a fancy name; it literally means two separate deployments, usually separated by a time window. It works like this:

Phase 1: Code adaptation

The first phase involves deploying new code that doesn't use the soon-to-be-deleted database column. However, the column still exists in the database at this stage. This allows both old and new code to run without errors, ensuring a smooth transition or rollback if needed.

Phase 2: Schema change

Once it’s safe, it’s time for the second phase. This involves deploying the change to delete the unused column.

Benefits of Two-Phased deployments

Implementing two-phased deployments ensures more reliable and reversible updates to applications. This approach offers several benefits:

It provides a smooth transition between versions
It allows the ability to do safe rollbacks
It significantly reduces the risk of data inconsistencies or application errors.

Beyond a table column deletion

While we've focused on column deletion, two-phased deployments can be generalized to implement any sort of changes, making the separation into phases a powerful concept for reliable engineering practices.

Conclusion

Two-phased deployments fill a critical gap between zero downtime and reliability, allowing teams to navigate complex updates in a simpler way. By separating the process into phases, they minimize risks and improve overall system stability. Ultimately, this approach ensures that applications can evolve gracefully, maintaining both performance and data integrity throughout the deployment process.

Database transactions edge cases

Sun, 05 Nov 2023 12:08:55 GMT

Transactions look simple. They're not. Especially when you mix database operations with external I/O like emails, APIs, or payment gateways. This is where things break.

Let me show you the progression from naive to robust — and why you might not need the complex solution.

The naive approach: I/O inside transactions

Consider a bank transfer: debit one account, credit another, send confirmation email.

try {
  begin_transaction()
  debit_source_account()
  credit_destination_account()
  send_email_notification()    // 💣
  commit_transaction()
} catch (error) {
  rollback_transaction()
}

Looks clean. But that email call is a landmine.

What goes wrong

By default, most databases use pessimistic locking — when you read or write a row inside a transaction, that row is locked until commit. Other transactions wait.

Now add a slow email server:

Row blocking: Other transactions trying to access those accounts wait for the lock to release.
Deadlocks: Transaction A waits for a row locked by Transaction B, while B waits for a row locked by A. Neither proceeds.
Connection exhaustion: Each transaction holds a database connection. Slow I/O means connections pile up. Pool hits zero. Everything stops.

One slow email server takes down your whole application. The domino effect is real.

The rollback problem

There's another issue. Consider an e-commerce checkout:

try {
  begin_transaction()
  update_inventory()
  charge_credit_card()         // 💣
  update_order_status()
  commit_transaction()
} catch (error) {
  rollback_transaction()
}

What if update_order_status fails? The database rolls back — but the credit card was already charged. You can't rollback an HTTP call. Now you have money taken and no order record. Manual cleanup required.

Better: Optimistic locking

Pessimistic locking: Lock rows during the transaction. Others wait.

Optimistic locking: Don't lock anything. Check for conflicts at commit time using version numbers.

Here's how it works:

Read records and note their version numbers
Do your work (including slow I/O)
At update time, check if versions changed
If unchanged, commit. If changed, retry.

try {
  begin_transaction()

  // Read with versions
  source = get_account(source_id)        // version: 5
  dest = get_account(dest_id)            // version: 3

  // Slow I/O — but no rows are locked
  send_email_notification()

  // Update with version check
  debit(source, amount, expected_version: 5)
  credit(dest, amount, expected_version: 3)

  commit_transaction()
} catch (version_conflict) {
  rollback_transaction()
  retry()
}

What this solves

No row blocking: Other transactions read and write freely
No deadlocks: Nothing to deadlock on
Better concurrency: Multiple transactions proceed in parallel

What this doesn't solve

You're still holding a database connection for the duration of the transaction. If the email server is slow and you have many concurrent requests, you can still exhaust your connection pool.

Optimistic locking works well for most applications. But if you need guaranteed delivery and maximum throughput, there's one more level.

Full solution: Transactional outbox

The outbox pattern separates "what needs to happen" from "making it happen."

try {
  begin_transaction()
  update_inventory()
  add_to_outbox("charge_credit_card", order_id)
  add_to_outbox("send_confirmation", order_id)
  commit_transaction()
} catch (error) {
  rollback_transaction()
}

The transaction is fast — just database writes. A separate process handles the I/O:

// Background worker
process_outbox() {
  for each pending_operation {
    execute(operation)
    mark_completed(operation)
  }
}

What this solves

Fast transactions: No I/O, no waiting
No connection exhaustion: Transactions complete quickly
Guaranteed delivery: Outbox persists until processed
Retry built-in: Failed operations stay in queue
Dead letter queue: Errors go somewhere visible

Trade-off

Eventual consistency. The email doesn't send immediately — it sends when the worker gets to it. For most use cases, a few seconds delay is fine. For some, it's not.

Choose wisely

Approach	Row blocking	Deadlocks	Connection exhaustion	Rollback safety	Complexity
Naive (pessimistic + I/O)	Yes	Yes	Yes	No	Low
Optimistic locking	No	No	Possible	No	Medium
Transactional outbox	No	No	No	Yes	High

Most apps: Just move I/O after the commit. Done.

High concurrency: Optimistic locking.

Mission-critical delivery: Outbox pattern.

Don't reach for the complex solution until you have the complex problem.

Readability through Conceptual Compression

Sat, 11 Jul 2020 00:00:00 GMT

Code is more read than written. I see programming as a two-fold task: giving machines instructions and making those instructions clear to people. You can tell a difference when someone wrote code to help you read it, they use semantic variable names, define the right boundaries, it's clean, basically a joy. But, what makes it feel this way? Let me explore in this post what I think makes code a pleasure to read:

What takes to easily understand a piece of code

Strong foundational knowledge in computer science, etc
Proficiency in the programming language
Knowledge about the business domain
Understanding of the problem
Capturing the intent of how the code solves the problem

For programmers, the key lies in the last three: how well your code bridges the business domain and the problem. The real challenge is to explain a solution via abstractions and their interactions, in other words, expressing intent. Your code needs to speak, and as a real book, it should be a consistent, balanced, and focused story.

Conceptual compression: Intro

One resource I use to express intent is Conceptual Compression. Conceptual compression is about reducing complexity through abstraction. It allows multiple elements of a concept to be encapsulated in a more manageable form so they can be reused without explaining it again. Like the term "Quidditch" in the Harry Potter, reusing the book analogy.

My argument is that compressing portions of code into concepts, and making then interact in the right abstraction level, helps to write code that it's easy to understand.

How can it be achieved with different language constructs?

Variable naming: A quick win

With variables, it's easy to create abstractions in small fractions of code.

The following piece of code uses some implementation details to compose an important business logic through a (complex) conditional:

const user =  /* ... */;
if (project.ownerId === user.userId || user.role === 'MANAGE_PROJECTS') { /* ... */ }

And here's an improvement using better abstractions:

const currentUser = /* ... */;
const canManageProjects = currentUser.role === 'MANAGE_PROJECTS';
const isProjectOwner = project.ownerId === currentUser.userId;

if (isProjectOwner || canManageProjects) { /* ... */ }

In the second version, the compressed concepts optimize the clarity of the conditional, making it look almost in plain English. This is powerful to make things easier to understand.

Functions for clarity

When abstractions need to travel across function boundaries, functions can also work for encapsulating them. Continuing with the previous example, what if now the code is part of an endpoint?

The following example makes use of variable concept compression, but it's not enough, there's a mix of abstraction levels making the whole function difficult to understand:

app.put('/project/:projectId', async ({ request, response }) => {
    const user = /* ... */;
    const project = /* ... */;
    const canManageProjects = user.role === 'MANAGE_PROJECTS';
    const isProjectOwner = project.ownerId === user.userId;

    if (isProjectOwner || canManageProjects) { /* ... */ }
});

My improved version:

function getCurrentUser(request) {
    /* ... */
}

function canManageProjects(user) {
    return user.role === 'MANAGE_PROJECTS';
}

function isProjectOwner(user, project) {
    return project.ownerId === user.userId;
}

app.put('/project/:projectId', async ({ request, response }) => {
    const user = getCurrentUser(request);
    const project = /* ... */;

    if (isProjectOwner(user, project) || canManageProjects(user)) { 
        /* ... */ 
    }
});

It's much better now! The endpoint implementation it's simpler, and it clearly expresses its purpose with unneeded details.

Modules for scalability

Encapsulating concepts in different files starts becoming a more complex problem. There are different levels in the strength of the connection between different files and their concepts (topic for a different post). Regardless of that, the need to compress similar concepts remains to define what goes together and what not.

Different languages have different constructs to abstract or encapsulate functionality in files, like packages, classes, or modules. I'll call them Modules for simplicity.

Here's a continuation of the hypothetical use case I analyzed before using modules.

// File: src/projects/policies/canUpdateProjectPolicy
function canManageProjects(actor) {
    return actor.role === 'MANAGE_PROJECTS';
}

function isProjectOwner(actor, project) {
    return project.ownerId === actor.userId;
}

export function canUpdateProjectPolicy({ actor, project }) {
    return isProjectOwner(actor, project) || canManageProjects(actor)) { 
}

//----------------------------------------------------------------
    
// File: src/projects/use-cases/update-project.use-case.js
import canUpdateProjectPolicy from 'src/projects/policies/can-update-project.policy';
import ProjectManager from 'src/projects/...<not-relevant-now>';

export async function updateProjectUseCase({ projectId, payload, actor }) {
  const project = ProjectManager.find(projectId);
  if (canUpdateProjectPolicy({ actor, project }) {
      return ProjectManager.update(id, params);
  }
  /* ... */;
};

//----------------------------------------------------------------

// File: src/projects/web/index.js
import updateProjectUseCase from 'src/projects/use-cases/update-project.use-case';
import { getCurrentUser } from 'src/shared/http/helpers';

app.put('/project/:projectId', async ({ request, response }) => {
  const currentUser = getCurrentUser(request);
  const projectUpdatePayload = /* ... */;
  const projectId = /* ... */;
  const updatedProject = await updateProjectUseCase({ 
    projectId, 
    payload: projectUpdatePayload, 
    actor: currentUser
  });
  /* ... */
});

In the example there's a project module containing all related functionality, it is composed of some smaller modules called policies, web, use-cases, and I hint that there could be different files inside them. The point is that to apply conceptual compression at module level, there are many layers of abstraction to deal with and this becomes a new challenge. Having a good balance of abstraction layers is essential.

Taking for example the `can-update-project.policy`, which main goal is to define when a user can update a project, if this function would have to deal with SQL queries it would've made it more complex and harder to understand.

Closing thoughts

Navigating the complexities of business logic often leads to intricate coding challenges. Conceptual compression serves as a powerful tool in this regard, helping to simplify how programs are structured by creating effective abstractions, making a better bridge between the problem and its solution, creating that joy in readers. This practice not only enhances code readability but also reduces complexity and promotes maintainability and scalability of your codebase.

But it also brings big challenges, such as deciding what concepts belong together, or what's the right connection strength that will break module's apart. Regardless of that, I argue that those challenges are there anyway in Software Development, applying Conceptual Compression it's only making them explicit. Feels like a good way to approach Software Design and architectural decisions in general.

Writing maintainable tests

Wed, 02 Oct 2019 00:00:00 GMT

Alright, so after a decade-plus in Ruby, I've gotten into some great debates about testing with colleagues. Through these conversations, I've come up with a testing style that's not exactly textbook, but it's been doing the job and doing it well. Good to get this down for my own clarity and future reference.

Anatomy of my ideal test

Here's a structured blueprint of how I craft my tests:

it 'provides a succinct description of the behavior' do
  setup
  exercise
  verify
  teardown
end

Key Characteristics

Verbose: Each test is self-explanatory, serving as its documentation.
State isolation: Avoid shared global state to prevent entanglement.
Four-step structure: The tests follow the Setup, Exercise, Verify, Teardown paradigm.
Co-location: All four steps reside within the same block.
Visual clarity: Steps are demarcated with empty lines for easier navigation.

Rationale

Modularity for future changes

In my observation, most test modifications are constrained to specific portions of the test file. This isolation eliminates the need to refactor unrelated tests, ensuring that changes are smooth and swift.

Avoiding shared state

Shared states in tests invite two vices: mutability and coupling. They often lead to complex setups and tear-downs that become increasingly harder to manage as your codebase evolves.

Advocacy for verbosity

Tests aren't just tests; they are documentation. A verbose test provides more context, aiding in both readability and maintainability.

Explicit test steps

Each of the four steps —Setup, Exercise, Verify, Teardown— is designed to be semantically clear. This separation enhances readability, serving as a reliable guide through complex test scenarios.

Concluding remarks

While Ruby’s philosophy often emphasizes principles like DRY (Don't Repeat Yourself), I've found that a different mindset is necessary for testing. Duplication is fine in this case in favor of an optimal test case isolation. I feel confident that this testing approach delivers a strong return on investment.

System back-pressure

Sat, 24 Feb 2018 00:00:00 GMT

What's back-pressure

Back-pressure happens when data comes in faster than your system can handle it. A slow database can't keep up with incoming requests, so the HTTP layer starts timing out. One bottleneck breaks everything upstream.

Understanding back-pressure is undervalued. It's not just about preventing failures — it's about adding capacity control mechanisms that let you control data flow speed across your entire system.

Back-pressure effects

When your database can't keep up with write requests, queries start piling up. The backlog hits the HTTP layer — requests timeout, errors spike, users complain. The breakdown spreads. One slow component takes down the whole system.

Not just a database issue

Databases are the usual suspect, but any component can cause back-pressure — a message bus, a cache, an external API. When something downstream slows down, everything upstream breaks.

Solutions

The obvious fix is to throw more resources at the database. But that's reactive and expensive.

A better approach: add a buffer between the HTTP layer and the database. Queue incoming requests and drain them at a pace the database can handle. This decouples request handling from database state — requests don't fail just because the database is temporarily slow.

This makes your system more reliable and cheaper. You're not paying for capacity you only need during spikes.

Pull systems

Push systems send data as fast as they can. Pull systems let the consumer request data when it's ready.

Pull is underrated. The consumer controls the pace, so it never gets overwhelmed. This adds reliability that push architectures often lack. It also means better resource utilization — you're not wasting compute on data you can't process yet.

Conclusion

Back-pressure breaks systems when data floods in faster than you can process it. The fix is simple: buffers or pull systems to control the flow. Prevents crashes. Saves money. Worth keeping in mind before you reach for the "scale up" button.