From the engineering blog.
How we think about intelligent systems, reliability, and shipping software that lasts.
Designing agents that fail safely
Autonomy is easy to demo and hard to trust. An AI agent that can call your tools can also call them wrong. The real question when building AI agents isn't "will the agent make a mistake?" — the answer is always yes — it's whether that mistake is contained, visible, and reversible. That's exactly what "failing safely" means, and it's an architectural design problem before it's a language-model problem. In this article we share the principles we apply at Grid when designing AI agents that are ready for production, not just for demos.Why the demo succeeds and production failsIn a demo, the team controls everything: the inputs are clean, the script is rehearsed, and the audience is forgiving. In production, the agent meets inputs nobody has seen before, systems that change without notice, and users who probe the edges of the system — sometimes in good faith, sometimes not. The gap between those two environments is where incidents are born.Successful teams don't try to build an agent that "never makes mistakes" — an impossible goal with probabilistic models. Instead, they build infrastructure that makes the cost of any single mistake small and contained. That shift in mindset — from preventing failure to containing it — is the foundation of everything that follows.Start from permissions, not promptsMost agent failures we see in real projects trace back to one cause: broader access than necessary. It's easy to hand an agent a full API key and write "never delete any data" in the prompt — but a prompt is not a guarantee, it's a wish. The model can be misdirected through prompt injection, misread the context, or simply hallucinate.The rule we hold to: scope every agent to the smallest set of tools and data it needs for its task, expressed as explicit system-level permissions — narrow OAuth scopes, restricted keys, and read-only database roles wherever that is enough.An agent should never be able to do something your permission model didn't already allow — regardless of what the prompt says.The advantage of this approach is that it's auditable: you can know precisely what the agent could do in the worst case by reading the permissions, not by guessing at model behavior.Design for blast radius: put humans at the dangerous edgesNot every step needs human approval — an agent that asks permission for every read is useless, and users learn to click "approve" without thinking, which is more dangerous than having no approval at all. The answer is to classify actions by blast radius:Low-impact actions — reading data, drafting content, proposing a plan: run autonomously without stopping.Medium-impact actions — sending an internal email, editing a single record: execute with an immediate notification and an undo path.High-impact actions — deleting data, moving money, messaging customers: always stop for explicit human approval.With this classification you get an agent that is fast in 90% of its work and careful in the 10% that deserves care. Most importantly: when the model misjudges how risky an action is, the safety net lives in the classification layer itself, not in the model's good intentions.Make every decision traceableWhen something goes wrong in production, "the model decided that" is not an answer that satisfies a client or an auditor. Every agent run should emit a complete, reviewable trace: the original inputs, the plan the agent formed, every tool call with its parameters and result, and the final output — all tied to a single run ID.That log is not an operational luxury; it's what turns an incident from a mystery into a lesson. Without it you know something went wrong, but not why, and you can't prevent it from happening again. And with regulators in sensitive sectors, an audit trail is table stakes, not a bonus feature.Make reversibility the first plan, not the emergency planA reversible action can be delegated to an agent with far more confidence than a final one. So we redesign the tools themselves before handing them to the agent: delete becomes restorable archiving, send becomes scheduling with a cancellation window, and edit keeps the previous version. The wider the "reversible" surface in your system, the more you can safely automate — without raising your risk ceiling.Evaluate before launch, monitor afterManual testing is not enough for a non-deterministic system. Before launch, we build every agent an evaluation suite of real cases drawn from the actual work, plus adversarial cases designed to break it: ambiguous inputs, prompt-injection attempts, and rare edge cases. Any change to the prompts or the model runs through this suite before it ships.After launch, monitoring continues on samples of real traffic: task success rates, approval-stop rates, and anomalous patterns in tool calls. A model that worked well last month can change behavior with a new update — continuous monitoring is what catches that before your customers do.A checklist before launching any agentDoes the agent hold the narrowest possible permissions for its task?Are actions classified by blast radius, with mandatory human approval for the sensitive ones?Does every run produce a complete audit trail you can go back to?Are dangerous actions reversible, or deferred with a cancellation window?Is there an evaluation suite that runs before every change, and continuous monitoring after release?The bottom lineA trustworthy agent is not one that never makes mistakes — it's one whose mistakes are contained by permissions, visible in the logs, and reversible by design. These are not limits on what the agent can do; they are precisely what lets you widen its permissions later with confidence, one step at a time.At Grid we build AI agents on these principles from day one: minimal permissions, approvals at the dangerous edges, complete audit trails, and continuous evaluation. If you're planning to bring agents into your business systems, get in touch — we'll help you start safely without sacrificing speed.
How we hold 99.98% across regions
99.98% is not a marketing slogan — it's less than two hours of downtime in an entire year. Reaching that number in system availability doesn't come from buying bigger servers; it comes from architectural decisions that assume every component will eventually fail: the server, the database, the network, even an entire cloud region. In this article we walk through the practical playbook we follow at Grid to hold high availability across multiple regions, from honest service-level objectives to deliberately testing failure.Start with an error budget, not a promiseThe first step toward real availability is an honest number. An SLA that promises 100% is a polite lie; mature teams define a measurable service-level objective (SLO) and derive an error budget from it: the number of minutes of degradation allowed before new feature releases pause in favor of stability work.The error budget turns reliability from an emotional debate into an engineering decision: burn the budget early and risk-taking freezes automatically; keep it healthy and you've earned safe room to experiment and ship fast.A single region is a single point of failure — no matter how bigThe best architecture inside one cloud region can vanish entirely due to a provider outage, a routing mistake, or a datacenter-level incident. That's why we design from day one around at least two regions: a primary serving traffic and a secondary ready to take over.The question is not "will the region fail?" but "how many minutes do we need to move traffic away when it does?"Effective failover needs three pieces working together: DNS routing with a short TTL (or a global load balancer), a fresh copy of the data in the secondary region, and automation that shifts traffic without waiting for a human decision at 3 a.m.Data is the hard partMoving web traffic between regions is relatively easy; data is what's hard. For every system we pin down two numbers explicitly: RPO (how many seconds of data we can afford to lose in the worst case) and RTO (how many minutes we need to be back up). Those two numbers dictate the replication mode:Synchronous replication — zero data loss, but it adds latency and limits the distance between regions.Semi-synchronous replication — the practical balance: a few seconds of lag in exchange for normal performance.Asynchronous replication + continuous backups — for systems that can tolerate losing a few minutes.What matters is that the choice is a conscious decision per system — not one default silently imposed on everything.Health checks that tell the truthPlenty of systems show "green" on the dashboard while being effectively down, because the health check only tests that the process is alive — not that it's serving users. An honest check exercises the real path: it queries the database, touches the cache, verifies critical external dependencies — then decides. And based on it, the load balancer pulls a server or a whole region out of service automatically.Test failure before it tests youA failover plan that has never been exercised isn't a plan — it's a wish. We schedule regular drills where we deliberately take down the primary region in production (or an environment that mirrors it) and measure in minutes: when did the system detect the failure? When did traffic move? Was any data lost? Every drill exposes a wrong assumption that would otherwise have surfaced during a real customer-facing incident.The most valuable side effect: the team rehearses the procedure until it becomes boring routine — which is exactly what you want in a real emergency.Incidents will happen — prepare the script, not the heroicsEven with all of the above, incidents will happen. The difference between a 5-minute outage and a 5-hour one is operational readiness: a clear runbook for every scenario, alerts that reach the right person the first time, a public status channel for customers, and a blameless post-incident review that produces root-cause fixes instead of patches.A high-availability checklistDo you have a written SLO and an error budget reviewed monthly?Does your service run from at least two regions, with failover tested in the last quarter?Have you defined RPO and RTO per data system — and does the actual replication meet them?Do your health checks exercise the real path, not just process liveness?Do you keep an up-to-date runbook for every major failure scenario?The bottom lineHigh availability is not a product you buy but a set of habits you build: honest targets that drive decisions, a second region always ready, consciously replicated data, health checks that tell the truth, and failure rehearsed on a schedule before it shows up unannounced. With those habits, a number like 99.98% becomes a natural outcome — not a miracle.At Grid we design and operate highly available cloud architectures across multiple regions for our clients. If your system can't afford to go down, get in touch — we'll help you build reliability that is measured in numbers, not promises.
Tokens as a contract between teams
Every digital product replays the same scene: a designer picks a shade in the design tool, a developer writes a close-enough value in code, and six months later the product has eleven different "blues" and nobody knows which one is right. Design tokens are the structural fix for that chaos — and the cheapest way we know to stay consistent as a product and a team grow. In this article we explain how we treat tokens at Grid: as a binding contract between design and engineering, not just a color file.What are design tokens?A token is a named design value: instead of writing #237F8C in twenty places, you define color-brand-primary once and reference it everywhere. The same applies to spacing, font sizes, radii, shadows, and motion durations. The value lives in one place; the name carries the meaning.The real power isn't the naming — it's the consequence: when the brand's primary color changes, it changes in one source and propagates automatically to the web, the app, and the marketing emails, instead of weeks of manual hunting.A token is a contract — and a contract has two partiesThe common mistake is treating tokens as a config file that belongs to developers. Successful tokens are a contract between design and engineering: the designer commits that every visual decision goes through a named token, and the developer commits that not a single raw value enters the codebase.Any visual value without an agreed name is visual technical debt — you'll pay for it at the first redesign.With that contract, the "is this the right gray?" debate happens in exactly one place, and code review turns from comparing hex numbers into reading clear intent.Three layers keep the chaos outFlat token lists collapse at the first dark mode or sub-brand. The structure we rely on has three layers:Core tokens — the raw palette: teal-600, space-4, font-size-16. Never used directly in interfaces.Semantic tokens — functional meaning: color-text-primary, surface-raised, border-focus. They point at the core layer.Component tokens — only where needed: button-bg pointing at a semantic token.With this structure, dark mode is not a new CSS file — it's remapping the semantic layer to different core values. A new sub-brand is a new core palette under the same semantic names.One source of truth — everything else is generatedThe contract breaks the moment it has two copies. We keep tokens in a single source (usually JSON files in the repository, or a tool like Tokens Studio wired to it) and generate every format from it automatically: CSS variables for the web, design-tool values, native platform constants. Tools like Style Dictionary make that generation an ordinary build step.The golden rule: edits happen in the source only. A manual tweak in any generated format is a breach of contract — and it will be erased by the next generation run anyway.Change governance: tokens have versionsChanging a token's value touches every screen in the product, so it deserves versioning discipline: changing only a value is a patch; renaming or removing a token is a breaking change that needs a transition window and a clear migration path. Token changes get a shared review — designer and developer together — because a contract is never amended by one side.A checklist for starting with tokensHave you inventoried the duplicated visual values in your current product (colors first)?Are tokens split into core and semantic layers — with interfaces consuming only the semantic ones?Is there a single source from which every format is generated automatically at build time?Does an automated CI check block new raw values from entering the code?Do token changes go through a joint design–engineering review?The bottom lineDesign tokens aren't a luxury for big design teams; they're the cheapest insurance against the consistency rot that hits every growing product. A clear contract, three layers, one source of truth, and lightweight governance — that's all it takes to keep your product's identity coherent across every platform, every team, and every redesign to come.At Grid we build design systems and token architectures for Arabic and English products alike — including the challenges of RTL and LTR. If your product suffers from accumulated visual chaos, get in touch and we'll help you build the system that stops it at the root.
Choosing a model-agnostic architecture
Every few months a new AI model arrives that beats its predecessor on capability, price, or both. A product that hardcoded one provider's API calls throughout its codebase faces two painful options: a sweeping rewrite, or staying on an aging model and paying for it in performance and cost. A model-agnostic architecture dissolves that dilemma at the root: you build your product once and swap models underneath it freely. In this article we explain how we design that layer at Grid and what we've learned from running it in real products.Why neutrality is a strategic decision, not a technical luxuryThree reasons make hard-coupling to a single provider a risk. First, the race is fierce — today's best model can be third place within months. Second, prices move constantly, and the cost gap between two models capable of the same task can reach 10x. Third, compliance and data-residency requirements may demand tomorrow a model that runs inside your geography or your own infrastructure.Couple your product to the task it solves, not to the model that happens to run it today.One abstraction layer separates the product from the providerRule one: product code never calls a provider directly. Every call goes through a single internal interface that translates requests to each provider's format, normalizes response shapes and tool calls, and handles errors, retries, and fallback when a provider goes down.The interface is defined in your product's terms — "summarize", "classify", "extract" — and beneath it lives a configuration file that maps each task to a model. Changing the model becomes a one-line config edit, not a rewrite project.Route each task to the model it deservesNot every call needs the strongest, most expensive model. Smart routing classifies the work:Complex tasks — deep analysis, code generation, multi-step reasoning: the strongest model.Volume tasks — classification, field extraction, short drafting: a small fast model at a fraction of the cost.Sensitive tasks — data that must not leave your infrastructure: a self-hosted model or one inside the required region.In many products, 80% of calls are volume tasks that a cheaper model serves at comparable quality — routing alone can cut the bill severalfold.Prompts are managed assets — not scattered stringsFree swapping assumes the instructions themselves are portable. We store prompts outside the code, versioned with a change history, and allow per-model variants where needed — because optimal phrasing differs between models. The result: trying a new model never touches product code at all.Evaluation is the safety valveFreedom to swap means nothing without a way to know the swap is safe. For each task we build an evaluation suite from real cases with clear success criteria, and any candidate model runs through it before touching production. The numbers settle the debate: equal or better quality, at lower cost, at acceptable latency — or no swap.After the swap, monitoring continues in production on real samples, because a model's behavior on your actual data can differ from its eval-suite results.A warning: don't over-engineerModel-agnostic doesn't mean building a huge platform on day one. If your product calls one model in two places, a single wrapper function and an organized prompt file are enough. Build the full layer when tasks and models genuinely multiply — neutrality is an architectural habit that starts small, not a big upfront project.A model-agnostic checklistDoes every model call go through one internal abstraction layer?Does a config file — not code — decide which model serves which task?Are your prompts stored with versions and per-model variants?Do you have an evaluation suite per task that settles swap decisions with numbers?Do you track cost and latency per task and per model in production?The bottom lineThe model market will keep moving for years, and the winner isn't whoever bets on the right horse — it's whoever builds a carriage whose horses swap easily. A clean abstraction layer, task-based routing, managed prompts, and evaluations that settle decisions: with these pieces, every new model becomes an opportunity you capture in days, not a threat that demands a rebuild.At Grid we design AI architectures for products that want to profit from the race instead of being burned by it. If your product is locked to a single provider and you want to decouple safely, get in touch.
Observability from day one
In most projects, observability gets added after the first painful incident: hours of guessing in the dark, then a belated decision that "we need better monitoring." The saner — and far cheaper — approach is to build observability from day one, as part of the definition of done for every new feature. In this article we walk through the three pillars we establish at Grid with every system we build, and why AI systems have made that foundation more urgent than ever.Watching charts is not observabilityThe difference is fundamental: traditional monitoring answers questions you decided on in advance — "what's the CPU usage?". Observability lets you ask questions you hadn't thought of when you built the system: "why do this specific customer's requests fail after 8 p.m.?". Real incidents are always the second kind — new questions nobody predicted.An observable system answers questions that haven't been asked yet — without shipping new code to collect the data.Pillar one: structured logs, not free textA text log line like "payment error occurred" is read by one human; a structured log (JSON with stable fields) is queried by machines across millions of lines. From day one we commit to: every log line carries a correlation ID that follows the request across all services, an accurate severity level, and field names that are consistent across the entire system.The payoff shows in the first investigation: instead of digging through scattered files, one query returns the request's full journey from the gateway to the database.Pillar two: metrics that matter to the userCPU and memory are useful, but they don't tell you whether users are suffering. The four golden signals are the foundation:Latency — specifically p95 and p99, not the average that flatters the picture.Traffic — requests per second, for context and peak awareness.Error rate — the share of failing requests, segmented by type and route.Saturation — how close resources are to their limits before they tip over.These four per service, with one dashboard bringing them together, surface 90% of problems before customers report them.Pillar three: distributed tracing ties the story togetherIn a multi-service system, logs and metrics tell you something is slow — distributed tracing tells you where. Every request carries a trace context that travels across services, so you see the journey as a waterfall: which service consumed the time, which call repeated needlessly. The OpenTelemetry standard has made this achievable with reasonable effort and no vendor lock-in.AI systems raise the stakesWith agents and language models, the question is no longer "which service is slow?" but "why did the agent make that decision?". So we extend tracing to cover every run: the prompts used and their version, every model call with its inputs and outputs, every tool call with its result — all tied to a single run ID. Without that trail, debugging agent behavior in production is pure guesswork; with it, every decision is explainable and reviewable.Alert on symptoms, investigate causesToo many alerts are more dangerous than too few — the team learns to ignore them. The rule: page only on what touches users (error rate over threshold, noticeable latency, full outage), and make every alert actionable with a direct link to the investigation dashboard or the runbook. Internal-cause warnings — a filling disk, a lagging replica — go to daily review boards, not to waking someone at dawn.A day-one checklistAre your logs structured with consistent fields and a correlation ID on every request?Do you measure the four golden signals per service, in percentiles rather than averages?Does distributed tracing flow through all your services via OpenTelemetry?Do your AI systems record every decision with its calls and outputs?Is every human-waking alert tied to real user impact and a clear action?The bottom lineObservability is an investment that compounds with every incident: what would have taken a night of guessing becomes a minutes-long query. Start it on day one — structured logs, golden signals, distributed tracing, and a full trail of AI decisions — and you'll find that most major incidents were small signals that showed up early and nobody saw.At Grid we build observability into every system we deliver — from day one, not after the first incident. If your system runs as a black box and you want to see inside it, get in touch.
Multi-tenant SaaS without the pain
Every successful SaaS platform reaches the same moment: dozens, then hundreds of tenants on the same infrastructure, and one question deciding the product's future — was multi-tenancy designed in from the start, or patched on later? The gap between those two is the gap between quiet growth and nights of cross-customer data-leak incidents. In this article we summarize the isolation, authentication, and billing patterns we apply at Grid when building multi-tenant platforms that scale without the pain.Choose your isolation model deliberately — not by accidentThe isolation decision is the platform's most important architectural choice, with three main patterns:Database per tenant — strongest isolation and easiest compliance, but operational cost multiplies with every customer. Right for large clients and strictly regulated sectors.Shared schema with a tenant ID — the most operationally efficient and fastest to grow, but it puts the entire isolation burden on code discipline.Hybrid — shared schema for the general tier, dedicated databases for those who pay for isolation. The pattern most mature platforms end up with.There is no "correct" model — there is the model that fits today's customers while leaving the door open for tomorrow's.The tenant ID flows through every layer — and defense has multiple linesIn the shared model, the worst possible incident is a query that forgot the tenant filter and showed one customer's data to another. Protection isn't left to human attention; it's built in layers: a mandatory tenant context on every request, a data-access layer that adds the filter automatically and can't be bypassed, and row-level security (RLS) in the database as a last line of defense that works even when the code is wrong.Then automated tests that deliberately attempt cross-tenant access — failing the build in CI before the hole ever reaches production.Authentication: identity is one thing, membership is anotherThe pattern that lasts: one identity per user, multiple tenant memberships, and roles per membership. That handles the real-world cases smoothly — a consultant working with three companies, an employee moving between teams — without duplicate accounts. And with your first enterprise customer comes the SSO request (SAML or OIDC); build the integration point early, because it's a deal requirement, not a luxury feature.Billing starts with meteringYou can't bill what you don't measure. From day one, record usage per tenant — active users, storage, API calls, or whatever unit reflects value in your product — in an event stream independent of the billing logic itself. Then you can lay any pricing model on top of the data: plans, pay-as-you-go, or a mix, and change it later without a rebuild.And connect plan limits to actual enforcement: a plan promising ten users must be enforced by the system automatically, with a polite upgrade prompt rather than a cryptic error.The noisy neighbor: resource fairnessOn shared infrastructure, one heavy tenant — a million-row import, a runaway integration — can slow the platform for everyone. Prevention: rate limits per tenant rather than per system, processing queues that separate heavy work from the interactive path, and per-tenant monitoring that shows who is consuming what before the others start complaining.Migrations across a fleet of databasesIf you chose database-per-tenant, every schema migration becomes a fleet operation: automation that rolls the migration out gradually, verifies success per database, and halts the rollout at the first failure. Without it, tenants drift across schema versions — a mess that's very hard to come back from.A pain-free multi-tenancy checklistIs your isolation model a documented, deliberate decision — not an inherited default?Does the tenant ID flow mandatorily through every layer, with RLS as the last line of defense?Is identity separate from membership, with an SSO integration point ready for enterprise customers?Do you meter usage per tenant from day one, with limits enforced automatically?Do you have per-tenant rate limits and segmented monitoring that exposes the noisy neighbor?The bottom lineMulti-tenancy isn't a feature you add later; it's a lens you design every layer through: isolation, identity, metering, and operational fairness. Platforms that build it in grow from ten customers to a thousand quietly; platforms that patch it on pay in incidents and customer trust. Early investment here is the cheapest insurance for your product's future.At Grid we design multi-tenant SaaS platforms from scratch and fix multi-tenancy pain in existing ones. If you're building your platform or preparing for your first enterprise customer, get in touch.
A dashboard people actually read
Most dashboards die a quiet death: built with enthusiasm, shown once at the launch meeting, and unopened two weeks later. The problem is rarely the data — it's design that shows everything and says nothing. A successful dashboard isn't a gallery of charts; it's a daily decision tool. In this article we distill the principles we use at Grid to design dashboards people open every morning because they actually answer their questions.Start from the decision, not the dataThe wrong question when building a dashboard: "what data do we have?". The right one: "what decisions does this dashboard's user make every week?". Every element must answer a specific question that leads to action — "do we need to intervene today?", "where are customers leaking?". A chart that changes no decision is noise, however beautiful.Every chart on the dashboard answers a question — if you can't write the question down, delete the chart.Data-ink: every pixel earns its placeEdward Tufte's famous principle: give as much "ink" as possible to the data itself, and as little as possible to decoration. In practice that means removing shaded backgrounds, heavy borders, gradients, drop shadows, dense gridlines, and 3D effects — they all compete with the data for the eye's attention and add no information. A clean dashboard isn't an aesthetic preference; it's reading speed.One accent color is enoughA dashboard using ten colors distinguishes nothing — when everything shouts, nothing is heard. The system that works: quiet grays for the base, one accent color for what matters most — the line that matters now, the segment that needs attention — and red reserved exclusively for genuine warnings. With that discipline, color guides the reader's eye to what deserves it within a second.Hierarchy guides the eyeReaders scan dashboards; they don't read them line by line. Design for that behavior: the critical numbers at the top, large and unmissable; supporting detail beneath; deep exploration at the last level or behind a click. And the one-screen rule: what needs long scrolling won't be seen — if sections multiply, split them into dashboards for different audiences instead of one endless screen.Choose the chart that matches the questionTrend over time — a line. Nothing beats it.Comparison across categories — bars, sorted largest to smallest.One critical number — a big number with its change indicator versus the previous period, not a speedometer gauge.Pie charts — rarely: two or three slices at most, otherwise bars are clearer.And watch for bar-chart axes that don't start at zero — they inflate differences and deceive the reader, and a dashboard's credibility is its most valuable asset.A number without context is not information"Revenue: 48,000" — is that good? Nobody knows without a comparison. Every number on the dashboard needs a reference: the previous period, the target, or the market average. Add the target line to the chart, the change percentage next to the number, and color the deviation — then the number turns from a statistic into a verdict: we're fine, or we need to act.For Arabic audiences: the dashboard reads from the rightBilingual dashboards are a challenge we handle daily: in the Arabic interface the entire layout mirrors (RTL), so the visual hierarchy starts at the top right — while time axes inside charts still run left-to-right as readers expect for numbers. Long numbers and dates need correct locale formatting. Test both versions — a dashboard that reads naturally in Arabic doesn't come for free from a foreign charting library.A pre-launch dashboard checklistDoes every chart answer a clear decision question you can write in one sentence?Have you removed every decoration that carries no information — backgrounds, shadows, 3D?Do you use one accent color, with red only for warnings?Is the most important content at the top, everything visible without long scrolling?Is every number paired with a comparison that gives it meaning — a target or a previous period?The bottom lineA good dashboard isn't the one with the most charts but the one that answers fastest: the decision-maker opens it and knows within seconds whether things are fine and where to look. Start from the questions, be frugal with ink and color, order by hierarchy, and give every number its context — and you'll get that rare thing: a dashboard people open every day by choice.At Grid we design bilingual dashboards and interactive reports for companies and organizations — from data modeling to an interface that actually gets read. If your dashboards get built but never opened, get in touch.
Get new posts by email.
Occasional, technical, no fluff. Unsubscribe any time.
One email when we publish. That's it.