Reproducibility
Infrastructure is defined as code so environments can be recreated and reviewed.
Rather than an environment whose current shape exists only in a console and in the memory of whoever last changed it.
Applied AI & Systems Lab
07 · Cloud & Infrastructure
Infrastructure engineered for when things go wrong
We design and engineer resilient cloud infrastructure, deployment systems, security controls, observability, and recovery paths for production systems that need to keep operating.
- Application
- Containers
- CI/CD
- Infrastructure as Code
- Cloud
- Networking
- Security
- Observability
- Recovery
01The Problem
Infrastructure becomes visible when it fails
Manual deployments, invisible failures, weak access controls, untested recovery plans, and fragile environments become expensive when production depends on them. Switch the two states below and follow the same four chains.
Four chains · one link each
Manual Deployments
Releases depend on people remembering fragile operational steps.
- Change
- Review
- Release steps
- Production
- Verified
Release steps — not a mechanismA sequence someone follows from memory or a wiki page, correctly, under time pressure, at the end of a long day.
The release becomes an event that needs a specific person awake for it.
Invisible Failures
Without observability, problems become visible only after users are affected.
- Degradation
- Signal
- Alert
- Owner
- Response
Signal — not a mechanismNothing is emitting the signal, so the first detector in the path is a user deciding the problem is worth reporting.
Detection time is set by your customers rather than by your engineering.
Security Gaps
Access and security controls are often added after the architecture is already established.
- Identity
- Permission
- Boundary
- Secret
- Audit
Boundary — not a mechanismOne broad network and one broad credential, because narrowing either of them after the fact means changing the architecture.
One compromised component reaches everything the component could reach.
Untested Recovery
A documented recovery plan is not enough if nobody has verified that it works.
- Failure
- Detection
- Restore
- Validate
- Service
Restore — not a mechanismA written procedure, never executed against the current system, referencing infrastructure that has since changed.
You discover which parts of the plan were wrong during the incident.
None of these chains is missing a discipline. Each is missing one link, and in every case the missing link is the one that was left to a person, a document, or an assumption.
02What We Engineer
The infrastructure behind production systems
Three bands, in order of dependency rather than order of interest. Select any of the seven for the decision it settles, and for what happens when it is left undecided.
Foundation
Decided first, because everything above inherits it
Delivery
How change reaches the foundation
Operations
What makes the two bands above survivable
Cloud Architecture
What it decides
Where the workload runs, how it is isolated, what it is allowed to reach, and which parts of it are allowed to fail independently.
Left undecided
The topology becomes whatever the first deployment needed, and every later requirement is fitted around that accident.
Which of these an engagement includes depends on the environment you already have. Most start with two or three of them, not all seven.
03Infrastructure Architecture
Every layer has a failure mode
Infrastructure is designed as a system. Application architecture, deployment, networking, security, observability, and recovery have to work together. Select a layer for its responsibility and the failure it owns.
How the application is packaged, configured, and started decides most of what the infrastructure below it can do for it. An application that holds state in local memory cannot be scaled horizontally, and one that reads configuration at build time cannot be promoted between environments — both are infrastructure constraints written in application code.
- Configuration
- Statelessness
- Startup & shutdown
- Health endpoints
04Engineering Approach
Design for the failure, not just the happy path
Five principles, each of which is mostly a statement about what it rules out. The happy path is the easy half of any infrastructure design; these are about the other one.
Automation
Deployments should reduce manual intervention and operational risk.
Rather than a release procedure whose correctness depends on a person executing eleven steps in order, without interruption.
Observability
System health should be visible before failures become incidents.
Rather than a system whose first reliable failure detector is a customer who decided it was worth reporting.
Least Privilege
Access should be limited to what each system and person actually needs.
Rather than one broad credential shared across services, because narrowing it later means changing the architecture.
Recovery
Recovery procedures should be tested rather than simply documented.
Rather than a plan whose first execution against the current architecture happens during the incident it was written for.
05Deployment
Make deployment routine, not risky
A production deployment should follow a repeatable path with automated validation and visible system health. Here is one change taking it, stage by stage.
One change reaching production
- 01Code Change
- 02Validation
- 03Security Checks
- 04Build
- 05Deployment
- 06Health Checks
- 07Observability
- 01Code ChangeVersion control
A change is proposed against a reviewed branch. Nothing about it is special — the point of the rest of this path is that it does not need to be.
- 02ValidationAutomated tests
Tests, linting, and type checks run on the change itself rather than on a developer's machine after the fact. A failure here stops the run and costs nothing.
- 03Security ChecksDependency & config scanning
Dependencies, container images, and infrastructure definitions are checked for known vulnerabilities and misconfiguration before anything is built from them.
- 04BuildReproducible artifact
One artifact is produced and versioned. The same artifact is promoted through every environment, so what was tested and what runs are the same object rather than two builds of the same commit.
- 05DeploymentProgressive rollout
The artifact is released using a strategy the workload can actually support — rolling, blue-green, or canary — with the previous version still available to return to.
- 06Health ChecksAutomated verification
The system verifies itself against defined health signals before the rollout is allowed to continue. Failing that verification reverses the deployment rather than notifying someone about it.
- 07ObservabilityPost-deploy signals
Error rates, latency, and resource behaviour are watched against the pre-deploy baseline, so a change that passed every gate and is still wrong is visible in minutes.
OutcomeThe change is in production, verified by the system rather than by a person watching it, and the previous version is still one step away.
The point of the path is that none of it is remarkable. A deployment that requires attention is a deployment that requires a specific person to be available, which is a constraint on the business rather than on the system.
06Resilience
A recovery plan is only real after it has been tested
We treat disaster recovery as an engineering system: define recovery paths, establish dependencies, test scenarios, measure results, and improve the plan. Below, a failure runs through one.
Failure · simulated recovery path
- 01Failure
- 02Detection
- 03Isolation
- 04Recovery
- 05Validation
- 06Return to Service
- 01FailureSomething has broken
A component stops behaving correctly. This is the assumption the rest of the architecture is built on, not the exception to it — every layer will eventually produce one of these.
- 02DetectionAutomated alerting
Health signals and thresholds surface the failure to a named owner. The measure that matters here is how long the system was wrong before anyone knew, and it is set by engineering rather than by chance.
- 03IsolationBlast-radius containment
The failing component is taken out of the serving path so it stops affecting the rest of the system. This is where segmentation and independent failure domains either exist or do not.
- 04RecoveryDefined restore path
The documented and rehearsed path runs: replace, restore, fail over, or roll back, in the dependency order the system actually requires rather than the order the diagram lists.
- 05ValidationVerified before reopening
The recovered component is checked against real health signals and data integrity before traffic returns to it. Reopening an unverified path turns one incident into two.
- 06Return to ServiceTraffic restored
Traffic is returned, usually gradually, and the system is watched at the same intensity as during the incident. What the run exposed then feeds back into the architecture rather than into a postmortem nobody reopens.
OutcomeService is back, the path that restored it has now been executed rather than described, and what the run exposed goes back into the architecture.
How long each stage takes depends entirely on your architecture, data volumes, and dependencies — which is why the sequence is shown and no times are attached to it. Recovery targets are established against your system, with you, not claimed here.
07Security
Security is part of the architecture
Security controls are designed into infrastructure rather than treated as a final checklist. Each control sits at the boundary it belongs to.
Perimeter
What can reach the environment at all
- Identity & Access
- Network Boundaries
Platform
What a reachable component is permitted to do
- Least Privilege
- Secrets Management
- Hardening
Runtime & Record
What is protected in place, and what is written down
- Encryption
- Audit Logging
- Security Monitoring
Infrastructure
Cloud environment, workloads, pipelines, data, and the paths between them — the thing all three boundaries exist to hold.
Controls are placed at the boundary they belong to rather than gathered into a checklist at the end. A control added after the architecture is settled can usually only be as narrow as the architecture already allows — which is why the ordering matters more than the list.
08Engagement Model
Engineer the foundation. Then keep it evolving
Two ways this work runs. Which one fits depends on whether the system above the infrastructure has stopped changing, and for most organisations running production software, it has not.
Project Engagement
For new infrastructure, cloud architecture, migration, deployment systems, security hardening, or resilience initiatives.
- Research
- Architecture
- Engineering
- Deploy
Engineering Retainer
For organizations that need continued infrastructure engineering as applications, traffic, integrations, and requirements change.
- Monitor
- Improve
- Secure
- Evolve
Neither of these is a package or a fixed scope. Both start with the same conversation about your environment, and what the engagement contains is decided from what that conversation finds.
09Existing Environment
Improve what you already have
Infrastructure work does not always mean starting over. We can assess and improve existing environments where the underlying architecture is still viable. What an assessment typically finds, and what changes.
AreaAs foundAfter engineering
Existing Cloud Environments
As foundResources created over time by different people for different reasons, with no single definition of the whole.
AfterA described environment with the accidental parts separated from the intentional ones, and a sequence for changing them.
Legacy Infrastructure
As foundWorking systems nobody wants to touch, usually because the consequences of touching them are unknown.
AfterKnown consequences: dependencies mapped, a rollback that exists, and change made possible again without a rewrite.
Existing CI/CD
As foundA pipeline that builds and deploys, with the checks that matter either missing or bypassable.
AfterThe same pipeline with mandatory validation, a real rollback path, and no default route around either.
Container Platforms
As foundOrchestration adopted for a scale the system has not reached, or outgrown by one it has.
AfterAn honest read on whether the platform is earning its operational cost, and the smaller or larger thing that would.
Security Posture
As foundBroad network access and broad credentials, because both were the fastest way to get the first version running.
AfterSegmentation and scoped credentials introduced in an order that does not require an outage to adopt.
Monitoring
As foundInfrastructure metrics that say the host is alive, and nothing that says the system is working.
AfterSignals tied to what users actually experience, with thresholds set against measured behaviour.
Recovery Planning
As foundA document, written once, referencing infrastructure that has since changed.
AfterA restore path executed against the current architecture, with what it exposed already fixed.
10Long-Term Engineering
Production infrastructure keeps changing
Applications evolve, traffic changes, vulnerabilities emerge, cloud environments shift, and operational requirements grow. Ongoing engineering keeps the foundation aligned with the system above it.
Infrastructure Optimization
Capacity, configuration, and cost revisited against how the system is actually used rather than how it was predicted to be.
Security Updates
Patching, dependency currency, credential rotation, and control changes as the environment and its exposure change.
Pipeline Improvements
New checks, faster feedback, and deployment strategies that keep pace with how often the team needs to release.
Observability Evolution
Signals and alerts adjusted as the system changes shape — an alert set for last year's architecture is noise in this one.
Capacity Changes
Scaling behaviour adjusted for real traffic, seasonality, and the workloads the business adds after launch.
Recovery Testing
Restore paths re-executed against the current architecture, because the plan ages every time the system changes.
Architecture Evolution
Restructuring what the original design has outgrown — early, while it is still a change and not a migration.
Operational Support
Engineering availability for the infrastructure questions and incidents that come with running a production system.
11Possible Outcomes
What the engagement can produce
The artefacts this work tends to leave behind. Which of them exist at the end of a given engagement is a function of where it started.
Cloud Architecture
The environment design, its boundaries, and the trade-offs behind it written down — including the options that were rejected.
Infrastructure as Code
Environment definitions in version control, reviewable and able to recreate what they describe.
CI/CD Pipeline
The build, validation, and deployment path, with its mandatory checks and its rollback.
Observability Setup
Logs, metrics, traces, dashboards, and alerting thresholds tied to real system behaviour.
Security Hardening
Access scoping, network boundaries, encryption, and secret handling implemented in the environment.
Recovery Plan
Restore paths and dependency order, with evidence from having run them.
Deployment Architecture
How releases reach production, and what happens automatically when one goes wrong.
Operational Documentation
What the team that owns this afterwards needs in order to actually own it.
To be exact — actual outputs depend on the existing environment, technical requirements, and engagement scope. This is what the work can produce, not a list of what every engagement includes.
12Common Questions
Questions worth asking
13Start With the Foundation
Is your infrastructure ready for what comes next?
Tell us where your infrastructure creates risk, friction, or uncertainty. We'll assess the existing environment and determine what actually needs to change.
