Commit → Build → Test → Security Scan → Deploy → Monitor
  ↓        ↓       ↓         ↓           ↓        ↓
  Git    Compile  Unit    SAST/DAST  Staging  Metrics
                  Integration           Prod

2000s: Manual deployments
  - FTP uploads
  - Manual testing
  - 1-2 releases per year
  - High risk, high effort

2010s: Continuous Integration
  - Jenkins, Travis CI emerge
  - Automated testing
  - Version control integration
  - 1-2 releases per month

2015+: Full CI/CD & DevOps
  - Containerization (Docker)
  - Kubernetes orchestration
  - GitOps principles
  - 10-100 releases per day
  - Infrastructure as Code

Choose Jenkins if:
- On-premises required
- Full customization needed
- Complex legacy systems
- High compliance requirements

Choose GitLab CI if:
- All-in-one DevOps platform needed
- Self-hosted or cloud option desired
- GitOps workflow preferred
- Container-native architecture

Choose GitHub Actions if:
- GitHub already primary repository
- Minimal setup desired
- Open source projects
- Cost-conscious, free tier sufficient

Choose Azure Pipelines if:
- Microsoft stack (Azure, Teams, Office 365)
- Enterprise Windows/C# development
- MSDN subscription available
- Integrated with Azure infrastructure

Choose GCP Cloud Build if:
- Google Cloud Platform primary
- Kubernetes (GKE) primary platform
- Multi-cloud build needed
- Container registry (GCR/Artifact Registry)

Git Commit → Webhook → Pipeline Triggered
└─ Branch strategy (main, develop, feature)
└─ PR approval gates
└─ Code review integration

Checkout Code → Compile → Package → Artifact Storage
└─ Language: Java, Python, Go, Node.js, C#, etc.
└─ Dependency resolution
└─ Version tagging
└─ Artifact: JAR, Docker image, ZIP, etc.

Unit Tests → Integration Tests → E2E Tests
├─ Minimum 80% code coverage
├─ Performance benchmarks
└─ Test data management

SAST → DAST → Dependency Scan → Container Scan
├─ Static analysis (SonarQube, Checkmarx)
├─ Dynamic analysis (OWASP ZAP)
├─ CVE vulnerability detection
└─ Bill of Materials (SBOM)

Docker Image → Registry Push
├─ Tagging: v1.2.3, latest, stable
├─ Registry: Docker Hub, ECR, GCR, ACR
└─ Immutable image reference

Deploy to QA → Smoke Tests → Approval Gate
├─ Infrastructure provisioning
├─ Configuration management
└─ Manual sign-off (optional)

Production Deployment → Health Checks → Monitoring
├─ Blue-green / Canary / Rolling
├─ Automated rollback
└─ Production monitoring

Logs → Metrics → Alerts → Incidents
├─ Application metrics (response time, error rate)
├─ Infrastructure metrics (CPU, memory, disk)
├─ Business metrics (user signups, conversion)
└─ Alert escalation

# Multi-stage build (optimized)
FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o app

# Final stage (small image)
FROM alpine:3.18
RUN apk add --no-cache ca-certificates
COPY --from=builder /app/app .
EXPOSE 8080
CMD ["./app"]

# Image size: 15MB (vs 1.2GB with full Go SDK)

Image Naming:
registry.example.com/namespace/service:tag
  ├─ registry: Docker Hub, ECR, GCR, ACR
  ├─ namespace: company, team, project
  ├─ service: app name
  └─ tag: v1.2.3, latest, stable, 2024-01-15

Tagging Strategy:
- Semantic: v1.2.3 (breaking.feature.patch)
- Timestamp: 2024-01-15-14-30-45
- SHA: git-abc1234567890def
- Branch: main, develop, feature-x
- Quality: latest, stable, canary

Registry Security:
- Image scanning on push (CVE detection)
- Retention policies (delete old images)
- Access control (IAM, service accounts)
- Signed images (Docker Content Trust)

Blue (Current)  →  Green (New)  →  Switch Traffic
Version 1.0          Version 2.0      to Green

Benefits:
✓ Zero-downtime deployments
✓ Easy rollback (switch back to Blue)
✓ Test in production-like environment
✓ User acceptance testing on Green

Challenges:
✗ Requires 2x infrastructure
✗ Database migration complexity
✗ Cache invalidation

When to use:
- Large systems with high availability
- Frequent deployments (multiple/day)
- Database schema changes infrequent

Old (95%)  +  New (5%)  →  Monitor  →  Gradual Shift
Version 1.0    Version 2.0   Metrics    5% → 25% → 50% → 100%

Benefits:
✓ Gradual rollout reduces risk
✓ Real user feedback early
✓ Automatic rollback if errors detected
✓ Minimal infrastructure overhead

Challenges:
✗ Complex monitoring setup
✗ Session affinity needed
✗ Stateful services difficult

When to use:
- Risk-averse deployments
- Gradual feature rollouts
- A/B testing scenarios

Instance 1 → Drain → Update → Bring Up
Instance 2 → Drain → Update → Bring Up
Instance 3 → Drain → Update → Bring Up

Benefits:
✓ No downtime
✓ Gradual resource recovery
✓ Load balancer handles traffic

Challenges:
✗ Multiple versions running
✗ Data migration complex
✗ Debugging harder

When to use:
- Kubernetes deployments
- Stateless services
- Frequent updates

1. Git is Source of Truth
   - All configuration in Git repo
   - Git history = audit trail
   - Rollback = git revert

2. Declarative Description
   - YAML defines desired state
   - System reconciles to match
   - No imperative scripts

3. Continuous Synchronization
   - Pull model (not push)
   - Watch for drift
   - Auto-correct or alert

4. Git Workflows
   - PR approval before deploy
   - Code review + CI/CD gates
   - Traceability for compliance

Before code runs, scan for vulnerabilities:

Tools:
- SonarQube (universal)
- Checkmarx (comprehensive)
- GitHub CodeQL (GitHub-native)
- Snyk (dependencies)
- Fortify (enterprise)

Checks:
✓ SQL injection patterns
✓ XSS vulnerabilities
✓ Buffer overflow risks
✓ Hardcoded secrets
✓ Insecure crypto
✓ Code quality metrics

Test running application:

Tools:
- OWASP ZAP (free)
- Burp Suite (commercial)
- Rapid7 InsightAppSec
- Qualys ASPM

Checks:
✓ Authentication bypass
✓ Injection attacks
✓ Broken access control
✓ API security
✓ Session management
✓ Encryption validation

WRONG (Do NOT):
- Hardcoded in code
- Committed to Git
- Stored in config files
- Visible in logs

RIGHT:
- Vault (HashiCorp)
- AWS Secrets Manager
- Azure Key Vault
- Google Secret Manager
- Environment variables (at runtime)

CI/CD Integration:
1. Pipeline needs secret
2. Request from secrets manager
3. Secret injected at runtime
4. Secret NOT logged/stored
5. Automatic rotation

Container Image Security:
├─ Build provenance tracking
├─ Signed images (Cosign)
├─ SBOM (Software Bill of Materials)
├─ Vulnerability scanning
├─ Registry access control
└─ Policy enforcement

Dependency Security:
├─ Lock files (go.sum, package-lock.json)
├─ Version pinning
├─ Automated updates (Dependabot)
├─ License scanning
└─ CVE monitoring

1. Latency
   - Request response time
   - P50, P95, P99 percentiles
   - SLA: < 200ms for 99th percentile

2. Traffic
   - Requests per second
   - Concurrent users
   - Data throughput

3. Errors
   - Error rate (4xx, 5xx)
   - Exception types
   - Error budget tracking

4. Saturation
   - CPU utilization
   - Memory usage
   - Disk I/O
   - Network bandwidth

Application Code
    ↓
(Prometheus/Datadog/New Relic)
    ↓
Metrics Store
    ↓
Visualization (Grafana/Kibana)
    ↓
Alerting (PagerDuty/Opsgenie)
    ↓
Incident Response

Deployment Dashboard:
- Deployment frequency
- Lead time for changes
- Mean time to recovery (MTTR)
- Change failure rate

Application Dashboard:
- Requests per second
- Error rate
- P99 latency
- Top slowest endpoints

Infrastructure Dashboard:
- CPU utilization (all instances)
- Memory usage
- Disk I/O
- Network throughput

Caching Strategy:
├─ Dependency cache (faster builds)
├─ Container layer cache (faster images)
├─ Build artifact cache
└─ Expected savings: 70-80% build time

Parallel Execution:
├─ Run independent jobs in parallel
├─ Fan-out/fan-in patterns
├─ Expected speedup: 4-8x with 8 parallel jobs

Build Resource Sizing:
├─ Use smaller instances for lightweight builds
├─ Use spot instances for non-production
├─ Scale down when idle
└─ Expected savings: 30-50% on build infrastructure

Docker Image Optimization:
├─ Multi-stage builds (reduce image size)
├─ Alpine base images (5MB vs 200MB)
├─ Remove build tools from final image
└─ Expected savings: 90% smaller images

Remove Redundant Steps:
├─ Skip tests for docs-only changes
├─ Skip deploy for failed builds
├─ Fail fast (stop early on first failure)

Workflow Optimization:
├─ Merge fast paths (unit tests on demand)
├─ Run expensive tests only on main branch
├─ Run E2E tests only before production

Expected Results:
- 50% faster feedback to developers
- 70% reduction in wasted compute
- $500K-2M annual savings (enterprise scale)

Dev → Staging → Production
├─ Dev: Personal development, minimal checks
├─ Staging: Full testing, security scanning
└─ Production: Manual approval, zero downtime

Configuration Management:
├─ Secrets: Dev, Staging, Prod (separate)
├─ Feature flags: Enable/disable in runtime
├─ Infrastructure: IaC with environment overlays
└─ Monitoring: Different alert thresholds per env

Service A (Build → Test → Deploy) → Registry
Service B (Build → Test → Deploy) → Registry
Service C (Build → Test → Deploy) → Registry
             ↓
         Orchestrator (Kubernetes)
             ↓
    Multi-service deployment

Challenges:
✗ Service dependencies
✗ Database migrations
✗ Distributed tracing
✗ API versioning

Solutions:
✓ Contract testing (Consumer-Driven)
✓ Feature flags for compatibility
✓ Backward compatibility requirements
✓ Service mesh (Istio) for traffic management

Traditional (Quarterly):
Jan → Apr → Jul → Oct (4 releases/year)

Agile (Sprint-based):
Every 2 weeks (26 releases/year)

Continuous Deployment:
Multiple times per day (100+ releases/year)

Requirements:
✓ Automated testing (80%+ coverage)
✓ Feature flags (control rollout)
✓ Monitoring (detect issues instantly)
✓ Rollback automation (revert in seconds)
✓ Small, focused changes (easier to debug)

1. Deployment Frequency
   Low:  < 1/month (bottleneck)
   Medium: 1-6/month (acceptable)
   High: Daily (competitive)
   Elite: Multiple daily (leading)

2. Lead Time for Changes
   Low:  > 6 months (major delays)
   Medium: 1-6 months (acceptable)
   High: < 1 month (good)
   Elite: < 1 day (industry leading)

3. Mean Time to Recovery (MTTR)
   Low:  > 6 months (crisis mode)
   Medium: 1-6 months (poor)
   High: < 1 month (good)
   Elite: < 1 hour (excellent)

4. Change Failure Rate
   Low:  > 50% (unreliable)
   Medium: 15-50% (acceptable)
   High: < 15% (good)
   Elite: < 5% (excellent)

Fortune 500 Company:
- Deployment frequency: 1/month
- Lead time: 2 months
- MTTR: 3 days
- Failure rate: 20%

Fast-growing SaaS Startup:
- Deployment frequency: Daily
- Lead time: 1 week
- MTTR: 4 hours
- Failure rate: 8%

Tech Leader (Google, Amazon, Netflix):
- Deployment frequency: Hourly (1000s/day)
- Lead time: Minutes
- MTTR: 15 minutes
- Failure rate: < 3%

Source (CodeCommit/GitHub) →
Build (CodeBuild) →
Deploy (CodeDeploy/ECS/EKS) →
Test (CodeBuild) →
Release (Manual Approval)

Strengths:
✓ Tight AWS integration
✓ Cheap (pay per execution)
✓ Scales automatically

Weaknesses:
✗ Minimal UI
✗ Steeper learning curve
✗ Limited free tier

YAML pipelines (code-as-config) →
Hosted agents or self-hosted →
Deploy to Azure/on-premises/multi-cloud →
Integrated with Azure DevOps

Strengths:
✓ Microsoft ecosystem integration
✓ Free for public/internal
✓ MSDN integration

Weaknesses:
✗ Primarily Azure-focused
✗ Steeper learning curve for non-Microsoft

Trigger from Cloud Source Repos/GitHub →
Build in container (fast startup) →
Push to Artifact Registry/GCR →
Deploy to Cloud Run/GKE/App Engine →
Integrated with GCP services

Strengths:
✓ Container-native (fast)
✓ Seamless GCP integration
✓ Pay-per-minute pricing

Weaknesses:
✗ GCP-centric
✗ Less feature-rich than Jenkins/GitLab

Issue: Intermittent test failures ("flaky tests")
Causes: Race conditions, timing issues, external dependencies
Solutions:
- Isolate tests (no shared state)
- Mock external services
- Increase timeout thresholds
- Retry flaky tests

Issue: Out of memory during builds
Causes: Large test suites, memory leaks, limited heap
Solutions:
- Increase runner memory
- Run tests in parallel (smaller batches)
- Profile memory usage
- Split tests across jobs

Issue: Deployment hangs
Causes: Waiting for resources, health checks timing out
Solutions:
- Check resource availability
- Increase timeout thresholds
- Review load balancer configuration
- Check application startup logs

Issue: Production downtime after deployment
Causes: Faulty update, insufficient testing, traffic surge
Solutions:
- Use blue-green deployment
- Automated rollback on failed health checks
- Canary deployment (risk reduction)
- Load testing before deployment

Issue: Build takes too long (> 30 minutes)
Causes: Sequential execution, no caching, slow tests
Solutions:
- Enable parallel execution
- Implement caching
- Skip unnecessary steps
- Use spot instances (faster hardware)

Issue: Slow feedback loop
Causes: Serial pipeline stages, waiting for resources
Solutions:
- Parallel execution
- Smaller build jobs
- Fast feedback (fail fast)
- Skip heavy tests on every commit

- Manual code merges
- Manual builds
- Manual testing
- Manual deployments
- No automation
- Deployment: 1-2x per quarter
- Incident response: Days

- Automated builds on commit
- Unit testing automated
- Artifact versioning
- Deployment: Still manual
- Deployment: 1-2x per month
- Incident response: Hours

- Full test suite automated (unit, integration, E2E)
- Automated security scanning
- Automated deployments to staging
- Manual production approval
- Deployment: Weekly
- Incident response: Minutes-hours

- Everything automated
- Continuous deployment to production
- Feature flags for gradual rollout
- Automated rollback
- Deployment: Daily
- Incident response: Minutes

- ML-powered deployment decisions
- Automated incident resolution
- Self-healing infrastructure
- Predictive alerts
- Deployment: Multiple daily
- Incident response: Automatic
- Manual: Rare exceptions only