qvest-task/docs/diagrams/update-workflow.md

Update Workflow

This diagram shows the complete automated update workflow for the Gitea deployment, including update detection, automatic and manual update paths, rollback procedures, and certificate renewal.

Overview

• Diun monitors for container updates weekly (Sunday 3:00 AM)
• Automatic updates for low-risk containers (nginx)
• Manual approval required for critical containers (gitea, postgres)
• Backup before update with automatic rollback on failure
• Certificate renewal runs separately (Sunday 3:30 AM)
• Email notifications for all significant events

Update Workflow Diagram

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e5e7eb','primaryTextColor':'#111827','primaryBorderColor':'#9ca3af','lineColor':'#111827','secondaryColor':'#d1d5db','tertiaryColor':'#f3f4f6','edgeLabelBackground':'#ffffff','mainBkg':'#f5f5f4','nodeBorder':'#9ca3af','background':'#f5f5f4','clusterBkg':'transparent'},'themeCSS':'.node rect, .node circle, .node ellipse, .node polygon, .node path { filter: none !important; box-shadow: none !important; } .cluster rect { filter: none !important; box-shadow: none !important; } svg { background-color: #f5f5f4 !important; } .cluster-label { background-color: #ffffff !important; padding: 6px 12px !important; border-radius: 4px !important; font-size: 16px !important; font-weight: 700 !important; box-shadow: 0 1px 3px rgba(0,0,0,0.12) !important; border: 1px solid #d1d5db !important; } .edgePath, .edgePath path, .flowchart-link { z-index: 1 !important; }'}}%%

flowchart TD
    Start([Sunday 3:00 AM<br/>Cron Trigger])
    Diun{Diun<br/>Check Updates}
    Policy{Update Policy?}
    
    %% Automatic Path (nginx)
    AutoEmail[📧 Email: nginx update available]
    AutoCron([auto-update.sh<br/>Cron Execution])
    AutoBackup[🗄️ Backup Database & Data<br/>to S3]
    AutoBackupFail{Backup<br/>Success?}
    AutoPull[📥 Pull New Image<br/>nginx:latest-version]
    AutoRecreate[🔄 Recreate Container<br/>docker compose up]
    AutoHealth{Health Check<br/>Pass?}
    AutoRollback[↩️ Rollback<br/>Restore Previous Image]
    AutoRollbackHealth{Rollback<br/>Health OK?}
    AutoSuccess[✅ Update Complete<br/>Log Success]
    AutoFailEmail[📧 Email: Update Failed<br/>System Rolled Back]
    
    %% Manual Path (gitea/postgres)
    ManualEmail[📧 Email: Critical Update Available<br/>gitea or postgres]
    OperatorReview{Operator<br/>Reviews & Approves}
    ManualRun([Operator runs<br/>manual-update.sh])
    ManualConfirm{Confirm<br/>Update?}
    ManualBackup[🗄️ Backup Database & Data<br/>to S3]
    ManualBackupFail{Backup<br/>Success?}
    ManualPull[📥 Pull New Image<br/>gitea:x.y.z or postgres:x.y]
    ManualRecreate[🔄 Recreate Container<br/>docker compose up]
    ManualHealth{Health Check<br/>Pass?}
    ManualRollback[↩️ Rollback<br/>Restore Previous Image]
    ManualRollbackHealth{Rollback<br/>Health OK?}
    ManualSuccess[✅ Update Complete<br/>Email Success]
    ManualFailEmail[📧 Email: Update Failed<br/>System Rolled Back]
    ManualAbort[❌ Update Aborted]
    
    %% Certificate Renewal Path
    CertStart([Sunday 3:30 AM<br/>Cron Trigger])
    CertRenew[🔐 Certbot Renew<br/>docker compose run certbot]
    CertCheck{Certificate<br/>Renewed?}
    CertRestart[🔄 Restart nginx<br/>docker compose restart]
    CertSuccess[✅ Certificate Valid]
    CertSkip[ℹ️ No Renewal Needed]
    
    %% Flow connections
    Start --> Diun
    Diun -->|Updates Found| Policy
    Diun -->|No Updates| End1[End]
    
    %% Automatic Path
    Policy -->|automatic<br/>nginx| AutoEmail
    AutoEmail --> AutoCron
    AutoCron --> AutoBackup
    AutoBackup --> AutoBackupFail
    AutoBackupFail -->|❌ Failed| AutoFailEmail
    AutoFailEmail --> End2[End]
    AutoBackupFail -->|✅ Success| AutoPull
    AutoPull --> AutoRecreate
    AutoRecreate --> AutoHealth
    AutoHealth -->|✅ Healthy| AutoSuccess
    AutoSuccess --> End3[End]
    AutoHealth -->|❌ Unhealthy| AutoRollback
    AutoRollback --> AutoRollbackHealth
    AutoRollbackHealth -->|✅ Healthy| AutoFailEmail
    AutoRollbackHealth -->|❌ Still Failed| AutoFailEmail
    
    %% Manual Path
    Policy -->|manual<br/>gitea/postgres| ManualEmail
    ManualEmail --> OperatorReview
    OperatorReview -->|Later| End4[End]
    OperatorReview -->|Now| ManualRun
    ManualRun --> ManualConfirm
    ManualConfirm -->|No| ManualAbort
    ManualAbort --> End5[End]
    ManualConfirm -->|Yes| ManualBackup
    ManualBackup --> ManualBackupFail
    ManualBackupFail -->|❌ Failed| ManualFailEmail
    ManualFailEmail --> End6[End]
    ManualBackupFail -->|✅ Success| ManualPull
    ManualPull --> ManualRecreate
    ManualRecreate --> ManualHealth
    ManualHealth -->|✅ Healthy| ManualSuccess
    ManualSuccess --> End7[End]
    ManualHealth -->|❌ Unhealthy| ManualRollback
    ManualRollback --> ManualRollbackHealth
    ManualRollbackHealth -->|✅ Healthy| ManualFailEmail
    ManualRollbackHealth -->|❌ Still Failed| ManualFailEmail
    
    %% Certificate Renewal Path (separate flow)
    CertStart --> CertRenew
    CertRenew --> CertCheck
    CertCheck -->|New Cert| CertRestart
    CertRestart --> CertSuccess
    CertSuccess --> End8[End]
    CertCheck -->|Not Due| CertSkip
    CertSkip --> End9[End]
    
    %% Styling
    classDef trigger fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
    classDef decision fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
    classDef action fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#ffffff
    classDef success fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
    classDef failure fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#ffffff
    classDef operator fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#ffffff
    classDef monitor fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
    classDef email fill:#6366F1,stroke:#4338CA,stroke-width:2px,color:#ffffff
    classDef backup fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#ffffff
    
    class Start,AutoCron,ManualRun,CertStart trigger
    class Diun,Policy,AutoBackupFail,AutoHealth,AutoRollbackHealth,ManualBackupFail,ManualHealth,ManualRollbackHealth,OperatorReview,ManualConfirm,CertCheck monitor
    class AutoBackup,AutoPull,AutoRecreate,AutoRollback,ManualBackup,ManualPull,ManualRecreate,ManualRollback,CertRenew,CertRestart action
    class AutoSuccess,ManualSuccess,CertSuccess,CertSkip success
    class AutoFailEmail,ManualFailEmail,ManualAbort failure
    class AutoEmail,ManualEmail email

Update Policies

Automatic (Low Risk)

• nginx: Reverse proxy with stateless configuration
• Process: Detected → Backup → Update → Health Check → Success/Rollback
• No operator intervention required

Manual (High Risk)

• gitea: Git hosting application with user data
• postgres: Database containing all repository data
• Process: Detected → Email → Operator Reviews → Approval → Backup → Update → Health Check → Success/Rollback

Safety Mechanisms

1. Pre-Update Backup: Database and Gitea data backed up to S3 before any changes
2. Health Checks: Services validated after update (container running, postgres responding, gitea accessible, nginx config valid)
3. Automatic Rollback: Failed health check triggers immediate rollback to previous image
4. Email Notifications: Operator notified of:
   • Available updates (manual containers)
   • Update failures (all containers)
   • Successful updates (manual containers only)

Certificate Renewal

Runs separately at 3:30 AM on Sundays:

• Certbot checks certificate expiration
• Renews if within 30 days of expiry
• Restarts nginx to load new certificate
• Process is idempotent (safe to run weekly)

Monitoring

Diun Configuration:

• Schedule: 0 3 * * 0 (Sunday 3:00 AM)
• Monitors: postgres, gitea, nginx, diun
• Email: Via AWS SES SMTP
• Labels: Containers marked with diun.enable=true and update.policy=automatic|manual