docs: update docs.
All checks were successful
Update Automation Tests / Integration Tests (pull_request) Successful in 33s
All checks were successful
Update Automation Tests / Integration Tests (pull_request) Successful in 33s
This commit is contained in:
parent
fbcd561ee8
commit
3821f45d63
46
ADR.md
46
ADR.md
@ -117,7 +117,7 @@ This document tracks all significant architectural decisions made during the pro
|
|||||||
|
|
||||||
## ADR-007: SSL Certificates - Let's Encrypt
|
## ADR-007: SSL Certificates - Let's Encrypt
|
||||||
|
|
||||||
**Date**: 2026-06-08
|
**Date**: 2026-06-08 (Updated 2026-06-11)
|
||||||
**Status**: Accepted
|
**Status**: Accepted
|
||||||
|
|
||||||
**Decision**: Let's Encrypt with certbot
|
**Decision**: Let's Encrypt with certbot
|
||||||
@ -130,6 +130,10 @@ This document tracks all significant architectural decisions made during the pro
|
|||||||
|
|
||||||
**Requirement**: Valid domain name pointing to server
|
**Requirement**: Valid domain name pointing to server
|
||||||
|
|
||||||
|
**Domain**: git.poll-streams.com (changed from gitea.poll-streams.com)
|
||||||
|
|
||||||
|
**Implementation Note**: Initially encountered Let's Encrypt rate limits (5 certificates per week). Resolved by migrating to a fresh domain identifier (git.poll-streams.com), allowing immediate production certificate issuance. Production certificates obtained successfully.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ADR-008: Update Automation - Diun + Custom Scripts
|
## ADR-008: Update Automation - Diun + Custom Scripts
|
||||||
@ -167,6 +171,43 @@ This document tracks all significant architectural decisions made during the pro
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## ADR-012: CI/CD - Gitea Actions with Self-Hosted Runners
|
||||||
|
|
||||||
|
**Date**: 2026-06-11
|
||||||
|
**Status**: Accepted
|
||||||
|
|
||||||
|
**Decision**: Use Gitea Actions with self-hosted runners for CI/CD
|
||||||
|
|
||||||
|
**Rationale**:
|
||||||
|
- Native integration with Gitea (no external CI service)
|
||||||
|
- Self-hosted runners provide full control and security
|
||||||
|
- GitHub Actions-compatible workflow syntax (familiar, well-documented)
|
||||||
|
- Enables automated testing before merging changes
|
||||||
|
- Demonstrates production-grade CI/CD practices
|
||||||
|
|
||||||
|
**Implementation**:
|
||||||
|
- **Runners**: 2x act_runner v0.2.10 instances as systemd services
|
||||||
|
- **Automation**: Ansible playbook (setup-runner.yml) for reproducible deployment
|
||||||
|
- **Runner Registration**: Automated via Gitea API with token from AWS Secrets Manager
|
||||||
|
- **Networking**: Host network mode for job containers to access Gitea
|
||||||
|
- **Registration URL**: https://git.poll-streams.com (public URL for git clone operations)
|
||||||
|
- **Workflow**: .gitea/workflows/test.yml runs integration tests on PRs
|
||||||
|
- **Features**: Docker layer caching, artifact uploads, workflow_dispatch support
|
||||||
|
|
||||||
|
**Technical Details**:
|
||||||
|
- Each runner has dedicated config directory (/etc/act_runner-{1,2})
|
||||||
|
- Configuration includes host networking to allow job containers to reach services
|
||||||
|
- Runners registered with public URL to avoid localhost connection issues
|
||||||
|
- Systemd manages runner lifecycle with automatic restart
|
||||||
|
|
||||||
|
**Benefits**:
|
||||||
|
- Automated quality gates before merging
|
||||||
|
- Consistent test environment (matches CI exactly)
|
||||||
|
- Fast feedback on code changes
|
||||||
|
- Self-contained solution (no external dependencies)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## ADR-009: Monitoring - Prometheus + Grafana
|
## ADR-009: Monitoring - Prometheus + Grafana
|
||||||
|
|
||||||
**Date**: 2026-06-08
|
**Date**: 2026-06-08
|
||||||
@ -236,7 +277,8 @@ This document tracks all significant architectural decisions made during the pro
|
|||||||
| **Reverse Proxy** | Nginx | Lightweight, standard |
|
| **Reverse Proxy** | Nginx | Lightweight, standard |
|
||||||
| **SSL** | Let's Encrypt | Free, automated, professional |
|
| **SSL** | Let's Encrypt | Free, automated, professional |
|
||||||
| **DNS** | Route 53 | AWS-native |
|
| **DNS** | Route 53 | AWS-native |
|
||||||
| **Updates** | Watchtower | Docker-native automation |
|
| **Updates** | Diun + Scripts | Per-container policies, backup/rollback |
|
||||||
|
| **CI/CD** | Gitea Actions | Self-hosted runners, native integration |
|
||||||
| **Backups** | Scripts + S3 | Custom, controlled |
|
| **Backups** | Scripts + S3 | Custom, controlled |
|
||||||
| **Monitoring** | Prometheus + Grafana | Industry standard |
|
| **Monitoring** | Prometheus + Grafana | Industry standard |
|
||||||
| **Logging** | Loki + Promtail | Lightweight, integrated |
|
| **Logging** | Loki + Promtail | Lightweight, integrated |
|
||||||
|
|||||||
23
ROADMAP.md
23
ROADMAP.md
@ -104,7 +104,8 @@ This phase implements the automated, reproducible Gitea installation.
|
|||||||
|
|
||||||
### 3.3 Reverse Proxy Configuration ✅
|
### 3.3 Reverse Proxy Configuration ✅
|
||||||
- ✅ Nginx 1.27-alpine deployed via Docker Compose
|
- ✅ Nginx 1.27-alpine deployed via Docker Compose
|
||||||
- ✅ Let's Encrypt SSL certificate obtained via certbot
|
- ✅ Let's Encrypt SSL certificate obtained via certbot (production)
|
||||||
|
- ✅ Domain: git.poll-streams.com (migrated to avoid rate limits)
|
||||||
- ✅ Two-stage nginx config (HTTP-only for ACME, then HTTPS)
|
- ✅ Two-stage nginx config (HTTP-only for ACME, then HTTPS)
|
||||||
- ✅ SSL termination at nginx, proxy to Gitea on port 3000
|
- ✅ SSL termination at nginx, proxy to Gitea on port 3000
|
||||||
- ✅ HTTP to HTTPS redirect configured
|
- ✅ HTTP to HTTPS redirect configured
|
||||||
@ -114,7 +115,7 @@ This phase implements the automated, reproducible Gitea installation.
|
|||||||
|
|
||||||
### 3.4 Testing ✅
|
### 3.4 Testing ✅
|
||||||
- ✅ HTTPS access verified: https://git.poll-streams.com
|
- ✅ HTTPS access verified: https://git.poll-streams.com
|
||||||
- ✅ Valid SSL certificate (Let's Encrypt)
|
- ✅ Valid SSL certificate (Let's Encrypt production)
|
||||||
- ✅ HTTP → HTTPS redirect working
|
- ✅ HTTP → HTTPS redirect working
|
||||||
- ✅ Gitea web interface accessible and functional
|
- ✅ Gitea web interface accessible and functional
|
||||||
- ✅ User account created, repository created
|
- ✅ User account created, repository created
|
||||||
@ -191,11 +192,24 @@ This phase implements automated update mechanisms for Gitea and related componen
|
|||||||
- ✅ Diun monitoring confirmed (4 containers)
|
- ✅ Diun monitoring confirmed (4 containers)
|
||||||
- ✅ Update workflow diagram created
|
- ✅ Update workflow diagram created
|
||||||
|
|
||||||
|
### 4.7 CI/CD Implementation ✅
|
||||||
|
- ✅ Gitea Actions enabled on instance
|
||||||
|
- ✅ Self-hosted runners deployed (2x act_runner v0.2.10)
|
||||||
|
- ✅ Runner automation via Ansible (setup-runner.yml)
|
||||||
|
- ✅ Systemd services for runner management
|
||||||
|
- ✅ Host networking configuration for job containers
|
||||||
|
- ✅ CI workflow created (.gitea/workflows/test.yml)
|
||||||
|
- ✅ Automated testing on pull requests
|
||||||
|
- ✅ Docker layer caching for performance
|
||||||
|
- ✅ Artifact upload on test failure
|
||||||
|
- ✅ Full CI/CD pipeline tested and operational
|
||||||
|
|
||||||
### Goals:
|
### Goals:
|
||||||
- ✅ Automated update system operational
|
- ✅ Automated update system operational
|
||||||
- ✅ Update process tested and validated on live system
|
- ✅ Update process tested and validated on live system
|
||||||
- ✅ Rollback procedure implemented and tested
|
- ✅ Rollback procedure implemented and tested
|
||||||
- ✅ Quality gate for CI/local environments
|
- ✅ Quality gate for CI/local environments
|
||||||
|
- ✅ CI/CD pipeline with self-hosted runners
|
||||||
- ✅ Documentation complete (workflow diagram)
|
- ✅ Documentation complete (workflow diagram)
|
||||||
|
|
||||||
**Implementation Summary:**
|
**Implementation Summary:**
|
||||||
@ -205,9 +219,10 @@ This phase implements automated update mechanisms for Gitea and related componen
|
|||||||
- Pre-update backups with automatic rollback on failure
|
- Pre-update backups with automatic rollback on failure
|
||||||
- Certificate renewal automation
|
- Certificate renewal automation
|
||||||
- Comprehensive testing framework
|
- Comprehensive testing framework
|
||||||
- Visual workflow documentation
|
- CI/CD with Gitea Actions and 2 self-hosted runners
|
||||||
|
- Visual workflow documentation (including CI/CD flow)
|
||||||
|
|
||||||
**Phase 4 Complete!** Update automation fully operational with safety mechanisms.
|
**Phase 4 Complete!** Update automation and CI/CD fully operational with safety mechanisms.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@ -12,46 +12,72 @@ graph TB
|
|||||||
subgraph EC2["EC2 Instance"]
|
subgraph EC2["EC2 Instance"]
|
||||||
subgraph Docker["Docker Compose"]
|
subgraph Docker["Docker Compose"]
|
||||||
Nginx[Nginx<br/>Port 80, 443]
|
Nginx[Nginx<br/>Port 80, 443]
|
||||||
Gitea[Gitea<br/>Port 3000]
|
Gitea[Gitea<br/>Port 3000, 2222]
|
||||||
Postgres[(PostgreSQL<br/>Port 5432)]
|
Postgres[(PostgreSQL<br/>Port 5432)]
|
||||||
Watchtower[Watchtower<br/>Auto-updater]
|
Certbot[Certbot<br/>SSL Renewal]
|
||||||
|
DIUN[DIUN<br/>Update Monitor]
|
||||||
|
|
||||||
Nginx -->|Reverse Proxy| Gitea
|
Nginx -->|Reverse Proxy| Gitea
|
||||||
Gitea -->|Database Connection| Postgres
|
Gitea -->|Database Connection| Postgres
|
||||||
Watchtower -.->|Monitors & Updates| Nginx
|
DIUN -.->|Monitors for Updates| Nginx
|
||||||
Watchtower -.->|Monitors & Updates| Gitea
|
DIUN -.->|Monitors for Updates| Gitea
|
||||||
|
DIUN -.->|Monitors for Updates| Postgres
|
||||||
|
Certbot -.->|Renews Certificates| Nginx
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Systemd["Systemd Services"]
|
||||||
|
Runner1[act_runner-1<br/>CI/CD Runner]
|
||||||
|
Runner2[act_runner-2<br/>CI/CD Runner]
|
||||||
|
|
||||||
|
Runner1 -.->|Executes Workflows| Gitea
|
||||||
|
Runner2 -.->|Executes Workflows| Gitea
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
User -->|HTTPS| Nginx
|
User -->|HTTPS| Nginx
|
||||||
LetsEncrypt -.->|Certbot Renewal| Nginx
|
User -->|Git SSH| Gitea
|
||||||
|
LetsEncrypt -.->|Certificate Authority| Certbot
|
||||||
|
|
||||||
style EC2 fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
style EC2 fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||||
style Docker fill:#d1d5db,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
style Docker fill:#d1d5db,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||||
|
style Systemd fill:#d1d5db,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||||
|
|
||||||
style Nginx fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
style Nginx fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
||||||
style Gitea fill:#3B82F6,stroke:#333,stroke-width:1px,color:#fff
|
style Gitea fill:#3B82F6,stroke:#333,stroke-width:1px,color:#fff
|
||||||
style Postgres fill:#8B5CF6,stroke:#333,stroke-width:1px,color:#fff
|
style Postgres fill:#8B5CF6,stroke:#333,stroke-width:1px,color:#fff
|
||||||
style Watchtower fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
style DIUN fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
||||||
|
style Certbot fill:#6366F1,stroke:#333,stroke-width:1px,color:#fff
|
||||||
|
style Runner1 fill:#EF4444,stroke:#333,stroke-width:1px,color:#fff
|
||||||
|
style Runner2 fill:#EF4444,stroke:#333,stroke-width:1px,color:#fff
|
||||||
```
|
```
|
||||||
|
|
||||||
## Components
|
## Components
|
||||||
|
|
||||||
|
### Docker Containers
|
||||||
- **Nginx**: Reverse proxy handling SSL termination and routing to Gitea
|
- **Nginx**: Reverse proxy handling SSL termination and routing to Gitea
|
||||||
- **Gitea**: Git server application (main service)
|
- **Gitea**: Git server application with Actions enabled (HTTP: 3000, SSH: 2222)
|
||||||
- **PostgreSQL**: Database storing repositories metadata, users, issues
|
- **PostgreSQL**: Database storing repositories metadata, users, issues
|
||||||
- **Watchtower**: Monitors Docker Hub for image updates, automatically pulls and restarts containers
|
- **DIUN**: Monitors Docker Hub for image updates, sends email notifications
|
||||||
|
- **Certbot**: Handles Let's Encrypt SSL certificate renewal
|
||||||
|
|
||||||
|
### Systemd Services
|
||||||
|
- **act_runner-1**: First Gitea Actions runner for CI/CD workflows
|
||||||
|
- **act_runner-2**: Second Gitea Actions runner for CI/CD workflows
|
||||||
|
|
||||||
## Container Communication
|
## Container Communication
|
||||||
|
|
||||||
- All containers in the same Docker network
|
- All containers in the same Docker network (`gitea-network`)
|
||||||
- Nginx proxies HTTPS requests to Gitea's internal port 3000
|
- Nginx proxies HTTPS requests to Gitea's internal port 3000
|
||||||
- Gitea connects to PostgreSQL via container name
|
- Gitea connects to PostgreSQL via container name (`postgres`)
|
||||||
- Watchtower runs on schedule, checking for updates
|
- DIUN monitors containers based on labels (`diun.enable=true`)
|
||||||
- Let's Encrypt certbot renews certificates automatically (via nginx container or separate container)
|
- Certbot shares volumes with nginx for certificate storage
|
||||||
|
- Runners connect to Gitea via `http://localhost:3000`
|
||||||
|
|
||||||
## Data Persistence
|
## Data Persistence
|
||||||
|
|
||||||
Docker volumes ensure data survives container restarts:
|
Docker volumes ensure data survives container restarts:
|
||||||
- `gitea_data`: Git repositories and uploads
|
- `gitea-data`: Git repositories and uploads
|
||||||
- `postgres_data`: Database files
|
- `gitea_postgres-data`: PostgreSQL database files
|
||||||
|
- `certbot-etc`: Let's Encrypt certificates
|
||||||
|
- `certbot-var`: Certbot working directory
|
||||||
|
- `web-root`: ACME challenge files for SSL verification
|
||||||
|
|||||||
@ -8,12 +8,17 @@ This diagram shows the high-level AWS resources and their relationships.
|
|||||||
graph TB
|
graph TB
|
||||||
Internet([Internet/Users])
|
Internet([Internet/Users])
|
||||||
Route53[Route 53<br/>DNS]
|
Route53[Route 53<br/>DNS]
|
||||||
EC2[EC2 Instance<br/>Docker Host]
|
EC2[EC2 Instance<br/>Docker Host + Runners]
|
||||||
S3[(S3 Bucket<br/>Backups)]
|
S3[(S3 Bucket<br/>Backups)]
|
||||||
|
Secrets[AWS Secrets Manager<br/>DB/Admin Credentials]
|
||||||
|
IAM[IAM Role<br/>EC2 Permissions]
|
||||||
|
|
||||||
Internet -->|HTTPS| Route53
|
Internet -->|HTTPS| Route53
|
||||||
Route53 -->|DNS Resolution| EC2
|
Route53 -->|DNS Resolution| EC2
|
||||||
EC2 -->|Backup Upload| S3
|
EC2 -->|Backup Upload| S3
|
||||||
|
EC2 -->|Fetch Credentials| Secrets
|
||||||
|
IAM -.->|Attached to| EC2
|
||||||
|
EC2 -->|Update Runner Token| Secrets
|
||||||
|
|
||||||
subgraph AWS["AWS Account"]
|
subgraph AWS["AWS Account"]
|
||||||
subgraph VPC["VPC"]
|
subgraph VPC["VPC"]
|
||||||
@ -21,6 +26,8 @@ graph TB
|
|||||||
end
|
end
|
||||||
Route53
|
Route53
|
||||||
S3
|
S3
|
||||||
|
Secrets
|
||||||
|
IAM
|
||||||
end
|
end
|
||||||
|
|
||||||
style AWS fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
style AWS fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||||
@ -29,18 +36,24 @@ graph TB
|
|||||||
style EC2 fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
style EC2 fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
||||||
style S3 fill:#F97316,stroke:#333,stroke-width:1px,color:#fff
|
style S3 fill:#F97316,stroke:#333,stroke-width:1px,color:#fff
|
||||||
style Route53 fill:#6366F1,stroke:#333,stroke-width:1px,color:#fff
|
style Route53 fill:#6366F1,stroke:#333,stroke-width:1px,color:#fff
|
||||||
|
style Secrets fill:#8B5CF6,stroke:#333,stroke-width:1px,color:#fff
|
||||||
|
style IAM fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
||||||
```
|
```
|
||||||
|
|
||||||
## Components
|
## Components
|
||||||
|
|
||||||
- **Route 53**: DNS service that points domain to EC2 instance
|
- **Route 53**: DNS service that points domain to EC2 instance
|
||||||
- **EC2 Instance**: Single VM running Docker with all application containers
|
- **EC2 Instance**: Single VM running Docker containers + 2 Gitea Actions runners (systemd services)
|
||||||
- **S3 Bucket**: Storage for database and application backups
|
- **S3 Bucket**: Storage for database and application backups (with versioning)
|
||||||
|
- **AWS Secrets Manager**: Stores DB credentials, admin credentials, SES SMTP credentials, runner tokens
|
||||||
|
- **IAM Role**: EC2 instance profile with permissions for S3, Secrets Manager read/update
|
||||||
- **VPC**: Isolated network containing EC2 instance
|
- **VPC**: Isolated network containing EC2 instance
|
||||||
|
|
||||||
## Traffic Flow
|
## Traffic Flow
|
||||||
|
|
||||||
1. User accesses `gitea.yourdomain.com`
|
1. User accesses `git.poll-streams.com`
|
||||||
2. Route 53 resolves to EC2 public IP
|
2. Route 53 resolves to EC2 public IP
|
||||||
3. Request hits EC2 (nginx handles SSL, proxies to Gitea)
|
3. Request hits EC2 (nginx handles SSL, proxies to Gitea)
|
||||||
4. EC2 regularly backs up data to S3
|
4. EC2 regularly backs up data to S3
|
||||||
|
5. Ansible fetches credentials from Secrets Manager during deployment
|
||||||
|
6. Gitea generates runner token via API, stored back in Secrets Manager
|
||||||
|
|||||||
242
docs/diagrams/ci-cd-workflow.md
Normal file
242
docs/diagrams/ci-cd-workflow.md
Normal file
@ -0,0 +1,242 @@
|
|||||||
|
# CI/CD Workflow with Gitea Actions
|
||||||
|
|
||||||
|
This diagram shows the complete CI/CD workflow using Gitea Actions with self-hosted runners, including the automated setup process.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
- **Gitea Actions**: GitHub Actions-compatible CI/CD built into Gitea
|
||||||
|
- **Self-hosted runners**: 2 act_runner instances running as systemd services
|
||||||
|
- **Automated setup**: Admin user, runner tokens, and registration fully automated via Ansible
|
||||||
|
- **Test workflow**: Integration tests run on every PR to main branch
|
||||||
|
|
||||||
|
## CI/CD Workflow Diagram
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e5e7eb','primaryTextColor':'#111827','primaryBorderColor':'#9ca3af','lineColor':'#111827','secondaryColor':'#d1d5db','tertiaryColor':'#f3f4f6','edgeLabelBackground':'#ffffff','mainBkg':'#f5f5f4','nodeBorder':'#9ca3af','background':'#f5f5f4','clusterBkg':'transparent'},'themeCSS':'.node rect, .node circle, .node ellipse, .node polygon, .node path { filter: none !important; box-shadow: none !important; } .cluster rect { filter: none !important; box-shadow: none !important; } svg { background-color: #f5f5f4 !important; } .cluster-label { background-color: #ffffff !important; padding: 6px 12px !important; border-radius: 4px !important; font-size: 16px !important; font-weight: 700 !important; box-shadow: 0 1px 3px rgba(0,0,0,0.12) !important; border: 1px solid #d1d5db !important; } .edgePath, .edgePath path, .flowchart-link { z-index: 1 !important; }'}}%%
|
||||||
|
|
||||||
|
flowchart TB
|
||||||
|
Dev([Developer])
|
||||||
|
|
||||||
|
subgraph Workflow["CI/CD Workflow"]
|
||||||
|
Push[Git Push / PR Created]
|
||||||
|
Trigger{Gitea Actions<br/>Workflow Trigger}
|
||||||
|
Queue[Job Queued]
|
||||||
|
|
||||||
|
subgraph Runners["Self-Hosted Runners"]
|
||||||
|
Runner1[act_runner-1<br/>systemd service]
|
||||||
|
Runner2[act_runner-2<br/>systemd service]
|
||||||
|
end
|
||||||
|
|
||||||
|
Pick{Runner<br/>Available?}
|
||||||
|
Checkout[📥 Checkout Code]
|
||||||
|
Cache[💾 Setup Docker Cache]
|
||||||
|
Pull[📥 Pre-pull Test Images<br/>postgres:18.4, nginx:1.27-alpine, alpine:3.19/3.20]
|
||||||
|
Test[🧪 Run Integration Tests<br/>scripts/test-update.sh]
|
||||||
|
TestResult{Tests<br/>Pass?}
|
||||||
|
Success[✅ Report Success<br/>PR can merge]
|
||||||
|
Failure[❌ Report Failure<br/>Upload test logs]
|
||||||
|
Artifact[📦 Upload Artifacts<br/>7-day retention]
|
||||||
|
end
|
||||||
|
|
||||||
|
Dev -->|git push| Push
|
||||||
|
Push --> Trigger
|
||||||
|
Trigger -->|PR to main| Queue
|
||||||
|
Trigger -->|workflow_dispatch| Queue
|
||||||
|
Queue --> Pick
|
||||||
|
Pick -->|Assigns Job| Runner1
|
||||||
|
Pick -->|Assigns Job| Runner2
|
||||||
|
Runner1 --> Checkout
|
||||||
|
Runner2 --> Checkout
|
||||||
|
Checkout --> Cache
|
||||||
|
Cache --> Pull
|
||||||
|
Pull --> Test
|
||||||
|
Test --> TestResult
|
||||||
|
TestResult -->|✅ All Pass| Success
|
||||||
|
TestResult -->|❌ Any Fail| Failure
|
||||||
|
Failure --> Artifact
|
||||||
|
|
||||||
|
style Dev fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||||
|
style Push fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Trigger fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||||
|
style Queue fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||||
|
style Pick fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||||
|
style Runner1 fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||||
|
style Runner2 fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||||
|
style Checkout fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Cache fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Pull fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Test fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style TestResult fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||||
|
style Success fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||||
|
style Failure fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||||
|
style Artifact fill:#6366F1,stroke:#4338CA,stroke-width:2px,color:#fff
|
||||||
|
```
|
||||||
|
|
||||||
|
## Automated Setup Flow
|
||||||
|
|
||||||
|
This diagram shows how the runner infrastructure is automatically provisioned and configured.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e5e7eb','primaryTextColor':'#111827','primaryBorderColor':'#9ca3af','lineColor':'#111827','secondaryColor':'#d1d5db','tertiaryColor':'#f3f4f6','edgeLabelBackground':'#ffffff','mainBkg':'#f5f5f4','nodeBorder':'#9ca3af','background':'#f5f5f4','clusterBkg':'transparent'},'themeCSS':'.node rect, .node circle, .node ellipse, .node polygon, .node path { filter: none !important; box-shadow: none !important; } .cluster rect { filter: none !important; box-shadow: none !important; } svg { background-color: #f5f5f4 !important; } .cluster-label { background-color: #ffffff !important; padding: 6px 12px !important; border-radius: 4px !important; font-size: 16px !important; font-weight: 700 !important; box-shadow: 0 1px 3px rgba(0,0,0,0.12) !important; border: 1px solid #d1d5db !important; } .edgePath, .edgePath path, .flowchart-link { z-index: 1 !important; }'}}%%
|
||||||
|
|
||||||
|
flowchart TD
|
||||||
|
Start([Terraform Apply])
|
||||||
|
Secrets[🔐 Create AWS Secrets<br/>DB credentials, Admin credentials]
|
||||||
|
EC2[🖥️ Provision EC2 Instance<br/>With IAM role for Secrets Manager]
|
||||||
|
|
||||||
|
Ansible([Ansible Playbook])
|
||||||
|
Deploy[📦 Deploy Gitea<br/>docker-compose up]
|
||||||
|
Wait[⏳ Wait for Gitea<br/>HTTP 200 response]
|
||||||
|
CreateUser[👤 Create Admin User<br/>docker exec gitea gitea admin user create]
|
||||||
|
DisableChange[🔓 Disable Password Change<br/>UPDATE user SET must_change_password=false]
|
||||||
|
GenToken[🎟️ Generate Runner Token<br/>GET /api/v1/admin/runners/registration-token]
|
||||||
|
UpdateSecret[💾 Store Token in Secrets Manager<br/>aws secretsmanager update-secret]
|
||||||
|
|
||||||
|
DownloadRunner[📥 Download act_runner v0.2.10]
|
||||||
|
CreateDirs[📁 Create /etc/act_runner-{1,2}]
|
||||||
|
FetchToken[🔍 Fetch Runner Token<br/>from Secrets Manager]
|
||||||
|
RegisterRunner[📝 Register Runners<br/>act_runner register --instance http://localhost:3000]
|
||||||
|
CreateService[⚙️ Create systemd services<br/>act_runner-1.service, act_runner-2.service]
|
||||||
|
StartService[▶️ Enable & Start Services]
|
||||||
|
|
||||||
|
Complete([✅ Ready for CI/CD])
|
||||||
|
|
||||||
|
Start --> Secrets
|
||||||
|
Secrets --> EC2
|
||||||
|
EC2 --> Ansible
|
||||||
|
|
||||||
|
Ansible --> Deploy
|
||||||
|
Deploy --> Wait
|
||||||
|
Wait --> CreateUser
|
||||||
|
CreateUser --> DisableChange
|
||||||
|
DisableChange --> GenToken
|
||||||
|
GenToken --> UpdateSecret
|
||||||
|
|
||||||
|
UpdateSecret --> DownloadRunner
|
||||||
|
DownloadRunner --> CreateDirs
|
||||||
|
CreateDirs --> FetchToken
|
||||||
|
FetchToken --> RegisterRunner
|
||||||
|
RegisterRunner --> CreateService
|
||||||
|
CreateService --> StartService
|
||||||
|
|
||||||
|
StartService --> Complete
|
||||||
|
|
||||||
|
style Start fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||||
|
style Secrets fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||||
|
style EC2 fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||||
|
style Ansible fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||||
|
style Deploy fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Wait fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style CreateUser fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style DisableChange fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style GenToken fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style UpdateSecret fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||||
|
style DownloadRunner fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style CreateDirs fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style FetchToken fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||||
|
style RegisterRunner fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style CreateService fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style StartService fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||||
|
style Complete fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workflow Configuration
|
||||||
|
|
||||||
|
The CI/CD workflow is defined in `.gitea/workflows/test.yml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
name: Integration Tests
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Docker Buildx
|
||||||
|
uses: docker/setup-buildx-action@v3
|
||||||
|
|
||||||
|
- name: Cache Docker layers
|
||||||
|
uses: actions/cache@v4
|
||||||
|
with:
|
||||||
|
path: /tmp/.buildx-cache
|
||||||
|
key: ${{ runner.os }}-buildx-${{ github.sha }}
|
||||||
|
restore-keys: ${{ runner.os }}-buildx-
|
||||||
|
|
||||||
|
- name: Pre-pull test images
|
||||||
|
run: |
|
||||||
|
docker pull postgres:18.4
|
||||||
|
docker pull nginx:1.27-alpine
|
||||||
|
docker pull alpine:3.19
|
||||||
|
docker pull alpine:3.20
|
||||||
|
|
||||||
|
- name: Run integration tests
|
||||||
|
run: ./scripts/test-update.sh
|
||||||
|
|
||||||
|
- name: Upload test logs
|
||||||
|
if: failure()
|
||||||
|
uses: actions/upload-artifact@v4
|
||||||
|
with:
|
||||||
|
name: test-logs
|
||||||
|
path: /tmp/test-*.log
|
||||||
|
retention-days: 7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test Suite
|
||||||
|
|
||||||
|
The `scripts/test-update.sh` integration test suite validates:
|
||||||
|
|
||||||
|
1. **Static validation** (2 tests):
|
||||||
|
- Script syntax and linting
|
||||||
|
- Required executables available
|
||||||
|
|
||||||
|
2. **Docker-based tests** (12 tests):
|
||||||
|
- PostgreSQL backup and restore
|
||||||
|
- Health check functionality
|
||||||
|
- Archive validation (SQL and tar formats)
|
||||||
|
- Update simulation workflow
|
||||||
|
- Container cleanup and resource management
|
||||||
|
|
||||||
|
All tests must pass for a PR to be mergeable.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
### Zero-Configuration CI/CD
|
||||||
|
- Runners automatically registered during initial deployment
|
||||||
|
- No manual token management needed
|
||||||
|
- Runner tokens stored securely in AWS Secrets Manager
|
||||||
|
- Complete automation from infrastructure provision to working CI/CD
|
||||||
|
|
||||||
|
### High Availability
|
||||||
|
- 2 concurrent runners for parallel job execution
|
||||||
|
- Automatic job distribution by Gitea
|
||||||
|
- Systemd ensures runners restart on failure
|
||||||
|
|
||||||
|
### Security
|
||||||
|
- Runners use local Gitea instance (`http://localhost:3000`)
|
||||||
|
- Admin credentials never exposed (CLI-based user creation)
|
||||||
|
- IAM roles for least-privilege access to AWS resources
|
||||||
|
- Runner tokens rotated on redeployment
|
||||||
|
|
||||||
|
### Docker Optimization
|
||||||
|
- Docker layer caching for faster builds
|
||||||
|
- Image pre-pulling reduces test execution time
|
||||||
|
- Shared Docker daemon for all tests
|
||||||
|
|
||||||
|
## Deployment Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Full deployment (includes runner setup)
|
||||||
|
make full-deploy
|
||||||
|
|
||||||
|
# Update only configuration (re-registers runners if needed)
|
||||||
|
make configure
|
||||||
|
|
||||||
|
# Run tests locally
|
||||||
|
make test
|
||||||
|
```
|
||||||
@ -58,17 +58,19 @@ graph TB
|
|||||||
**EC2 Security Group**:
|
**EC2 Security Group**:
|
||||||
- **Inbound Rules**:
|
- **Inbound Rules**:
|
||||||
- Port 22 (SSH): From admin IP only (for management)
|
- Port 22 (SSH): From admin IP only (for management)
|
||||||
- Port 80 (HTTP): From 0.0.0.0/0 (redirects to HTTPS)
|
- Port 80 (HTTP): From 0.0.0.0/0 (redirects to HTTPS, ACME challenge)
|
||||||
- Port 443 (HTTPS): From 0.0.0.0/0 (Gitea access)
|
- Port 443 (HTTPS): From 0.0.0.0/0 (Gitea web access)
|
||||||
|
- Port 2222 (Git SSH): From 0.0.0.0/0 (Git push/pull via SSH)
|
||||||
- **Outbound Rules**:
|
- **Outbound Rules**:
|
||||||
- All traffic: To 0.0.0.0/0 (for updates, backups to S3)
|
- All traffic: To 0.0.0.0/0 (for updates, backups to S3, Secrets Manager)
|
||||||
|
|
||||||
## Security Considerations
|
## Security Considerations
|
||||||
|
|
||||||
1. **SSH Access**: Restricted to specific admin IP address (your IP)
|
1. **SSH Access**: Restricted to specific admin IP address (your IP)
|
||||||
2. **HTTP/HTTPS**: Open to internet (required for Gitea web access)
|
2. **HTTP/HTTPS**: Open to internet (required for Gitea web access)
|
||||||
3. **No Direct Gitea Access**: Port 3000 not exposed; only nginx on 80/443
|
3. **Git SSH**: Port 2222 exposed for Git operations over SSH
|
||||||
4. **Outbound**: Allowed for Docker image pulls, package updates, S3 backups
|
4. **No Direct Gitea HTTP Access**: Port 3000 not exposed; only nginx on 80/443
|
||||||
|
5. **Outbound**: Allowed for Docker image pulls, package updates, S3 backups, AWS API calls
|
||||||
|
|
||||||
## Traffic Flow
|
## Traffic Flow
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user