docs: update docs.
All checks were successful
Update Automation Tests / Integration Tests (pull_request) Successful in 33s
All checks were successful
Update Automation Tests / Integration Tests (pull_request) Successful in 33s
This commit is contained in:
aviyadeveloper
parent
fbcd561ee8
commit
3821f45d63
46
ADR.md
46
ADR.md
@ -117,7 +117,7 @@ This document tracks all significant architectural decisions made during the pro
|
||||
|
||||
## ADR-007: SSL Certificates - Let's Encrypt
|
||||
|
||||
**Date**: 2026-06-08
|
||||
**Date**: 2026-06-08 (Updated 2026-06-11)
|
||||
**Status**: Accepted
|
||||
|
||||
**Decision**: Let's Encrypt with certbot
|
||||
@ -130,6 +130,10 @@ This document tracks all significant architectural decisions made during the pro
|
||||
|
||||
**Requirement**: Valid domain name pointing to server
|
||||
|
||||
**Domain**: git.poll-streams.com (changed from gitea.poll-streams.com)
|
||||
|
||||
**Implementation Note**: Initially encountered Let's Encrypt rate limits (5 certificates per week). Resolved by migrating to a fresh domain identifier (git.poll-streams.com), allowing immediate production certificate issuance. Production certificates obtained successfully.
|
||||
|
||||
---
|
||||
|
||||
## ADR-008: Update Automation - Diun + Custom Scripts
|
||||
@ -167,6 +171,43 @@ This document tracks all significant architectural decisions made during the pro
|
||||
|
||||
---
|
||||
|
||||
## ADR-012: CI/CD - Gitea Actions with Self-Hosted Runners
|
||||
|
||||
**Date**: 2026-06-11
|
||||
**Status**: Accepted
|
||||
|
||||
**Decision**: Use Gitea Actions with self-hosted runners for CI/CD
|
||||
|
||||
**Rationale**:
|
||||
- Native integration with Gitea (no external CI service)
|
||||
- Self-hosted runners provide full control and security
|
||||
- GitHub Actions-compatible workflow syntax (familiar, well-documented)
|
||||
- Enables automated testing before merging changes
|
||||
- Demonstrates production-grade CI/CD practices
|
||||
|
||||
**Implementation**:
|
||||
- **Runners**: 2x act_runner v0.2.10 instances as systemd services
|
||||
- **Automation**: Ansible playbook (setup-runner.yml) for reproducible deployment
|
||||
- **Runner Registration**: Automated via Gitea API with token from AWS Secrets Manager
|
||||
- **Networking**: Host network mode for job containers to access Gitea
|
||||
- **Registration URL**: https://git.poll-streams.com (public URL for git clone operations)
|
||||
- **Workflow**: .gitea/workflows/test.yml runs integration tests on PRs
|
||||
- **Features**: Docker layer caching, artifact uploads, workflow_dispatch support
|
||||
|
||||
**Technical Details**:
|
||||
- Each runner has dedicated config directory (/etc/act_runner-{1,2})
|
||||
- Configuration includes host networking to allow job containers to reach services
|
||||
- Runners registered with public URL to avoid localhost connection issues
|
||||
- Systemd manages runner lifecycle with automatic restart
|
||||
|
||||
**Benefits**:
|
||||
- Automated quality gates before merging
|
||||
- Consistent test environment (matches CI exactly)
|
||||
- Fast feedback on code changes
|
||||
- Self-contained solution (no external dependencies)
|
||||
|
||||
---
|
||||
|
||||
## ADR-009: Monitoring - Prometheus + Grafana
|
||||
|
||||
**Date**: 2026-06-08
|
||||
@ -236,7 +277,8 @@ This document tracks all significant architectural decisions made during the pro
|
||||
| **Reverse Proxy** | Nginx | Lightweight, standard |
|
||||
| **SSL** | Let's Encrypt | Free, automated, professional |
|
||||
| **DNS** | Route 53 | AWS-native |
|
||||
| **Updates** | Watchtower | Docker-native automation |
|
||||
| **Updates** | Diun + Scripts | Per-container policies, backup/rollback |
|
||||
| **CI/CD** | Gitea Actions | Self-hosted runners, native integration |
|
||||
| **Backups** | Scripts + S3 | Custom, controlled |
|
||||
| **Monitoring** | Prometheus + Grafana | Industry standard |
|
||||
| **Logging** | Loki + Promtail | Lightweight, integrated |
|
||||
|
||||
23
ROADMAP.md
23
ROADMAP.md
@ -104,7 +104,8 @@ This phase implements the automated, reproducible Gitea installation.
|
||||
|
||||
### 3.3 Reverse Proxy Configuration ✅
|
||||
- ✅ Nginx 1.27-alpine deployed via Docker Compose
|
||||
- ✅ Let's Encrypt SSL certificate obtained via certbot
|
||||
- ✅ Let's Encrypt SSL certificate obtained via certbot (production)
|
||||
- ✅ Domain: git.poll-streams.com (migrated to avoid rate limits)
|
||||
- ✅ Two-stage nginx config (HTTP-only for ACME, then HTTPS)
|
||||
- ✅ SSL termination at nginx, proxy to Gitea on port 3000
|
||||
- ✅ HTTP to HTTPS redirect configured
|
||||
@ -114,7 +115,7 @@ This phase implements the automated, reproducible Gitea installation.
|
||||
|
||||
### 3.4 Testing ✅
|
||||
- ✅ HTTPS access verified: https://git.poll-streams.com
|
||||
- ✅ Valid SSL certificate (Let's Encrypt)
|
||||
- ✅ Valid SSL certificate (Let's Encrypt production)
|
||||
- ✅ HTTP → HTTPS redirect working
|
||||
- ✅ Gitea web interface accessible and functional
|
||||
- ✅ User account created, repository created
|
||||
@ -191,11 +192,24 @@ This phase implements automated update mechanisms for Gitea and related componen
|
||||
- ✅ Diun monitoring confirmed (4 containers)
|
||||
- ✅ Update workflow diagram created
|
||||
|
||||
### 4.7 CI/CD Implementation ✅
|
||||
- ✅ Gitea Actions enabled on instance
|
||||
- ✅ Self-hosted runners deployed (2x act_runner v0.2.10)
|
||||
- ✅ Runner automation via Ansible (setup-runner.yml)
|
||||
- ✅ Systemd services for runner management
|
||||
- ✅ Host networking configuration for job containers
|
||||
- ✅ CI workflow created (.gitea/workflows/test.yml)
|
||||
- ✅ Automated testing on pull requests
|
||||
- ✅ Docker layer caching for performance
|
||||
- ✅ Artifact upload on test failure
|
||||
- ✅ Full CI/CD pipeline tested and operational
|
||||
|
||||
### Goals:
|
||||
- ✅ Automated update system operational
|
||||
- ✅ Update process tested and validated on live system
|
||||
- ✅ Rollback procedure implemented and tested
|
||||
- ✅ Quality gate for CI/local environments
|
||||
- ✅ CI/CD pipeline with self-hosted runners
|
||||
- ✅ Documentation complete (workflow diagram)
|
||||
|
||||
**Implementation Summary:**
|
||||
@ -205,9 +219,10 @@ This phase implements automated update mechanisms for Gitea and related componen
|
||||
- Pre-update backups with automatic rollback on failure
|
||||
- Certificate renewal automation
|
||||
- Comprehensive testing framework
|
||||
- Visual workflow documentation
|
||||
- CI/CD with Gitea Actions and 2 self-hosted runners
|
||||
- Visual workflow documentation (including CI/CD flow)
|
||||
|
||||
**Phase 4 Complete!** Update automation fully operational with safety mechanisms.
|
||||
**Phase 4 Complete!** Update automation and CI/CD fully operational with safety mechanisms.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@ -12,46 +12,72 @@ graph TB
|
||||
subgraph EC2["EC2 Instance"]
|
||||
subgraph Docker["Docker Compose"]
|
||||
Nginx[Nginx<br/>Port 80, 443]
|
||||
Gitea[Gitea<br/>Port 3000]
|
||||
Gitea[Gitea<br/>Port 3000, 2222]
|
||||
Postgres[(PostgreSQL<br/>Port 5432)]
|
||||
Watchtower[Watchtower<br/>Auto-updater]
|
||||
Certbot[Certbot<br/>SSL Renewal]
|
||||
DIUN[DIUN<br/>Update Monitor]
|
||||
|
||||
Nginx -->|Reverse Proxy| Gitea
|
||||
Gitea -->|Database Connection| Postgres
|
||||
Watchtower -.->|Monitors & Updates| Nginx
|
||||
Watchtower -.->|Monitors & Updates| Gitea
|
||||
DIUN -.->|Monitors for Updates| Nginx
|
||||
DIUN -.->|Monitors for Updates| Gitea
|
||||
DIUN -.->|Monitors for Updates| Postgres
|
||||
Certbot -.->|Renews Certificates| Nginx
|
||||
end
|
||||
|
||||
subgraph Systemd["Systemd Services"]
|
||||
Runner1[act_runner-1<br/>CI/CD Runner]
|
||||
Runner2[act_runner-2<br/>CI/CD Runner]
|
||||
|
||||
Runner1 -.->|Executes Workflows| Gitea
|
||||
Runner2 -.->|Executes Workflows| Gitea
|
||||
end
|
||||
end
|
||||
|
||||
User -->|HTTPS| Nginx
|
||||
LetsEncrypt -.->|Certbot Renewal| Nginx
|
||||
User -->|Git SSH| Gitea
|
||||
LetsEncrypt -.->|Certificate Authority| Certbot
|
||||
|
||||
style EC2 fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||
style Docker fill:#d1d5db,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||
style Systemd fill:#d1d5db,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||
|
||||
style Nginx fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Gitea fill:#3B82F6,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Postgres fill:#8B5CF6,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Watchtower fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
||||
style DIUN fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Certbot fill:#6366F1,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Runner1 fill:#EF4444,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Runner2 fill:#EF4444,stroke:#333,stroke-width:1px,color:#fff
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### Docker Containers
|
||||
- **Nginx**: Reverse proxy handling SSL termination and routing to Gitea
|
||||
- **Gitea**: Git server application (main service)
|
||||
- **Gitea**: Git server application with Actions enabled (HTTP: 3000, SSH: 2222)
|
||||
- **PostgreSQL**: Database storing repositories metadata, users, issues
|
||||
- **Watchtower**: Monitors Docker Hub for image updates, automatically pulls and restarts containers
|
||||
- **DIUN**: Monitors Docker Hub for image updates, sends email notifications
|
||||
- **Certbot**: Handles Let's Encrypt SSL certificate renewal
|
||||
|
||||
### Systemd Services
|
||||
- **act_runner-1**: First Gitea Actions runner for CI/CD workflows
|
||||
- **act_runner-2**: Second Gitea Actions runner for CI/CD workflows
|
||||
|
||||
## Container Communication
|
||||
|
||||
- All containers in the same Docker network
|
||||
- All containers in the same Docker network (`gitea-network`)
|
||||
- Nginx proxies HTTPS requests to Gitea's internal port 3000
|
||||
- Gitea connects to PostgreSQL via container name
|
||||
- Watchtower runs on schedule, checking for updates
|
||||
- Let's Encrypt certbot renews certificates automatically (via nginx container or separate container)
|
||||
- Gitea connects to PostgreSQL via container name (`postgres`)
|
||||
- DIUN monitors containers based on labels (`diun.enable=true`)
|
||||
- Certbot shares volumes with nginx for certificate storage
|
||||
- Runners connect to Gitea via `http://localhost:3000`
|
||||
|
||||
## Data Persistence
|
||||
|
||||
Docker volumes ensure data survives container restarts:
|
||||
- `gitea_data`: Git repositories and uploads
|
||||
- `postgres_data`: Database files
|
||||
- `gitea-data`: Git repositories and uploads
|
||||
- `gitea_postgres-data`: PostgreSQL database files
|
||||
- `certbot-etc`: Let's Encrypt certificates
|
||||
- `certbot-var`: Certbot working directory
|
||||
- `web-root`: ACME challenge files for SSL verification
|
||||
|
||||
@ -8,12 +8,17 @@ This diagram shows the high-level AWS resources and their relationships.
|
||||
graph TB
|
||||
Internet([Internet/Users])
|
||||
Route53[Route 53<br/>DNS]
|
||||
EC2[EC2 Instance<br/>Docker Host]
|
||||
EC2[EC2 Instance<br/>Docker Host + Runners]
|
||||
S3[(S3 Bucket<br/>Backups)]
|
||||
Secrets[AWS Secrets Manager<br/>DB/Admin Credentials]
|
||||
IAM[IAM Role<br/>EC2 Permissions]
|
||||
|
||||
Internet -->|HTTPS| Route53
|
||||
Route53 -->|DNS Resolution| EC2
|
||||
EC2 -->|Backup Upload| S3
|
||||
EC2 -->|Fetch Credentials| Secrets
|
||||
IAM -.->|Attached to| EC2
|
||||
EC2 -->|Update Runner Token| Secrets
|
||||
|
||||
subgraph AWS["AWS Account"]
|
||||
subgraph VPC["VPC"]
|
||||
@ -21,6 +26,8 @@ graph TB
|
||||
end
|
||||
Route53
|
||||
S3
|
||||
Secrets
|
||||
IAM
|
||||
end
|
||||
|
||||
style AWS fill:#e5e7eb,stroke:#4b5563,stroke-width:2px,stroke-dasharray: 5 5
|
||||
@ -29,18 +36,24 @@ graph TB
|
||||
style EC2 fill:#10B981,stroke:#333,stroke-width:1px,color:#fff
|
||||
style S3 fill:#F97316,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Route53 fill:#6366F1,stroke:#333,stroke-width:1px,color:#fff
|
||||
style Secrets fill:#8B5CF6,stroke:#333,stroke-width:1px,color:#fff
|
||||
style IAM fill:#F59E0B,stroke:#333,stroke-width:1px,color:#fff
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
- **Route 53**: DNS service that points domain to EC2 instance
|
||||
- **EC2 Instance**: Single VM running Docker with all application containers
|
||||
- **S3 Bucket**: Storage for database and application backups
|
||||
- **EC2 Instance**: Single VM running Docker containers + 2 Gitea Actions runners (systemd services)
|
||||
- **S3 Bucket**: Storage for database and application backups (with versioning)
|
||||
- **AWS Secrets Manager**: Stores DB credentials, admin credentials, SES SMTP credentials, runner tokens
|
||||
- **IAM Role**: EC2 instance profile with permissions for S3, Secrets Manager read/update
|
||||
- **VPC**: Isolated network containing EC2 instance
|
||||
|
||||
## Traffic Flow
|
||||
|
||||
1. User accesses `gitea.yourdomain.com`
|
||||
1. User accesses `git.poll-streams.com`
|
||||
2. Route 53 resolves to EC2 public IP
|
||||
3. Request hits EC2 (nginx handles SSL, proxies to Gitea)
|
||||
4. EC2 regularly backs up data to S3
|
||||
5. Ansible fetches credentials from Secrets Manager during deployment
|
||||
6. Gitea generates runner token via API, stored back in Secrets Manager
|
||||
|
||||
242
docs/diagrams/ci-cd-workflow.md
Normal file
242
docs/diagrams/ci-cd-workflow.md
Normal file
@ -0,0 +1,242 @@
|
||||
# CI/CD Workflow with Gitea Actions
|
||||
|
||||
This diagram shows the complete CI/CD workflow using Gitea Actions with self-hosted runners, including the automated setup process.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Gitea Actions**: GitHub Actions-compatible CI/CD built into Gitea
|
||||
- **Self-hosted runners**: 2 act_runner instances running as systemd services
|
||||
- **Automated setup**: Admin user, runner tokens, and registration fully automated via Ansible
|
||||
- **Test workflow**: Integration tests run on every PR to main branch
|
||||
|
||||
## CI/CD Workflow Diagram
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e5e7eb','primaryTextColor':'#111827','primaryBorderColor':'#9ca3af','lineColor':'#111827','secondaryColor':'#d1d5db','tertiaryColor':'#f3f4f6','edgeLabelBackground':'#ffffff','mainBkg':'#f5f5f4','nodeBorder':'#9ca3af','background':'#f5f5f4','clusterBkg':'transparent'},'themeCSS':'.node rect, .node circle, .node ellipse, .node polygon, .node path { filter: none !important; box-shadow: none !important; } .cluster rect { filter: none !important; box-shadow: none !important; } svg { background-color: #f5f5f4 !important; } .cluster-label { background-color: #ffffff !important; padding: 6px 12px !important; border-radius: 4px !important; font-size: 16px !important; font-weight: 700 !important; box-shadow: 0 1px 3px rgba(0,0,0,0.12) !important; border: 1px solid #d1d5db !important; } .edgePath, .edgePath path, .flowchart-link { z-index: 1 !important; }'}}%%
|
||||
|
||||
flowchart TB
|
||||
Dev([Developer])
|
||||
|
||||
subgraph Workflow["CI/CD Workflow"]
|
||||
Push[Git Push / PR Created]
|
||||
Trigger{Gitea Actions<br/>Workflow Trigger}
|
||||
Queue[Job Queued]
|
||||
|
||||
subgraph Runners["Self-Hosted Runners"]
|
||||
Runner1[act_runner-1<br/>systemd service]
|
||||
Runner2[act_runner-2<br/>systemd service]
|
||||
end
|
||||
|
||||
Pick{Runner<br/>Available?}
|
||||
Checkout[📥 Checkout Code]
|
||||
Cache[💾 Setup Docker Cache]
|
||||
Pull[📥 Pre-pull Test Images<br/>postgres:18.4, nginx:1.27-alpine, alpine:3.19/3.20]
|
||||
Test[🧪 Run Integration Tests<br/>scripts/test-update.sh]
|
||||
TestResult{Tests<br/>Pass?}
|
||||
Success[✅ Report Success<br/>PR can merge]
|
||||
Failure[❌ Report Failure<br/>Upload test logs]
|
||||
Artifact[📦 Upload Artifacts<br/>7-day retention]
|
||||
end
|
||||
|
||||
Dev -->|git push| Push
|
||||
Push --> Trigger
|
||||
Trigger -->|PR to main| Queue
|
||||
Trigger -->|workflow_dispatch| Queue
|
||||
Queue --> Pick
|
||||
Pick -->|Assigns Job| Runner1
|
||||
Pick -->|Assigns Job| Runner2
|
||||
Runner1 --> Checkout
|
||||
Runner2 --> Checkout
|
||||
Checkout --> Cache
|
||||
Cache --> Pull
|
||||
Pull --> Test
|
||||
Test --> TestResult
|
||||
TestResult -->|✅ All Pass| Success
|
||||
TestResult -->|❌ Any Fail| Failure
|
||||
Failure --> Artifact
|
||||
|
||||
style Dev fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||
style Push fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Trigger fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||
style Queue fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||
style Pick fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||
style Runner1 fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||
style Runner2 fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||
style Checkout fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Cache fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Pull fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Test fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style TestResult fill:#F97316,stroke:#C2410C,stroke-width:2px,color:#111827
|
||||
style Success fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||
style Failure fill:#EF4444,stroke:#B91C1C,stroke-width:2px,color:#fff
|
||||
style Artifact fill:#6366F1,stroke:#4338CA,stroke-width:2px,color:#fff
|
||||
```
|
||||
|
||||
## Automated Setup Flow
|
||||
|
||||
This diagram shows how the runner infrastructure is automatically provisioned and configured.
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e5e7eb','primaryTextColor':'#111827','primaryBorderColor':'#9ca3af','lineColor':'#111827','secondaryColor':'#d1d5db','tertiaryColor':'#f3f4f6','edgeLabelBackground':'#ffffff','mainBkg':'#f5f5f4','nodeBorder':'#9ca3af','background':'#f5f5f4','clusterBkg':'transparent'},'themeCSS':'.node rect, .node circle, .node ellipse, .node polygon, .node path { filter: none !important; box-shadow: none !important; } .cluster rect { filter: none !important; box-shadow: none !important; } svg { background-color: #f5f5f4 !important; } .cluster-label { background-color: #ffffff !important; padding: 6px 12px !important; border-radius: 4px !important; font-size: 16px !important; font-weight: 700 !important; box-shadow: 0 1px 3px rgba(0,0,0,0.12) !important; border: 1px solid #d1d5db !important; } .edgePath, .edgePath path, .flowchart-link { z-index: 1 !important; }'}}%%
|
||||
|
||||
flowchart TD
|
||||
Start([Terraform Apply])
|
||||
Secrets[🔐 Create AWS Secrets<br/>DB credentials, Admin credentials]
|
||||
EC2[🖥️ Provision EC2 Instance<br/>With IAM role for Secrets Manager]
|
||||
|
||||
Ansible([Ansible Playbook])
|
||||
Deploy[📦 Deploy Gitea<br/>docker-compose up]
|
||||
Wait[⏳ Wait for Gitea<br/>HTTP 200 response]
|
||||
CreateUser[👤 Create Admin User<br/>docker exec gitea gitea admin user create]
|
||||
DisableChange[🔓 Disable Password Change<br/>UPDATE user SET must_change_password=false]
|
||||
GenToken[🎟️ Generate Runner Token<br/>GET /api/v1/admin/runners/registration-token]
|
||||
UpdateSecret[💾 Store Token in Secrets Manager<br/>aws secretsmanager update-secret]
|
||||
|
||||
DownloadRunner[📥 Download act_runner v0.2.10]
|
||||
CreateDirs[📁 Create /etc/act_runner-{1,2}]
|
||||
FetchToken[🔍 Fetch Runner Token<br/>from Secrets Manager]
|
||||
RegisterRunner[📝 Register Runners<br/>act_runner register --instance http://localhost:3000]
|
||||
CreateService[⚙️ Create systemd services<br/>act_runner-1.service, act_runner-2.service]
|
||||
StartService[▶️ Enable & Start Services]
|
||||
|
||||
Complete([✅ Ready for CI/CD])
|
||||
|
||||
Start --> Secrets
|
||||
Secrets --> EC2
|
||||
EC2 --> Ansible
|
||||
|
||||
Ansible --> Deploy
|
||||
Deploy --> Wait
|
||||
Wait --> CreateUser
|
||||
CreateUser --> DisableChange
|
||||
DisableChange --> GenToken
|
||||
GenToken --> UpdateSecret
|
||||
|
||||
UpdateSecret --> DownloadRunner
|
||||
DownloadRunner --> CreateDirs
|
||||
CreateDirs --> FetchToken
|
||||
FetchToken --> RegisterRunner
|
||||
RegisterRunner --> CreateService
|
||||
CreateService --> StartService
|
||||
|
||||
StartService --> Complete
|
||||
|
||||
style Start fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||
style Secrets fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||
style EC2 fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||
style Ansible fill:#F59E0B,stroke:#B45309,stroke-width:2px,color:#111827
|
||||
style Deploy fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Wait fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style CreateUser fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style DisableChange fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style GenToken fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style UpdateSecret fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||
style DownloadRunner fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style CreateDirs fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style FetchToken fill:#8B5CF6,stroke:#6D28D9,stroke-width:2px,color:#fff
|
||||
style RegisterRunner fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style CreateService fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style StartService fill:#3B82F6,stroke:#1D4ED8,stroke-width:2px,color:#fff
|
||||
style Complete fill:#10B981,stroke:#047857,stroke-width:2px,color:#111827
|
||||
```
|
||||
|
||||
## Workflow Configuration
|
||||
|
||||
The CI/CD workflow is defined in `.gitea/workflows/test.yml`:
|
||||
|
||||
```yaml
|
||||
name: Integration Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Cache Docker layers
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: /tmp/.buildx-cache
|
||||
key: ${{ runner.os }}-buildx-${{ github.sha }}
|
||||
restore-keys: ${{ runner.os }}-buildx-
|
||||
|
||||
- name: Pre-pull test images
|
||||
run: |
|
||||
docker pull postgres:18.4
|
||||
docker pull nginx:1.27-alpine
|
||||
docker pull alpine:3.19
|
||||
docker pull alpine:3.20
|
||||
|
||||
- name: Run integration tests
|
||||
run: ./scripts/test-update.sh
|
||||
|
||||
- name: Upload test logs
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-logs
|
||||
path: /tmp/test-*.log
|
||||
retention-days: 7
|
||||
```
|
||||
|
||||
## Test Suite
|
||||
|
||||
The `scripts/test-update.sh` integration test suite validates:
|
||||
|
||||
1. **Static validation** (2 tests):
|
||||
- Script syntax and linting
|
||||
- Required executables available
|
||||
|
||||
2. **Docker-based tests** (12 tests):
|
||||
- PostgreSQL backup and restore
|
||||
- Health check functionality
|
||||
- Archive validation (SQL and tar formats)
|
||||
- Update simulation workflow
|
||||
- Container cleanup and resource management
|
||||
|
||||
All tests must pass for a PR to be mergeable.
|
||||
|
||||
## Key Features
|
||||
|
||||
### Zero-Configuration CI/CD
|
||||
- Runners automatically registered during initial deployment
|
||||
- No manual token management needed
|
||||
- Runner tokens stored securely in AWS Secrets Manager
|
||||
- Complete automation from infrastructure provision to working CI/CD
|
||||
|
||||
### High Availability
|
||||
- 2 concurrent runners for parallel job execution
|
||||
- Automatic job distribution by Gitea
|
||||
- Systemd ensures runners restart on failure
|
||||
|
||||
### Security
|
||||
- Runners use local Gitea instance (`http://localhost:3000`)
|
||||
- Admin credentials never exposed (CLI-based user creation)
|
||||
- IAM roles for least-privilege access to AWS resources
|
||||
- Runner tokens rotated on redeployment
|
||||
|
||||
### Docker Optimization
|
||||
- Docker layer caching for faster builds
|
||||
- Image pre-pulling reduces test execution time
|
||||
- Shared Docker daemon for all tests
|
||||
|
||||
## Deployment Commands
|
||||
|
||||
```bash
|
||||
# Full deployment (includes runner setup)
|
||||
make full-deploy
|
||||
|
||||
# Update only configuration (re-registers runners if needed)
|
||||
make configure
|
||||
|
||||
# Run tests locally
|
||||
make test
|
||||
```
|
||||
@ -58,17 +58,19 @@ graph TB
|
||||
**EC2 Security Group**:
|
||||
- **Inbound Rules**:
|
||||
- Port 22 (SSH): From admin IP only (for management)
|
||||
- Port 80 (HTTP): From 0.0.0.0/0 (redirects to HTTPS)
|
||||
- Port 443 (HTTPS): From 0.0.0.0/0 (Gitea access)
|
||||
- Port 80 (HTTP): From 0.0.0.0/0 (redirects to HTTPS, ACME challenge)
|
||||
- Port 443 (HTTPS): From 0.0.0.0/0 (Gitea web access)
|
||||
- Port 2222 (Git SSH): From 0.0.0.0/0 (Git push/pull via SSH)
|
||||
- **Outbound Rules**:
|
||||
- All traffic: To 0.0.0.0/0 (for updates, backups to S3)
|
||||
- All traffic: To 0.0.0.0/0 (for updates, backups to S3, Secrets Manager)
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **SSH Access**: Restricted to specific admin IP address (your IP)
|
||||
2. **HTTP/HTTPS**: Open to internet (required for Gitea web access)
|
||||
3. **No Direct Gitea Access**: Port 3000 not exposed; only nginx on 80/443
|
||||
4. **Outbound**: Allowed for Docker image pulls, package updates, S3 backups
|
||||
3. **Git SSH**: Port 2222 exposed for Git operations over SSH
|
||||
4. **No Direct Gitea HTTP Access**: Port 3000 not exposed; only nginx on 80/443
|
||||
5. **Outbound**: Allowed for Docker image pulls, package updates, S3 backups, AWS API calls
|
||||
|
||||
## Traffic Flow
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user