Architecture & Design Patterns Chapter 05
The 1Engage multitenant backend is a Go microservices monorepo built on clean architecture principles. This chapter covers the full system architecture — service structure, layering, the API gateway, multitenancy implementation, configuration, database strategy, inter-service communication, graceful shutdown, observability, and every API endpoint.
On This Page
5.1 Monorepo Structure
Monorepo ConventionService Directory Convention
Every service under apps/ follows an identical directory layout. This convention
means a developer who understands one service can navigate any other without a learning curve.
apps/{service-name}/
├── cmd/
│ ├── server/main.go # HTTP server entry point
│ ├── worker/main.go # Kafka consumer (if needed)
│ ├── seed/main.go # Database seeder
│ └── scheduler/main.go # CronJob entry point
├── config/
│ └── config.go # Env var loading → typed Config struct
├── internal/
│ ├── domain/ # Interface contracts (Service + Repository)
│ ├── model/ # GORM models / DTOs
│ ├── repository/ # Data access (GORM)
│ ├── service/ # Business logic
│ ├── handler/ # HTTP handlers
│ ├── db/ # Database connection factory
│ ├── worker/ # Job dispatcher (broadcast-service only)
│ ├── ratelimit/ # Rate limiting (broadcast-service only)
│ └── job/ # Job definitions (broadcast-service only)
└── migrations/ # SQL migrations (goose)
internal/ directory has compiler-enforced visibility.
Code inside apps/auth-service/internal/ cannot be imported by apps/admin-service/.
This prevents accidental cross-service coupling at the language level.
Shared Packages (pkg/)
Cross-service code lives in pkg/ and is imported by any service that needs it.
Each package has a single, well-defined responsibility:
| Package | Purpose | Used By |
|---|---|---|
pkg/auth |
JWT validation, TenantContext extraction, role checking middleware |
All services |
pkg/crypto |
AES-GCM encryption/decryption for sensitive data at rest | auth-service, admin-service |
pkg/eventbus |
Kafka producer/consumer abstraction, event types, idempotency, retry | All services with Kafka |
pkg/middleware |
Tenant identifier, request ID, CORS middleware | gateway-service |
pkg/shared |
Generic HTTP helpers (SendJSON, SendPaginated), validators, converters |
All services |
pkg/tracing |
OpenTelemetry setup, OTLP exporter config, trace middleware | All services |
pkg/version |
Build-time version injection via -ldflags |
All services |
internal/. If it's used by two or more services, it belongs in pkg/.
This prevents premature abstraction while enabling genuine reuse.
5.2 Layered Architecture (Clean Architecture)
Clean Architecture Dependency InversionThe 4-Layer Flow
Every request passes through exactly four layers, each with a single responsibility:
| Layer | Responsibility | Depends On | Example File |
|---|---|---|---|
| Handler | HTTP parsing, validation, response formatting | domain.Service (interface) |
internal/handler/ |
| Service | Business logic, orchestration, event publishing | domain.Repository (interface) |
internal/service/ |
| Repository | Data access, SQL queries, GORM operations | *gorm.DB (concrete) |
internal/repository/ |
| Domain | Interface contracts — defines what each layer must implement | Nothing (leaf package) | internal/domain/ |
Real Wiring: auth-service
Dependencies are assembled in main.go using plain constructor injection —
no DI framework, no reflection, no magic:
// 1. Create the concrete repository (depends on *gorm.DB)
userRepo := repository.NewUserRepository(gormDB)
// 2. Create the service (depends on domain.UserRepository interface)
userService := service.NewUserService(userRepo, emailWorker, feUrl, kafkaProducer, internalUrl)
// 3. Create the handler (depends on domain.UserService interface)
userHandler := handler.NewUserHandler(userService)
Dependency inversion in action: The Handler depends on domain.UserService
(an interface), not the concrete service.userService struct. The Service depends on
domain.UserRepository (an interface), not the concrete repository. Only
main.go knows the concrete types — the rest of the code is testable in isolation
with mock implementations.
Domain Interface Contract
Interfaces are defined in the domain/ package, which has zero dependencies:
// internal/domain/user.go
type UserService interface {
Register(ctx context.Context, req model.RegisterRequest) (*model.User, error)
Login(ctx context.Context, req model.LoginRequest) (*model.TokenPair, error)
FindAll(ctx context.Context, tenantID string) (*[]model.User, error)
// ...
}
type UserRepository interface {
Create(user *model.User) error
FindByEmail(email string) (*model.User, error)
FindAll(tenantID string) (*[]model.User, error)
// ...
}
5.3 API Gateway Pattern
Gateway Reverse Proxy
The gateway-service is the single entry point for all client traffic. It acts as a
reverse proxy that handles authentication, tenant identification, role-based access control,
and request routing — so downstream services don't have to.
1. Route Configuration
Routes are defined declaratively in YAML, mapping URL prefixes to backend services:
apps/gateway-service/config/routes.yamlroutes:
- path: "/api/v1/auth"
target_url: "http://localhost:8081"
auth_required: false
- path: "/api/v1/admin"
target_url: "http://localhost:8082"
auth_required: true
roles: ["Platform Admin"]
- path: "/api/v1/broadcast"
target_url: "http://localhost:8083"
auth_required: true
roles for coarse-grained access control
at the gateway level.
2. Router.ServeHTTP Flow
The gateway's router implements http.Handler with prefix matching, JWT validation,
role checking, and reverse proxying:
func (rtr *Router) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// 1. Find matching route by URL prefix
route := rtr.findRoute(r.URL.Path)
if route == nil {
http.Error(w, "Not Found", 404)
return
}
// 2. If route requires auth, validate JWT and inject tenant headers
if route.AuthRequired {
tenantCtx := rtr.validateAndInjectTenantHeaders(w, r)
if tenantCtx == nil {
return // 401 already sent
}
// 3. Check role-based access if route specifies allowed roles
if len(route.Roles) > 0 && !containsRole(route.Roles, tenantCtx.Role) {
http.Error(w, "Forbidden", 403)
return
}
}
// 4. Proxy the request to the target service
rtr.proxyRequest(w, r, route) // uses httputil.ReverseProxy
}
3. Middleware Stack
The gateway applies middleware in order (outermost executes first):
5.4 Multitenancy Implementation
Multitenancy Row-Level Isolation3 Layers of Tenant Identification
Tenant context is established through three complementary layers, each adding more information:
1 Subdomain Extraction
The gateway extracts the tenant subdomain from the Host header and forwards it:
Host: acme.app.com → "acme" extracted → X-Subdomain header set
pkg/middleware/tenant.go
2 JWT-Based Tenant Context
The JWT contains a tenant_id claim. After validation, a TenantContext struct is injected into the Go context:
JWT → { tenant_id, user_id, role }
pkg/auth/context.go
3 Gateway Header Injection
After JWT validation, the gateway sets downstream headers so services don't need JWT parsing:
X-Tenant-ID, X-User-ID, X-User-Role, X-Subdomain
apps/gateway-service/internal/handler/router.go
Data Isolation — Row-Level Filtering
Every repository query filters by tenant_id. There is no database-per-tenant
isolation; instead, all tenants share the same database with row-level filtering:
func (r *UserRepository) FindAll(tenantID string) (*[]model.User, error) {
var users []model.User
query := r.db.Preload("Role")
if tenantID != "" {
// Tenant user: only see own tenant's data
query = query.Where("tenant_id = ?", tenantID)
} else {
// Platform admin: see platform-level records
query = query.Where("tenant_id IS NULL OR tenant_id = '00000000-0000-0000-0000-000000000000'")
}
return &users, query.Find(&users).Error
}
tenant_id. Forgetting this filter leaks data across tenants. This is enforced
by code review convention, not by a database-level mechanism.
Platform Admin
A special "zero UUID" tenant represents the platform operator. Platform admins can access any tenant's data via ownership bypass:
const PlatformAdminTenantID = "00000000-0000-0000-0000-000000000000"
func IsPlatformAdmin(tc TenantContext) bool {
return tc.TenantID == PlatformAdminTenantID
}
func RequireTenantOwnership(tc TenantContext, targetID string) error {
if IsPlatformAdmin(tc) {
return nil // platform admin bypasses ownership check
}
if tc.TenantID != targetID {
return Forbidden("access denied")
}
return nil
}
Kafka Event Isolation
Kafka events carry TenantID as the partition key. This ensures:
- Ordered delivery per tenant — all events for tenant "acme" go to the same partition, preserving message order
- Fair scheduling — high-volume tenants don't starve others across partitions
- Consumer filtering — workers can optionally filter events by tenant
5.5 Configuration Pattern
Config Environment Variables
Each service loads its configuration from environment variables using godotenv
for local development and plain os.Getenv for production:
func Load() (*Config, error) {
godotenv.Load() // optional: loads .env file if present
return &Config{
Port: getEnvOrDefault("PORT", "8081"),
DatabaseURL: os.Getenv("DATABASE_URL"),
Secret: os.Getenv("JWT_SIGNING_KEY"),
}, nil
}
func getEnvOrDefault(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
DATABASE_URL) cause a fatal
exit if missing. Optional variables (like PORT) have sensible defaults.
This fails fast at startup rather than silently misbehaving at runtime.
Key Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
DATABASE_URL |
Yes | — | PostgreSQL / CockroachDB connection string |
PORT |
No | 8081 |
HTTP server listen port |
JWT_SIGNING_KEY |
Yes | — | HMAC-SHA256 secret for JWT signing/verification |
KAFKA_BROKERS |
Conditional | — | Comma-separated Kafka broker addresses |
REDIS_ADDRESS |
Conditional | — | Redis connection address (for idempotency store) |
OTEL_SERVICE_NAME |
No | Service name | OpenTelemetry service identifier |
OTEL_EXPORTER_OTLP_ENDPOINT |
No | — | OTLP HTTP exporter endpoint (e.g. HyperDX) |
BASE_DOMAIN |
Gateway only | — | Base domain for subdomain extraction (e.g. app.com) |
FRONTEND_URL |
No | — | Frontend URL for email links and CORS |
ENCRYPTION_KEY |
Conditional | — | AES-256 key for encrypting sensitive data at rest |
5.6 Database & Migrations
PostgreSQL GORM gooseDatabase Strategy
- ORM: GORM with the PostgreSQL driver (
gorm.io/driver/postgres) - Production DB: CockroachDB (PostgreSQL wire-compatible, distributed)
- Isolation: Database per service — each service has its own schema namespace
| Service | Database | Primary Tables |
|---|---|---|
| auth-service | authdb |
users, roles, permissions, sessions |
| admin-service | admindb |
tenants, tenant_configs, role_menus |
| broadcast-service | broadcastdb |
campaigns, recipients, templates, message_logs |
Migrations with goose
SQL migrations live in apps/{service}/migrations/ and are managed by
goose. Each migration file is a timestamped SQL file:
# Run migrations for a specific service
make migrate app=auth-service
# Create a new migration
make migration app=auth-service name=add_users_table
Connection Pool Tuning
The broadcast-service configures aggressive pool settings for high-throughput message processing:
sqlDB, _ := gormDB.DB()
sqlDB.SetMaxOpenConns(25) // max concurrent connections
sqlDB.SetMaxIdleConns(10) // keep warm connections
sqlDB.SetConnMaxLifetime(5 * time.Minute) // prevent stale connections
5.7 Inter-Service Communication
Sync HTTP Async KafkaSystem Architecture
┌──────────────────┐
Client ──────────► Gateway Service │
│ (Reverse Proxy) │
└──┬───┬───┬───────┘
│ │ │
┌────────────┘ │ └────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Auth │◄──►│ Admin │ │ Broadcast │
│ Service │HTTP│ Service │ │ Service │
└────┬─────┘ └────┬─────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌──────────────┐
│PostgreSQL│ │PostgreSQL│ │ PostgreSQL │
└─────────┘ └─────────┘ └──────────────┘
│
┌─────────┐ ┌──────────────┐
│ Redis │ │ Kafka │
└─────────┘ └──┬───────────┘
│
┌─────┴──────┐
▼ ▼
┌──────────┐ ┌──────────────┐
│Broadcast │ │Meta Webhook │
│ Worker │ │ Service │
└──────────┘ └──────────────┘
Synchronous Communication (HTTP)
Direct HTTP calls are used for request-response operations that need immediate results:
| Caller | Target | Purpose | Example |
|---|---|---|---|
| admin-service | auth-service | Create user accounts during tenant provisioning | POST /api/v1/auth/internal/users |
| admin-service | auth-service | Assign roles to newly created users | POST /api/v1/auth/internal/roles |
| gateway-service | All services | Reverse-proxy client requests | httputil.ReverseProxy |
Asynchronous Communication (Kafka)
Events are published to Kafka for operations that don't require an immediate response:
| Producer | Topic | Consumer | Purpose |
|---|---|---|---|
| auth-service | user.registered |
admin-service | Sync new user to admin records |
| admin-service | tenant.created |
auth-service | Provision tenant admin user |
| broadcast-service | broadcast.send |
broadcast-worker | Dispatch WhatsApp/SMS messages |
| meta-webhook | webhook.received |
broadcast-service | Process delivery receipts and inbound messages |
5.8 Graceful Shutdown Pattern
Shutdown Signal HandlingEvery service implements the same graceful shutdown pattern to prevent data loss and connection leaks during deployments or restarts:
// 1. Start the HTTP server in the background
go func() {
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
slog.Error("server error", "error", err)
}
}()
// 2. Wait for termination signal
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
slog.Info("shutting down...")
// 3. Ordered cleanup (reverse of initialization)
cancelConsumer() // Stop Kafka consumer first (stop accepting work)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
server.Shutdown(ctx) // Drain in-flight HTTP requests
kafkaProducer.Close() // Flush pending Kafka messages
shutdown(ctx) // Flush OpenTelemetry spans
Order matters: Stop accepting new work first (consumer, server), then flush outgoing data (Kafka, telemetry). This ensures in-flight requests complete, pending messages are delivered, and trace data isn't lost. The 10-second timeout prevents hanging on unresponsive dependencies.
Shutdown Sequence
| Step | Action | Why This Order |
|---|---|---|
| 1 | Cancel Kafka consumer context | Stop accepting new messages from queue |
| 2 | server.Shutdown(ctx) |
Stop accepting new HTTP requests; wait for in-flight to finish |
| 3 | kafkaProducer.Close() |
Flush any pending events to Kafka brokers |
| 4 | tracerProvider.Shutdown(ctx) |
Flush trace spans to the collector before exit |
5.9 Observability (OpenTelemetry)
OpenTelemetry Distributed TracingThe entire system is instrumented with OpenTelemetry for distributed tracing, providing end-to-end visibility across services.
Components
| Component | Implementation | Purpose |
|---|---|---|
| Trace Exporter | OTLP HTTP exporter | Send spans to observability backend |
| Structured Logging | otelslog bridge |
Correlate logs with trace/span IDs |
| HTTP Propagation | Tracing middleware (pkg/tracing) |
Extract/inject W3C trace context headers |
| Kafka Propagation | Event header injection | Carry trace context across async message boundaries |
| Backend | HyperDX | Query traces, view service maps, correlate logs |
Trace Context Flow
Client Request
→ Gateway (creates root span, injects W3C traceparent header)
→ Auth Service (extracts traceparent, creates child span)
→ Kafka Event (trace context injected into event headers)
→ Worker (extracts trace context from headers, creates child span)
→ External API call (child span recorded)
Setup Pattern
pkg/tracing/tracing.gofunc InitTracer(ctx context.Context, serviceName string) (func(context.Context) error, error) {
exporter, err := otlptracehttp.New(ctx)
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String(serviceName),
)),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
return tp.Shutdown, nil // returns shutdown function for graceful cleanup
}
5.10 Middleware Stack Summary
Middleware Chi RouterEach service has its own middleware chain, applied in order. The gateway handles cross-cutting concerns; backend services handle service-specific concerns.
gateway-service
auth-service
Protected routes additionally use JWTMiddleware and Permission(menu, action) per route group.
admin-service
All admin routes require authentication. Per-route Permission middleware enforces menu+action checks.
broadcast-service
Kafka consumer middleware: RetryableHandler → IdempotentHandler → business handler.
Middleware Responsibility Matrix
| Middleware | Gateway | Auth | Admin | Broadcast |
|---|---|---|---|---|
tracing.Middleware |
✓ | ✓ | ✓ | ✓ |
TenantIdentifierMiddleware |
✓ | — | — | — |
RequestIDMiddleware |
✓ | — | — | — |
CORSMiddleware |
✓ | — | — | — |
chi.Logger |
— | ✓ | ✓ | ✓ |
chi.Recoverer |
— | ✓ | ✓ | ✓ |
JWTMiddleware |
— | Per-group | ✓ | ✓ |
Permission(menu, action) |
— | Per-route | Per-route | Per-route |
5.11 Complete API Endpoints
REST API Endpoints
All routes are prefixed through the gateway. Internal routes (marked with /internal/)
are not exposed through the gateway and are only callable service-to-service.
auth-service :8081
| Method | Path | Auth | Description |
|---|---|---|---|
POST |
/api/v1/auth/register |
No | Register a new user account |
POST |
/api/v1/auth/login |
No | Authenticate and receive JWT token pair |
POST |
/api/v1/auth/refresh |
No | Refresh access token using refresh token |
POST |
/api/v1/auth/forgot-password |
No | Send password reset email |
POST |
/api/v1/auth/reset-password |
No | Reset password with token |
GET |
/api/v1/auth/verify-email |
No | Verify email address with token |
GET |
/api/v1/auth/me |
JWT | Get current user profile |
PUT |
/api/v1/auth/me |
JWT | Update current user profile |
GET |
/api/v1/auth/users |
JWT + Permission | List users (filtered by tenant) |
GET |
/api/v1/auth/users/:id |
JWT + Permission | Get user by ID |
PUT |
/api/v1/auth/users/:id |
JWT + Permission | Update user by ID |
DELETE |
/api/v1/auth/users/:id |
JWT + Permission | Delete user by ID |
GET |
/api/v1/auth/roles |
JWT + Permission | List available roles |
POST |
/api/v1/auth/internal/users |
Internal | Create user (called by admin-service) |
POST |
/api/v1/auth/internal/roles |
Internal | Assign role (called by admin-service) |
admin-service :8082
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/api/v1/admin/tenants |
JWT + Platform Admin | List all tenants |
POST |
/api/v1/admin/tenants |
JWT + Platform Admin | Create a new tenant |
GET |
/api/v1/admin/tenants/:id |
JWT + Ownership | Get tenant details |
PUT |
/api/v1/admin/tenants/:id |
JWT + Ownership | Update tenant |
DELETE |
/api/v1/admin/tenants/:id |
JWT + Platform Admin | Delete tenant |
GET |
/api/v1/admin/tenants/:id/config |
JWT + Ownership | Get tenant configuration |
PUT |
/api/v1/admin/tenants/:id/config |
JWT + Ownership | Update tenant configuration (WhatsApp keys, etc.) |
GET |
/api/v1/admin/roles |
JWT | List roles |
POST |
/api/v1/admin/roles |
JWT + Permission | Create a new role |
PUT |
/api/v1/admin/roles/:id |
JWT + Permission | Update role |
DELETE |
/api/v1/admin/roles/:id |
JWT + Permission | Delete role |
GET |
/api/v1/admin/menus |
JWT | List menu permissions |
POST |
/api/v1/admin/menus |
JWT + Permission | Create menu permission |
broadcast-service :8083
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/api/v1/broadcast/campaigns |
JWT | List broadcast campaigns |
POST |
/api/v1/broadcast/campaigns |
JWT + Permission | Create a new broadcast campaign |
GET |
/api/v1/broadcast/campaigns/:id |
JWT | Get campaign details |
PUT |
/api/v1/broadcast/campaigns/:id |
JWT + Permission | Update campaign |
DELETE |
/api/v1/broadcast/campaigns/:id |
JWT + Permission | Delete campaign |
POST |
/api/v1/broadcast/campaigns/:id/send |
JWT + Permission | Trigger broadcast sending |
GET |
/api/v1/broadcast/recipients |
JWT | List recipients |
POST |
/api/v1/broadcast/recipients |
JWT + Permission | Add recipients |
POST |
/api/v1/broadcast/recipients/import |
JWT + Permission | Bulk import recipients from CSV/Excel |
DELETE |
/api/v1/broadcast/recipients/:id |
JWT + Permission | Delete recipient |
GET |
/api/v1/broadcast/templates |
JWT | List message templates |
POST |
/api/v1/broadcast/templates |
JWT + Permission | Create message template |
PUT |
/api/v1/broadcast/templates/:id |
JWT + Permission | Update message template |
DELETE |
/api/v1/broadcast/templates/:id |
JWT + Permission | Delete message template |
GET |
/api/v1/broadcast/logs |
JWT | View message delivery logs |
gateway-service :8080
| Method | Path | Description |
|---|---|---|
GET |
/health |
Health check endpoint |
* |
/api/v1/auth/* |
Proxy to auth-service (:8081) |
* |
/api/v1/admin/* |
Proxy to admin-service (:8082), requires auth + Platform Admin role |
* |
/api/v1/broadcast/* |
Proxy to broadcast-service (:8083), requires auth |