Architecture & Design Patterns Chapter 05

The 1Engage multitenant backend is a Go microservices monorepo built on clean architecture principles. This chapter covers the full system architecture — service structure, layering, the API gateway, multitenancy implementation, configuration, database strategy, inter-service communication, graceful shutdown, observability, and every API endpoint.

5.1 Monorepo Structure

Monorepo Convention

Service Directory Convention

Every service under apps/ follows an identical directory layout. This convention means a developer who understands one service can navigate any other without a learning curve.

apps/{service-name}/
├── cmd/
│   ├── server/main.go     # HTTP server entry point
│   ├── worker/main.go     # Kafka consumer (if needed)
│   ├── seed/main.go       # Database seeder
│   └── scheduler/main.go  # CronJob entry point
├── config/
│   └── config.go          # Env var loading → typed Config struct
├── internal/
│   ├── domain/            # Interface contracts (Service + Repository)
│   ├── model/             # GORM models / DTOs
│   ├── repository/        # Data access (GORM)
│   ├── service/           # Business logic
│   ├── handler/           # HTTP handlers
│   ├── db/                # Database connection factory
│   ├── worker/            # Job dispatcher (broadcast-service only)
│   ├── ratelimit/         # Rate limiting (broadcast-service only)
│   └── job/               # Job definitions (broadcast-service only)
└── migrations/            # SQL migrations (goose)
Key insight: Go's internal/ directory has compiler-enforced visibility. Code inside apps/auth-service/internal/ cannot be imported by apps/admin-service/. This prevents accidental cross-service coupling at the language level.

Shared Packages (pkg/)

Cross-service code lives in pkg/ and is imported by any service that needs it. Each package has a single, well-defined responsibility:

Package Purpose Used By
pkg/auth JWT validation, TenantContext extraction, role checking middleware All services
pkg/crypto AES-GCM encryption/decryption for sensitive data at rest auth-service, admin-service
pkg/eventbus Kafka producer/consumer abstraction, event types, idempotency, retry All services with Kafka
pkg/middleware Tenant identifier, request ID, CORS middleware gateway-service
pkg/shared Generic HTTP helpers (SendJSON, SendPaginated), validators, converters All services
pkg/tracing OpenTelemetry setup, OTLP exporter config, trace middleware All services
pkg/version Build-time version injection via -ldflags All services
Design rule: If code is used by exactly one service, it belongs in that service's internal/. If it's used by two or more services, it belongs in pkg/. This prevents premature abstraction while enabling genuine reuse.

5.2 Layered Architecture (Clean Architecture)

Clean Architecture Dependency Inversion

The 4-Layer Flow

Every request passes through exactly four layers, each with a single responsibility:

HTTP Request Handler Service Repository Database
Layer Responsibility Depends On Example File
Handler HTTP parsing, validation, response formatting domain.Service (interface) internal/handler/
Service Business logic, orchestration, event publishing domain.Repository (interface) internal/service/
Repository Data access, SQL queries, GORM operations *gorm.DB (concrete) internal/repository/
Domain Interface contracts — defines what each layer must implement Nothing (leaf package) internal/domain/

Real Wiring: auth-service

Dependencies are assembled in main.go using plain constructor injection — no DI framework, no reflection, no magic:

apps/auth-service/cmd/server/main.go
// 1. Create the concrete repository (depends on *gorm.DB)
userRepo := repository.NewUserRepository(gormDB)

// 2. Create the service (depends on domain.UserRepository interface)
userService := service.NewUserService(userRepo, emailWorker, feUrl, kafkaProducer, internalUrl)

// 3. Create the handler (depends on domain.UserService interface)
userHandler := handler.NewUserHandler(userService)

Dependency inversion in action: The Handler depends on domain.UserService (an interface), not the concrete service.userService struct. The Service depends on domain.UserRepository (an interface), not the concrete repository. Only main.go knows the concrete types — the rest of the code is testable in isolation with mock implementations.

Domain Interface Contract

Interfaces are defined in the domain/ package, which has zero dependencies:

// internal/domain/user.go
type UserService interface {
    Register(ctx context.Context, req model.RegisterRequest) (*model.User, error)
    Login(ctx context.Context, req model.LoginRequest) (*model.TokenPair, error)
    FindAll(ctx context.Context, tenantID string) (*[]model.User, error)
    // ...
}

type UserRepository interface {
    Create(user *model.User) error
    FindByEmail(email string) (*model.User, error)
    FindAll(tenantID string) (*[]model.User, error)
    // ...
}

5.3 API Gateway Pattern

Gateway Reverse Proxy

The gateway-service is the single entry point for all client traffic. It acts as a reverse proxy that handles authentication, tenant identification, role-based access control, and request routing — so downstream services don't have to.

1. Route Configuration

Routes are defined declaratively in YAML, mapping URL prefixes to backend services:

apps/gateway-service/config/routes.yaml
routes:
  - path: "/api/v1/auth"
    target_url: "http://localhost:8081"
    auth_required: false

  - path: "/api/v1/admin"
    target_url: "http://localhost:8082"
    auth_required: true
    roles: ["Platform Admin"]

  - path: "/api/v1/broadcast"
    target_url: "http://localhost:8083"
    auth_required: true
Benefits: Adding a new service requires only a YAML entry — no code changes to the gateway. Routes support optional roles for coarse-grained access control at the gateway level.

2. Router.ServeHTTP Flow

The gateway's router implements http.Handler with prefix matching, JWT validation, role checking, and reverse proxying:

apps/gateway-service/internal/handler/router.go
func (rtr *Router) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // 1. Find matching route by URL prefix
    route := rtr.findRoute(r.URL.Path)
    if route == nil {
        http.Error(w, "Not Found", 404)
        return
    }

    // 2. If route requires auth, validate JWT and inject tenant headers
    if route.AuthRequired {
        tenantCtx := rtr.validateAndInjectTenantHeaders(w, r)
        if tenantCtx == nil {
            return  // 401 already sent
        }

        // 3. Check role-based access if route specifies allowed roles
        if len(route.Roles) > 0 && !containsRole(route.Roles, tenantCtx.Role) {
            http.Error(w, "Forbidden", 403)
            return
        }
    }

    // 4. Proxy the request to the target service
    rtr.proxyRequest(w, r, route)  // uses httputil.ReverseProxy
}

3. Middleware Stack

The gateway applies middleware in order (outermost executes first):

tracing.Middleware TenantIdentifierMiddleware RequestIDMiddleware CORSMiddleware Router
Separation of concerns: The gateway handles cross-cutting concerns (auth, tracing, CORS, tenant identification). Backend services receive pre-validated requests with tenant context injected as HTTP headers — they never parse JWTs themselves.

5.4 Multitenancy Implementation

Multitenancy Row-Level Isolation

3 Layers of Tenant Identification

Tenant context is established through three complementary layers, each adding more information:

1 Subdomain Extraction

The gateway extracts the tenant subdomain from the Host header and forwards it:

Host: acme.app.com"acme" extracted → X-Subdomain header set

pkg/middleware/tenant.go

2 JWT-Based Tenant Context

The JWT contains a tenant_id claim. After validation, a TenantContext struct is injected into the Go context:

JWT → { tenant_id, user_id, role }

pkg/auth/context.go

3 Gateway Header Injection

After JWT validation, the gateway sets downstream headers so services don't need JWT parsing:

X-Tenant-ID, X-User-ID, X-User-Role, X-Subdomain

apps/gateway-service/internal/handler/router.go

Data Isolation — Row-Level Filtering

Every repository query filters by tenant_id. There is no database-per-tenant isolation; instead, all tenants share the same database with row-level filtering:

func (r *UserRepository) FindAll(tenantID string) (*[]model.User, error) {
    var users []model.User
    query := r.db.Preload("Role")

    if tenantID != "" {
        // Tenant user: only see own tenant's data
        query = query.Where("tenant_id = ?", tenantID)
    } else {
        // Platform admin: see platform-level records
        query = query.Where("tenant_id IS NULL OR tenant_id = '00000000-0000-0000-0000-000000000000'")
    }

    return &users, query.Find(&users).Error
}
Critical invariant: Every query that returns user data MUST filter by tenant_id. Forgetting this filter leaks data across tenants. This is enforced by code review convention, not by a database-level mechanism.

Platform Admin

A special "zero UUID" tenant represents the platform operator. Platform admins can access any tenant's data via ownership bypass:

const PlatformAdminTenantID = "00000000-0000-0000-0000-000000000000"

func IsPlatformAdmin(tc TenantContext) bool {
    return tc.TenantID == PlatformAdminTenantID
}

func RequireTenantOwnership(tc TenantContext, targetID string) error {
    if IsPlatformAdmin(tc) {
        return nil  // platform admin bypasses ownership check
    }
    if tc.TenantID != targetID {
        return Forbidden("access denied")
    }
    return nil
}

Kafka Event Isolation

Kafka events carry TenantID as the partition key. This ensures:

  • Ordered delivery per tenant — all events for tenant "acme" go to the same partition, preserving message order
  • Fair scheduling — high-volume tenants don't starve others across partitions
  • Consumer filtering — workers can optionally filter events by tenant

5.5 Configuration Pattern

Config Environment Variables

Each service loads its configuration from environment variables using godotenv for local development and plain os.Getenv for production:

func Load() (*Config, error) {
    godotenv.Load()  // optional: loads .env file if present

    return &Config{
        Port:        getEnvOrDefault("PORT", "8081"),
        DatabaseURL: os.Getenv("DATABASE_URL"),
        Secret:      os.Getenv("JWT_SIGNING_KEY"),
    }, nil
}

func getEnvOrDefault(key, fallback string) string {
    if v := os.Getenv(key); v != "" {
        return v
    }
    return fallback
}
Convention: Required variables (like DATABASE_URL) cause a fatal exit if missing. Optional variables (like PORT) have sensible defaults. This fails fast at startup rather than silently misbehaving at runtime.

Key Environment Variables

Variable Required Default Description
DATABASE_URL Yes PostgreSQL / CockroachDB connection string
PORT No 8081 HTTP server listen port
JWT_SIGNING_KEY Yes HMAC-SHA256 secret for JWT signing/verification
KAFKA_BROKERS Conditional Comma-separated Kafka broker addresses
REDIS_ADDRESS Conditional Redis connection address (for idempotency store)
OTEL_SERVICE_NAME No Service name OpenTelemetry service identifier
OTEL_EXPORTER_OTLP_ENDPOINT No OTLP HTTP exporter endpoint (e.g. HyperDX)
BASE_DOMAIN Gateway only Base domain for subdomain extraction (e.g. app.com)
FRONTEND_URL No Frontend URL for email links and CORS
ENCRYPTION_KEY Conditional AES-256 key for encrypting sensitive data at rest

5.6 Database & Migrations

PostgreSQL GORM goose

Database Strategy

  • ORM: GORM with the PostgreSQL driver (gorm.io/driver/postgres)
  • Production DB: CockroachDB (PostgreSQL wire-compatible, distributed)
  • Isolation: Database per service — each service has its own schema namespace
Service Database Primary Tables
auth-service authdb users, roles, permissions, sessions
admin-service admindb tenants, tenant_configs, role_menus
broadcast-service broadcastdb campaigns, recipients, templates, message_logs

Migrations with goose

SQL migrations live in apps/{service}/migrations/ and are managed by goose. Each migration file is a timestamped SQL file:

# Run migrations for a specific service
make migrate app=auth-service

# Create a new migration
make migration app=auth-service name=add_users_table

Connection Pool Tuning

The broadcast-service configures aggressive pool settings for high-throughput message processing:

sqlDB, _ := gormDB.DB()
sqlDB.SetMaxOpenConns(25)                  // max concurrent connections
sqlDB.SetMaxIdleConns(10)                   // keep warm connections
sqlDB.SetConnMaxLifetime(5 * time.Minute)  // prevent stale connections
Why tune? The broadcast worker processes thousands of messages concurrently. Without pool tuning, it would either exhaust connections (too many open) or suffer cold-start latency (too few idle). These values are calibrated for the expected concurrent worker count.

5.7 Inter-Service Communication

Sync HTTP Async Kafka

System Architecture

                    ┌──────────────────┐
   Client ──────────►  Gateway Service  │
                    │  (Reverse Proxy)  │
                    └──┬───┬───┬───────┘
                       │   │   │
          ┌────────────┘   │   └────────────┐
          ▼                ▼                ▼
   ┌──────────┐    ┌──────────┐    ┌──────────────┐
   │   Auth   │◄──►│  Admin   │    │  Broadcast   │
   │ Service  │HTTP│ Service  │    │   Service    │
   └────┬─────┘    └────┬─────┘    └──────┬───────┘
        │               │                 │
        ▼               ▼                 ▼
   ┌─────────┐   ┌─────────┐     ┌──────────────┐
   │PostgreSQL│   │PostgreSQL│     │  PostgreSQL   │
   └─────────┘   └─────────┘     └──────────────┘
                                       │
           ┌─────────┐           ┌──────────────┐
           │  Redis   │           │    Kafka     │
           └─────────┘           └──┬───────────┘
                                    │
                              ┌─────┴──────┐
                              ▼            ▼
                       ┌──────────┐  ┌──────────────┐
                       │Broadcast │  │Meta Webhook  │
                       │ Worker   │  │  Service     │
                       └──────────┘  └──────────────┘

Synchronous Communication (HTTP)

Direct HTTP calls are used for request-response operations that need immediate results:

Caller Target Purpose Example
admin-service auth-service Create user accounts during tenant provisioning POST /api/v1/auth/internal/users
admin-service auth-service Assign roles to newly created users POST /api/v1/auth/internal/roles
gateway-service All services Reverse-proxy client requests httputil.ReverseProxy

Asynchronous Communication (Kafka)

Events are published to Kafka for operations that don't require an immediate response:

Producer Topic Consumer Purpose
auth-service user.registered admin-service Sync new user to admin records
admin-service tenant.created auth-service Provision tenant admin user
broadcast-service broadcast.send broadcast-worker Dispatch WhatsApp/SMS messages
meta-webhook webhook.received broadcast-service Process delivery receipts and inbound messages
Rule of thumb: Use sync HTTP when the caller needs the result to continue (e.g., tenant provisioning creates a user and needs the user ID). Use async Kafka when the caller can fire-and-forget (e.g., sending a broadcast message batch).

5.8 Graceful Shutdown Pattern

Shutdown Signal Handling

Every service implements the same graceful shutdown pattern to prevent data loss and connection leaks during deployments or restarts:

// 1. Start the HTTP server in the background
go func() {
    if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
        slog.Error("server error", "error", err)
    }
}()

// 2. Wait for termination signal
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
slog.Info("shutting down...")

// 3. Ordered cleanup (reverse of initialization)
cancelConsumer()           // Stop Kafka consumer first (stop accepting work)

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

server.Shutdown(ctx)       // Drain in-flight HTTP requests
kafkaProducer.Close()      // Flush pending Kafka messages
shutdown(ctx)              // Flush OpenTelemetry spans

Order matters: Stop accepting new work first (consumer, server), then flush outgoing data (Kafka, telemetry). This ensures in-flight requests complete, pending messages are delivered, and trace data isn't lost. The 10-second timeout prevents hanging on unresponsive dependencies.

Shutdown Sequence

Step Action Why This Order
1 Cancel Kafka consumer context Stop accepting new messages from queue
2 server.Shutdown(ctx) Stop accepting new HTTP requests; wait for in-flight to finish
3 kafkaProducer.Close() Flush any pending events to Kafka brokers
4 tracerProvider.Shutdown(ctx) Flush trace spans to the collector before exit

5.9 Observability (OpenTelemetry)

OpenTelemetry Distributed Tracing

The entire system is instrumented with OpenTelemetry for distributed tracing, providing end-to-end visibility across services.

Components

Component Implementation Purpose
Trace Exporter OTLP HTTP exporter Send spans to observability backend
Structured Logging otelslog bridge Correlate logs with trace/span IDs
HTTP Propagation Tracing middleware (pkg/tracing) Extract/inject W3C trace context headers
Kafka Propagation Event header injection Carry trace context across async message boundaries
Backend HyperDX Query traces, view service maps, correlate logs

Trace Context Flow

Client Request
  → Gateway (creates root span, injects W3C traceparent header)
    → Auth Service (extracts traceparent, creates child span)
      → Kafka Event (trace context injected into event headers)
        → Worker (extracts trace context from headers, creates child span)
          → External API call (child span recorded)
End-to-end trace: A single user action (e.g., "send broadcast") produces a trace that spans the HTTP request through the gateway, the service logic, the Kafka event, the worker processing, and the external WhatsApp API call — all correlated under one trace ID.

Setup Pattern

pkg/tracing/tracing.go
func InitTracer(ctx context.Context, serviceName string) (func(context.Context) error, error) {
    exporter, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return tp.Shutdown, nil  // returns shutdown function for graceful cleanup
}

5.10 Middleware Stack Summary

Middleware Chi Router

Each service has its own middleware chain, applied in order. The gateway handles cross-cutting concerns; backend services handle service-specific concerns.

gateway-service

tracing.Middleware TenantIdentifier RequestID CORS Router (proxy)

auth-service

tracing.Middleware chi.Logger chi.Recoverer Route Handlers

Protected routes additionally use JWTMiddleware and Permission(menu, action) per route group.

admin-service

tracing.Middleware chi.Logger chi.Recoverer JWTMiddleware Route Handlers

All admin routes require authentication. Per-route Permission middleware enforces menu+action checks.

broadcast-service

tracing.Middleware chi.Logger chi.Recoverer JWTMiddleware Route Handlers

Kafka consumer middleware: RetryableHandlerIdempotentHandler → business handler.

Middleware Responsibility Matrix

Middleware Gateway Auth Admin Broadcast
tracing.Middleware
TenantIdentifierMiddleware
RequestIDMiddleware
CORSMiddleware
chi.Logger
chi.Recoverer
JWTMiddleware Per-group
Permission(menu, action) Per-route Per-route Per-route

5.11 Complete API Endpoints

REST API Endpoints

All routes are prefixed through the gateway. Internal routes (marked with /internal/) are not exposed through the gateway and are only callable service-to-service.

auth-service :8081

Method Path Auth Description
POST /api/v1/auth/register No Register a new user account
POST /api/v1/auth/login No Authenticate and receive JWT token pair
POST /api/v1/auth/refresh No Refresh access token using refresh token
POST /api/v1/auth/forgot-password No Send password reset email
POST /api/v1/auth/reset-password No Reset password with token
GET /api/v1/auth/verify-email No Verify email address with token
GET /api/v1/auth/me JWT Get current user profile
PUT /api/v1/auth/me JWT Update current user profile
GET /api/v1/auth/users JWT + Permission List users (filtered by tenant)
GET /api/v1/auth/users/:id JWT + Permission Get user by ID
PUT /api/v1/auth/users/:id JWT + Permission Update user by ID
DELETE /api/v1/auth/users/:id JWT + Permission Delete user by ID
GET /api/v1/auth/roles JWT + Permission List available roles
POST /api/v1/auth/internal/users Internal Create user (called by admin-service)
POST /api/v1/auth/internal/roles Internal Assign role (called by admin-service)

admin-service :8082

Method Path Auth Description
GET /api/v1/admin/tenants JWT + Platform Admin List all tenants
POST /api/v1/admin/tenants JWT + Platform Admin Create a new tenant
GET /api/v1/admin/tenants/:id JWT + Ownership Get tenant details
PUT /api/v1/admin/tenants/:id JWT + Ownership Update tenant
DELETE /api/v1/admin/tenants/:id JWT + Platform Admin Delete tenant
GET /api/v1/admin/tenants/:id/config JWT + Ownership Get tenant configuration
PUT /api/v1/admin/tenants/:id/config JWT + Ownership Update tenant configuration (WhatsApp keys, etc.)
GET /api/v1/admin/roles JWT List roles
POST /api/v1/admin/roles JWT + Permission Create a new role
PUT /api/v1/admin/roles/:id JWT + Permission Update role
DELETE /api/v1/admin/roles/:id JWT + Permission Delete role
GET /api/v1/admin/menus JWT List menu permissions
POST /api/v1/admin/menus JWT + Permission Create menu permission

broadcast-service :8083

Method Path Auth Description
GET /api/v1/broadcast/campaigns JWT List broadcast campaigns
POST /api/v1/broadcast/campaigns JWT + Permission Create a new broadcast campaign
GET /api/v1/broadcast/campaigns/:id JWT Get campaign details
PUT /api/v1/broadcast/campaigns/:id JWT + Permission Update campaign
DELETE /api/v1/broadcast/campaigns/:id JWT + Permission Delete campaign
POST /api/v1/broadcast/campaigns/:id/send JWT + Permission Trigger broadcast sending
GET /api/v1/broadcast/recipients JWT List recipients
POST /api/v1/broadcast/recipients JWT + Permission Add recipients
POST /api/v1/broadcast/recipients/import JWT + Permission Bulk import recipients from CSV/Excel
DELETE /api/v1/broadcast/recipients/:id JWT + Permission Delete recipient
GET /api/v1/broadcast/templates JWT List message templates
POST /api/v1/broadcast/templates JWT + Permission Create message template
PUT /api/v1/broadcast/templates/:id JWT + Permission Update message template
DELETE /api/v1/broadcast/templates/:id JWT + Permission Delete message template
GET /api/v1/broadcast/logs JWT View message delivery logs

gateway-service :8080

Method Path Description
GET /health Health check endpoint
* /api/v1/auth/* Proxy to auth-service (:8081)
* /api/v1/admin/* Proxy to admin-service (:8082), requires auth + Platform Admin role
* /api/v1/broadcast/* Proxy to broadcast-service (:8083), requires auth

← 04 · Go Language Features 06 · Authentication & Authorization →