Ultra-Complex System Coding: Technical Guidelines

Last updated 14 Dec 2025, 10:10

Core engineering principles for the Zenpower ecosystem. These guidelines govern all development decisions.

1. Architecture & Structure

  • Design for evolution, not completion: Prefer modular, orthogonal, decoupled components; avoid monolith logic clusters.
  • Define invariants early: System-wide truths that must always hold; enforce via types, assertions, and CI checks.
  • Separate policy from mechanism: High-level orchestration independent from low-level execution.
  • Use stable interfaces, volatile implementations: Interfaces rarely change; internals evolve aggressively.
  • Treat concurrency as a first-class concern: Formalize thread/async models; design deterministic event flows.

2. Complexity Management

  • Minimize conceptual surface area: Fewer mental models, consistent patterns, predictable idioms.
  • Limit degrees of freedom: Provide one blessed way to do a critical task.
  • Decompose by responsibility, not by domain nouns; aim for "semantic minimal units."
  • Progressive disclosure of complexity: Advanced capabilities remain hidden behind simple baselines.

3. Code Quality & Design

  • Fail loudly, early, and deterministically: Reject invalid state instead of patching over it.
  • Prefer explicit over implicit: Avoid hidden side effects, global state, and magic mutable containers.
  • Data structures > algorithms > code: The choice of structure eliminates entire classes of bugs.
  • Immutability where possible; controlled mutability with clear ownership where not.
  • Idempotence and reversibility: Functions should behave predictably under repetition or rollback.

4. Performance Engineering

  • Optimize architecture first, code second: No micro-optimization can save a broken design.
  • Measure, don't guess: Always profile on representative workloads with production-like data.
  • Prefer O(1)/O(log n) over O(n) in key paths; avoid accidental N² explosions (especially in async contexts).
  • Cache carefully: Explicit invalidation strategy; avoid cache-induced complexity leaks.
  • Eliminate unnecessary allocations, serialization, and context-switches.

5. Reliability & Fault Tolerance

  • Fail-safe defaults: Safe mode > broken mode.
  • Define blast radius boundaries: Use bulkheads, rate-limits, circuit breakers.
  • Plan for partial failure: Every remote call is unreliable; design retry/backoff semantics intentionally.
  • Observability built-in: Structured logs, metrics, traces, correlation IDs.
  • No state corruption under failure: Atomicity, journaling, compensating actions.

6. Security & Safety

  • Zero-trust by default: Everything is hostile unless proven otherwise.
  • Principle of least privilege everywhere: processes, modules, threads, objects.
  • Validate all inputs, sanitize all outputs.
  • Deterministic, audit-friendly behavior: Record why decisions are made, not just outcomes.
  • Constant-time critical operations to avoid timing leaks.

7. Testing & Validation

  • Test architecture, not just code: invariants, contracts, boundaries, concurrency, failure modes.
  • Golden-path tests + edge-case batteries: Cover typical flows and pathological states.
  • Prefer property-based testing for anything mathematical, generative, or algorithmic.
  • Chaos & fuzz testing for distributed or concurrent systems.
  • Regression tests for every bug—never fix without locking the fix into the test suite.

8. Tooling & DevOps

  • Automate everything: build, test, deploy, rollback, lint, format, schema evolution.
  • Static analysis mandatory (type checks, lint rules, contract verifiers).
  • Continuous integration should enforce invariants: no merge without stable state.
  • Feature flags over long-lived branches.
  • Repeatable environments: containerized, deterministic dependencies.

9. Documentation & Knowledge Transfer

  • Document invariants, decisions, and guarantees—not code line-by-line.
  • Architecture decision records (ADRs) for all nontrivial choices.
  • Operational playbooks for SRE/dev teams: failure handling, debugging, diagnosis.
  • Low-friction onboarding docs: diagrams over prose; examples over theory.

10. Mindset & Team Practices

  • Complexity is a tax: Always assume future maintenance by someone less familiar.
  • Bias toward simplicity: The simplest correct design is the best design.
  • Continuous refactoring: Do not let entropy accumulate.
  • Encourage code reviews that target correctness, clarity, and invariants.
  • Design with humility: Assume every component will be misused, extended, or pushed beyond intended limits.

Zenpower-Specific Applications

Service Design

  • Each service owns its data; cross-service access via APIs only
  • Use RQ for async tasks with idempotent job handlers
  • Redis for caching with explicit TTLs; PostgreSQL for state

API Contracts

  • FastAPI with Pydantic models for automatic validation
  • Versioned endpoints; deprecation notices before removal
  • Rate limiting via Traefik middlewares

Security Boundaries

  • SIWE wallet auth as primary identity
  • Forward-auth middleware for protected routes
  • zengroup for secret sharing; root-only for encryption keys

Observability

  • Prometheus metrics on /metrics endpoints
  • Structured JSON logging with correlation IDs
  • Grafana dashboards for service health

Last updated: 2025-12-14 Reference: docs/dev/TECHNICAL_GUIDELINES.md