Capacity Planning

This guide provides sizing models, reference benchmarks, and recommendations for provisioning DataSynth deployments.

Performance Characteristics

DataSynth is CPU-bound during generation and I/O-bound during output. Key characteristics:

• Throughput: 100K+ journal entries per second on a single core
• Scaling: Near-linear scaling with CPU cores for batch generation
• Memory: Proportional to active dataset size (companies, accounts, master data)
• Disk: Output size depends on format, compression, and enabled modules
• Network: REST/gRPC overhead is minimal; bulk generation is the bottleneck

Sizing Model

CPU

DataSynth uses Rayon for parallel generation and Tokio for async I/O. The relationship between CPU cores and throughput:

These numbers are for journal entry generation with balanced debit/credit lines. Enabling additional modules (document flows, subledgers, master data, anomaly injection) reduces throughput by 30-60% due to cross-referencing overhead.

Memory

Memory usage depends on the active generation context:

Sizing formula (approximate):

Memory (MB) = 100 + (companies * master_data_per_company) + (buffer_entries * 0.5)

Recommended Memory by Config Complexity

DataSynth includes built-in memory guards that trigger graceful degradation before OOM. See Runbook - Memory Issues for degradation levels.

Disk Sizing

Output Size by Format

The output size depends on the number of entries, enabled modules, and output format:

These estimates cover journal entries only. Enabling all modules (master data, document flows, subledgers, audit trails, etc.) can multiply total output by 5-10x.

Output Files by Module

When all modules are enabled, a typical generation produces 60+ output files:

Disk Provisioning Formula

Disk (GB) = entries_millions * format_multiplier * module_multiplier * safety_margin

Where:
  format_multiplier:  CSV=0.25, JSON=0.50, Parquet=0.05  (per million entries)
  module_multiplier:  JE only=1.0, all modules=5.0
  safety_margin:      1.5 (for temp files, logs, etc.)

Example: 1M entries, all modules, CSV format:

Disk = 1 * 0.25 * 5.0 * 1.5 = 1.875 GB (round up to 2 GB)

Reference Benchmarks

Benchmarks run on c5.2xlarge (8 vCPU, 16 GB RAM):

Container Resource Recommendations

Docker / Single Host

Kubernetes

Scaling Guidelines

Vertical Scaling (Single Node)

Vertical scaling is effective up to 16 cores. Beyond that, returns diminish due to lock contention in the shared ServerState. Recommendations:

1. Start with the “Standard” Kubernetes profile.
2. Monitor synth_entries_per_second in Grafana.
3. If throughput plateaus at high CPU, add replicas instead.

Horizontal Scaling (Multi-Replica)

DataSynth is stateless – each pod generates data independently. Horizontal scaling considerations:

1. Enable Redis for shared rate limiting across replicas.
2. Use deterministic seeds per replica to avoid duplicate data (seed = base_seed + replica_index).
3. Route bulk generation requests to specific replicas if output deduplication matters.
4. WebSocket streams are per-connection and do not share state across replicas.

Scaling Decision Tree

Is throughput below target?
  |
  +-- Yes: Is CPU utilization > 70%?
  |    |
  |    +-- Yes: Add more replicas (horizontal)
  |    +-- No:  Is memory > 80%?
  |         |
  |         +-- Yes: Increase memory limit
  |         +-- No:  Check I/O (disk throughput, network)
  |
  +-- No: Current sizing is adequate

Network Bandwidth

DataSynth’s network requirements are modest:

Network is rarely the bottleneck. A 1 Gbps link supports hundreds of concurrent clients.