What is SyntheticData?

SyntheticData is a configurable synthetic data generator that produces realistic, interconnected enterprise financial data. It generates General Ledger journal entries, Chart of Accounts, SAP HANA-compatible ACDOCA event logs, document flows, subledger records, banking/KYC/AML transactions, OCEL 2.0 process mining data, audit workpapers, and ML-ready graph exports at scale.

The generator produces statistically accurate data based on empirical research from real-world general ledger patterns, ensuring that synthetic datasets exhibit the same characteristics as production data—including Benford’s Law compliance, temporal patterns, and document flow integrity.

New in v0.5.0: LLM-augmented generation (vendor names, descriptions, anomaly explanations), diffusion model backend (statistical denoising, hybrid generation), causal & counterfactual generation (SCMs, do-calculus interventions), federated fingerprinting, synthetic data certificates, and ecosystem integrations (Airflow, dbt, MLflow, Spark).

v0.3.0: ACFE-aligned fraud taxonomy, collusion modeling, industry-specific transactions (Manufacturing, Retail, Healthcare), and ML benchmarks.

v0.2.x: Privacy-preserving fingerprinting, accounting/audit standards (US GAAP, IFRS, ISA, SOX), streaming output API.

Quick Links

Key Features

Core Data Generation

Enterprise Simulation

• Master Data Management: Vendors, customers, materials, fixed assets, employees with temporal validity
• Document Flow Engine: Complete P2P (Procure-to-Pay) and O2C (Order-to-Cash) processes with three-way matching
• Intercompany Transactions: IC matching, transfer pricing, consolidation eliminations
• Balance Coherence: Opening balances, running balance tracking, trial balance generation
• Subledger Simulation: AR, AP, Fixed Assets, Inventory with GL reconciliation
• Currency & FX: Realistic exchange rates (Ornstein-Uhlenbeck process), currency translation, CTA generation
• Period Close Engine: Monthly close, depreciation runs, accruals, year-end closing
• Banking/KYC/AML: Customer personas, KYC profiles, AML typologies (structuring, funnel, mule, layering, round-tripping)
• Process Mining: OCEL 2.0 event logs with object-centric relationships
• Audit Simulation: ISA-compliant engagements, workpapers, findings, risk assessments, professional judgments

Fraud Patterns & Industry-Specific Features

• ACFE-Aligned Fraud Taxonomy: Asset Misappropriation, Corruption, Financial Statement Fraud calibrated to ACFE statistics
• Collusion & Conspiracy Modeling: Multi-party fraud networks with 9 ring types and role-based conspirators
• Management Override: Senior-level fraud with fraud triangle modeling (Pressure, Opportunity, Rationalization)
• Red Flag Generation: 40+ probabilistic fraud indicators with Bayesian probabilities
• Industry-Specific Transactions: Manufacturing (BOM, WIP), Retail (POS, shrinkage), Healthcare (ICD-10, claims)
• Industry-Specific Anomalies: Authentic fraud patterns per industry (upcoding, sweethearting, yield manipulation)

Machine Learning & Analytics

• Graph Export: PyTorch Geometric, Neo4j, DGL, and RustGraph formats with train/val/test splits
• Anomaly Injection: 60+ fraud types, errors, process issues with full labeling
• Data Quality Variations: Missing values (MCAR, MAR, MNAR), format variations, duplicates, typos
• Evaluation Framework: Auto-tuning with configuration recommendations based on metric gaps
• ACFE Benchmarks: ML benchmarks calibrated to ACFE fraud statistics
• Industry Benchmarks: Pre-configured benchmarks for fraud detection by industry

AI & ML-Powered Generation

• LLM-Augmented Generation: Use LLMs to generate realistic vendor names, transaction descriptions, memo fields, and anomaly explanations via pluggable provider abstraction (Mock, OpenAI, Anthropic, Custom)
• Natural Language Configuration: Generate YAML configs from natural language descriptions (init --from-description "Generate 1 year of retail data for a mid-size US company")
• Diffusion Model Backend: Statistical diffusion with configurable noise schedules (linear, cosine, sigmoid) for learned distribution capture
• Hybrid Generation: Blend rule-based and diffusion outputs using interpolation, selection, or ensemble strategies
• Causal Generation: Define Structural Causal Models (SCMs) with interventional (“what-if”) and counterfactual generation
• Built-in Causal Templates: Pre-configured fraud_detection and revenue_cycle causal graphs

Privacy-Preserving Fingerprinting

• Fingerprint Extraction: Extract statistical properties from real data into .dsf files
• Differential Privacy: Laplace and Gaussian mechanisms with configurable epsilon budget
• K-Anonymity: Suppression of rare categorical values below configurable threshold
• Privacy Audit Trail: Complete logging of all privacy decisions and epsilon spent
• Fidelity Evaluation: Validate synthetic data matches original fingerprint (KS, Wasserstein, Benford MAD)
• Gaussian Copula: Preserve multivariate correlations during synthesis
• Federated Fingerprinting: Extract fingerprints from distributed data sources without centralization using secure aggregation (weighted average, median, trimmed mean)
• Synthetic Data Certificates: Cryptographic proof of DP guarantees with HMAC-SHA256 signing, embeddable in Parquet metadata and JSON output
• Privacy-Utility Pareto Frontier: Automated exploration of optimal epsilon values for given utility targets

Production Features

• REST & gRPC APIs: Streaming generation with authentication and rate limiting
• Desktop UI: Cross-platform Tauri/SvelteKit application with 15+ configuration pages
• Resource Guards: Memory, disk, and CPU monitoring with graceful degradation
• Graceful Degradation: Progressive feature reduction under resource pressure (Normal→Reduced→Minimal→Emergency)
• Deterministic Generation: Seeded RNG (ChaCha8) for reproducible output
• Python Wrapper: Programmatic access with blueprints and config validation

Performance

Use Cases

Quick Start

# Install from source
git clone https://github.com/ey-asu-rnd/SyntheticData.git
cd SyntheticData
cargo build --release

# Run demo mode
./target/release/datasynth-data generate --demo --output ./output

# Or create a custom configuration
./target/release/datasynth-data init --industry manufacturing --complexity medium -o config.yaml
./target/release/datasynth-data generate --config config.yaml --output ./output

Fingerprinting (New in v0.2.0)

# Extract fingerprint from real data with privacy protection
./target/release/datasynth-data fingerprint extract \
    --input ./real_data.csv \
    --output ./fingerprint.dsf \
    --privacy-level standard

# Validate fingerprint integrity
./target/release/datasynth-data fingerprint validate ./fingerprint.dsf

# View fingerprint details
./target/release/datasynth-data fingerprint info ./fingerprint.dsf --detailed

# Evaluate synthetic data fidelity
./target/release/datasynth-data fingerprint evaluate \
    --fingerprint ./fingerprint.dsf \
    --synthetic ./synthetic_data/ \
    --threshold 0.8

LLM-Augmented Generation (New in v0.5.0)

# Generate config from natural language description
./target/release/datasynth-data init \
    --from-description "Generate 1 year of retail data for a mid-size US company with fraud patterns" \
    -o config.yaml

# Generate with LLM enrichment (uses mock provider by default)
./target/release/datasynth-data generate --config config.yaml --output ./output

Causal Generation (New in v0.5.0)

# Generate data with causal structure (fraud_detection template)
./target/release/datasynth-data causal generate \
    --template fraud_detection \
    --samples 10000 \
    --output ./causal_output

# Run intervention ("what-if" scenario)
./target/release/datasynth-data causal intervene \
    --template fraud_detection \
    --variable transaction_amount \
    --value 50000 \
    --samples 5000 \
    --output ./intervention_output

Diffusion Model Training (New in v0.5.0)

# Train a diffusion model on fingerprint data
./target/release/datasynth-data diffusion train \
    --fingerprint ./fingerprint.dsf \
    --output ./model.json

# Evaluate diffusion model fit
./target/release/datasynth-data diffusion evaluate \
    --model ./model.json \
    --samples 5000

Python Wrapper

from datasynth_py import DataSynth
from datasynth_py.config import blueprints

config = blueprints.retail_small(companies=4, transactions=10000)
synth = DataSynth()
result = synth.generate(config=config, output={"format": "csv", "sink": "temp_dir"})
print(result.output_dir)

Architecture

SyntheticData is organized as a Rust workspace with 15 modular crates:

datasynth-cli          Command-line interface (binary: datasynth-data)
datasynth-server       REST/gRPC/WebSocket server with auth and rate limiting
datasynth-ui           Tauri/SvelteKit desktop application
    │
datasynth-runtime      Orchestration layer (parallel execution, resource guards)
    │
datasynth-generators   Data generators (JE, documents, subledgers, anomalies, audit)
datasynth-banking      KYC/AML banking transaction generator
datasynth-ocpm         Object-Centric Process Mining (OCEL 2.0)
datasynth-fingerprint  Privacy-preserving fingerprint extraction and synthesis
datasynth-standards    Accounting/audit standards (US GAAP, IFRS, ISA, SOX)
    │
datasynth-graph        Graph/network export (PyTorch Geometric, Neo4j, DGL)
datasynth-eval         Evaluation framework with auto-tuning
    │
datasynth-config       Configuration schema, validation, industry presets
    │
datasynth-core         Domain models, traits, distributions, resource guards
    │
datasynth-output       Output sinks (CSV, JSON, Parquet, streaming)
datasynth-test-utils   Test utilities, fixtures, mocks

License

Copyright 2024-2026 Michael Ivertowski, Ernst & Young Ltd., Zurich, Switzerland

Licensed under the Apache License, Version 2.0. See LICENSE for details.

Support

Commercial support, custom development, and enterprise licensing are available upon request. Please contact the author at michael.ivertowski@ch.ey.com for inquiries.

SyntheticData is provided “as is” without warranty of any kind. It is intended for testing, development, and educational purposes. Generated data should not be used as a substitute for real financial records.

Getting Started

Welcome to SyntheticData! This section will help you get up and running quickly.

What You’ll Learn

• Installation: Set up SyntheticData on your system
• Quick Start: Generate your first synthetic dataset
• Demo Mode: Explore SyntheticData with built-in demo presets

Prerequisites

Before you begin, ensure you have:

• Rust 1.88+: SyntheticData is written in Rust and requires the Rust toolchain
• Git: For cloning the repository
• (Optional) Node.js 18+: Required only for the desktop UI

Installation Overview

# Clone and build
git clone https://github.com/ey-asu-rnd/SyntheticData.git
cd SyntheticData
cargo build --release

# The binary is at target/release/datasynth-data

First Steps

The fastest way to explore SyntheticData is through demo mode:

datasynth-data generate --demo --output ./demo-output

This generates a complete set of synthetic financial data using sensible defaults.

Architecture at a Glance

SyntheticData generates interconnected financial data:

┌─────────────────────────────────────────────────────────────┐
│                    Configuration (YAML)                      │
├─────────────────────────────────────────────────────────────┤
│                    Generation Pipeline                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │  Master  │→│ Document │→│  Journal │→│  Output  │     │
│  │   Data   │  │  Flows   │  │ Entries  │  │  Files   │     │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘    │
├─────────────────────────────────────────────────────────────┤
│  Output: CSV, JSON, Neo4j, PyTorch Geometric, ACDOCA        │
└─────────────────────────────────────────────────────────────┘

Next Steps

1. Follow the Installation Guide to set up your environment
2. Work through the Quick Start Tutorial
3. Explore Demo Mode for a hands-on introduction
4. Review the CLI Reference for all available commands

Getting Help

• Check the User Guide for detailed usage instructions
• Review Configuration for all available options
• See Use Cases for real-world examples

Installation

This guide covers installing SyntheticData from source.

Prerequisites

Required

Optional

Installing Rust

If you don’t have Rust installed, use rustup:

# Linux/macOS
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Windows
# Download and run rustup-init.exe from https://rustup.rs

# Verify installation
rustc --version
cargo --version

Building from Source

Clone the Repository

git clone https://github.com/ey-asu-rnd/SyntheticData.git
cd SyntheticData

Build Release Binary

# Build optimized release binary
cargo build --release

# The binary is at target/release/datasynth-data

Verify Installation

# Check version
./target/release/datasynth-data --version

# View help
./target/release/datasynth-data --help

# Run quick validation
./target/release/datasynth-data info

Adding to PATH

To run datasynth-data from anywhere:

Linux/macOS

# Option 1: Symlink to local bin
ln -s $(pwd)/target/release/datasynth-data ~/.local/bin/datasynth-data

# Option 2: Copy to system bin (requires sudo)
sudo cp target/release/datasynth-data /usr/local/bin/

# Option 3: Add target/release to PATH in ~/.bashrc or ~/.zshrc
export PATH="$PATH:/path/to/SyntheticData/target/release"

Windows

Add the target/release directory to your system PATH environment variable.

Building the Desktop UI

The desktop UI requires additional setup:

# Navigate to UI crate
cd crates/datasynth-ui

# Install Node.js dependencies
npm install

# Run in development mode
npm run tauri dev

# Build production release
npm run tauri build

Platform-Specific Dependencies

Linux (Ubuntu/Debian):

sudo apt-get install libwebkit2gtk-4.1-dev \
    libgtk-3-dev \
    libayatana-appindicator3-dev \
    librsvg2-dev

macOS: No additional dependencies required.

Windows: Install WebView2 runtime (usually pre-installed on Windows 10/11).

Running Tests

Verify your installation by running the test suite:

# Run all tests
cargo test

# Run tests for a specific crate
cargo test -p datasynth-core
cargo test -p datasynth-generators

# Run with output
cargo test -- --nocapture

Development Setup

For development work:

# Check code without building
cargo check

# Format code
cargo fmt

# Run lints
cargo clippy

# Build documentation
cargo doc --workspace --no-deps --open

Running Benchmarks

# Run all benchmarks
cargo bench

# Run specific benchmark
cargo bench --bench generation_throughput

Troubleshooting

Build Failures

Missing C compiler:

# Ubuntu/Debian
sudo apt-get install build-essential

# macOS
xcode-select --install

# Fedora/RHEL
sudo dnf install gcc

Out of memory during build:

# Limit parallel jobs
cargo build --release -j 2

Runtime Issues

Permission denied:

chmod +x target/release/datasynth-data

Library not found (Linux):

# Check for missing dependencies
ldd target/release/datasynth-data

Next Steps

• Follow the Quick Start Guide to generate your first dataset
• Explore Demo Mode for a hands-on introduction
• Review the CLI Reference for all commands

Quick Start

This guide walks you through generating your first synthetic financial dataset.

Overview

The typical workflow is:

1. Initialize a configuration file
2. Validate the configuration
3. Generate synthetic data
4. Review the output

Step 1: Initialize Configuration

Create a configuration file for your industry and complexity needs:

# Manufacturing company with medium complexity (~400 accounts)
datasynth-data init --industry manufacturing --complexity medium -o config.yaml

Available Industry Presets

Complexity Levels

Step 2: Review Configuration

Open config.yaml to review and customize:

global:
  seed: 42                        # For reproducible generation
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 12
  group_currency: USD

companies:
  - code: "1000"
    name: "Headquarters"
    currency: USD
    country: US
    volume_weight: 1.0

transactions:
  target_count: 100000            # Number of journal entries

fraud:
  enabled: true
  fraud_rate: 0.005               # 0.5% fraud transactions

output:
  format: csv
  compression: none

See the Configuration Guide for all options.

Step 3: Validate Configuration

Check your configuration for errors:

datasynth-data validate --config config.yaml

The validator checks:

• Required fields are present
• Values are within valid ranges
• Distribution weights sum to 1.0
• Dates are consistent

Step 4: Generate Data

Run the generation:

datasynth-data generate --config config.yaml --output ./output

You’ll see a progress bar:

Generating synthetic data...
[████████████████████████████████] 100000/100000 entries
Generation complete in 1.2s

Step 5: Explore Output

The output directory contains organized subdirectories:

output/
├── master_data/
│   ├── vendors.csv
│   ├── customers.csv
│   ├── materials.csv
│   └── employees.csv
├── transactions/
│   ├── journal_entries.csv
│   ├── acdoca.csv
│   ├── purchase_orders.csv
│   └── vendor_invoices.csv
├── subledgers/
│   ├── ar_open_items.csv
│   └── ap_open_items.csv
├── period_close/
│   └── trial_balances/
├── labels/
│   ├── anomaly_labels.csv
│   └── fraud_labels.csv
└── controls/
    └── internal_controls.csv

Common Customizations

Generate More Data

transactions:
  target_count: 1000000           # 1 million entries

Enable Graph Export

graph_export:
  enabled: true
  formats:
    - pytorch_geometric
    - neo4j

Add Anomaly Injection

anomaly_injection:
  enabled: true
  total_rate: 0.02                # 2% anomaly rate
  generate_labels: true           # For ML training

Multiple Companies

companies:
  - code: "1000"
    name: "Headquarters"
    currency: USD
    volume_weight: 0.6

  - code: "2000"
    name: "European Subsidiary"
    currency: EUR
    volume_weight: 0.4

Next Steps

• Explore Demo Mode for built-in presets
• Learn the CLI Reference
• Review Output Formats
• See Configuration for all options

Quick Reference

# Common commands
datasynth-data init --industry <industry> --complexity <level> -o config.yaml
datasynth-data validate --config config.yaml
datasynth-data generate --config config.yaml --output ./output
datasynth-data generate --demo --output ./demo-output
datasynth-data info                   # Show available presets

Demo Mode

Demo mode provides a quick way to explore SyntheticData without creating a configuration file. It uses sensible defaults to generate a complete synthetic dataset.

Running Demo Mode

datasynth-data generate --demo --output ./demo-output

What Demo Mode Generates

Demo mode creates a comprehensive dataset with:

Demo Configuration

Demo mode uses these defaults:

global:
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 3
  group_currency: USD

companies:
  - code: "1000"
    name: "Demo Company"
    currency: USD
    country: US

chart_of_accounts:
  complexity: medium              # ~400 accounts

transactions:
  target_count: 10000

fraud:
  enabled: true
  fraud_rate: 0.005

anomaly_injection:
  enabled: true
  total_rate: 0.01
  generate_labels: true

output:
  format: csv

Output Structure

After running demo mode, explore the output:

tree demo-output/

demo-output/
├── master_data/
│   ├── chart_of_accounts.csv     # GL accounts
│   ├── vendors.csv               # Vendor master
│   ├── customers.csv             # Customer master
│   ├── materials.csv             # Material/product master
│   └── employees.csv             # Employee/user master
├── transactions/
│   ├── journal_entries.csv       # Main JE file
│   ├── acdoca.csv                # SAP HANA format
│   ├── purchase_orders.csv       # P2P documents
│   ├── goods_receipts.csv
│   ├── vendor_invoices.csv
│   ├── payments.csv
│   ├── sales_orders.csv          # O2C documents
│   ├── deliveries.csv
│   ├── customer_invoices.csv
│   └── customer_receipts.csv
├── subledgers/
│   ├── ar_open_items.csv
│   ├── ap_open_items.csv
│   └── inventory_positions.csv
├── period_close/
│   └── trial_balances/
│       ├── 2024_01.csv
│       ├── 2024_02.csv
│       └── 2024_03.csv
├── labels/
│   ├── anomaly_labels.csv        # For ML training
│   └── fraud_labels.csv
└── controls/
    ├── internal_controls.csv
    └── sod_rules.csv

Exploring the Data

Journal Entries

head -5 demo-output/transactions/journal_entries.csv

Key fields:

• document_id: Unique transaction identifier
• posting_date: When the entry was posted
• company_code: Company identifier
• account_number: GL account
• debit_amount / credit_amount: Entry amounts
• is_fraud: Fraud label (true/false)
• is_anomaly: Anomaly label

Fraud Labels

# View fraud transactions
grep "true" demo-output/labels/fraud_labels.csv | head

Trial Balance

# Check balance coherence
head demo-output/period_close/trial_balances/2024_01.csv

Customizing Demo Output

You can combine demo mode with some options:

# Change output directory
datasynth-data generate --demo --output ./my-demo

# Use demo as starting point, then create config
datasynth-data init --industry manufacturing --complexity medium -o config.yaml
# Edit config.yaml as needed
datasynth-data generate --config config.yaml --output ./output

Use Cases for Demo Mode

Quick Exploration

Test SyntheticData’s capabilities before creating a custom configuration.

Development Testing

Generate test data quickly for development purposes.

Training & Workshops

Provide sample data for training sessions without complex setup.

Benchmarking

Establish baseline performance metrics.

Moving Beyond Demo Mode

When you’re ready for more control:

1. Create a configuration file:

   datasynth-data init --industry <your-industry> -o config.yaml
   

2. Customize settings:

   • Adjust transaction volume
   • Configure multiple companies
   • Enable graph export
   • Fine-tune fraud/anomaly rates

3. Generate with your config:

   datasynth-data generate --config config.yaml --output ./output
   

Next Steps

• Review Quick Start for custom configurations
• Learn the CLI Reference
• Explore Configuration Options
• See Use Cases for real-world examples

User Guide

This section covers the different ways to use SyntheticData.

Interfaces

SyntheticData offers three interfaces:

Quick Comparison

CLI Overview

The command-line interface (datasynth-data) is ideal for:

• Batch generation
• CI/CD pipelines
• Scripting and automation
• Server environments

datasynth-data generate --config config.yaml --output ./output

Server Overview

The server (datasynth-server) provides:

• REST API for configuration and control
• gRPC for high-performance integration
• WebSocket for real-time streaming

cargo run -p datasynth-server -- --port 3000

Desktop UI Overview

The desktop application offers:

• Visual configuration editor
• Industry preset selector
• Real-time generation monitoring
• Cross-platform support (Windows, macOS, Linux)

cd crates/datasynth-ui && npm run tauri dev

Output Formats

SyntheticData produces various output formats:

• CSV: Standard tabular data
• JSON: Structured data with nested objects
• ACDOCA: SAP HANA Universal Journal format
• PyTorch Geometric: ML-ready graph tensors
• Neo4j: Graph database import format

See Output Formats for details.

Choosing an Interface

Use the CLI if you:

• Need to automate generation
• Work in headless/server environments
• Prefer command-line tools
• Want to integrate with shell scripts

Use the Server if you:

• Build applications that consume synthetic data
• Need streaming generation
• Want API-based control
• Integrate with microservices

Use the Desktop UI if you:

• Prefer visual configuration
• Want to explore options interactively
• Need real-time monitoring
• Are new to SyntheticData

Next Steps

• CLI Reference - Complete command documentation
• Server API - REST, gRPC, and WebSocket endpoints
• Desktop UI - Desktop application guide
• Output Formats - Detailed output file documentation

CLI Reference

The datasynth-data command-line tool provides commands for generating synthetic financial data and extracting fingerprints from real data.

Installation

After building the project, the binary is at target/release/datasynth-data.

cargo build --release
./target/release/datasynth-data --help

Global Options

Commands

generate

Generate synthetic financial data.

datasynth-data generate [OPTIONS]

Options:

Examples:

# Generate with configuration file
datasynth-data generate --config config.yaml --output ./output

# Use demo mode
datasynth-data generate --demo --output ./demo-output

# Override seed for reproducibility
datasynth-data generate --config config.yaml --output ./output --seed 12345

# JSON output format
datasynth-data generate --config config.yaml --output ./output --format json

init

Create a new configuration file from industry presets.

datasynth-data init [OPTIONS]

Options:

Available Industries:

• manufacturing
• retail
• financial_services
• healthcare
• technology

Examples:

# Create manufacturing config
datasynth-data init --industry manufacturing --complexity medium -o config.yaml

# Create large retail config
datasynth-data init --industry retail --complexity large -o retail.yaml

validate

Validate a configuration file.

datasynth-data validate --config <PATH>

Options:

Example:

datasynth-data validate --config config.yaml

Validation Checks:

• Required fields present
• Value ranges (period_months: 1-120)
• Distribution weights sum to 1.0 (±0.01 tolerance)
• Date consistency
• Company code uniqueness
• Compression level: 1-9 when enabled
• All rate/percentage fields: 0.0-1.0
• Approval thresholds: strictly ascending order

info

Display available presets and configuration options.

datasynth-data info

Output includes:

• Available industry presets
• Complexity levels
• Supported output formats
• Feature capabilities

fingerprint

Privacy-preserving fingerprint extraction and evaluation. This command has several subcommands.

datasynth-data fingerprint <SUBCOMMAND>

fingerprint extract

Extract a fingerprint from real data with privacy controls.

datasynth-data fingerprint extract [OPTIONS]

Options:

Privacy Levels:

Examples:

# Extract with standard privacy
datasynth-data fingerprint extract \
    --input ./real_data.csv \
    --output ./fingerprint.dsf \
    --privacy-level standard

# Extract with custom privacy parameters
datasynth-data fingerprint extract \
    --input ./real_data.csv \
    --output ./fingerprint.dsf \
    --epsilon 0.75 \
    --k 8

fingerprint validate

Validate a fingerprint file’s integrity and structure.

datasynth-data fingerprint validate <PATH>

Arguments:

Validation Checks:

• DSF file structure (ZIP archive with required components)
• SHA-256 checksums for all components
• Required fields in manifest, schema, statistics
• Privacy audit completeness

Example:

datasynth-data fingerprint validate ./fingerprint.dsf

fingerprint info

Display fingerprint metadata and statistics.

datasynth-data fingerprint info <PATH> [OPTIONS]

Arguments:

Options:

Examples:

# Basic info
datasynth-data fingerprint info ./fingerprint.dsf

# Detailed statistics
datasynth-data fingerprint info ./fingerprint.dsf --detailed

# JSON output for scripting
datasynth-data fingerprint info ./fingerprint.dsf --json

fingerprint diff

Compare two fingerprints.

datasynth-data fingerprint diff <PATH1> <PATH2>

Arguments:

Output includes:

• Schema differences (columns added/removed/changed)
• Statistical distribution changes
• Correlation matrix differences

Example:

datasynth-data fingerprint diff ./fp_v1.dsf ./fp_v2.dsf

fingerprint evaluate

Evaluate synthetic data fidelity against a fingerprint.

datasynth-data fingerprint evaluate [OPTIONS]

Options:

Fidelity Metrics:

• Statistical: KS statistic, Wasserstein distance, Benford MAD
• Correlation: Correlation matrix RMSE
• Schema: Column type match, row count ratio
• Rules: Balance equation compliance rate

Examples:

# Basic evaluation
datasynth-data fingerprint evaluate \
    --fingerprint ./fingerprint.dsf \
    --synthetic ./synthetic_data/ \
    --threshold 0.8

# Generate HTML report
datasynth-data fingerprint evaluate \
    --fingerprint ./fingerprint.dsf \
    --synthetic ./synthetic_data/ \
    --threshold 0.85 \
    --report ./fidelity_report.html

diffusion (v0.5.0)

Train and evaluate diffusion models for statistical data generation.

diffusion train

Train a diffusion model from a fingerprint file.

datasynth-data diffusion train \
    --fingerprint ./fingerprint.dsf \
    --output ./model.json \
    --n-steps 1000 \
    --schedule cosine

diffusion evaluate

Evaluate a trained diffusion model’s fit quality.

datasynth-data diffusion evaluate \
    --model ./model.json \
    --samples 5000

causal (v0.5.0)

Generate data with causal structure, run interventions, and produce counterfactuals.

causal generate

Generate data following a causal graph structure.

datasynth-data causal generate \
    --template fraud_detection \
    --samples 10000 \
    --seed 42 \
    --output ./causal_output

causal intervene

Run do-calculus interventions (“what-if” scenarios).

datasynth-data causal intervene \
    --template fraud_detection \
    --variable transaction_amount \
    --value 50000 \
    --samples 5000 \
    --output ./intervention_output

causal validate

Validate that generated data preserves causal structure.

datasynth-data causal validate \
    --data ./causal_output \
    --template fraud_detection

fingerprint federated (v0.5.0)

Aggregate fingerprints from multiple distributed sources without centralizing raw data.

datasynth-data fingerprint federated \
    --sources ./source_a.dsf ./source_b.dsf ./source_c.dsf \
    --output ./aggregated.dsf \
    --method weighted_average \
    --max-epsilon 5.0

init –from-description (v0.5.0)

Generate configuration from a natural language description using LLM.

datasynth-data init \
    --from-description "Generate 1 year of retail data for a mid-size US company with fraud patterns" \
    -o config.yaml

Uses the configured LLM provider (defaults to Mock) to parse the description and generate an appropriate YAML configuration.

generate –certificate (v0.5.0)

Attach a synthetic data certificate to the generated output.

datasynth-data generate \
    --config config.yaml \
    --output ./output \
    --certificate

Produces a certificate.json in the output directory containing DP guarantees, quality metrics, and an HMAC-SHA256 signature.

Signal Handling (Unix)

On Unix systems, you can pause and resume generation:

# Start generation in background
datasynth-data generate --config config.yaml --output ./output &

# Pause generation
kill -USR1 $(pgrep datasynth-data)

# Resume generation (send SIGUSR1 again)
kill -USR1 $(pgrep datasynth-data)

Exit Codes

Environment Variables

Example:

SYNTH_DATA_LOG=debug datasynth-data generate --config config.yaml --output ./output

Configuration File Location

The tool looks for configuration files in this order:

1. Path specified with --config
2. ./datasynth-data.yaml in current directory
3. ~/.config/datasynth-data/config.yaml

Output Directory Structure

Generation creates this structure:

output/
├── master_data/          Vendors, customers, materials, assets, employees
├── transactions/         Journal entries, purchase orders, invoices, payments
├── subledgers/           AR, AP, FA, inventory detail records
├── period_close/         Trial balances, accruals, closing entries
├── consolidation/        Eliminations, currency translation
├── fx/                   Exchange rates, CTA adjustments
├── banking/              KYC profiles, bank transactions, AML typology labels
├── process_mining/       OCEL 2.0 event logs, process variants
├── audit/                Engagements, workpapers, findings, risk assessments
├── graphs/               PyTorch Geometric, Neo4j, DGL exports (if enabled)
├── labels/               Anomaly, fraud, and data quality labels for ML
└── controls/             Internal control mappings, SoD rules

Scripting Examples

Batch Generation

#!/bin/bash
for industry in manufacturing retail healthcare; do
    datasynth-data init --industry $industry --complexity medium -o ${industry}.yaml
    datasynth-data generate --config ${industry}.yaml --output ./output/${industry}
done

CI/CD Pipeline

# GitHub Actions example
- name: Generate Test Data
  run: |
    cargo build --release
    ./target/release/datasynth-data generate --demo --output ./test-data

- name: Validate Generation
  run: |
    # Check output files exist
    test -f ./test-data/transactions/journal_entries.csv
    test -f ./test-data/master_data/vendors.csv

Reproducible Generation

# Same seed produces identical output
datasynth-data generate --config config.yaml --output ./run1 --seed 42
datasynth-data generate --config config.yaml --output ./run2 --seed 42
diff -r run1 run2  # No differences

Fingerprint Pipeline

#!/bin/bash
# Extract fingerprint from real data
datasynth-data fingerprint extract \
    --input ./real_data.csv \
    --output ./fingerprint.dsf \
    --privacy-level high

# Validate the fingerprint
datasynth-data fingerprint validate ./fingerprint.dsf

# Generate synthetic data matching the fingerprint
# (fingerprint informs config generation)
datasynth-data generate --config generated_config.yaml --output ./synthetic

# Evaluate fidelity
datasynth-data fingerprint evaluate \
    --fingerprint ./fingerprint.dsf \
    --synthetic ./synthetic \
    --threshold 0.85 \
    --report ./fidelity_report.html

Troubleshooting

Common Issues

“Configuration file not found”

# Check file path
ls -la config.yaml
# Use absolute path
datasynth-data generate --config /full/path/to/config.yaml --output ./output

“Invalid configuration”

# Validate first
datasynth-data validate --config config.yaml

“Permission denied”

# Check output directory permissions
mkdir -p ./output
chmod 755 ./output

“Out of memory”

The generator includes memory guards that prevent OOM conditions. If you still encounter issues:

• Reduce transaction count in configuration
• The system will automatically reduce batch sizes under memory pressure
• Check memory_guard settings in configuration

“Fingerprint validation failed”

# Check DSF file integrity
datasynth-data fingerprint validate ./fingerprint.dsf

# View detailed info
datasynth-data fingerprint info ./fingerprint.dsf --detailed

“Low fidelity score”

If synthetic data fidelity is below threshold:

• Review the fidelity report for specific metrics
• Adjust configuration to better match fingerprint statistics
• Consider using the evaluation framework’s auto-tuning recommendations

See Also

• Quick Start
• Configuration Reference
• Output Formats
• Fingerprinting Guide
• Python Wrapper

Server API

SyntheticData provides a server component with REST, gRPC, and WebSocket APIs for application integration.

Starting the Server

cargo run -p datasynth-server -- --port 3000 --worker-threads 4

Options:

Authentication

When --api-key is set, include it in requests:

curl -H "X-API-Key: your-api-key" http://localhost:3000/api/config

REST API

Configuration Endpoints

GET /api/config

Retrieve current configuration.

curl http://localhost:3000/api/config

Response:

{
  "global": {
    "seed": 42,
    "industry": "manufacturing",
    "start_date": "2024-01-01",
    "period_months": 12
  },
  "transactions": {
    "target_count": 100000
  }
}

POST /api/config

Update configuration.

curl -X POST http://localhost:3000/api/config \
  -H "Content-Type: application/json" \
  -d '{"transactions": {"target_count": 50000}}'

POST /api/config/validate

Validate configuration without applying.

curl -X POST http://localhost:3000/api/config/validate \
  -H "Content-Type: application/json" \
  -d @config.json

Stream Control Endpoints

POST /api/stream/start

Start data generation.

curl -X POST http://localhost:3000/api/stream/start

Response:

{
  "status": "started",
  "stream_id": "abc123"
}

POST /api/stream/stop

Stop current generation.

curl -X POST http://localhost:3000/api/stream/stop

POST /api/stream/pause

Pause generation.

curl -X POST http://localhost:3000/api/stream/pause

POST /api/stream/resume

Resume paused generation.

curl -X POST http://localhost:3000/api/stream/resume

Pattern Trigger Endpoints

POST /api/stream/trigger/

Trigger special event patterns.

Available patterns:

• month_end - Month-end close activities
• quarter_end - Quarter-end activities
• year_end - Year-end close activities

curl -X POST http://localhost:3000/api/stream/trigger/month_end

Health Check

GET /health

curl http://localhost:3000/health

Response:

{
  "status": "healthy",
  "uptime_seconds": 3600
}

WebSocket API

Connect to receive real-time events during generation.

Connection

const ws = new WebSocket('ws://localhost:3000/ws/events');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log(data);
};

Event Types

Progress Event:

{
  "type": "progress",
  "current": 50000,
  "total": 100000,
  "percent": 50.0,
  "rate": 85000.5
}

Entry Event:

{
  "type": "entry",
  "data": {
    "document_id": "abc-123",
    "posting_date": "2024-03-15",
    "account": "1100",
    "debit": "1000.00",
    "credit": "0.00"
  }
}

Error Event:

{
  "type": "error",
  "message": "Memory limit exceeded"
}

Complete Event:

{
  "type": "complete",
  "total_entries": 100000,
  "duration_ms": 1200
}

gRPC API

Proto Definition

syntax = "proto3";

package synth;

service SynthService {
  rpc GetConfig(Empty) returns (Config);
  rpc SetConfig(Config) returns (Status);
  rpc StartGeneration(GenerationRequest) returns (stream Entry);
  rpc StopGeneration(Empty) returns (Status);
}

message Config {
  string yaml = 1;
}

message GenerationRequest {
  optional int64 count = 1;
}

message Entry {
  string document_id = 1;
  string posting_date = 2;
  string company_code = 3;
  repeated LineItem lines = 4;
}

message LineItem {
  string account = 1;
  string debit = 2;
  string credit = 3;
}

Client Example (Rust)

use synth::synth_client::SynthClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = SynthClient::connect("http://localhost:50051").await?;

    let request = tonic::Request::new(GenerationRequest { count: Some(1000) });
    let mut stream = client.start_generation(request).await?.into_inner();

    while let Some(entry) = stream.message().await? {
        println!("Entry: {:?}", entry.document_id);
    }

    Ok(())
}

Rate Limiting

The server implements sliding window rate limiting:

Exceeding the limit returns 429 Too Many Requests:

{
  "error": "rate_limit_exceeded",
  "retry_after": 30
}

Memory Management

The server enforces memory limits:

# Set memory limit (bytes)
cargo run -p datasynth-server -- --memory-limit 1073741824  # 1GB

When memory is low:

• Generation pauses automatically
• WebSocket sends warning event
• New requests may be rejected

Error Responses

Error Response Format:

{
  "error": "error_code",
  "message": "Human readable description",
  "details": {}
}

Integration Examples

Python Client

import requests
import websocket
import json

BASE_URL = "http://localhost:3000"

# Set configuration
config = {
    "transactions": {"target_count": 10000}
}
requests.post(f"{BASE_URL}/api/config", json=config)

# Start generation
requests.post(f"{BASE_URL}/api/stream/start")

# Monitor via WebSocket
ws = websocket.create_connection(f"ws://localhost:3000/ws/events")
while True:
    event = json.loads(ws.recv())
    if event["type"] == "complete":
        break
    print(f"Progress: {event.get('percent', 0)}%")

Node.js Client

const axios = require('axios');
const WebSocket = require('ws');

const BASE_URL = 'http://localhost:3000';

async function generate() {
    // Configure
    await axios.post(`${BASE_URL}/api/config`, {
        transactions: { target_count: 10000 }
    });

    // Start
    await axios.post(`${BASE_URL}/api/stream/start`);

    // Monitor
    const ws = new WebSocket('ws://localhost:3000/ws/events');
    ws.on('message', (data) => {
        const event = JSON.parse(data);
        console.log(event);
    });
}

See Also

• CLI Reference
• Configuration
• Performance Tuning

Desktop UI

SyntheticData includes a cross-platform desktop application built with Tauri and SvelteKit.

Overview

The desktop UI provides:

• Visual configuration editing
• Industry preset selection
• Real-time generation monitoring
• Configuration validation feedback

Installation

Prerequisites

Platform Dependencies

Linux (Ubuntu/Debian):

sudo apt-get install libwebkit2gtk-4.1-dev \
    libgtk-3-dev \
    libayatana-appindicator3-dev \
    librsvg2-dev

macOS: No additional dependencies required.

Windows: WebView2 runtime (usually pre-installed on Windows 10/11).

Running in Development

cd crates/datasynth-ui
npm install
npm run tauri dev

Building for Production

cd crates/datasynth-ui
npm run tauri build

Build outputs are in crates/datasynth-ui/src-tauri/target/release/bundle/.

Application Layout

Dashboard

The main dashboard provides:

• Quick stats overview
• Recent generation history
• System status

Configuration Editor

Access via the sidebar. Configuration is organized into sections:

Configuration Sections

Global Settings

• Industry: Select from presets (manufacturing, retail, etc.)
• Start Date: Beginning of simulation period
• Period Months: Duration (1-120 months)
• Group Currency: Base currency for consolidation
• Random Seed: For reproducible generation

Chart of Accounts

• Complexity: Small (~100), Medium (~400), Large (~2500) accounts
• Structure: Industry-specific account hierarchies

Transactions

• Target Count: Number of journal entries to generate
• Line Item Distribution: Configure line count probabilities
• Amount Distribution: Log-normal parameters, round number bias

Master Data

Configure generation parameters for:

• Vendors (count, payment terms, intercompany flags)
• Customers (count, credit terms, payment behavior)
• Materials (count, valuation methods)
• Fixed Assets (count, depreciation methods)
• Employees (count, hierarchy depth)

Document Flows

• P2P (Procure-to-Pay): PO → GR → Invoice → Payment rates
• O2C (Order-to-Cash): SO → Delivery → Invoice → Receipt rates
• Three-Way Match: Tolerance settings

Financial Settings

• Balance: Opening balance configuration
• Subledger: AR, AP, FA, Inventory settings
• FX: Currency pairs, rate volatility
• Period Close: Accrual, depreciation, closing settings

Compliance

• Fraud: Enable/disable, fraud rate, fraud types
• Controls: Internal control definitions
• Approval: Threshold configuration, SoD rules

Analytics

• Graph Export: Format selection (PyTorch Geometric, Neo4j, DGL)
• Anomaly Injection: Rate, types, labeling
• Data Quality: Missing values, format variations, duplicates

Output Settings

• Format: CSV or JSON
• Compression: None, gzip, or zstd
• File Organization: Directory structure options

Preset Selector

Quickly load industry presets:

1. Click “Load Preset” in the header
2. Select industry
3. Choose complexity level
4. Click “Apply”

Real-time Streaming

During generation, view:

• Progress bar with percentage
• Entries per second
• Memory usage
• Recent entries table

Access streaming view via “Generate” → “Stream”.

Validation

The UI validates configuration in real-time:

• Required fields are highlighted
• Invalid values show error messages
• Distribution weights are checked
• Constraints are enforced

Keyboard Shortcuts

Configuration Files

The UI stores configurations in:

Exporting Configuration

To use your configuration with the CLI:

1. Configure in the UI
2. Click “Export” → “Export YAML”
3. Save the .yaml file
4. Use with CLI: datasynth-data generate --config exported.yaml --output ./output

Development

Project Structure

crates/datasynth-ui/
├── src/                      # SvelteKit frontend
│   ├── routes/               # Page routes
│   │   ├── +page.svelte      # Dashboard
│   │   ├── generate/         # Generation views
│   │   └── config/           # Configuration pages
│   └── lib/
│       ├── stores/           # State management
│       └── components/       # Reusable components
├── src-tauri/                # Rust backend
│   └── src/
│       └── main.rs           # Tauri commands
├── package.json
└── tauri.conf.json

Adding a Configuration Page

1. Create route in src/routes/config/<section>/+page.svelte
2. Add form components
3. Connect to config store
4. Add navigation link

Debugging

# Enable Tauri dev tools
npm run tauri dev

# View browser console (Ctrl/Cmd + Shift + I in dev mode)

Troubleshooting

UI Doesn’t Start

# Check Node dependencies
npm install

# Rebuild native modules
npm run tauri clean
npm run tauri build

Configuration Not Saving

Check file permissions in the config directory.

WebSocket Connection Failed

Ensure the server is running if using streaming features:

cargo run -p datasynth-server -- --port 3000

See Also

• CLI Reference
• Server API
• Configuration

Output Formats

SyntheticData generates multiple file types organized into categories.

Output Directory Structure

output/
├── master_data/          # Entity master records
├── transactions/         # Journal entries and documents
├── subledgers/           # Subsidiary ledger records
├── period_close/         # Trial balances and closing
├── consolidation/        # Elimination entries
├── fx/                   # Exchange rates
├── banking/              # KYC profiles and bank transactions
├── process_mining/       # OCEL 2.0 event logs
├── audit/                # Audit engagements and workpapers
├── graphs/               # ML-ready graph exports
├── labels/               # Anomaly, fraud, and quality labels
└── controls/             # Internal control mappings

File Formats

CSV

Default format with standard conventions:

• UTF-8 encoding
• Comma-separated values
• Header row included
• Quoted strings when needed
• Decimal values serialized as strings (prevents floating-point artifacts)

Example (journal_entries.csv):

document_id,posting_date,company_code,account,description,debit,credit,is_fraud
abc-123,2024-01-15,1000,1100,Customer payment,"1000.00","0.00",false
abc-123,2024-01-15,1000,1200,Cash receipt,"0.00","1000.00",false

JSON

Structured format with nested objects:

Example (journal_entries.json):

[
  {
    "header": {
      "document_id": "abc-123",
      "posting_date": "2024-01-15",
      "company_code": "1000",
      "source": "Manual",
      "is_fraud": false
    },
    "lines": [
      {
        "account": "1100",
        "description": "Customer payment",
        "debit": "1000.00",
        "credit": "0.00"
      },
      {
        "account": "1200",
        "description": "Cash receipt",
        "debit": "0.00",
        "credit": "1000.00"
      }
    ]
  }
]

ACDOCA (SAP HANA)

SAP Universal Journal format with simulation fields:

Master Data Files

chart_of_accounts.csv

vendors.csv

customers.csv

materials.csv

employees.csv

Transaction Files

journal_entries.csv

Line Items (embedded or separate)

Document Flow Files

purchase_orders.csv:

• Order header with vendor, dates, status
• Line items with materials, quantities, prices

goods_receipts.csv:

• Receipt linked to PO
• Quantities received, variances

vendor_invoices.csv:

• Invoice with three-way match status
• Payment terms, due date

payments.csv:

• Payment documents
• Bank references, cleared invoices

document_references.csv:

• Links between documents (FollowOn, Payment, Reversal)
• Ensures complete document chains

Subledger Files

ar_open_items.csv

ap_open_items.csv

Similar structure for payables.

fa_register.csv

inventory_positions.csv

Period Close Files

trial_balances/YYYY_MM.csv

accruals.csv

Accrual entries with reversal dates.

depreciation.csv

Monthly depreciation entries per asset.

Banking Files

banking_customers.csv

bank_accounts.csv

bank_transactions.csv

kyc_profiles.csv

aml_typology_labels.csv

entity_risk_labels.csv

Process Mining Files (OCEL 2.0)

event_log.json

OCEL 2.0 format event log:

{
  "ocel:global-log": {
    "ocel:version": "2.0",
    "ocel:ordering": "timestamp"
  },
  "ocel:events": {
    "e1": {
      "ocel:activity": "Create Purchase Order",
      "ocel:timestamp": "2024-01-15T10:30:00Z",
      "ocel:typedOmap": [
        {"ocel:oid": "PO-001", "ocel:qualifier": "order"}
      ]
    }
  },
  "ocel:objects": {
    "PO-001": {
      "ocel:type": "PurchaseOrder",
      "ocel:attributes": {
        "vendor": "VEND-001",
        "amount": "10000.00"
      }
    }
  }
}

objects.json

Object instances with types and attributes.

events.json

Event records with object relationships.

process_variants.csv

Audit Files

audit_engagements.csv

audit_workpapers.csv

audit_evidence.csv

audit_risks.csv

audit_findings.csv

audit_judgments.csv

Graph Export Files

PyTorch Geometric

graphs/transaction_network/pytorch_geometric/
├── node_features.pt    # [num_nodes, features]
├── edge_index.pt       # [2, num_edges]
├── edge_attr.pt        # [num_edges, edge_features]
├── labels.pt           # Node/edge labels
├── train_mask.pt       # Training split
├── val_mask.pt         # Validation split
└── test_mask.pt        # Test split

Neo4j

graphs/entity_relationship/neo4j/
├── nodes_account.csv
├── nodes_entity.csv
├── nodes_user.csv
├── edges_transaction.csv
├── edges_approval.csv
└── import.cypher        # Import script

DGL (Deep Graph Library)

graphs/transaction_network/dgl/
├── graph.bin           # DGL binary format
├── node_features.npy   # NumPy arrays
└── edge_features.npy

Label Files

anomaly_labels.csv

fraud_labels.csv

quality_labels.csv

Control Files

internal_controls.csv

control_account_mappings.csv

sod_rules.csv

Segregation of duties conflict definitions.

sod_conflict_pairs.csv

Actual SoD violations detected in generated data.

Parquet Format

Apache Parquet columnar format for large analytical datasets:

output:
  format: parquet
  compression: snappy      # snappy, gzip, zstd

Benefits:

• Columnar storage — efficient for queries touching few columns
• Built-in compression — typically 5-10x smaller than CSV
• Schema embedding — self-describing files with full type information
• Predicate pushdown — query engines skip irrelevant row groups

Use with: Apache Spark, DuckDB, Polars, pandas, BigQuery, Snowflake, Databricks.

ERP-Specific Formats

SyntheticData can export in native ERP table schemas:

See ERP Output Formats for field mappings and configuration.

Compression Options

Configuration

output:
  format: csv              # csv, json, jsonl, parquet, sap, oracle, netsuite
  compression: none        # none, gzip, zstd (CSV/JSON) or snappy/gzip/zstd (Parquet)
  compression_level: 6     # 1-9 (if compression enabled)
  streaming: false         # Enable streaming mode for large outputs

See Also

• ERP Output Formats — SAP, Oracle, NetSuite table exports
• Streaming Output — Real-time streaming sinks
• Configuration — Output settings reference
• Graph Export
• Anomaly Injection
• AML/KYC Testing
• Process Mining

ERP Output Formats

SyntheticData can export data in native ERP table formats, enabling direct load testing and integration validation against SAP S/4HANA, Oracle EBS, and NetSuite environments.

Overview

The datasynth-output crate provides three ERP-specific exporters alongside the standard CSV/JSON/Parquet sinks. Each exporter transforms the internal data model into the target ERP’s table schema with correct field names, data types, and referential integrity.

SAP S/4HANA

Supported Tables

BKPF Fields (Document Header)

BSEG Fields (Line Items)

ACDOCA Fields (Universal Journal)

The ACDOCA format includes all standard SAP Universal Journal fields plus simulation metadata:

Configuration

output:
  format: sap
  sap:
    tables:
      - bkpf
      - bseg
      - acdoca
      - lfa1
      - kna1
      - mara
    client: "100"
    ledger: "0L"

Oracle EBS

Supported Tables

GL_JE_HEADERS Fields

GL_JE_LINES Fields

Configuration

output:
  format: oracle
  oracle:
    ledger_id: 1
    set_of_books_id: 1

NetSuite

Journal Entry Fields

NetSuite export includes support for subsidiaries, multi-book accounting, and custom fields:

NetSuite Line Fields

Configuration

output:
  format: netsuite
  netsuite:
    subsidiary_id: 1
    include_custom_fields: true

Usage Examples

SAP Load Testing

Generate data for SAP S/4HANA load testing with full table coverage:

global:
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 12

transactions:
  target_count: 100000

output:
  format: sap
  sap:
    tables: [bkpf, bseg, acdoca, lfa1, kna1, mara, csks, cepc]
    client: "100"

Oracle EBS Migration Validation

Generate journal entries in Oracle EBS format for migration testing:

output:
  format: oracle
  oracle:
    ledger_id: 1

NetSuite Integration Testing

Generate multi-subsidiary data with custom fields:

output:
  format: netsuite
  netsuite:
    subsidiary_id: 1
    include_custom_fields: true

Output Files

See Also

• Output Formats — Standard CSV/JSON/Parquet output
• Streaming Output — Real-time streaming sinks
• Output Settings — Configuration reference
• ERP Load Testing — ERP testing use case
• datasynth-output — Crate reference

Streaming Output

SyntheticData provides streaming output sinks for real-time data generation, enabling memory-efficient export of large datasets without loading everything into memory at once.

Overview

The streaming module in datasynth-output implements the StreamingSink trait for four output formats:

All streaming sinks accept StreamEvent values through the process() method:

#![allow(unused)]
fn main() {
pub enum StreamEvent<T> {
    Data(T),       // A data record to write
    Flush,         // Force flush to disk
    Close,         // Close the stream
}
}

StreamingSink Trait

All streaming sinks implement:

#![allow(unused)]
fn main() {
pub trait StreamingSink<T: Serialize + Send> {
    /// Process a single stream event (data, flush, or close).
    fn process(&mut self, event: StreamEvent<T>) -> SynthResult<()>;

    /// Close the stream and flush remaining data.
    fn close(&mut self) -> SynthResult<()>;

    /// Return the number of items written so far.
    fn items_written(&self) -> u64;

    /// Return the number of bytes written so far.
    fn bytes_written(&self) -> u64;
}
}

When to Use Streaming vs Batch

CSV Streaming

#![allow(unused)]
fn main() {
use datasynth_output::streaming::CsvStreamingSink;
use datasynth_core::traits::StreamEvent;

let mut sink = CsvStreamingSink::<JournalEntry>::new("output.csv".into())?;

// Write records one at a time (memory efficient)
for entry in generate_entries() {
    sink.process(StreamEvent::Data(entry))?;
}

// Periodic flush (optional — ensures data is on disk)
sink.process(StreamEvent::Flush)?;

// Close when done
sink.close()?;

println!("Wrote {} items ({} bytes)", sink.items_written(), sink.bytes_written());
}

Headers are written automatically on the first Data event.

JSON Streaming

Pretty-printed JSON Array

#![allow(unused)]
fn main() {
use datasynth_output::streaming::JsonStreamingSink;

let mut sink = JsonStreamingSink::<JournalEntry>::new("output.json".into())?;
for entry in entries {
    sink.process(StreamEvent::Data(entry))?;
}
sink.close()?;  // Writes closing bracket
}

Output:

[
  { "document_id": "abc-001", ... },
  { "document_id": "abc-002", ... }
]

Newline-Delimited JSON (NDJSON)

#![allow(unused)]
fn main() {
use datasynth_output::streaming::NdjsonStreamingSink;

let mut sink = NdjsonStreamingSink::<JournalEntry>::new("output.jsonl".into())?;
for entry in entries {
    sink.process(StreamEvent::Data(entry))?;
}
sink.close()?;
}

Output:

{"document_id":"abc-001",...}
{"document_id":"abc-002",...}

NDJSON is ideal for streaming consumers that process records line by line (e.g., jq, Kafka, log aggregators).

Parquet Streaming

Apache Parquet provides columnar compression, making it ideal for large analytical datasets:

#![allow(unused)]
fn main() {
use datasynth_output::streaming::ParquetStreamingSink;

let mut sink = ParquetStreamingSink::<JournalEntry>::new("output.parquet".into())?;
for entry in entries {
    sink.process(StreamEvent::Data(entry))?;
}
sink.close()?;
}

Parquet benefits:

• Columnar storage: Efficient for analytical queries that touch few columns
• Built-in compression: Snappy, Gzip, or Zstd per column group
• Schema embedding: Self-describing files with full type information
• Predicate pushdown: Query engines can skip irrelevant row groups

Configuration

Streaming output is enabled when using the server API or when the runtime detects memory pressure:

output:
  format: csv           # csv, json, jsonl, parquet
  streaming: true       # Enable streaming mode
  compression: none     # none, gzip, zstd (CSV/JSON) or snappy/gzip/zstd (Parquet)

Server Streaming

The server API uses streaming sinks for the /api/stream/ endpoints:

# Start streaming generation
curl -X POST http://localhost:3000/api/stream/start \
  -H "Content-Type: application/json" \
  -d '{"config": {...}, "format": "ndjson"}'

# WebSocket streaming
wscat -c ws://localhost:3000/ws/events

Backpressure

Streaming sinks monitor write throughput and provide backpressure signals:

• items_written() / bytes_written(): Track progress for rate limiting
• Flush events: Force periodic disk writes to bound memory usage
• Disk space monitoring: The runtime’s DiskGuard can pause generation when disk space runs low

Performance

Throughput varies with record complexity and disk speed.

See Also

• Output Formats — Batch output format details
• ERP Output Formats — SAP/Oracle/NetSuite formats
• Output Settings — Configuration reference
• Server API — Streaming via REST/WebSocket
• datasynth-output — Crate reference

Python Wrapper Specification (In-Memory Configs)

This document specifies a Python wrapper that makes DataSynth usable out of the box without requiring persisted configuration files. The wrapper focuses on rich, structured configuration objects and reusable configuration blueprints so developers can generate data entirely in memory while still benefiting from the full DataSynth configuration model.

Goals

• Zero-file setup: Instantiate and run DataSynth without writing YAML/JSON to disk.
• Rich configuration: Offer a Pythonic API that maps cleanly to the full DataSynth configuration schema.
• Blueprints: Provide reusable, parameterized configuration templates for common scenarios.
• Interoperable: Allow optional export to YAML/JSON for debugging or CLI parity.
• Composable: Enable programmatic composition, overrides, and validation.

Non-goals

• Replacing the DataSynth CLI or server API.
• Hiding the underlying schema; the wrapper should expose all configuration knobs.
• Managing persistence beyond optional explicit export helpers.

Package layout

packages/
  datasynth_py/
    __init__.py
    client.py             # entrypoint wrapper
    config/
      __init__.py
      models.py           # typed config objects
      blueprints.py       # blueprint registry + builders
      validation.py       # schema validation helpers
    runtime/
      __init__.py
      session.py          # in-memory execution

Core API surface

DataSynth entrypoint

from datasynth_py import DataSynth

synth = DataSynth()

Responsibilities

• Provide a generate() method that accepts rich configuration objects.
• Provide blueprints registry access for common starting points.
• Manage execution in memory, including optional output sinks.

generate() signature

result = synth.generate(
    config=Config(...),
    output=OutputSpec(...),
    seed=42,
)

Behavior

• Validates configuration objects.
• Converts configuration to DataSynth schema (internal model or JSON/YAML in-memory string).
• Executes the generator and returns result handles (paths, in-memory tables, or streams).

Optional output handling

OutputSpec can include:

• format (e.g., parquet, csv, jsonl)
• sink (memory, temp_dir, path)
• compression settings

When sink="memory", the wrapper returns in-memory table objects (pandas DataFrames by default).

Configuration model

Typed configuration objects

Provide typed dataclasses/Pydantic models mirroring the DataSynth YAML schema:

• GlobalSettings
• CompanySettings
• TransactionSettings
• MasterDataSettings
• ComplianceSettings
• OutputSettings

Example:

from datasynth_py.config import Config, GlobalSettings, CompanySettings

config = Config(
    global_settings=GlobalSettings(
        locale="en_US",
        fiscal_year_start="2024-01-01",
        periods=12,
    ),
    companies=CompanySettings(count=5, industry="retail"),
)

Overrides and layering

Allow configuration layering to support incremental overrides:

config = base_config.override(
    companies={"count": 10},
    output={"format": "parquet"},
)

The wrapper merges overrides deeply, preserving nested settings.

Blueprints

Blueprints provide preconfigured setups with parameters. The wrapper ships with a registry:

from datasynth_py import blueprints

config = blueprints.retail_small(companies=3, transactions=5000)

Blueprint characteristics

• Parameterized: Each blueprint accepts keyword overrides for key metrics.
• Composable: A blueprint can extend or wrap another blueprint.
• Discoverable: Registry lists available blueprints and metadata.

blueprints.list()
# ["retail_small", "banking_medium", "saas_subscription", ...]

Execution model

The wrapper runs the Rust engine in-process via FFI or uses the DataSynth runtime API:

• In-memory config: converted to serialized config strings without writing to disk.
• Transient workspace: uses a temporary directory only if required by runtime internals.
• Deterministic runs: seed controls RNG.

Streaming generation

The wrapper exposes a streaming session that connects to datasynth-server over WebSockets while using REST endpoints to start, pause, resume, and stop streams.

Examples

Example 1: Minimal generation in memory

from datasynth_py import DataSynth
from datasynth_py.config import Config, GlobalSettings, CompanySettings

config = Config(
    global_settings=GlobalSettings(locale="en_US", fiscal_year_start="2024-01-01"),
    companies=CompanySettings(count=2),
)

synth = DataSynth()
result = synth.generate(config=config, output={"format": "csv", "sink": "memory"})

# result.tables -> dict[str, pandas.DataFrame]
print(result.tables["transactions"].head())

Example 2: Use a blueprint with overrides

from datasynth_py import DataSynth, blueprints

synth = DataSynth()
config = blueprints.retail_small(companies=4, transactions=15000)

result = synth.generate(
    config=config,
    output={"format": "parquet", "sink": "temp_dir"},
    seed=7,
)

print(result.output_dir)

Example 3: Layering overrides for a custom scenario

from datasynth_py import DataSynth
from datasynth_py.config import Config, GlobalSettings, TransactionSettings

base = Config(global_settings=GlobalSettings(locale="en_GB"))
custom = base.override(
    transactions=TransactionSettings(
        count=25000,
        currency="GBP",
        anomaly_rate=0.02,
    )
)

synth = DataSynth()
result = synth.generate(config=custom, output={"format": "jsonl", "sink": "memory"})

Example 4: Export configuration for debugging

from datasynth_py import DataSynth
from datasynth_py.config import Config

synth = DataSynth()
config = Config(...)

print(config.to_yaml())
print(config.to_json())

Example 5: Streaming events

import asyncio

from datasynth_py import DataSynth
from datasynth_py.config import blueprints


async def main() -> None:
    synth = DataSynth(server_url="http://localhost:3000")
    config = blueprints.retail_small(companies=2, transactions=5000)
    session = synth.stream(config=config, events_per_second=50)

    async for event in session.events():
        print(event)
        break


asyncio.run(main())

Decisions

• In-memory table format: pandas DataFrames are the default return type for memory sinks.
• Validation errors: configuration validation raises ConfigValidationError with structured error details.

Python Wrapper Guide

This guide explains how to use the DataSynth Python wrapper for in-memory configuration, local CLI generation, and streaming generation through the server API.

Installation

The wrapper lives in the repository under python/. Install it in development mode:

cd python
pip install -e ".[all]"

Or install just the core with specific extras:

pip install -e ".[cli]"      # For CLI generation (requires PyYAML)
pip install -e ".[memory]"   # For in-memory tables (requires pandas)
pip install -e ".[streaming]" # For streaming (requires websockets)

Quick start (CLI generation)

from datasynth_py import DataSynth, CompanyConfig, Config, GlobalSettings, ChartOfAccountsSettings

config = Config(
    global_settings=GlobalSettings(
        industry="retail",
        start_date="2024-01-01",
        period_months=12,
    ),
    companies=[
        CompanyConfig(code="C001", name="Retail Corp", currency="USD", country="US"),
    ],
    chart_of_accounts=ChartOfAccountsSettings(complexity="small"),
)

synth = DataSynth()
result = synth.generate(config=config, output={"format": "csv", "sink": "temp_dir"})

print(result.output_dir)  # Path to generated files

Using blueprints

Blueprints provide preconfigured templates for common scenarios:

from datasynth_py import DataSynth
from datasynth_py.config import blueprints

# List available blueprints
print(blueprints.list())
# ['retail_small', 'banking_medium', 'manufacturing_large',
#  'banking_aml', 'ml_training', 'with_graph_export']

# Create a retail configuration with 4 companies
config = blueprints.retail_small(companies=4, transactions=10000)

# Banking/AML focused configuration
config = blueprints.banking_aml(customers=1000, typologies=True)

# ML training optimized configuration
config = blueprints.ml_training(
    industry="manufacturing",
    anomaly_ratio=0.05,
)

# Add graph export to any configuration
config = blueprints.with_graph_export(
    base_config=blueprints.retail_small(),
    formats=["pytorch_geometric", "neo4j"],
)

synth = DataSynth()
result = synth.generate(config=config, output={"format": "parquet", "sink": "path", "path": "./output"})

Configuration model

The configuration model matches the CLI schema:

from datasynth_py import (
    ChartOfAccountsSettings,
    CompanyConfig,
    Config,
    FraudSettings,
    GlobalSettings,
)

config = Config(
    global_settings=GlobalSettings(
        industry="manufacturing",      # Industry sector
        start_date="2024-01-01",       # Simulation start date
        period_months=12,              # Number of months to simulate
        seed=42,                       # Random seed for reproducibility
        group_currency="USD",          # Base currency
    ),
    companies=[
        CompanyConfig(
            code="M001",
            name="Manufacturing Co",
            currency="USD",
            country="US",
            annual_transaction_volume="ten_k",  # Volume preset
        ),
        CompanyConfig(
            code="M002",
            name="Manufacturing EU",
            currency="EUR",
            country="DE",
            annual_transaction_volume="hundred_k",
        ),
    ],
    chart_of_accounts=ChartOfAccountsSettings(
        complexity="medium",           # small, medium, or large
    ),
    fraud=FraudSettings(
        enabled=True,
        rate=0.01,                     # 1% fraud rate
    ),
)

Valid industry values

• manufacturing
• retail
• financial_services
• healthcare
• technology
• professional_services
• energy
• transportation
• real_estate
• telecommunications

Transaction volume presets

• ten_k - 10,000 transactions/year
• hundred_k - 100,000 transactions/year
• one_m - 1,000,000 transactions/year
• ten_m - 10,000,000 transactions/year
• hundred_m - 100,000,000 transactions/year

Extended configuration

Additional configuration sections for specialized scenarios:

from datasynth_py.config.models import (
    Config,
    GlobalSettings,
    BankingSettings,
    ScenarioSettings,
    TemporalDriftSettings,
    DataQualitySettings,
    GraphExportSettings,
)

config = Config(
    global_settings=GlobalSettings(industry="financial_services"),

    # Banking/KYC/AML configuration
    banking=BankingSettings(
        enabled=True,
        retail_customers=1000,
        business_customers=200,
        typologies_enabled=True,  # Structuring, layering, mule patterns
    ),

    # ML training scenario
    scenario=ScenarioSettings(
        tags=["ml_training", "fraud_detection"],
        ml_training=True,
        target_anomaly_ratio=0.05,
    ),

    # Temporal drift for concept drift testing
    temporal=TemporalDriftSettings(
        enabled=True,
        amount_mean_drift=0.02,
        drift_type="gradual",  # gradual, sudden, recurring
    ),

    # Data quality issues for DQ model training
    data_quality=DataQualitySettings(
        enabled=True,
        missing_rate=0.05,
        typo_rate=0.02,
    ),

    # Graph export for GNN training
    graph_export=GraphExportSettings(
        enabled=True,
        formats=["pytorch_geometric", "neo4j"],
    ),
)

Configuration layering

Override configuration values:

from datasynth_py import Config, GlobalSettings

base = Config(global_settings=GlobalSettings(industry="retail", start_date="2024-01-01"))
custom = base.override(
    fraud={"enabled": True, "rate": 0.02},
)

Validation

Validation raises ConfigValidationError with structured error details:

from datasynth_py import Config, GlobalSettings
from datasynth_py.config.validation import ConfigValidationError

try:
    Config(global_settings=GlobalSettings(period_months=0)).validate()
except ConfigValidationError as exc:
    for error in exc.errors:
        print(error.path, error.message, error.value)

Output options

Control where and how data is generated:

from datasynth_py import DataSynth, OutputSpec

synth = DataSynth()

# Write to a specific path
result = synth.generate(
    config=config,
    output=OutputSpec(format="csv", sink="path", path="./output"),
)

# Write to a temporary directory
result = synth.generate(
    config=config,
    output=OutputSpec(format="parquet", sink="temp_dir"),
)
print(result.output_dir)  # Temp directory path

# Load into memory (requires pandas)
result = synth.generate(
    config=config,
    output=OutputSpec(format="csv", sink="memory"),
)
print(result.tables["journal_entries"].head())

Fingerprint Operations

The Python wrapper provides access to fingerprint extraction, validation, and evaluation:

from datasynth_py import DataSynth

synth = DataSynth()

# Extract fingerprint from real data
synth.fingerprint.extract(
    input_path="./real_data/",
    output_path="./fingerprint.dsf",
    privacy_level="standard"  # minimal, standard, high, maximum
)

# Validate fingerprint file
is_valid, errors = synth.fingerprint.validate("./fingerprint.dsf")
if not is_valid:
    print(f"Validation errors: {errors}")

# Get fingerprint info
info = synth.fingerprint.info("./fingerprint.dsf", detailed=True)
print(f"Privacy level: {info.privacy_level}")
print(f"Epsilon spent: {info.epsilon_spent}")
print(f"Tables: {info.tables}")

# Evaluate synthetic data fidelity
report = synth.fingerprint.evaluate(
    fingerprint_path="./fingerprint.dsf",
    synthetic_path="./synthetic_data/",
    threshold=0.8
)
print(f"Overall score: {report.overall_score}")
print(f"Statistical fidelity: {report.statistical_fidelity}")
print(f"Correlation fidelity: {report.correlation_fidelity}")
print(f"Passes threshold: {report.passes}")

FidelityReport Fields

Streaming generation

Streaming uses the DataSynth server for real-time event generation. Start the server first:

cargo run -p datasynth-server -- --port 3000

Then stream events:

import asyncio

from datasynth_py import DataSynth
from datasynth_py.config import blueprints


async def main() -> None:
    synth = DataSynth(server_url="http://localhost:3000")
    config = blueprints.retail_small(companies=2)
    session = synth.stream(config=config, events_per_second=100)

    async for event in session.events():
        print(event)
        break


asyncio.run(main())

Stream controls

session.pause()
session.resume()
session.stop()

Pattern triggers

Trigger specific patterns during streaming to simulate real-world scenarios:

# Trigger temporal patterns
session.trigger_month_end()    # Month-end volume spike
session.trigger_year_end()     # Year-end closing entries
session.trigger_pattern("quarter_end_spike")

# Trigger anomaly patterns
session.trigger_fraud_cluster()  # Clustered fraud transactions
session.trigger_pattern("dormant_account_activity")

# Available patterns:
# - period_end_spike
# - quarter_end_spike
# - year_end_spike
# - fraud_cluster
# - error_burst
# - dormant_account_activity

Synchronous event consumption

For simpler use cases without async/await:

def process_event(event):
    print(f"Received: {event['document_id']}")

session.sync_events(callback=process_event, max_events=1000)

Runtime requirements

The wrapper shells out to the datasynth-data CLI for batch generation. Ensure the binary is available:

cargo build --release
export DATASYNTH_BINARY=target/release/datasynth-data

Alternatively, pass binary_path when creating the client:

synth = DataSynth(binary_path="/path/to/datasynth-data")

Troubleshooting

• MissingDependencyError: Install the required optional dependency (PyYAML, pandas, or websockets).
• CLI not found: Build the datasynth-data binary and set DATASYNTH_BINARY or pass binary_path.
• ConfigValidationError: Check the error details for invalid configuration values.
• Streaming errors: Verify the server is running and reachable at the configured URL.

Ecosystem Integrations (v0.5.0)

DataSynth includes optional integrations with popular data engineering and ML platforms. Install with:

pip install datasynth-py[integrations]
# Or install specific integrations
pip install datasynth-py[airflow,dbt,mlflow,spark]

Apache Airflow

Use the DataSynthOperator to generate data as part of Airflow DAGs:

from datasynth_py.integrations import DataSynthOperator, DataSynthSensor, DataSynthValidateOperator

# Generate data
generate = DataSynthOperator(
    task_id="generate_data",
    config=config,
    output_path="/data/synthetic/output",
)

# Wait for completion
sensor = DataSynthSensor(
    task_id="wait_for_data",
    output_path="/data/synthetic/output",
)

# Validate config
validate = DataSynthValidateOperator(
    task_id="validate_config",
    config_path="/data/configs/config.yaml",
)

dbt Integration

Generate dbt sources and seeds from synthetic data:

from datasynth_py.integrations import DbtSourceGenerator, create_dbt_project

gen = DbtSourceGenerator()

# Generate sources.yml for dbt
sources_path = gen.generate_sources_yaml("./output", "./my_dbt_project")

# Generate seed CSVs
seeds_dir = gen.generate_seeds("./output", "./my_dbt_project")

# Create complete dbt project from synthetic output
project = create_dbt_project("./output", "my_dbt_project")

MLflow Tracking

Track generation runs as MLflow experiments:

from datasynth_py.integrations import DataSynthMlflowTracker

tracker = DataSynthMlflowTracker(experiment_name="synthetic_data_runs")

# Track a generation run
run_info = tracker.track_generation("./output", config=cfg)

# Log quality metrics
tracker.log_quality_metrics({
    "completeness": 0.98,
    "benford_mad": 0.008,
    "correlation_preservation": 0.95,
})

# Compare recent runs
comparison = tracker.compare_runs(n=5)

Apache Spark

Read synthetic data as Spark DataFrames:

from datasynth_py.integrations import DataSynthSparkReader

reader = DataSynthSparkReader()

# Read a single table
df = reader.read_table(spark, "./output", "journal_entries")

# Read all tables
tables = reader.read_all_tables(spark, "./output")

# Create temporary views for SQL queries
views = reader.create_temp_views(spark, "./output")
spark.sql("SELECT * FROM journal_entries WHERE amount > 10000").show()

For comprehensive integration documentation, see the Ecosystem Integrations guide.

Ecosystem Integrations

New in v0.5.0

DataSynth’s Python wrapper includes optional integrations with popular data engineering and ML platforms for seamless pipeline orchestration.

Installation

# Install all integrations
pip install datasynth-py[integrations]

# Install specific integrations
pip install datasynth-py[airflow]
pip install datasynth-py[dbt]
pip install datasynth-py[mlflow]
pip install datasynth-py[spark]

Apache Airflow

The Airflow integration provides custom operators and sensors for orchestrating synthetic data generation in Airflow DAGs.

DataSynthOperator

Generates synthetic data as an Airflow task:

from datasynth_py.integrations import DataSynthOperator

generate = DataSynthOperator(
    task_id="generate_synthetic_data",
    config={
        "global": {"industry": "retail", "start_date": "2024-01-01", "period_months": 12},
        "transactions": {"target_count": 50000},
        "output": {"format": "csv"},
    },
    output_path="/data/synthetic/{{ ds }}",
)

DataSynthSensor

Waits for synthetic data generation to complete:

from datasynth_py.integrations import DataSynthSensor

wait = DataSynthSensor(
    task_id="wait_for_data",
    output_path="/data/synthetic/{{ ds }}",
    poke_interval=30,
    timeout=600,
)

DataSynthValidateOperator

Validates a configuration file before generation:

from datasynth_py.integrations import DataSynthValidateOperator

validate = DataSynthValidateOperator(
    task_id="validate_config",
    config_path="/configs/retail.yaml",
)

Complete DAG Example

from airflow import DAG
from airflow.utils.dates import days_ago
from datasynth_py.integrations import (
    DataSynthOperator,
    DataSynthSensor,
    DataSynthValidateOperator,
)

with DAG(
    "weekly_synthetic_data",
    start_date=days_ago(1),
    schedule_interval="@weekly",
    catchup=False,
) as dag:

    validate = DataSynthValidateOperator(
        task_id="validate",
        config_path="/configs/retail.yaml",
    )

    generate = DataSynthOperator(
        task_id="generate",
        config_path="/configs/retail.yaml",
        output_path="/data/synthetic/{{ ds }}",
    )

    wait = DataSynthSensor(
        task_id="wait",
        output_path="/data/synthetic/{{ ds }}",
    )

    validate >> generate >> wait

dbt Integration

Generate dbt-compatible project structures from synthetic data output.

DbtSourceGenerator

from datasynth_py.integrations import DbtSourceGenerator

gen = DbtSourceGenerator()

Generate sources.yml

Creates a dbt sources.yml file pointing to synthetic data tables:

sources_path = gen.generate_sources_yaml(
    output_dir="./synthetic_output",
    dbt_project_dir="./my_dbt_project",
)
# Creates ./my_dbt_project/models/sources.yml

Generate Seeds

Copies synthetic CSV files as dbt seeds:

seeds_dir = gen.generate_seeds(
    output_dir="./synthetic_output",
    dbt_project_dir="./my_dbt_project",
)
# Copies CSVs to ./my_dbt_project/seeds/

create_dbt_project

Creates a complete dbt project structure from synthetic output:

from datasynth_py.integrations import create_dbt_project

project = create_dbt_project(
    output_dir="./synthetic_output",
    project_name="synthetic_test",
)

This creates:

synthetic_test/
├── dbt_project.yml
├── models/
│   └── sources.yml
├── seeds/
│   ├── journal_entries.csv
│   ├── vendors.csv
│   ├── customers.csv
│   └── ...
└── tests/

Usage with dbt CLI

cd synthetic_test
dbt seed      # Load synthetic CSVs
dbt run       # Run transformations
dbt test      # Run data tests

MLflow Integration

Track synthetic data generation runs as MLflow experiments for comparison and reproducibility.

DataSynthMlflowTracker

from datasynth_py.integrations import DataSynthMlflowTracker

tracker = DataSynthMlflowTracker(experiment_name="synthetic_data_experiments")

Track a Generation Run

run_info = tracker.track_generation(
    output_dir="./output",
    config=config,
)
# Logs: config parameters, output file counts, generation metadata

Log Quality Metrics

tracker.log_quality_metrics({
    "completeness": 0.98,
    "benford_mad": 0.008,
    "correlation_preservation": 0.95,
    "statistical_fidelity": 0.92,
})

Compare Runs

comparison = tracker.compare_runs(n=5)
for run in comparison:
    print(f"Run {run['run_id']}: {run['metrics']}")

Experiment Comparison

Use MLflow to compare different generation configurations:

import mlflow

configs = {
    "baseline": baseline_config,
    "with_diffusion": diffusion_config,
    "high_fraud": high_fraud_config,
}

for name, cfg in configs.items():
    with mlflow.start_run(run_name=name):
        result = synth.generate(config=cfg, output={"format": "csv", "sink": "temp_dir"})
        tracker.track_generation(result.output_dir, config=cfg)
        tracker.log_quality_metrics(evaluate_quality(result.output_dir))

View results in the MLflow UI:

mlflow ui --port 5000
# Open http://localhost:5000

Apache Spark

Read synthetic data output directly as Spark DataFrames for large-scale analysis.

DataSynthSparkReader

from datasynth_py.integrations import DataSynthSparkReader

reader = DataSynthSparkReader()

Read a Single Table

df = reader.read_table(spark, "./output", "journal_entries")
df.printSchema()
df.show(5)

Read All Tables

tables = reader.read_all_tables(spark, "./output")
for name, df in tables.items():
    print(f"{name}: {df.count()} rows, {len(df.columns)} columns")

Create Temporary Views

views = reader.create_temp_views(spark, "./output")

# Now use SQL
spark.sql("""
    SELECT
        v.vendor_id,
        v.name,
        COUNT(p.document_id) as payment_count,
        SUM(p.amount) as total_paid
    FROM vendors v
    JOIN payments p ON v.vendor_id = p.vendor_id
    GROUP BY v.vendor_id, v.name
    ORDER BY total_paid DESC
    LIMIT 10
""").show()

Spark + DataSynth Pipeline

from pyspark.sql import SparkSession
from datasynth_py import DataSynth
from datasynth_py.config import blueprints
from datasynth_py.integrations import DataSynthSparkReader

# Generate
synth = DataSynth()
config = blueprints.retail_small(transactions=100000)
result = synth.generate(config=config, output={"format": "csv", "sink": "temp_dir"})

# Load into Spark
spark = SparkSession.builder.appName("SyntheticAnalysis").getOrCreate()
reader = DataSynthSparkReader()
reader.create_temp_views(spark, result.output_dir)

# Analyze
spark.sql("""
    SELECT fiscal_period, COUNT(*) as entry_count, SUM(amount) as total_amount
    FROM journal_entries
    GROUP BY fiscal_period
    ORDER BY fiscal_period
""").show()

Integration Dependencies

All integrations are optional — install only what you need.

See Also

• Python Wrapper Guide
• Pipeline Orchestration Use Case
• Server API

Configuration

SyntheticData uses YAML configuration files to control all aspects of data generation.

Quick Start

# Create configuration from preset
datasynth-data init --industry manufacturing --complexity medium -o config.yaml

# Validate configuration
datasynth-data validate --config config.yaml

# Generate with configuration
datasynth-data generate --config config.yaml --output ./output

Configuration Sections

Reference

• Complete YAML Schema
• Industry Presets

Minimal Configuration

global:
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 12

transactions:
  target_count: 10000

output:
  format: csv

Full Configuration Example

global:
  seed: 42
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 12
  group_currency: USD

companies:
  - code: "1000"
    name: "Headquarters"
    currency: USD
    country: US
    volume_weight: 0.6
  - code: "2000"
    name: "European Subsidiary"
    currency: EUR
    country: DE
    volume_weight: 0.4

chart_of_accounts:
  complexity: medium

transactions:
  target_count: 100000
  line_items:
    distribution: empirical
  amounts:
    min: 100
    max: 1000000

master_data:
  vendors:
    count: 200
  customers:
    count: 500
  materials:
    count: 1000

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.3
  o2c:
    enabled: true
    flow_rate: 0.3

fraud:
  enabled: true
  fraud_rate: 0.005

anomaly_injection:
  enabled: true
  total_rate: 0.02
  generate_labels: true

graph_export:
  enabled: true
  formats:
    - pytorch_geometric
    - neo4j

# AI & ML Features (v0.5.0)
diffusion:
  enabled: true
  n_steps: 1000
  schedule: "cosine"
  sample_size: 1000

causal:
  enabled: true
  template: "fraud_detection"
  sample_size: 1000
  validate: true

certificates:
  enabled: true
  issuer: "DataSynth"
  include_quality_metrics: true

# Enterprise Process Chains (v0.6.0)
source_to_pay:
  enabled: true
  projects_per_period: 5
  avg_bids_per_rfx: 4
  contract_award_rate: 0.75
  catalog_items_per_contract: 10

financial_reporting:
  enabled: true
  generate_balance_sheet: true
  generate_income_statement: true
  generate_cash_flow: true
  generate_changes_in_equity: true
  management_kpis:
    enabled: true
  budgets:
    enabled: true
    variance_threshold: 0.10

hr:
  enabled: true
  payroll_frequency: monthly
  time_tracking: true
  expense_reports: true

manufacturing:
  enabled: true
  production_orders_per_period: 20
  quality_inspection_rate: 0.30
  cycle_count_frequency: quarterly

sales_quotes:
  enabled: true
  quotes_per_period: 15
  conversion_rate: 0.35

output:
  format: csv
  compression: none

Configuration Loading

Configuration can be loaded from:

1. YAML file (recommended):

   datasynth-data generate --config config.yaml --output ./output
   

2. JSON file:

   datasynth-data generate --config config.json --output ./output
   

3. Demo preset:

   datasynth-data generate --demo --output ./output
   

Validation

The configuration is validated for:

Run validation:

datasynth-data validate --config config.yaml

Overriding Values

Command-line options override configuration file values:

# Override seed
datasynth-data generate --config config.yaml --seed 12345 --output ./output

# Override format
datasynth-data generate --config config.yaml --format json --output ./output

Environment Variables

Some settings can be controlled via environment variables:

See Also

• Quick Start
• CLI Reference
• datasynth-config Crate

YAML Schema Reference

Complete reference for all configuration options.

Schema Overview

global:                    # Global settings
companies:                 # Company definitions
chart_of_accounts:         # COA structure
transactions:              # Transaction settings
master_data:               # Master data settings
document_flows:            # P2P, O2C flows
intercompany:              # IC settings
balance:                   # Balance settings
subledger:                 # Subledger settings
fx:                        # FX settings
period_close:              # Period close settings
fraud:                     # Fraud injection
internal_controls:         # SOX controls
anomaly_injection:         # Anomaly injection
data_quality:              # Data quality variations
graph_export:              # Graph export settings
output:                    # Output settings
business_processes:        # Process distribution
templates:                 # External templates
approval:                  # Approval thresholds
departments:               # Department distribution
source_to_pay:             # Source-to-Pay (v0.6.0)
financial_reporting:       # Financial statements & KPIs (v0.6.0)
hr:                        # HR / payroll / expenses (v0.6.0)
manufacturing:             # Production orders & costing (v0.6.0)
sales_quotes:              # Quote-to-order pipeline (v0.6.0)

global

global:
  seed: 42                           # u64, optional - RNG seed
  industry: manufacturing            # string - industry preset
  start_date: 2024-01-01             # date - generation start
  period_months: 12                  # u32, 1-120 - duration
  group_currency: USD                # string - base currency
  worker_threads: 4                  # usize, optional - parallelism
  memory_limit: 2147483648           # u64, optional - bytes

Industries: manufacturing, retail, financial_services, healthcare, technology, energy, telecom, transportation, hospitality

companies

companies:
  - code: "1000"                     # string - unique code
    name: "Headquarters"             # string - display name
    currency: USD                    # string - local currency
    country: US                      # string - ISO country code
    volume_weight: 0.6               # f64, 0-1 - transaction weight
    is_parent: true                  # bool - consolidation parent
    parent_code: null                # string, optional - parent ref

Constraints:

• volume_weight across all companies must sum to 1.0
• code must be unique

chart_of_accounts

chart_of_accounts:
  complexity: medium                 # small, medium, large
  industry_specific: true            # bool - use industry COA
  custom_accounts: []                # list - additional accounts

Complexity levels:

• small: ~100 accounts
• medium: ~400 accounts
• large: ~2500 accounts

transactions

transactions:
  target_count: 100000               # u64 - total JEs to generate

  line_items:
    distribution: empirical          # empirical, uniform, custom
    min_lines: 2                     # u32 - minimum line items
    max_lines: 20                    # u32 - maximum line items
    custom_distribution:             # only if distribution: custom
      2: 0.6068
      3: 0.0524
      4: 0.1732

  amounts:
    min: 100                         # f64 - minimum amount
    max: 1000000                     # f64 - maximum amount
    distribution: log_normal         # log_normal, uniform, custom
    round_number_bias: 0.15          # f64, 0-1 - round number preference

  sources:                           # transaction source weights
    manual: 0.3
    automated: 0.5
    recurring: 0.15
    adjustment: 0.05

  benford:
    enabled: true                    # bool - Benford's Law compliance

  temporal:
    month_end_spike: 2.5             # f64 - month-end volume multiplier
    quarter_end_spike: 3.0           # f64 - quarter-end multiplier
    year_end_spike: 4.0              # f64 - year-end multiplier
    working_hours_only: true         # bool - restrict to business hours

master_data

master_data:
  vendors:
    count: 200                       # u32 - number of vendors
    intercompany_ratio: 0.05         # f64, 0-1 - IC vendor ratio

  customers:
    count: 500                       # u32 - number of customers
    intercompany_ratio: 0.05         # f64, 0-1 - IC customer ratio

  materials:
    count: 1000                      # u32 - number of materials

  fixed_assets:
    count: 100                       # u32 - number of assets

  employees:
    count: 50                        # u32 - number of employees
    hierarchy_depth: 4               # u32 - org chart depth

document_flows

document_flows:
  p2p:                               # Procure-to-Pay
    enabled: true
    flow_rate: 0.3                   # f64, 0-1 - JE percentage
    completion_rate: 0.95            # f64, 0-1 - full flow rate
    three_way_match:
      quantity_tolerance: 0.02       # f64, 0-1 - qty variance allowed
      price_tolerance: 0.01          # f64, 0-1 - price variance allowed

  o2c:                               # Order-to-Cash
    enabled: true
    flow_rate: 0.3                   # f64, 0-1 - JE percentage
    completion_rate: 0.95            # f64, 0-1 - full flow rate

intercompany

intercompany:
  enabled: true
  transaction_types:                 # weights must sum to 1.0
    goods_sale: 0.4
    service_provided: 0.2
    loan: 0.15
    dividend: 0.1
    management_fee: 0.1
    royalty: 0.05

  transfer_pricing:
    method: cost_plus                # cost_plus, resale_minus, comparable
    markup_range:
      min: 0.03
      max: 0.10

balance

balance:
  opening_balance:
    enabled: true
    total_assets: 10000000           # f64 - opening balance sheet size

  coherence_check:
    enabled: true                    # bool - verify A = L + E
    tolerance: 0.01                  # f64 - allowed imbalance

subledger

subledger:
  ar:
    enabled: true
    aging_buckets: [30, 60, 90]      # list of days

  ap:
    enabled: true
    aging_buckets: [30, 60, 90]

  fixed_assets:
    enabled: true
    depreciation_methods:
      - straight_line
      - declining_balance

  inventory:
    enabled: true
    valuation_methods:
      - fifo
      - weighted_average

fx

fx:
  enabled: true
  base_currency: USD

  currency_pairs:                    # currencies to generate
    - EUR
    - GBP
    - CHF
    - JPY

  volatility: 0.01                   # f64 - daily volatility

  translation:
    method: current_rate             # current_rate, temporal

period_close

period_close:
  enabled: true

  monthly:
    accruals: true
    depreciation: true

  quarterly:
    intercompany_elimination: true

  annual:
    closing_entries: true
    retained_earnings: true

fraud

fraud:
  enabled: true
  fraud_rate: 0.005                  # f64, 0-1 - fraud percentage

  types:                             # weights must sum to 1.0
    fictitious_transaction: 0.15
    revenue_manipulation: 0.10
    expense_capitalization: 0.10
    split_transaction: 0.15
    round_tripping: 0.05
    kickback_scheme: 0.10
    ghost_employee: 0.05
    duplicate_payment: 0.15
    unauthorized_discount: 0.10
    suspense_abuse: 0.05

internal_controls

internal_controls:
  enabled: true

  controls:
    - id: "CTL-001"
      name: "Payment Approval"
      type: preventive
      frequency: continuous

  sod_rules:
    - conflict_type: create_approve
      processes: [ap_invoice, ap_payment]

anomaly_injection

anomaly_injection:
  enabled: true
  total_rate: 0.02                   # f64, 0-1 - total anomaly rate
  generate_labels: true              # bool - output ML labels

  categories:                        # weights must sum to 1.0
    fraud: 0.25
    error: 0.40
    process_issue: 0.20
    statistical: 0.10
    relational: 0.05

  temporal_pattern:
    year_end_spike: 1.5              # f64 - year-end multiplier

  clustering:
    enabled: true
    cluster_probability: 0.2

data_quality

data_quality:
  enabled: true

  missing_values:
    rate: 0.01                       # f64, 0-1
    pattern: mcar                    # mcar, mar, mnar, systematic

  format_variations:
    date_formats: true
    amount_formats: true

  duplicates:
    rate: 0.001                      # f64, 0-1
    types: [exact, near, fuzzy]

  typos:
    rate: 0.005                      # f64, 0-1
    keyboard_aware: true

graph_export

graph_export:
  enabled: true

  formats:
    - pytorch_geometric
    - neo4j
    - dgl

  graphs:
    - transaction_network
    - approval_network
    - entity_relationship

  split:
    train: 0.7
    val: 0.15
    test: 0.15
    stratify: is_anomaly

  features:
    temporal: true
    amount: true
    structural: true
    categorical: true

output

output:
  format: csv                        # csv, json
  compression: none                  # none, gzip, zstd
  compression_level: 6               # u32, 1-9 (if compression enabled)

  files:
    journal_entries: true
    acdoca: true
    master_data: true
    documents: true
    subledgers: true
    trial_balances: true
    labels: true
    controls: true

Validation Summary

Diffusion Configuration (v0.5.0)

diffusion:
  enabled: false                    # Enable diffusion model backend
  n_steps: 1000                     # Number of diffusion steps (default: 1000)
  schedule: "linear"                # Noise schedule: "linear", "cosine", "sigmoid"
  sample_size: 1000                 # Number of samples to generate (default: 1000)

Causal Configuration (v0.5.0)

causal:
  enabled: false                    # Enable causal generation
  template: "fraud_detection"       # Built-in template or custom graph path
  sample_size: 1000                 # Number of samples to generate
  validate: true                    # Validate causal structure in output

Built-in Causal Templates

Certificate Configuration (v0.5.0)

certificates:
  enabled: false                    # Enable synthetic data certificates
  issuer: "DataSynth"              # Certificate issuer name
  include_quality_metrics: true     # Include quality metrics in certificate

Source-to-Pay Configuration (v0.6.0)

source_to_pay:
  enabled: false                       # Enable source-to-pay generation

  spend_analysis:
    hhi_threshold: 2500.0              # f64 - HHI threshold for sourcing trigger
    contract_coverage_target: 0.80     # f64, 0-1 - target spend under contracts

  sourcing:
    projects_per_year: 10              # u32 - sourcing projects per year
    renewal_horizon_months: 3          # u32 - months before expiry to trigger renewal
    project_duration_months: 4         # u32 - average project duration

  qualification:
    pass_rate: 0.75                    # f64, 0-1 - qualification pass rate
    validity_days: 365                 # u32 - qualification validity in days
    financial_weight: 0.25             # f64 - financial stability weight
    quality_weight: 0.30               # f64 - quality management weight
    delivery_weight: 0.25              # f64 - delivery performance weight
    compliance_weight: 0.20            # f64 - compliance weight

  rfx:
    rfi_threshold: 100000.0            # f64 - spend above which RFI required
    min_invited_vendors: 3             # u32 - minimum vendors per RFx
    max_invited_vendors: 8             # u32 - maximum vendors per RFx
    response_rate: 0.70                # f64, 0-1 - vendor response rate
    default_price_weight: 0.40         # f64 - price weight in evaluation
    default_quality_weight: 0.35       # f64 - quality weight in evaluation
    default_delivery_weight: 0.25      # f64 - delivery weight in evaluation

  contracts:
    min_duration_months: 12            # u32 - minimum contract duration
    max_duration_months: 36            # u32 - maximum contract duration
    auto_renewal_rate: 0.40            # f64, 0-1 - auto-renewal rate
    amendment_rate: 0.20               # f64, 0-1 - contracts with amendments
    type_distribution:
      fixed_price: 0.40               # f64 - fixed price contracts
      blanket: 0.30                    # f64 - blanket/framework agreements
      time_and_materials: 0.15         # f64 - T&M contracts
      service_agreement: 0.15          # f64 - service agreements

  catalog:
    preferred_vendor_flag_rate: 0.70   # f64, 0-1 - items marked as preferred
    multi_source_rate: 0.25            # f64, 0-1 - items with multiple sources

  scorecards:
    frequency: "quarterly"             # string - review frequency
    on_time_delivery_weight: 0.30      # f64 - OTD weight in score
    quality_weight: 0.30               # f64 - quality weight in score
    price_weight: 0.25                 # f64 - price competitiveness weight
    responsiveness_weight: 0.15        # f64 - responsiveness weight
    grade_a_threshold: 90.0            # f64 - grade A threshold
    grade_b_threshold: 75.0            # f64 - grade B threshold
    grade_c_threshold: 60.0            # f64 - grade C threshold

  p2p_integration:
    off_contract_rate: 0.15            # f64, 0-1 - maverick purchase rate
    price_tolerance: 0.02              # f64 - contract price variance allowed
    catalog_enforcement: false          # bool - enforce catalog ordering

Financial Reporting Configuration (v0.6.0)

financial_reporting:
  enabled: false                       # Enable financial reporting generation
  generate_balance_sheet: true         # bool - generate balance sheet
  generate_income_statement: true      # bool - generate income statement
  generate_cash_flow: true             # bool - generate cash flow statement
  generate_changes_in_equity: true     # bool - generate changes in equity
  comparative_periods: 1               # u32 - number of comparative periods

  management_kpis:
    enabled: false                     # bool - enable KPI generation
    frequency: "monthly"               # string - monthly, quarterly

  budgets:
    enabled: false                     # bool - enable budget generation
    revenue_growth_rate: 0.05          # f64 - expected revenue growth rate
    expense_inflation_rate: 0.03       # f64 - expected expense inflation rate
    variance_noise: 0.10               # f64 - noise for budget vs actual

HR Configuration (v0.6.0)

hr:
  enabled: false                       # Enable HR generation

  payroll:
    enabled: true                      # bool - enable payroll generation
    pay_frequency: "monthly"           # string - monthly, biweekly, weekly
    salary_ranges:
      staff_min: 50000.0               # f64 - staff level minimum salary
      staff_max: 70000.0               # f64 - staff level maximum salary
      manager_min: 80000.0             # f64 - manager level minimum salary
      manager_max: 120000.0            # f64 - manager level maximum salary
      director_min: 120000.0           # f64 - director level minimum salary
      director_max: 180000.0           # f64 - director level maximum salary
      executive_min: 180000.0          # f64 - executive level minimum salary
      executive_max: 350000.0          # f64 - executive level maximum salary
    tax_rates:
      federal_effective: 0.22          # f64 - federal effective tax rate
      state_effective: 0.05            # f64 - state effective tax rate
      fica: 0.0765                     # f64 - FICA/social security rate
    benefits_enrollment_rate: 0.60     # f64, 0-1 - benefits enrollment rate
    retirement_participation_rate: 0.45 # f64, 0-1 - retirement plan participation

  time_attendance:
    enabled: true                      # bool - enable time tracking
    overtime_rate: 0.10                # f64, 0-1 - employees with overtime

  expenses:
    enabled: true                      # bool - enable expense report generation
    submission_rate: 0.30              # f64, 0-1 - employees submitting per month
    policy_violation_rate: 0.08        # f64, 0-1 - rate of policy violations

Manufacturing Configuration (v0.6.0)

manufacturing:
  enabled: false                       # Enable manufacturing generation

  production_orders:
    orders_per_month: 50               # u32 - production orders per month
    avg_batch_size: 100                # u32 - average batch size
    yield_rate: 0.97                   # f64, 0-1 - production yield rate
    make_to_order_rate: 0.20           # f64, 0-1 - MTO vs MTS ratio
    rework_rate: 0.03                  # f64, 0-1 - rework rate

  costing:
    labor_rate_per_hour: 35.0          # f64 - labor rate per hour
    overhead_rate: 1.50                # f64 - overhead multiplier on direct labor
    standard_cost_update_frequency: "quarterly"  # string - cost update cycle

  routing:
    avg_operations: 4                  # u32 - average operations per routing
    setup_time_hours: 1.5              # f64 - average setup time in hours
    run_time_variation: 0.15           # f64 - run time variation coefficient

Sales Quotes Configuration (v0.6.0)

sales_quotes:
  enabled: false                       # Enable sales quote generation
  quotes_per_month: 30                 # u32 - quotes generated per month
  win_rate: 0.35                       # f64, 0-1 - quote-to-order conversion
  validity_days: 30                    # u32 - default quote validity period

See Also

• Configuration Overview
• Industry Presets
• datasynth-config Crate

Industry Presets

SyntheticData includes pre-configured settings for common industries.

Using Presets

# Create configuration from preset
datasynth-data init --industry manufacturing --complexity medium -o config.yaml

Available Industries

Complexity Levels

Manufacturing

Characteristics:

• High P2P activity (procurement, production)
• Significant inventory and WIP
• Fixed asset intensive
• Cost accounting emphasis

Key Settings:

global:
  industry: manufacturing

transactions:
  sources:
    manual: 0.2
    automated: 0.6
    recurring: 0.15
    adjustment: 0.05

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.4          # 40% of JEs from P2P
  o2c:
    enabled: true
    flow_rate: 0.25         # 25% of JEs from O2C

master_data:
  materials:
    count: 1000
  fixed_assets:
    count: 200

subledger:
  inventory:
    enabled: true
    valuation_methods:
      - weighted_average
      - fifo

Typical Account Distribution:

• 45% expense accounts (production costs)
• 25% asset accounts (inventory, equipment)
• 15% liability accounts
• 10% revenue accounts
• 5% equity accounts

Retail

Characteristics:

• High transaction volume
• Strong seasonal patterns
• High O2C activity
• Inventory turnover focus

Key Settings:

global:
  industry: retail

transactions:
  target_count: 500000      # High volume
  temporal:
    month_end_spike: 1.5
    quarter_end_spike: 2.0
    year_end_spike: 5.0     # Holiday season

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.25
  o2c:
    enabled: true
    flow_rate: 0.45         # High sales activity

master_data:
  customers:
    count: 2000
  materials:
    count: 5000

subledger:
  ar:
    enabled: true
    aging_buckets: [30, 60, 90, 120]

Seasonal Pattern:

• Q4 volume: 200-300% of Q1-Q3 average
• Black Friday/holiday spikes
• Post-holiday returns

Financial Services

Characteristics:

• Complex intercompany structures
• High regulatory requirements
• Sophisticated controls
• Mark-to-market adjustments

Key Settings:

global:
  industry: financial_services

transactions:
  sources:
    automated: 0.7          # High automation
    adjustment: 0.15        # MTM adjustments

intercompany:
  enabled: true
  transaction_types:
    loan: 0.3
    service_provided: 0.25
    dividend: 0.2
    management_fee: 0.15
    royalty: 0.1

internal_controls:
  enabled: true
  controls:
    - id: "SOX-001"
      type: preventive
      frequency: continuous

fx:
  enabled: true
  currency_pairs:
    - EUR
    - GBP
    - CHF
    - JPY
    - CNY
  volatility: 0.015

Control Requirements:

• SOX 404 compliance mandatory
• High SoD enforcement
• Continuous monitoring

Healthcare

Characteristics:

• Complex revenue recognition (insurance)
• Regulatory compliance (HIPAA)
• Seasonal patterns (flu season, open enrollment)
• High accounts receivable

Key Settings:

global:
  industry: healthcare

transactions:
  amounts:
    min: 50
    max: 500000
    distribution: log_normal

document_flows:
  o2c:
    enabled: true
    flow_rate: 0.5          # Revenue cycle focus

master_data:
  customers:
    count: 1000             # Patient/payer mix

subledger:
  ar:
    enabled: true
    aging_buckets: [30, 60, 90, 120, 180]  # Extended aging

fraud:
  types:
    fictitious_transaction: 0.2
    revenue_manipulation: 0.3   # Upcoding focus
    duplicate_payment: 0.2

Seasonal Pattern:

• Q1 spike (insurance deductible reset)
• Flu season (Oct-Feb)
• Open enrollment (Nov-Dec)

Technology

Characteristics:

• SaaS/subscription revenue
• R&D capitalization
• Stock compensation
• Deferred revenue management

Key Settings:

global:
  industry: technology

transactions:
  sources:
    automated: 0.65
    recurring: 0.25         # Subscription billing
    manual: 0.08
    adjustment: 0.02

document_flows:
  o2c:
    enabled: true
    flow_rate: 0.35

subledger:
  ar:
    enabled: true

# Additional technology-specific
deferred_revenue:
  enabled: true
  recognition_period: monthly

capitalization:
  r_and_d:
    enabled: true
    threshold: 50000

Revenue Pattern:

• Monthly recurring revenue (MRR)
• Annual contract billing (ACV)
• Usage-based components

Process Chain Defaults (v0.6.0)

Starting in v0.6.0, all five industry presets include default settings for the new enterprise process chains. When you generate a configuration with datasynth-data init, the preset populates sensible defaults for each new section, though they remain disabled until explicitly turned on.

Manufacturing presets emphasize production orders, routing, and costing. Retail presets increase sales quote volume and quote-to-order win rates. Financial Services presets focus on financial reporting with comprehensive KPIs and budgets. Healthcare and Technology presets provide balanced defaults.

Each preset configures the following when you set enabled: true:

• source_to_pay: Sourcing projects, RFx events, contract management, catalogs, and vendor scorecards that feed into the existing P2P document flow.
• financial_reporting: Balance sheets, income statements, cash flow statements, management KPIs, and budget vs. actual variance analysis.
• hr: Payroll runs based on employee master data, time and attendance tracking, and expense report generation with policy violation injection.
• manufacturing: Production orders, WIP tracking, standard costing with labor and overhead, and routing operations.
• sales_quotes: Quote-to-order pipeline that feeds into the existing O2C document flow.

Customizing Presets

Start with a preset and customize:

# Generate preset
datasynth-data init --industry manufacturing -o config.yaml

# Edit config.yaml
# - Adjust transaction counts
# - Add companies
# - Enable additional features

# Validate and generate
datasynth-data validate --config config.yaml
datasynth-data generate --config config.yaml --output ./output

Combining Industries

For conglomerates, use multiple companies with different characteristics:

companies:
  - code: "1000"
    name: "Manufacturing Division"
    volume_weight: 0.5

  - code: "2000"
    name: "Retail Division"
    volume_weight: 0.3

  - code: "3000"
    name: "Services Division"
    volume_weight: 0.2

See Also

• Configuration Overview
• Global Settings
• Companies

Global Settings

Global settings control overall generation behavior.

Configuration

global:
  seed: 42                           # Random seed for reproducibility
  industry: manufacturing            # Industry preset
  start_date: 2024-01-01             # Generation start date
  period_months: 12                  # Duration in months
  group_currency: USD                # Base/reporting currency
  worker_threads: 4                  # Parallel workers (optional)
  memory_limit: 2147483648           # Memory limit in bytes (optional)

Fields

seed

Random number generator seed for reproducible output.

global:
  seed: 42  # Same seed = same output

Use cases:

• Reproducible test datasets
• Debugging
• Consistent benchmarks

industry

Industry preset for domain-specific settings.

Available industries:

start_date

Beginning date for generated data.

global:
  start_date: 2024-01-01

Notes:

• First transaction will be on or after this date
• Combined with period_months to determine date range

period_months

Duration of generation period.

global:
  period_months: 12    # One year
  period_months: 36    # Three years
  period_months: 1     # One month

Considerations:

• Longer periods = more data
• Period close features require at least 1 month
• Year-end close requires at least 12 months

group_currency

Base currency for consolidation and reporting.

global:
  group_currency: USD
  group_currency: EUR
  group_currency: CHF

Used for:

• Currency translation
• Consolidation
• Intercompany eliminations

worker_threads

Number of parallel worker threads.

global:
  worker_threads: 4    # Use 4 threads
  worker_threads: 1    # Single-threaded

Guidance:

• Default (CPU cores) is usually optimal
• Reduce for memory-constrained systems
• Increase may not improve performance beyond CPU cores

memory_limit

Maximum memory usage in bytes.

global:
  memory_limit: 1073741824    # 1 GB
  memory_limit: 2147483648    # 2 GB
  memory_limit: 4294967296    # 4 GB

Behavior:

• Soft limit: Generation slows down
• Hard limit: Generation pauses until memory freed
• Streaming output to reduce memory pressure

Environment Variable Overrides

SYNTH_DATA_SEED=12345 datasynth-data generate --config config.yaml --output ./output

Examples

Minimal

global:
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 12
  group_currency: USD

Full Control

global:
  seed: 42
  industry: financial_services
  start_date: 2023-01-01
  period_months: 36
  group_currency: USD
  worker_threads: 8
  memory_limit: 8589934592  # 8 GB

Development/Testing

global:
  seed: 42                # Reproducible
  industry: manufacturing
  start_date: 2024-01-01
  period_months: 1        # Short period
  group_currency: USD
  worker_threads: 1       # Single thread for debugging

Validation

See Also

• Industry Presets
• Companies
• Performance Tuning

Companies

Company configuration defines the legal entities for data generation.

Configuration

companies:
  - code: "1000"
    name: "Headquarters"
    currency: USD
    country: US
    volume_weight: 0.6
    is_parent: true
    parent_code: null

  - code: "2000"
    name: "European Subsidiary"
    currency: EUR
    country: DE
    volume_weight: 0.4
    is_parent: false
    parent_code: "1000"

Fields

code

Unique identifier for the company.

companies:
  - code: "1000"      # Four-digit SAP-style
  - code: "US01"      # Region-based
  - code: "HQ"        # Abbreviated

name

Display name for the company.

companies:
  - name: "Headquarters"
  - name: "European Operations GmbH"
  - name: "Asia Pacific Holdings"

currency

Local currency for the company.

companies:
  - currency: USD
  - currency: EUR
  - currency: CHF
  - currency: JPY

Used for:

• Transaction amounts
• Local reporting
• FX translation

country

Country code for the company.

companies:
  - country: US
  - country: DE
  - country: CH
  - country: JP

Affects:

• Holiday calendars
• Tax calculations
• Regional templates

volume_weight

Relative transaction volume for this company.

companies:
  - code: "1000"
    volume_weight: 0.5    # 50% of transactions

  - code: "2000"
    volume_weight: 0.3    # 30% of transactions

  - code: "3000"
    volume_weight: 0.2    # 20% of transactions

is_parent

Whether this company is the consolidation parent.

companies:
  - code: "1000"
    is_parent: true       # Consolidation parent

  - code: "2000"
    is_parent: false      # Subsidiary

Notes:

• Exactly one company should be is_parent: true for consolidation
• Parent receives elimination entries

parent_code

Reference to parent company for subsidiaries.

companies:
  - code: "1000"
    is_parent: true
    parent_code: null     # No parent (is the parent)

  - code: "2000"
    is_parent: false
    parent_code: "1000"   # Owned by 1000

  - code: "3000"
    is_parent: false
    parent_code: "2000"   # Owned by 2000 (nested)

Examples

Single Company

companies:
  - code: "1000"
    name: "Demo Company"
    currency: USD
    country: US
    volume_weight: 1.0

Multi-National

companies:
  - code: "1000"
    name: "Global Holdings Inc"
    currency: USD
    country: US
    volume_weight: 0.4
    is_parent: true

  - code: "2000"
    name: "European Operations GmbH"
    currency: EUR
    country: DE
    volume_weight: 0.25
    parent_code: "1000"

  - code: "3000"
    name: "UK Limited"
    currency: GBP
    country: GB
    volume_weight: 0.15
    parent_code: "2000"

  - code: "4000"
    name: "Asia Pacific Pte Ltd"
    currency: SGD
    country: SG
    volume_weight: 0.2
    parent_code: "1000"

Regional Structure

companies:
  - code: "HQ"
    name: "Headquarters"
    currency: USD
    country: US
    volume_weight: 0.3
    is_parent: true

  - code: "NA01"
    name: "North America Operations"
    currency: USD
    country: US
    volume_weight: 0.3
    parent_code: "HQ"

  - code: "EU01"
    name: "EMEA Operations"
    currency: EUR
    country: DE
    volume_weight: 0.25
    parent_code: "HQ"

  - code: "AP01"
    name: "APAC Operations"
    currency: JPY
    country: JP
    volume_weight: 0.15
    parent_code: "HQ"

Validation

Intercompany Implications

When multiple companies exist:

• Intercompany transactions generated between companies
• FX rates generated for currency pairs
• Elimination entries created for parent
• Transfer pricing applied

See Intercompany Processing for details.

See Also

• Global Settings
• Intercompany Processing
• FX Settings

Transactions

Transaction settings control journal entry generation.

Configuration

transactions:
  target_count: 100000

  line_items:
    distribution: empirical
    min_lines: 2
    max_lines: 20

  amounts:
    min: 100
    max: 1000000
    distribution: log_normal
    round_number_bias: 0.15

  sources:
    manual: 0.3
    automated: 0.5
    recurring: 0.15
    adjustment: 0.05

  benford:
    enabled: true

  temporal:
    month_end_spike: 2.5
    quarter_end_spike: 3.0
    year_end_spike: 4.0
    working_hours_only: true

Fields

target_count

Total number of journal entries to generate.

transactions:
  target_count: 10000      # Small dataset
  target_count: 100000     # Medium dataset
  target_count: 1000000    # Large dataset

line_items

Controls the number of line items per journal entry.

distribution

Empirical distribution (default):

• 2 lines: 60.68%
• 3 lines: 5.24%
• 4 lines: 17.32%
• Even counts: 88% preference

line_items:
  distribution: empirical

Custom distribution:

line_items:
  distribution: custom
  custom_distribution:
    2: 0.50
    3: 0.10
    4: 0.20
    5: 0.10
    6: 0.10

min_lines / max_lines

line_items:
  min_lines: 2
  max_lines: 10

amounts

Controls transaction amounts.

min / max

amounts:
  min: 100           # Minimum amount
  max: 1000000       # Maximum amount

distribution

amounts:
  distribution: log_normal

round_number_bias

Preference for round numbers (100, 500, 1000, etc.).

amounts:
  round_number_bias: 0.15    # 15% round numbers
  round_number_bias: 0.0     # No round number bias

sources

Transaction source distribution (weights must sum to 1.0).

sources:
  manual: 0.3
  automated: 0.5
  recurring: 0.15
  adjustment: 0.05

benford

Benford’s Law compliance for first-digit distribution.

benford:
  enabled: true       # Follow P(d) = log10(1 + 1/d)
  enabled: false      # Disable Benford compliance

Expected distribution (enabled):

temporal

Temporal patterns for transaction timing.

Spikes

Volume multipliers for period ends:

temporal:
  month_end_spike: 2.5    # 2.5x volume at month end
  quarter_end_spike: 3.0  # 3.0x at quarter end
  year_end_spike: 4.0     # 4.0x at year end

Working Hours

Restrict transactions to business hours:

temporal:
  working_hours_only: true   # Mon-Fri, 8am-6pm
  working_hours_only: false  # Any time

Examples

High Volume Retail

transactions:
  target_count: 500000

  line_items:
    distribution: empirical
    min_lines: 2
    max_lines: 6

  amounts:
    min: 10
    max: 50000
    distribution: log_normal
    round_number_bias: 0.3

  sources:
    manual: 0.1
    automated: 0.8
    recurring: 0.08
    adjustment: 0.02

  temporal:
    month_end_spike: 1.5
    quarter_end_spike: 2.0
    year_end_spike: 5.0

Low Volume Manual

transactions:
  target_count: 5000

  line_items:
    distribution: empirical

  amounts:
    min: 1000
    max: 10000000

  sources:
    manual: 0.6
    automated: 0.2
    recurring: 0.1
    adjustment: 0.1

  temporal:
    month_end_spike: 3.0
    quarter_end_spike: 4.0
    year_end_spike: 5.0
    working_hours_only: true

Testing/Development

transactions:
  target_count: 1000

  line_items:
    distribution: uniform
    min_lines: 2
    max_lines: 4

  amounts:
    min: 100
    max: 10000
    distribution: uniform
    round_number_bias: 0.0

  sources:
    manual: 1.0

  benford:
    enabled: false

Validation

See Also

• Configuration Overview
• Document Flows
• Anomaly Injection

Master Data

Master data settings control generation of business entities.

Configuration

master_data:
  vendors:
    count: 200
    intercompany_ratio: 0.05

  customers:
    count: 500
    intercompany_ratio: 0.05

  materials:
    count: 1000

  fixed_assets:
    count: 100

  employees:
    count: 50
    hierarchy_depth: 4

Vendors

Supplier master data configuration.

master_data:
  vendors:
    count: 200                    # Number of vendors
    intercompany_ratio: 0.05      # IC vendor percentage

    payment_terms:
      - code: "NET30"
        days: 30
        weight: 0.5
      - code: "NET60"
        days: 60
        weight: 0.3
      - code: "NET10"
        days: 10
        weight: 0.2

    behavior:
      late_payment_rate: 0.1      # % with late payment tendency
      discount_usage_rate: 0.3    # % that take early pay discounts

Generated Fields

Customers

Customer master data configuration.

master_data:
  customers:
    count: 500                    # Number of customers
    intercompany_ratio: 0.05      # IC customer percentage

    credit_rating:
      - code: "AAA"
        limit_multiplier: 10.0
        weight: 0.1
      - code: "AA"
        limit_multiplier: 5.0
        weight: 0.2
      - code: "A"
        limit_multiplier: 2.0
        weight: 0.4
      - code: "B"
        limit_multiplier: 1.0
        weight: 0.3

    payment_behavior:
      on_time_rate: 0.7           # % that pay on time
      early_payment_rate: 0.1     # % that pay early
      late_payment_rate: 0.2      # % that pay late

Generated Fields

Materials

Product/material master data.

master_data:
  materials:
    count: 1000                   # Number of materials

    types:
      raw_material: 0.3
      work_in_progress: 0.1
      finished_goods: 0.4
      services: 0.2

    valuation:
      - method: fifo
        weight: 0.3
      - method: weighted_average
        weight: 0.5
      - method: standard_cost
        weight: 0.2

Generated Fields

Fixed Assets

Capital asset master data.

master_data:
  fixed_assets:
    count: 100                    # Number of assets

    categories:
      buildings: 0.1
      machinery: 0.3
      vehicles: 0.2
      furniture: 0.2
      it_equipment: 0.2

    depreciation:
      - method: straight_line
        weight: 0.7
      - method: declining_balance
        weight: 0.2
      - method: units_of_production
        weight: 0.1

Generated Fields

Employees

User/employee master data.

master_data:
  employees:
    count: 50                     # Number of employees
    hierarchy_depth: 4            # Org chart depth

    roles:
      - name: "AP Clerk"
        approval_limit: 5000
        weight: 0.3
      - name: "AP Manager"
        approval_limit: 50000
        weight: 0.1
      - name: "AR Clerk"
        approval_limit: 5000
        weight: 0.3
      - name: "Controller"
        approval_limit: 500000
        weight: 0.1
      - name: "CFO"
        approval_limit: 999999999
        weight: 0.05

    transaction_codes:
      - "FB01"     # Post document
      - "FB50"     # Enter GL
      - "F-28"     # Post incoming payment
      - "F-53"     # Post outgoing payment

Generated Fields

HR and Payroll Integration (v0.6.0)

Employee master data serves as the foundation for the hr configuration section introduced in v0.6.0. When the HR module is enabled, each employee record drives downstream generation:

• Payroll: Salary, tax withholdings, benefits deductions, and retirement contributions are computed per employee based on their role and the salary ranges defined in hr.payroll.salary_ranges. The pay_frequency setting (monthly, biweekly, or weekly) determines how many payroll runs are generated per period.
• Time and Attendance: Time entries are generated for each employee according to working days in the period. The overtime_rate controls how many employees have overtime hours in a given period.
• Expense Reports: A subset of employees (controlled by hr.expenses.submission_rate) generate expense reports each month. Policy violations are injected at the configured policy_violation_rate.

The employees.count and employees.hierarchy_depth settings in master_data directly determine the population size for all HR outputs. Increasing the employee count will proportionally increase payroll journal entries, time records, and expense reports.

master_data:
  employees:
    count: 200                     # Drives payroll and HR volume
    hierarchy_depth: 5

hr:
  enabled: true                    # Activates payroll, time, and expenses
  payroll:
    pay_frequency: "biweekly"      # 26 pay periods per year
  expenses:
    submission_rate: 0.40          # 40% of employees submit per month

Examples

Small Company

master_data:
  vendors:
    count: 50
  customers:
    count: 100
  materials:
    count: 200
  fixed_assets:
    count: 20
  employees:
    count: 10
    hierarchy_depth: 2

Large Enterprise

master_data:
  vendors:
    count: 2000
    intercompany_ratio: 0.1
  customers:
    count: 10000
    intercompany_ratio: 0.1
  materials:
    count: 50000
  fixed_assets:
    count: 5000
  employees:
    count: 500
    hierarchy_depth: 8

Validation

See Also

• Document Flows
• Compliance
• datasynth-generators

Document Flows

Document flow settings control P2P (Procure-to-Pay) and O2C (Order-to-Cash) process generation, including document types, three-way matching, credit checks, and document chain management.

Configuration

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.3
    completion_rate: 0.95
    three_way_match:
      quantity_tolerance: 0.02
      price_tolerance: 0.01

  o2c:
    enabled: true
    flow_rate: 0.3
    completion_rate: 0.95

Procure-to-Pay (P2P)

Flow

Purchase     Purchase    Goods      Vendor     Three-Way
Requisition → Order   → Receipt  → Invoice  → Match    → Payment
                │                     │          │
                │                ┌────┘          │
                ▼                ▼               ▼
           AP Open Item ← Match Result      AP Aging

Purchase Order Types

SyntheticData models 6 PO types, each with different downstream behavior:

Goods Receipt Movement Types

Goods receipts use SAP-style movement type codes:

Three-Way Match

The three-way match validator compares Purchase Order, Goods Receipt, and Vendor Invoice to detect variances before payment.

Algorithm

For each invoice line item:
  1. Find matching PO line (by PO reference + line number)
  2. Sum GR quantities for that PO line (supports multiple partial GRs)
  3. Compare:
     a. PO quantity vs GR quantity → QuantityPoGr variance
     b. GR quantity vs Invoice quantity → QuantityGrInvoice variance
     c. PO unit price vs Invoice unit price → PricePoInvoice variance
     d. PO total vs Invoice total → TotalAmount variance
  4. Apply tolerances:
     - Quantity: ±quantity_tolerance (default 2%)
     - Price: ±price_tolerance (default 5%)
     - Absolute: ±absolute_amount_tolerance (default $0.01)
  5. Check over-delivery:
     - If GR qty > PO qty and allow_over_delivery=true:
       allow up to max_over_delivery_pct (default 10%)

Variance Types

Match Outcomes

Configuration

document_flows:
  p2p:
    three_way_match:
      enabled: true
      price_tolerance: 0.05              # 5% price variance allowed
      quantity_tolerance: 0.02            # 2% quantity variance allowed
      absolute_amount_tolerance: 0.01     # $0.01 rounding tolerance
      allow_over_delivery: true
      max_over_delivery_pct: 0.10         # 10% over-delivery allowed

P2P Stage Configuration

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.3                       # 30% of JEs from P2P
    completion_rate: 0.95                # 95% complete full flow

    stages:
      po_approval_rate: 0.9             # 90% of POs approved
      gr_rate: 0.98                     # 98% of POs get goods receipts
      invoice_rate: 0.95                # 95% of GRs get invoices
      payment_rate: 0.92                # 92% of invoices get paid

    timing:
      po_to_gr_days:
        min: 1
        max: 30
      gr_to_invoice_days:
        min: 1
        max: 14
      invoice_to_payment_days:
        min: 10
        max: 60

P2P Journal Entries

Order-to-Cash (O2C)

Flow

Sales     Credit   Delivery    Customer    Customer
Order   → Check  → (Pick/   → Invoice   → Receipt
  │                Pack/         │           │
  │                Ship)         │           │
  │                  │           ▼           ▼
  │                  │      AR Open Item   AR Aging
  │                  │           │
  │                  │           └→ Dunning (if overdue)
  │                  ▼
  │            Inventory Issue
  │            (COGS posting)
  ▼
Revenue Recognition
(ASC 606 / IFRS 15)

Sales Order Types

SyntheticData models 9 SO types:

Delivery Types

6 delivery types model different fulfillment scenarios:

Customer Invoice Types

7 invoice types with different accounting treatment:

Credit Check

Sales orders pass through credit verification before delivery:

document_flows:
  o2c:
    credit_check:
      enabled: true
      check_credit_limit: true          # Verify customer limit
      check_overdue: true               # Check for past-due AR
      block_threshold: 0.9              # Block if >90% of limit used

O2C Stage Configuration

document_flows:
  o2c:
    enabled: true
    flow_rate: 0.3                       # 30% of JEs from O2C
    completion_rate: 0.95                # 95% complete full flow

    stages:
      so_approval_rate: 0.95            # 95% of SOs approved
      credit_check_pass_rate: 0.9       # 90% pass credit check
      delivery_rate: 0.98               # 98% of SOs get deliveries
      invoice_rate: 0.95                # 95% of deliveries get invoices
      collection_rate: 0.85             # 85% of invoices collected

    timing:
      so_to_delivery_days:
        min: 1
        max: 14
      delivery_to_invoice_days:
        min: 0
        max: 3
      invoice_to_payment_days:
        min: 15
        max: 90

O2C Journal Entries

Document Chain Manager

The document chain manager maintains referential integrity across the complete document flow by tracking references between documents.

Reference Types

Document Chain Output

PO-001 ─→ GR-001 ─→ INV-001 ─→ PAY-001
   │          │          │          │
   └──────────┴──────────┴──────────┘
              Document Chain

The document_references.csv output file records all links:

Complex Scenario Examples

Partial Deliveries with Split Invoice

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.4
    completion_rate: 0.90           # 10% incomplete (partial deliveries)
    three_way_match:
      quantity_tolerance: 0.05      # 5% tolerance for partials
      allow_over_delivery: true
      max_over_delivery_pct: 0.10
    timing:
      po_to_gr_days: { min: 3, max: 45 }    # Longer lead times
      gr_to_invoice_days: { min: 1, max: 21 }
      invoice_to_payment_days: { min: 30, max: 90 }

High-Volume Retail O2C

document_flows:
  o2c:
    enabled: true
    flow_rate: 0.5                  # 50% of JEs from O2C
    completion_rate: 0.98           # High completion rate
    stages:
      so_approval_rate: 0.99       # Auto-approved
      credit_check_pass_rate: 0.95
      delivery_rate: 0.99
      invoice_rate: 0.99
      collection_rate: 0.92
    timing:
      so_to_delivery_days: { min: 0, max: 3 }     # Fast fulfillment
      delivery_to_invoice_days: { min: 0, max: 0 } # Immediate invoice
      invoice_to_payment_days: { min: 10, max: 45 }

Combined Manufacturing P2P + O2C

document_flows:
  p2p:
    enabled: true
    flow_rate: 0.35
    completion_rate: 0.95
    three_way_match:
      quantity_tolerance: 0.02
      price_tolerance: 0.01
    timing:
      po_to_gr_days: { min: 5, max: 30 }
      gr_to_invoice_days: { min: 1, max: 10 }
      invoice_to_payment_days: { min: 20, max: 45 }

  o2c:
    enabled: true
    flow_rate: 0.35
    completion_rate: 0.90
    credit_check:
      enabled: true
      block_threshold: 0.85
    timing:
      so_to_delivery_days: { min: 3, max: 21 }
      delivery_to_invoice_days: { min: 0, max: 2 }
      invoice_to_payment_days: { min: 30, max: 60 }

Validation

See Also

• Subledgers — AR/AP records generated by document flows
• FX & Currency — Multi-currency document flows
• Master Data — Vendor and customer master records
• Process Chains — Enterprise process chain architecture
• Process Mining — OCEL 2.0 event logs from document flows
• datasynth-generators — Generator crate reference

Subledgers

SyntheticData generates subsidiary ledger records for Accounts Receivable (AR), Accounts Payable (AP), Fixed Assets (FA), and Inventory, with automatic GL reconciliation and document flow linking.

Overview

Subledger generators produce detailed records that reconcile back to GL control accounts:

Configuration

subledger:
  enabled: true
  ar:
    enabled: true
    aging_buckets: [30, 60, 90, 120]    # Days
    dunning_levels: 3
    credit_memo_rate: 0.05               # 5% of invoices get credit memos
  ap:
    enabled: true
    aging_buckets: [30, 60, 90, 120]
    early_payment_discount_rate: 0.02
    payment_scheduling: true
  fa:
    enabled: true
    depreciation_methods:
      - straight_line
      - declining_balance
      - sum_of_years_digits
    disposal_rate: 0.03                  # 3% of assets disposed per year
  inventory:
    enabled: true
    valuation_method: standard_cost      # standard_cost, moving_average, fifo, lifo
    cycle_count_frequency: monthly

Accounts Receivable (AR)

Record Types

The AR subledger generates:

• Open Items: Outstanding customer invoices with aging classification
• Receipts: Customer payments applied to invoices (full, partial, on-account)
• Credit Memos: Credits issued for returns, disputes, or pricing adjustments
• Aging Reports: Aged balances by customer and aging bucket
• Dunning Notices: Automated collection notices at configurable levels

Open Item Fields

Aging Buckets

Default aging buckets classify receivables by days past due:

Dunning Process

Dunning generates progressively urgent collection notices:

Document Flow Integration

AR open items are created from O2C customer invoices:

Sales Order → Delivery → Customer Invoice → AR Open Item → Customer Receipt
                                                 │
                                                 └→ Dunning Notice (if overdue)

Accounts Payable (AP)

Record Types

The AP subledger generates:

• Open Items: Outstanding vendor invoices with aging and payment scheduling
• Payments: Vendor payment runs (check, wire, ACH)
• Debit Memos: Deductions for quality issues, returns, pricing errors
• Aging Reports: Aged payables by vendor
• Payment Scheduling: Planned payments considering cash flow and discounts

Open Item Fields

Early Payment Discounts

The AP generator models cash discount optimization:

Payment Terms: 2/10 Net 30
  → Pay within 10 days: 2% discount
  → Pay by day 30: full amount
  → Past day 30: overdue

early_payment_discount_rate: 0.02   # Take 2% discount when offered

Payment Scheduling

When enabled, the AP generator creates a payment schedule that optimizes:

• Discount capture: Prioritize invoices with expiring discounts
• Cash flow: Spread payments across the period
• Vendor priority: Pay critical vendors first

Document Flow Integration

AP open items are created from P2P vendor invoices:

Purchase Order → Goods Receipt → Vendor Invoice → Three-Way Match → AP Open Item → Payment
                                                                          │
                                                                          └→ Debit Memo (if variance)

Fixed Assets (FA)

Record Types

The FA subledger generates:

• Asset Register: Master record for each fixed asset
• Depreciation Schedule: Monthly depreciation entries per asset
• Acquisitions: New asset additions (from PO or direct capitalization)
• Disposals: Asset retirements, sales, scrapping
• Transfers: Inter-company or inter-department transfers
• Impairment: Write-downs when fair value drops below book value

Asset Register Fields

Depreciation Methods

Depreciation Journal Entries

Each period, the FA generator creates depreciation entries:

Disposal Accounting

When an asset is disposed: