# CipherStream Enterprise Data Extraction Platform **Customer API Documentation & Testing Collection** --- ## About CipherStream **CipherStream** is Centaur Software's proprietary enterprise bulk data extraction platform, designed for high-performance data extraction from cloud-hosted databases with intelligent processing optimisation. ### Key Features - **39 Specialised Procedures**: Complete coverage of dental practice management data - **Intelligent Processing**: Auto-switching between streaming (≤100K rows) and job processing (>100K rows) - **Enterprise Security**: AES-256-GCM encryption, API key authentication, IP whitelisting - **Multiple Formats**: JSON, NDJSON, CSV with optional GZIP/ZIP compression - **Real-time & Batch**: Sub-second streaming for small datasets, S3-backed jobs for large datasets - **Webhook Integration**: Real-time notifications for job completion --- ## Authentication ### API Key Types | Key Type | Format | Access Level | Use Case | |----------|--------|--------------|----------| | **Customer Live** | `cs_live_*` | Production API access | Live data extraction | | **Customer Demo** | `cs_demo_*` | Demo/testing access | Testing and evaluation | ### Headers Required ```http Authorization: Bearer Content-Type: application/json ``` --- ## Execution Modes CipherStream automatically determines the optimal processing method based on data volume: ### Auto Mode (Recommended) - **≤100K rows**: Streaming response (immediate) - **>100K rows**: Background job with S3 download - **>50MB estimated**: Automatic job mode regardless of row count - **System decides**: Based on current load and data characteristics ### Stream Mode - **Force streaming**: Real-time response for all requests - **Recommended limit**: ≤100K rows for optimal performance - **Timeout**: 30 seconds maximum - **Use case**: Small datasets requiring immediate response ### Job Mode - **Force background processing**: All requests create jobs - **S3 storage**: Secure file storage with presigned URLs - **Extended timeout**: 180 seconds for large extractions - **Use case**: Large datasets, scheduled extractions --- ## Date Modifiers Control which records are returned based on their creation and modification dates: ### Available Options - **All** (default): Returns all records within the date range regardless of creation/update dates - **Created**: Returns only records created within the specified date range - **Updated**: Returns only records updated within the specified date range ### Usage Examples ```json { "date_modifier": "Created", "from_date": "2024-01-01", "to_date": "2024-12-31" } ``` ### Behavior by Procedure Type - **Reference Data**: Date modifiers not applicable (always returns current data) - **Data Procedures**: Uses ts_4_insert (Created) and ts_4_update (Updated) fields - **Special Procedures**: Applies to underlying table timestamp fields --- ## Output Formats ### NDJSON (Default) - **Format**: Newline-delimited JSON - **Structure**: One JSON object per line - **Benefits**: Streaming-friendly, memory efficient - **Use case**: Large datasets, real-time processing ``` {"id": 1, "name": "John Doe", "date": "2024-01-01"} {"id": 2, "name": "Jane Smith", "date": "2024-01-02"} {"metadata": {"execution_time": 2.5, "record_count": 2}} ``` ### JSON - **Format**: Standard JSON array - **Structure**: Array of objects with metadata - **Benefits**: Standard format, easy parsing - **Use case**: Small to medium datasets, standard integrations ```json { "data": [ {"id": 1, "name": "John Doe", "date": "2024-01-01"}, {"id": 2, "name": "Jane Smith", "date": "2024-01-02"} ], "metadata": {"execution_time": 2.5, "record_count": 2} } ``` ### CSV - **Format**: Comma-separated values - **Structure**: Header row followed by data rows - **Benefits**: Universal compatibility, Excel-friendly - **Use case**: Reporting, data analysis, spreadsheet import ```csv id,name,date 1,John Doe,2024-01-01 2,Jane Smith,2024-01-02 ``` --- ## Compression Options ### None (Default) - **No compression**: Raw output format - **Use case**: Small datasets, immediate processing ### GZIP - **Compression ratio**: 70-90% size reduction - **Format**: .gz compressed files - **Use case**: Large datasets, bandwidth optimisation ### ZIP - **Compression ratio**: 60-80% size reduction - **Format**: .zip archive files - **Use case**: Multiple files, Windows compatibility --- ## Security & Encryption ### Data in Transit - **TLS 1.3**: All API communications encrypted - **Certificate pinning**: Enhanced security for production - **HSTS**: HTTP Strict Transport Security enabled ### Data at Rest - **AES-256-GCM**: S3 bucket encryption for job results - **Presigned URLs**: Time-limited access (12 hours) - **Automatic cleanup**: Files removed after expiry ### Authentication - **Bearer tokens**: API key-based authentication - **IP whitelisting**: Optional IP restriction - **Rate limiting**: Prevents abuse and ensures fair usage --- ## Webhook System ### Supported Events - **job.completed**: Job finished successfully with download URL - **job.failed**: Job encountered an error with details - **job.cancelled**: Job was manually cancelled ### Security Features - **HMAC-SHA256**: Payload signature verification - **Automatic retries**: Up to 3 attempts with exponential backoff - **Delivery tracking**: Monitor success/failure rates - **Timeout handling**: 30-second response timeout ### Webhook Payload Example ```json { "event": "job.completed", "timestamp": "2024-09-26T10:02:15Z", "signature": "sha256=abc123...", "data": { "job_id": "appointments_1758630144658_c7511999", "customer_id": "your-customer-id", "status": "completed", "rows_processed": 45678, "execution_time_seconds": 64.75, "s3_url": "https://secure-download-url", "expires_at": "2024-09-26T22:02:15Z", "file_size_bytes": 2048576, "output_format": "ndjson", "compression": "gzip" } } ``` --- ## Job Management Lifecycle ### Job States 1. **Queued**: Job created and waiting for processing 2. **Running**: Data extraction in progress with real-time progress updates 3. **Completed**: Data available for download via S3 presigned URL 4. **Failed**: Error occurred during processing with detailed error information 5. **Cancelled**: Job manually cancelled by user or system timeout ### Job Features - **Progress tracking**: Real-time percentage completion (0-100%) - **Row counting**: Live count of processed records - **Time estimation**: Estimated completion time based on current progress - **Resource monitoring**: Memory and CPU usage tracking - **Error handling**: Detailed error messages and recovery suggestions ### Download Management - **S3 presigned URLs**: Secure, time-limited download links - **12-hour expiry**: URLs automatically expire for security - **Resume support**: Partial download recovery for large files - **Bandwidth optimisation**: CDN-accelerated downloads --- ## Data Procedures Overview ### Reference Data (16 procedures) - **Always streamed**: Immediate response (no job creation) - **5-minute cache**: Optimal performance for frequently accessed data - **No parameters required**: Simple, consistent access pattern - **Small datasets**: Typically <1000 records per procedure - **Use cases**: Dropdown population, validation, system configuration ### Data Procedures (22 procedures) - **Intelligent mode selection**: Auto streaming/job based on volume - **Date range filtering**: Flexible from_date/to_date parameters - **Date modifier support**: Created, Updated, or All records - **High-volume capable**: Handles millions of records efficiently - **Use cases**: Bulk data extraction, reporting, analytics ### Special Procedures (1 procedure) - **CALL syntax**: Advanced stored procedure execution - **Direct table access**: Extract from any accessible database table - **Extended timeout**: 180-second timeout for large extractions - **Flexible parameters**: Custom table names and date filtering - **Use cases**: Custom extractions, ad-hoc queries, data migration --- ## Performance Characteristics ### Streaming Response - **Latency**: <2 seconds for reference data - **Throughput**: Up to 10,000 records/second - **Memory usage**: Constant memory footprint - **Concurrent requests**: Up to 10 simultaneous streams ### Job Processing - **Throughput**: Up to 100,000 records/second - **Scalability**: Auto-scaling based on queue depth - **Resource allocation**: Dedicated processing resources - **Monitoring**: Real-time progress and performance metrics --- ## Error Handling ### HTTP Status Codes - **200**: Success - Data returned or job created - **400**: Bad Request - Invalid parameters or format - **401**: Unauthorized - Invalid or missing API key - **403**: Forbidden - Access denied or rate limited - **404**: Not Found - Endpoint or resource not found - **429**: Too Many Requests - Rate limit exceeded - **500**: Internal Server Error - System error occurred ### Error Response Format ```json { "error": { "code": "INVALID_DATE_RANGE", "message": "The specified date range is invalid", "details": "from_date must be earlier than to_date", "timestamp": "2024-09-26T10:00:00Z", "request_id": "req_abc123" } } ``` --- ## Environment URLs - **Production**: https://api.cipherstream.com - **UAT**: https://cipherstream-uat.centaur.software --- ## Support **Centaur Software** - Enterprise Support - **Email**: support@centaur.software - **Documentation**: docs.cipherstream.com - **Professional Services**: Custom integration and optimisation - **SLA**: 99.9% uptime guarantee for production environment --- **© 2024 Centaur Software. CipherStream is a proprietary enterprise platform.**

Patient Management

Core patient data and analytics for comprehensive patient relationship management.

Characteristics:

• Patient demographics - Core patient information and contact details
• Patient analytics - Lifetime value and business intelligence
• Follow-up management - Recall scheduling and tracking

Use Cases:

• Patient management systems
• Contact information updates
• Demographics analysis
• Marketing and communication
• Patient retention analysis
• Follow-up scheduling

Scheduling & Appointments

Complete appointment management including scheduling, status tracking, and optimisation.

Characteristics:

• Core appointments - Main appointment records with intelligent processing
• Status management - Appointment statuses and types for workflow control
• Schedule optimisation - Cancellation tracking and waitlist management
• Communication - Reminder and notification systems

Use Cases:

• Appointment booking systems
• Schedule management
• Cancellation analysis
• Waitlist optimisation
• Patient communication
• Scheduling templates

Financial Management

Comprehensive financial data including transactions, payments, invoicing, and accounts receivable.

Characteristics:

• Core financials - Main financial transaction records
• Payment processing - Receipts, deposits, and payment allocations
• Billing management - Invoices and discount tracking
• Accounts receivable - Debtor management and outstanding balances
• Pricing structure - Fee schedules and payment types

Use Cases:

• Financial reporting and analysis
• Revenue tracking
• Accounting system integration
• Business intelligence dashboards
• Payment reconciliation
• Debt collection workflows

Clinical & Treatment

Clinical data management including completed treatments and treatment planning.

Characteristics:

• Treatment records - Completed procedures and clinical work
• Treatment planning - Proposed treatments and care plans
• Service catalogue - Available procedures and service items

Use Cases:

• Clinical reporting and analysis
• Treatment outcome tracking
• Care plan management
• Service utilisation analysis
• Clinical decision support
• Quality assurance

Business Operations

Business expense management and operational cost tracking.

Characteristics:

• Expense tracking - Business expense records with date filtering
• Expense categorisation - Expense categories for financial reporting

Use Cases:

• Expense tracking and reporting
• Budget analysis
• Tax preparation
• Cost centre reporting
• Financial planning
• Operational efficiency analysis

Financial Data: Extract expense records and categorisations.

Use Cases:

• Expense tracking and reporting
• Budget analysis
• Tax preparation
• Cost centre reporting

Reference Data: Retrieve expense category classifications for financial reporting.

Use Cases:

• Expense tracking applications
• Financial reporting categorisation
• Budget planning tools

Practice Setup & Configuration

Practice management configuration including staff, locations, and system setup.

Characteristics:

• Staff management - Practitioners and users with role definitions
• Location management - Practice locations and facilities
• System configuration - User roles and access control
• External relationships - Health funds, third parties, and referral sources
• Recall management - Recall types and follow-up configurations

Use Cases:

• Practice setup and configuration
• Staff management systems
• Location and facility management
• User access control
• External partner integration
• Follow-up system configuration

Advanced Tools

Advanced data extraction capabilities for power users and custom requirements.

Characteristics:

• Direct table access - Extract from any accessible database table
• CALL syntax - Advanced stored procedure execution
• Extended timeout - 180-second timeout for large extractions
• Flexible parameters - Custom table names and date filtering

Use Cases:

• Custom data extractions
• Ad-hoc reporting requirements
• Data migration projects
• Specialised analytics queries
• Direct database access
• Custom integration needs

Job Management

Background job management for large data extractions and processing with comprehensive monitoring and control.

When Jobs Are Created:

• Large datasets: >100,000 rows automatically trigger job processing
• File size threshold: >50MB estimated output size
• Explicit job mode: User specifically requests job processing
• System load balancing: High system load triggers job queuing
• Complex queries: Resource-intensive extractions
• Scheduled extractions: Automated recurring data pulls

Job Lifecycle States:

1. Queued - Job created and waiting for available processing resources
2. Running - Data extraction in progress with real-time progress updates
3. Completed - Data successfully extracted and available for download via S3
4. Failed - Error occurred during processing with detailed error information
5. Cancelled - Job manually cancelled by user or system timeout

Job Processing Features:

• Real-time progress tracking - Live percentage completion (0-100%)
• Row count monitoring - Current number of processed records
• Time estimation - Estimated completion time based on current progress
• Resource allocation - Dedicated processing resources for optimal performance
• Error handling - Detailed error messages and recovery suggestions
• Automatic retries - Built-in retry logic for transient failures

Download Management:

• S3 secure storage - Enterprise-grade cloud storage with encryption
• Presigned URLs - Time-limited, secure download links
• 12-hour expiry - URLs automatically expire for security
• Resume support - Partial download recovery for large files
• CDN acceleration - Global content delivery for faster downloads
• Bandwidth optimisation - Compressed files for efficient transfer

Monitoring & Notifications:

• Webhook integration - Real-time job completion alerts
• Email notifications - Optional email alerts for job status changes
• Progress callbacks - Periodic progress updates via webhooks
• Performance metrics - Execution time, throughput, and resource usage
• Audit logging - Complete job history and user actions

Security & Compliance:

• Data encryption - AES-256-GCM encryption for stored files
• Access control - Customer-specific job isolation
• Audit trail - Complete job lifecycle logging
• Automatic cleanup - Files removed after expiry for data protection
• IP restrictions - Optional IP-based access control

Integrations

Advanced webhook system for real-time notifications, system integration, and automated workflow triggers.

Supported Events:

• job.completed - Job finished successfully with download URL and metadata
• job.failed - Job encountered an error with detailed failure information
• job.cancelled - Job was manually cancelled or timed out
• job.progress - Periodic progress updates during job execution (optional)
• system.maintenance - Scheduled maintenance notifications
• api.rate_limit - Rate limit threshold warnings

Webhook Security Features:

• HMAC-SHA256 signature - Cryptographic payload verification using shared secret
• Timestamp validation - Prevents replay attacks with time-based verification
• IP whitelisting - Optional source IP restrictions for enhanced security
• TLS encryption - All webhook deliveries use HTTPS/TLS 1.3
• Signature verification - Complete payload integrity checking

Delivery & Reliability:

• Automatic retries - Up to 3 retry attempts with exponential backoff
• Delivery tracking - Complete success/failure monitoring and logging
• Timeout handling - 30-second response timeout with configurable settings
• Dead letter queue - Failed deliveries stored for manual retry
• Circuit breaker - Automatic endpoint disabling for persistent failures
• Rate limiting - Configurable delivery rate limits to prevent overwhelming

Webhook Payload Structure:

{
  "event": "job.completed",
  "timestamp": "2024-09-26T10:02:15Z",
  "signature": "sha256=abc123def456...",
  "delivery_id": "del_1758630144658",
  "attempt": 1,
  "data": {
    "job_id": "appointments_1758630144658_c7511999",
    "customer_id": "your-customer-id",
    "procedure_name": "appointments",
    "status": "completed",
    "rows_processed": 45678,
    "execution_time_seconds": 64.75,
    "file_size_bytes": 2048576,
    "s3_url": "https://secure-download-url",
    "expires_at": "2024-09-26T22:02:15Z",
    "output_format": "ndjson",
    "compression": "gzip",
    "metadata": {
      "from_date": "2024-01-01",
      "to_date": "2024-12-31",
      "date_modifier": "Created"
    }
  }
}

Signature Verification:

import hmac
import hashlib

def verify_webhook_signature(payload, signature, secret):
    expected_signature = hmac.new(
        secret.encode('utf-8'),
        payload.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(f"sha256={expected_signature}", signature)

Configuration Options:

• Event filtering - Subscribe to specific events only
• Custom headers - Add custom HTTP headers to webhook requests
• Retry configuration - Customize retry attempts and backoff strategy
• Timeout settings - Configure response timeout values
• Batch delivery - Group multiple events into single webhook call

Monitoring & Debugging:

• Delivery logs - Complete webhook delivery history
• Response tracking - HTTP status codes and response times
• Error analysis - Detailed failure reasons and troubleshooting
• Performance metrics - Delivery success rates and latency statistics
• Test endpoints - Webhook testing and validation tools

Use Cases:

• Automated workflows - Trigger business processes when jobs complete
• Real-time notifications - Instant alerts when data extraction finishes
• System integration - Connect CipherStream to other business systems
• Data pipeline automation - Chain multiple data processing steps
• Monitoring and alerting - Track job completion and system health
• Business intelligence - Trigger report generation and dashboard updates
• Customer notifications - Inform end users when their data is ready

Documentation

API documentation and OpenAPI schemas for integration and development.

Available Documentation:

• Interactive Docs - Swagger UI for testing endpoints
• OpenAPI Schema - Machine-readable API specification
• Integration Guides - Code examples and best practices

Use Cases:

• Interactive API testing and exploration
• Code generation for client libraries
• API documentation for development teams
• Integration planning and validation
• Request/response format reference