Creators Agency Docs

Data Sources & Integrations ​

Mission Control aggregates data from multiple internal and external sources to provide comprehensive operational visibility. This page documents all data sources, their update frequencies, and integration patterns.

Architecture Overview ​

Primary Data Sources ​

1. Convex Real-time Database ​

Purpose: Live operational data and user interactions
Connection: Environment variable NEXT_PUBLIC_CONVEX_SITE_URL
Authentication: ACTIVITY_LOG_SECRET for write operations

Tables:

revenue_snapshots ​

  • Purpose: Historical revenue data across all sources
  • Update frequency: Daily (6am CT via cron)
  • Key fields: totalYtdUsd, projectedAnnualUsd, sources, snapshotDate
  • Used by: Revenue tracker, progress bars, source breakdown

needs_apple ​

  • Purpose: Items requiring Apple's attention
  • Update frequency: Every 2 hours (8am-10pm CT)
  • Key fields: fromAgent, description, context, resolved
  • Used by: Hero section blockers, Apple's morning brief

activity_events ​

  • Purpose: System activity timeline
  • Update frequency: Real-time as events occur
  • Key fields: eventType, agent, status, timestamp, details
  • Used by: Recent activity feed, system health

system_metrics ​

  • Purpose: Infrastructure health data
  • Update frequency: Every 15 minutes
  • Key fields: cpu, memory, disk, services, timestamp
  • Used by: System health indicators, infrastructure page

2. File-based Data Sources ​

Location: /Users/aurora/.openclaw/workspace/data/

Agent Handoffs (agent-handoffs.jsonl) ​

  • Format: JSONL (one JSON object per line)
  • Purpose: Inter-agent communication and coordination
  • Update pattern: Appended to as agents communicate
  • Key fields: from, to, type, summary, details, action_needed
  • Used by: Needs Apple parsing, agent coordination

Pending Work (memory/pending-work.md) ​

  • Format: Markdown with checkboxes and status indicators
  • Purpose: Open tasks and blockers across all projects
  • Update pattern: Manual updates by agents
  • Parsed for: Blocked items, Apple dependencies
  • Used by: Needs Apple system, project status

Projects Data (public/data/projects.json) ​

  • Format: Static JSON array
  • Purpose: All agency projects with status and metrics
  • Update pattern: Manual updates as projects progress
  • Used by: Projects page, home dashboard kanban, active projects

Cron Execution Logs ​

  • Location: Various log files in /var/log/ and data/cron_logs/
  • Purpose: Scheduled job execution history and status
  • Update pattern: Generated after each cron run
  • Used by: Cron health monitoring, system health scores

3. External API Sources ​

Close CRM ​

  • Purpose: Sponsorship deal tracking and revenue
  • API: https://api.close.com/api/v1/
  • Authentication: Basic auth with API key from .secrets/close_api_key.txt
  • Update frequency: Daily via revenue collector
  • Data used: Won opportunities for revenue calculation
  • Rate limits: 600 requests per minute

Impact Affiliate Network ​

  • Purpose: Affiliate earnings and performance data
  • API: https://api.impact.com/Mediapartners/{account_sid}/
  • Authentication: API key from .secrets/impact_api_key.txt
  • Update frequency: Daily via revenue collector
  • Data used: partner_performance_by_day reports for YTD earnings
  • Rate limits: 1000 requests per hour

RedVentures ​

  • Purpose: Commission from RV Media Network
  • API: https://reporting-api.rvmedianetwork.com/
  • Authentication: OAuth2 client credentials flow
  • Credentials: Client ID and API key from .secrets/
  • Update frequency: Daily via revenue collector
  • Limitations: 31-day maximum window per request

AdsByMoney ​

  • Purpose: Publisher earnings from CPC campaigns
  • Method: Automated web scraping (no official API)
  • Authentication: Session-based login with stored credentials
  • Update frequency: Daily via revenue collector
  • Fallback: Manual data file when scraping fails

GitHub API ​

  • Purpose: Repository status, deployment tracking
  • API: https://api.github.com/
  • Authentication: GitHub token (when available)
  • Data used: Build status, deployment information, commit activity
  • Used by: Infrastructure health, development metrics

Vercel API ​

  • Purpose: Application deployment and performance metrics
  • API: https://api.vercel.com/
  • Authentication: Team-scoped API token
  • Current status: 403 errors since Mar 23 (needs Apple to fix permissions)
  • Data used: Build health, deployment status, performance metrics

Data Update Frequencies ​

Real-time (< 1 minute) ​

  • Convex database updates
  • User interactions (resolve buttons, status changes)
  • System alerts and notifications
  • Agent heartbeat monitoring

Frequent (5-15 minutes) ​

  • System health metrics (CPU, memory, disk)
  • Service status monitoring
  • Process monitoring
  • Network connectivity checks

Regular (1-2 hours) ​

  • Needs Apple sync from pending work and handoffs
  • Agent status updates
  • Recent activity aggregation
  • Error log analysis

Daily (6am CT) ​

  • Revenue collection from all sources
  • Financial metrics calculation
  • YTD progress updates
  • Revenue source breakdown

Manual/On-demand ​

  • Project status updates
  • MSN revenue entry (monthly around 21st)
  • Infrastructure changes
  • New project creation

Data Processing Pipeline ​

Revenue Collection Flow ​

Health Monitoring Flow ​

Data Quality and Reliability ​

Error Handling ​

  • API timeouts: Automatic retry with exponential backoff
  • Authentication failures: Credential refresh and retry
  • Rate limiting: Queue requests and respect limits
  • Data validation: Schema validation before storage
  • Fallback data: Manual data sources when automation fails

Monitoring and Alerts ​

  • Missing data alerts — When expected data doesn't arrive
  • Stale data warnings — When data hasn't updated recently
  • API failure notifications — When external APIs are down
  • Data quality checks — Validation of ranges and formats

Backup and Recovery ​

  • Database backups: Convex automatic backups
  • File backups: Git version control for configuration
  • Manual data preservation: Critical manual entries backed up
  • Recovery procedures: Documented steps for data restoration

Performance Considerations ​

Caching Strategy ​

  • Dashboard data: 5-minute cache for expensive queries
  • Static data: Browser caching for projects and configuration
  • API responses: Temporary caching to reduce external calls
  • Computed metrics: Pre-calculated values stored in database

Rate Limit Management ​

  • Close CRM: Batch requests, respect 600/minute limit
  • Impact API: Queue requests, stay under 1000/hour
  • RedVentures: OAuth token refresh as needed
  • GitHub: Use conditional requests when available

Data Volume Management ​

  • Historical data retention: Archive old metrics after 1 year
  • Log rotation: Automatic cleanup of old log files
  • Database cleanup: Remove obsolete records
  • File system monitoring: Prevent disk space issues

Security and Access Control ​

API Credentials ​

  • Storage: Encrypted in .secrets/ directory
  • Access: Restricted to specific scripts and services
  • Rotation: Regular credential updates
  • Monitoring: Track API usage and detect anomalies

Database Security ​

  • Authentication: Secret-based access for write operations
  • Encryption: Data encrypted in transit and at rest
  • Access logs: Track all database modifications
  • Backup security: Encrypted backup storage

Data Privacy ​

  • PII handling: No personal information stored in health metrics
  • Financial data: Revenue aggregates only, no individual transactions
  • Access control: Role-based access to sensitive data
  • Audit trails: Track who accessed what data when

Future Enhancements ​

Additional Data Sources ​

  • Google Analytics — Website traffic and conversion metrics
  • Slack Analytics — Team communication patterns
  • Calendar API — Meeting and schedule optimization
  • Email metrics — Communication effectiveness tracking

Real-time Capabilities ​

  • WebSocket connections — Live dashboard updates
  • Push notifications — Real-time alerts
  • Live data streaming — Continuous metric updates
  • Real-time collaboration — Multi-user dashboard interactions

Advanced Analytics ​

  • Predictive modeling — Forecast revenue and capacity needs
  • Anomaly detection — Identify unusual patterns automatically
  • Trend analysis — Long-term pattern recognition
  • Performance benchmarking — Compare against historical baselines