Actor Monitoring System

A comprehensive monitoring and reporting system for Apify actors and tasks. This actor generates detailed reports comparing recent runs and provides comprehensive analysis over configurable monthly periods.

🚀 Features

• Configurable Runs Comparison: Compare the recent two runs of and Actor or Task, or any 2 runs you wish
• Flexible Monthly Reports: Generate detailed Excel reports for any period (1-24 months) with data categorization
• Direct Actor Runs: Fetch runs directly from actors without going through tasks for simplified monitoring
• Email Notifications: Automated email delivery of reports to specified recipients
• Advanced Filtering: Include/exclude actors and tasks based on name patterns
• Performance Metrics: Track runtime, costs, and efficiency across all runs
• Geographic Analysis: Categorize data by country and region
• Excel Export: Generate professional Excel reports with multiple sheets

🏗️ Clean Architecture & Code Quality

🎯 Modular Service Architecture

The Actor Monitoring System follows clean architecture principles with a comprehensive service layer that eliminates code duplication and provides consistent patterns:

Core Services Layer

• 📧 EmailValidationService: Centralized email validation with consistent messaging
• 🌐 ApiClientWrapper: Unified error handling for all Apify API calls
• 📝 LogHelper: Standardized logging patterns with emoji formatting
• ✅ InputValidator: Comprehensive input validation matching interface contracts
• 🎨 EmailStyles: Modular CSS management with theme support (default, dark, minimal)
• 🔽 FilteringService: Unified include/exclude pattern matching across actors and tasks
• 📊 SheetFactory: Standardized Excel sheet creation with consistent formatting

Architecture Benefits

• Code Quality: 475+ lines of duplicated code eliminated through DRY principles
• Maintainability: Single sources of truth for common operations
• Consistency: Uniform patterns for error handling, logging, and validation
• Testability: Smaller, focused functions with clear responsibilities
• Extensibility: Easy to add new themes, validation rules, or data formats

📋 Report Types

1. Runs Comparison Report (compare)

Compares the configurable number of recent refresh runs (2-10) for each task to identify:

• Changes in product counts
• Performance variations
• Potential issues or improvements
• Automatically sends alerts for significant changes (>5% difference)
• Uses configurable time period for analysis (1-24 months)

📧 Email Grouping Options

• Consolidated Emails (default): All comparisons combined into a single email with one table
• Individual Emails: Separate email sent for each actor/task comparison
• Configurable via combineEmails parameter

2. Monthly Comprehensive Report (monthly)

Generates comprehensive reports for a configurable time period (1-24 months, defaults to 3) including:

• Refresh Runs: Regular data collection activities
• Historical Runs: Historical data processing activities
• Performance Analytics: Runtime, cost, and efficiency metrics
• Geographic Distribution: Country-wise data breakdown
• Excel Files: Professional reports with categorized data

🧠 Intelligent Field Comparison

Enhanced Analysis Features

The actor monitoring system now includes intelligent field comparison with data-driven strategy selection:

🔍 Automatic Strategy Detection

The system automatically analyzes your scraped data and selects the most meaningful comparison approach:

• Numeric Fields: Automatically detects price, rating, and metric fields
  • Average Comparison: For price-like data (e.g., $45.99 → $52.10 (+13.29%))
  • Sum Comparison: For aggregate metrics and large datasets
• Array Fields: Compares total counts (e.g., reviews, images, tags)
• Data Quality: Monitors field coverage for sparse or optional fields
• Categorical Fields: Tracks item counts for complete datasets

📊 Smart Field Classification

Instead of relying on field names, the system analyzes actual data characteristics:

• Value Range Analysis: Detects price ranges, ratings (0-5, 0-10), percentages
• Data Distribution: Identifies complete vs sparse fields (< 80% coverage)
• Type Detection: Distinguishes between numeric aggregates and categorical data
• Context Awareness: Provides business-relevant descriptions in reports

🎯 Meaningful Business Metrics

The system generates business-focused insights that directly relate to scraping success:

"Total reviews count: 1,875"  
"Customer rating average: 4.2 stars"
"Product availability: 95% in stock"

📈 Advanced Insights

Each comparison now includes additional context:

• Dataset size changes
• Unique value diversity trends
• Data completeness improvements
• Performance impact analysis

🎯 User-Defined Comparison Strategies

You can now override the automatic strategy detection by explicitly defining how each field should be compared:

📝 Strategy Configuration

Use the fieldsAndCompareStrategy parameter to specify custom comparison strategies:

{
  "actorMode": "compare",
  "fieldsAndCompareStrategy": {
    "product.price": "average",
    "reviews": "array_length", 
    "product.description": "coverage",
    "total_sales": "sum"
  }
}

📊 Available Strategies

Strategy	Description	Best For	Example Output
count	Compare number of items containing the field	Categorical data, item counts	Items with category (150 → 175)
sum	Compare total sum of all numeric values	Revenue, totals, aggregate metrics	Total sales sum (15,750.00 → 18,240.00)
average	Compare average of all numeric values	Prices, ratings, scores	Average price ($45.99 → $52.10)
array_length	Compare total length of all arrays combined	Reviews, images, tags, lists	Total reviews count (1,250 → 1,875)
coverage	Compare percentage of items with field populated	Data quality, optional fields	Data completeness (75% → 83% populated)

🔄 Strategy Selection Priority

1. User-Defined Strategy: Uses your explicit strategy choice
2. Automatic Detection: Falls back to intelligent auto-detection if no strategy specified
3. Legacy Fields: Falls back to auto-detection for fields not in strategy map

💡 Strategy Selection Tips

• Prices/Ratings: Use average for meaningful price tracking
• Aggregates/Totals: Use sum for revenue, sales, or count totals
• Data Quality: Use coverage to monitor field completeness
• Arrays/Lists: Use array_length for counting elements across items
• Categories: Use count for tracking items by category

⚙️ Hybrid Approach

You can mix user-defined and auto-detected strategies:

{
  "fieldsAndCompareStrategy": {
    "price": "average",        // Explicit: track average price
    "reviews": "array_length", // Explicit: count total reviews
    // "category" and "optional_field" will use auto-detection
  }
}

🔧 Configuration

Input Parameters

Parameter	Type	Required	Description
actorMode	string	✅	Report type: compare or monthly
useDirectActorRuns	boolean	❌	If enabled, fetches runs directly from actors instead of through tasks (defaults to true)
monthsToAnalyze	integer	❌	Number of months to analyze for reports (1-24, defaults to 3)
maxRuns	integer	❌	Maximum number of runs to analyze per actor/task in monthly reports (1-1000, defaults to 100)
compareRunId1	string	❌	First specific run ID for comparison (only used in compare mode)
compareRunId2	string	❌	Second specific run ID for comparison (only used in compare mode)
combineEmails	boolean	❌	Combine all comparisons into single email (defaults to true)
emailRecipients	array	❌	List of email addresses for report delivery
actorNameInclude	array	❌	Include actors containing these substrings
actorNameExclude	array	❌	Exclude actors containing these substrings
taskNameInclude	array	❌	Include tasks containing these substrings (ignored if useDirectActorRuns is true)
taskNameExclude	array	❌	Exclude tasks containing these substrings (ignored if useDirectActorRuns is true)
token	string	❌	Custom Apify API token (uses default if not provided)
debug	boolean	❌	Enable detailed debug logging (defaults to false)
fieldsAndCompareStrategy	object	❌	Custom strategy mapping for field comparisons (auto-detection used if not provided)
filterRunsByStatus	array	❌	Run statuses to filter by in monthly reports (defaults to [] - no filtering)

Configuration Examples

1. Basic Monthly Report (Default)

{
    "actorMode": "monthly"
}

This configuration will:

• Generate a 3-month comprehensive report (default)
• Use direct actor runs mode (default)
• Analyze up to 100 runs per actor (default)
• Include only SUCCEEDED runs (default)
• No email notifications (no recipients specified)

2. Extended 12-Month Analysis with Email Notifications

{
    "actorMode": "monthly",
    "monthsToAnalyze": 12,
    "maxRuns": 200,
    "emailRecipients": ["quarterly-reports@company.com", "team-lead@company.com"],
    "actorNameInclude": ["production-"],
    "filterRunsByStatus": ["SUCCEEDED", "FAILED"],
    "debug": false
}

This configuration will:

• Generate a 12-month comprehensive report
• Analyze up to 200 runs per actor for more comprehensive data
• Include both successful and failed runs for complete analysis
• Filter to only actors with "production-" in their names
• Send email reports to specified recipients
• Use standard logging level

3. Detailed Comparison of Last 2 Runs (Combined Email)

{
    "actorMode": "compare",
    "combineEmails": true,
    "emailRecipients": ["monitoring@company.com"],
    "actorNameInclude": ["data-scraper", "product-monitor"],
    "actorNameExclude": ["test", "dev"],
    "fieldsAndCompareStrategy": {
        "product.price": "average",
        "reviews": "array_length",
        "details.rating": "coverage",
        "total_items": "sum"
    },
    "debug": true
}

This configuration will:

• Compare the latest 2 runs from each matching actor
• Combine all comparisons into a single email report
• Use custom field comparison strategies for intelligent analysis
• Filter actors to include specific patterns and exclude test/dev actors
• Enable debug logging for detailed execution information
• Send consolidated email to monitoring team

4. Individual Email per Comparison (Task-Based Mode)

{
    "actorMode": "compare",
    "useDirectActorRuns": false,
    "combineEmails": false,
    "emailRecipients": ["individual-alerts@company.com"],
    "taskNameInclude": ["production", "main"],
    "taskNameExclude": ["backup", "archive"],
    "debug": true
}

This configuration will:

• Use task-based mode instead of direct actor runs
• Send separate email for each task comparison
• Filter tasks to include specific patterns and exclude backup/archive tasks
• Enable debug logging for troubleshooting
• Send individual email alerts for focused monitoring

5. Specific Run Comparison

{
    "actorMode": "compare",
    "compareRunId1": "abcd1234efgh5678",
    "compareRunId2": "wxyz9876ijkl5432",
    "emailRecipients": ["analysis-team@company.com"],
    "fieldsAndCompareStrategy": {
        "dataset_items": "count",
        "processing_time": "sum",
        "error_rate": "coverage"
    }
}

This configuration will:

• Compare two specific runs by their IDs
• Use custom comparison strategies for specific metrics
• Send detailed analysis to the analysis team
• Ignore actor/task filtering (specific runs mode)

6. Comprehensive Production Monitoring

{
    "actorMode": "monthly",
    "monthsToAnalyze": 6,
    "maxRuns": 500,
    "useDirectActorRuns": true,
    "emailRecipients": ["production-team@company.com", "management@company.com"],
    "actorNameInclude": ["prod-", "live-"],
    "actorNameExclude": ["staging", "test", "dev"],
    "filterRunsByStatus": ["SUCCEEDED", "FAILED", "ABORTED"],
    "debug": false
}

This configuration will:

• Generate a comprehensive 6-month production report
• Analyze up to 500 runs per actor for maximum data coverage
• Include all run statuses for complete operational visibility
• Filter to production actors only
• Include custom fields in Excel reports for detailed analysis
• Send reports to both technical team and management
• Use standard logging for production stability

📧 Email Combination Feature

The system now supports two email delivery modes for comparison reports:

🔗 Combined Emails (Default - combineEmails: true)

• Single Email: All detected comparisons combined into one email
• Unified Table: One table containing all actor/task comparisons
• Cleaner Inbox: Reduces email volume significantly
• Better Overview: See all changes at once in a single view

📧 Individual Emails (combineEmails: false)

• Separate Emails: One email per actor/task comparison
• Focused Content: Each email contains only one comparison
• Legacy Behavior: Maintains the original email delivery pattern
• Individual Alerts: Better for specific monitoring setups

Note: In compare mode, the system automatically fetches the most recent runs for comparison, ensuring you get the latest data. When using specific run IDs (compareRunId1 and compareRunId2), all filtering parameters are ignored and only those specific runs are compared.

Direct Actor Runs vs Task-Based Runs

The system supports two modes for fetching run data:

🎭 Direct Actor Runs Mode (useDirectActorRuns: true)

• Purpose: Monitor actor performance directly without task-level organization
• Use Case: When you want to analyze all runs from specific actors regardless of how they were triggered
• Benefits:
  • Simpler setup - no need to configure task filters
  • Captures all actor runs including those not associated with tasks
  • Better for actor-level performance analysis
• Note: Task filtering parameters (taskNameInclude, taskNameExclude) are ignored

📋 Task-Based Mode (useDirectActorRuns: false - default)

• Purpose: Monitor runs organized by tasks for better categorization
• Use Case: When you want detailed task-level analysis and organization
• Benefits:
  • Better data organization by task categories
  • More granular filtering capabilities
  • Ideal for task-specific performance monitoring

Example: Direct Actor Runs Configuration

{
    "actorMode": "compare",
    "useDirectActorRuns": true,
    "actorNameInclude": ["data-scraper", "product-monitor"],
    "emailRecipients": ["team@company.com"],
    "debug": true
}

📋 Complete Usage Scenarios

Scenario 1: Daily Production Monitoring

Use Case: Monitor critical production actors daily for performance issues

{
    "actorMode": "compare",
    "useDirectActorRuns": true,
    "combineEmails": true,
    "emailRecipients": ["devops@company.com", "alerts@company.com"],
    "actorNameInclude": ["prod-"],
    "actorNameExclude": ["staging", "test"],
    "fieldsAndCompareStrategy": {
        "total_items": "count",
        "runtime_seconds": "sum",
        "memory_mb": "average",
        "cost_usd": "sum"
    },
    "debug": false
}

Expected Output: Single email with comparison table showing all production actors, their performance metrics, and any significant changes.

Scenario 2: Weekly Quality Assurance Reports

Use Case: Generate comprehensive weekly reports for QA team review

{
    "actorMode": "monthly",
    "monthsToAnalyze": 1,
    "maxRuns": 50,
    "useDirectActorRuns": false,
    "emailRecipients": ["qa-team@company.com"],
    "taskNameInclude": ["QA", "testing"],
    "filterRunsByStatus": ["SUCCEEDED", "FAILED"],
    "debug": true
}

Expected Output: Excel report with one month of QA runs, including custom test metrics and success/failure analysis.

Scenario 3: Monthly Executive Dashboard

Use Case: High-level monthly reports for executives showing business metrics

{
    "actorMode": "monthly",
    "monthsToAnalyze": 3,
    "maxRuns": 1000,
    "useDirectActorRuns": true,
    "emailRecipients": ["ceo@company.com", "cto@company.com", "operations@company.com"],
    "actorNameInclude": ["revenue-", "customer-", "product-"],
    "filterRunsByStatus": ["SUCCEEDED"],
    "debug": false
}

Expected Output: Professional Excel report with 3-month business metrics, geographic analysis, and performance trends.

Scenario 4: Incident Investigation

Use Case: Compare specific runs when investigating performance issues

{
    "actorMode": "compare",
    "compareRunId1": "good_run_id_12345",
    "compareRunId2": "problem_run_id_67890",
    "emailRecipients": ["engineering@company.com"],
    "fieldsAndCompareStrategy": {
        "error_count": "sum",
        "timeout_count": "sum",
        "memory_usage": "average",
        "cpu_usage": "average",
        "dataset_quality": "coverage"
    },
    "debug": true
}

Expected Output: Detailed comparison email showing exactly what changed between the good run and the problematic run.

Scenario 5: A/B Testing Analysis

Use Case: Compare two different versions of an actor to measure performance improvements

{
    "actorMode": "compare",
    "useDirectActorRuns": true,
    "combineEmails": false,
    "emailRecipients": ["data-science@company.com"],
    "actorNameInclude": ["experiment-a", "experiment-b"],
    "fieldsAndCompareStrategy": {
        "processing_speed": "average",
        "accuracy_score": "average",
        "resource_efficiency": "sum",
        "data_completeness": "coverage"
    },
    "debug": true
}

Expected Output: Separate email for each experiment actor comparison, allowing detailed analysis of A/B test results.

Scenario 6: Client Reporting

Use Case: Generate client-facing reports showing data collection performance

{
    "actorMode": "monthly",
    "monthsToAnalyze": 6,
    "maxRuns": 100,
    "useDirectActorRuns": false,
    "emailRecipients": ["client-success@company.com"],
    "taskNameInclude": ["client-alpha", "client-beta"],
    "filterRunsByStatus": ["SUCCEEDED"],
    "debug": false
}

Expected Output: Clean Excel report showing 6 months of data collection performance for specific clients.

Scenario 7: Performance Regression Detection

Use Case: Automated monitoring to catch performance regressions early

{
    "actorMode": "compare",
    "combineEmails": true,
    "emailRecipients": ["performance-team@company.com"],
    "actorNameInclude": ["critical-"],
    "fieldsAndCompareStrategy": {
        "response_time": "average",
        "success_rate": "coverage",
        "throughput": "sum",
        "error_rate": "coverage"
    },
    "debug": false
}

Expected Output: Consolidated email highlighting any performance regressions across critical actors.

Scenario 8: Comprehensive System Health Check

Use Case: Monthly system-wide health assessment

{
    "actorMode": "monthly",
    "monthsToAnalyze": 1,
    "maxRuns": 200,
    "useDirectActorRuns": true,
    "emailRecipients": ["sre-team@company.com", "management@company.com"],
    "filterRunsByStatus": ["SUCCEEDED", "FAILED", "ABORTED", "RUNNING"],
    "debug": false
}

Expected Output: Comprehensive system health report including all run statuses and custom reliability metrics.

📊 Data Flow

graph TD
    A[Input Configuration] --> B[Initialize Apify Client]
    B --> C{Report Mode}
    
    C -->|compare| D{Direct Actor Runs?}
    C -->|monthly| E{Direct Actor Runs?}
    
    D -->|Yes| D1[Fetch Actor Runs Directly]
    D -->|No| D2[Fetch Task Runs]
    E -->|Yes| E1[Fetch Actor Runs Directly]
    E -->|No| E2[Fetch Task Runs]
    
    D1 --> F[Compare Recent Runs]
    D2 --> F
    F --> G[Generate Comparison Report]
    G --> H[Send Email Alert]
    
    E1 --> I[Categorize by Month/Country]
    E2 --> I
    I --> J[Create Excel Reports]
    J --> K[Store in Key-Value Store]
    K --> L[Send Comprehensive Email]
    
    H --> M[Complete]
    L --> M

📧 Email Reports

Enhanced Comparison Report Email

• Subject: Dynamic subject based on comparison type (Dataset Comparison or Run Comparison)

• Intelligent Analysis: Automatically detects and switches between run-level and dataset-level comparisons

• Email Grouping: Choose between consolidated (single email) or individual emails per comparison

• Content Structure:

  📊 Dataset-Level Analysis Mode (when dataset items available)

  • Analysis Overview Panel: Explains dataset comparison methodology
  • Enhanced Summary Cards: Dataset item counts, growth percentages, field-specific changes
  • Smart Field Analysis: Automatic strategy selection (coverage, sums, array lengths)
  • Data Quality Insights: Field coverage percentages, unique value counts
  • Direct Run Links: Quick access to both compared runs in Apify Console

  ⚡ Run-Level Metrics Mode (fallback)

  • Performance Overview Panel: Explains run metrics comparison
  • Standard Summary Cards: Runtime, cost, memory, CPU, item counts
  • Efficiency Trends: Performance and resource usage analysis
  • Direct Run Links: Quick access to compared runs

  🔧 Universal Features

  • Dynamic Field Support: Any field path with dot notation (details.rating, reviews.0.text)
  • Smart Field Formatting: Converts snake_case to "Title Case Change"
  • Adaptive Insights: Context-aware recommendations based on analysis type
  • Responsive Design: Mobile-friendly HTML with modern styling
  • Type-Aware Comparisons: Handles strings (coverage), numbers (sums), arrays (lengths)
  • Run Links: Direct links to both new and previous runs in the Apify Console
  • Consolidated Table: When multiple comparisons exist, all data combined in one table

Example Dataset Analysis:

• Retailer Name Coverage: 98.5% → 99.1% (+0.6%)
• Details Rating Sum: 6,125.5 → 6,958.0 (+13.6%)
• Reviews Array Length: 3,450 → 4,120 (+19.4%)
• Questions And Answers Length: 890 → 950 (+6.7%)

Monthly Report Email

• Subject: Monthly comprehensive report
• Content:
  • Executive summary
  • Download links for Excel files
  • Raw data access URLs
  • Usage instructions

🆕 Custom Dimensions for Monthly Reports

The system supports custom categorization of runs in monthly reports using the customDimensions input property. This allows you to group and organize runs by any field(s) in your data, such as actor name, status, region, or custom metrics.

Input Property: customDimensions

• Type: Array of objects
• Each object:
  • field (string, required): Path to extract values from run data (e.g. actorName, status, stats.runTimeSecs)
  • label (string, optional): Human-readable label for this dimension
  • fallback (string, optional): Default value when field is missing or null

Example

{
  "customDimensions": [
    { "field": "actorName", "label": "Actor Name", "fallback": "Unknown Actor" },
    { "field": "status", "label": "Run Status", "fallback": "Unknown Status" }
  ]
}

Use Cases

• Group runs by actor names
• Organize by success/failure status
• Categorize by runtime, cost, or other metrics
• Group by region or market data

---

📈 Performance

Optimization Features

• Parallel processing with configurable concurrency
• Efficient data pagination
• Memory-optimized Excel generation
• Batch processing for large datasets

Scalability

• Handles hundreds of actors and thousands of runs
• Optimized for configurable monthly reporting periods
• Configurable processing limits

Email Delivery Failures

• Cause: Invalid email addresses or service issues
• Solution: Validate email list and check service status