Scheduling & Automation

Automate container updates, cleanups, and maintenance tasks with cron-based scheduling

Overview

Dockhand’s unified scheduler service manages all automated tasks using cron expressions with timezone support. The scheduler handles container auto-updates, Git stack synchronization, system cleanup, and custom maintenance tasks.

Scheduler Architecture

The scheduler is built on Croner, a robust cron implementation with timezone support:

/**
 * Unified Scheduler Service
 *
 * Manages all scheduled tasks using croner with automatic job lifecycle:
 * - System cleanup jobs (static cron schedules)
 * - Container auto-updates (dynamic schedules from database)
 * - Git stack auto-sync (dynamic schedules from database)
 *
 * All execution logic is in separate task files for clean architecture.
 */
import { Cron } from 'croner';

const activeJobs: Map<string, Cron> = new Map();

export async function startScheduler(): Promise<void> {
  console.log('[Scheduler] Starting scheduler service...');
  
  // Get default timezone from database
  const defaultTimezone = await getDefaultTimezone();
  
  // Start system cleanup jobs
  cleanupJob = new Cron(scheduleCleanupCron, { timezone: defaultTimezone }, async () => {
    await runScheduleCleanupJob();
  });
  
  // Register all dynamic schedules from database
  await refreshAllSchedules();
  
  console.log('[Scheduler] Service started');
}

Cron Expression Format

Dockhand uses standard cron expressions with optional seconds:

# Format: [second] minute hour day month weekday
# Ranges: 0-59   0-59  0-23 1-31 1-12  0-7

# Examples:
"0 3 * * *"        # Daily at 3:00 AM
"*/15 * * * *"     # Every 15 minutes
"0 */6 * * *"      # Every 6 hours
"0 0 * * 0"        # Weekly on Sunday at midnight
"0 2 1 * *"        # Monthly on the 1st at 2:00 AM
"0 0 1 1 *"        # Yearly on January 1st at midnight

Advanced Expressions

# Multiple values
"0 0 8,12,16 * * *"     # At 8 AM, 12 PM, and 4 PM

# Ranges
"0 0 9-17 * * 1-5"      # Every hour from 9 AM to 5 PM, Monday-Friday

# Steps
"*/30 9-17 * * 1-5"     # Every 30 min during work hours on weekdays

# Day of week names
"0 9 * * MON-FRI"       # 9 AM Monday through Friday

Timezone Support

Every schedule can have its own timezone:

// Environment-specific timezone
{
  "cronExpression": "0 2 * * *",
  "timezone": "America/New_York"  // 2 AM Eastern Time
}

// Default timezone for system jobs
{
  "defaultTimezone": "Europe/Warsaw"  // Stored in settings
}

Supported Timezones

Any IANA timezone is supported:

• UTC
• America/New_York
• Europe/London
• Asia/Tokyo
• Australia/Sydney

Container Auto-Updates

Configuration

Configure automatic updates for any container:

POST /api/auto-update/settings

{
  "containerName": "myapp",
  "environmentId": 1,
  "enabled": true,
  "cronExpression": "0 3 * * *",
  "vulnerabilityCriteria": "critical_high"
}

Vulnerability Criteria

Block updates based on vulnerability scans:

type VulnerabilityCriteria = 
  | 'never'              // Never block updates (no scanning)
  | 'any'                // Block if any vulnerabilities found
  | 'critical_high'      // Block if critical or high vulnerabilities
  | 'critical'           // Block only on critical vulnerabilities
  | 'more_than_current'; // Block if worse than current image

Update Process

The auto-update task performs intelligent updates:

export async function runContainerUpdate(
  settingId: number,
  containerName: string,
  environmentId: number | null | undefined,
  triggeredBy: ScheduleTrigger
): Promise<void> {
  const log = (message: string) => {
    console.log(`[Auto-update] ${message}`);
    appendScheduleExecutionLog(execution.id, `[${new Date().toISOString()}] ${message}`);
  };

  log(`Checking container: ${containerName}`);
  
  // Check registry for updates
  const registryCheck = await checkImageUpdateAvailable(imageNameFromConfig, currentImageId, envId);
  
  if (!registryCheck.hasUpdate) {
    log(`Already up-to-date: ${containerName}`);
    return;
  }

  log(`Update available! Registry digest: ${registryCheck.registryDigest}`);
  
  // Pull new image
  await pullImage(imageNameFromConfig, undefined, envId);
  
  // Scan for vulnerabilities if enabled
  if (shouldScan) {
    const scanOutcome = await scanAndCheckBlock({
      newImageId,
      currentImageId,
      envId,
      vulnerabilityCriteria,
      log
    });
    
    if (scanOutcome.blocked) {
      log(`UPDATE BLOCKED: ${scanOutcome.reason}`);
      await sendEventNotification('auto_update_blocked', {
        title: 'Auto-update blocked',
        message: `Container "${containerName}" update blocked: ${scanOutcome.reason}`,
        type: 'warning'
      }, envId);
      return;
    }
  }
  
  // Recreate container with new image
  log(`Recreating container with full config passthrough...`);
  await recreateContainer(containerName, envId, log, imageNameFromConfig);
  
  log(`Successfully updated container: ${containerName}`);
}

Skip Conditions

Auto-updates are automatically skipped for:

1. Digest-pinned images: nginx@sha256:abc123...
2. Local images: Images not available in a registry
3. System containers: Dockhand and Hawser agents
4. Already up-to-date: Current image matches registry
5. Failed vulnerability scan: When criteria not met

Git Stack Sync

Scheduled Sync

Automatically sync Git stacks on a schedule:

{
  "stackName": "production-app",
  "autoUpdate": true,
  "autoUpdateCron": "0 */6 * * *"  // Every 6 hours
}

Smart Sync Behavior

The scheduler only redeploys when changes are detected:

export async function runGitStackSync(
  stackId: number,
  stackName: string,
  environmentId: number | null | undefined,
  triggeredBy: ScheduleTrigger
): Promise<void> {
  log(`Starting sync for stack: ${stackName}`);

  // Deploy the git stack (only if there are changes)
  const result = await deployGitStack(stackId, { force: false });

  if (result.success) {
    if (result.skipped) {
      log(`No changes detected for stack: ${stackName}, skipping redeploy`);
      await sendEventNotification('git_sync_skipped', {
        title: 'Git sync skipped',
        message: `Stack "${stackName}" sync skipped: no changes detected`,
        type: 'info'
      }, envId);
    } else {
      log(`Successfully deployed stack: ${stackName}`);
      await sendEventNotification('git_sync_success', {
        title: 'Git stack deployed',
        message: `Stack "${stackName}" was synced and deployed successfully`,
        type: 'success'
      }, envId);
    }
  }
}

System Cleanup Jobs

Schedule Execution Cleanup

Automatically remove old execution logs:

{
  "scheduleCleanupEnabled": true,
  "scheduleCleanupCron": "0 2 * * *",      // Daily at 2 AM
  "scheduleRetentionDays": 30               // Keep 30 days
}

Event Cleanup

Clean up container event logs:

{
  "eventCleanupEnabled": true,
  "eventCleanupCron": "0 3 * * *",         // Daily at 3 AM
  "eventRetentionDays": 7                   // Keep 7 days
}

Volume Helper Cleanup

Automatic cleanup of temporary volume browser containers:

// Runs every 30 minutes automatically
volume HelperCleanupJob = new Cron('*/30 * * * *', { timezone: defaultTimezone }, async () => {
  await runVolumeHelperCleanupJob('cron', volumeCleanupFns);
});

Environment Update Checks

Schedule periodic checks for available updates across an entire environment:

{
  "environmentId": 1,
  "enabled": true,
  "cron": "0 8 * * *",  // Daily at 8 AM
  "notifyOnAvailable": true
}

Image Pruning

Automate Docker image cleanup:

{
  "environmentId": 1,
  "enabled": true,
  "cronExpression": "0 4 * * 0",  // Weekly on Sunday at 4 AM
  "pruneAll": false,
  "pruneFilters": {
    "dangling": true,
    "until": "720h"  // 30 days
  }
}

Schedule Management

Register a Schedule

export async function registerSchedule(
  scheduleId: number,
  type: 'container_update' | 'git_stack_sync' | 'env_update_check' | 'image_prune',
  environmentId: number | null
): Promise<boolean> {
  const key = `${type}-${scheduleId}`;

  // Get timezone for this environment
  const timezone = environmentId ? await getEnvironmentTimezone(environmentId) : 'UTC';

  // Create new Cron instance with timezone
  const job = new Cron(cronExpression, { timezone }, async () => {
    // Execute the task
    if (type === 'container_update') {
      await runContainerUpdate(scheduleId, containerName, environmentId, 'cron');
    } else if (type === 'git_stack_sync') {
      await runGitStackSync(scheduleId, stackName, environmentId, 'cron');
    }
  });

  // Store in active jobs map
  activeJobs.set(key, job);
  console.log(`[Scheduler] Registered ${type} schedule ${scheduleId}: ${cronExpression} [${timezone}]`);
  return true;
}

Unregister a Schedule

export function unregisterSchedule(
  scheduleId: number,
  type: 'container_update' | 'git_stack_sync' | 'env_update_check' | 'image_prune'
): void {
  const key = `${type}-${scheduleId}`;
  const job = activeJobs.get(key);

  if (job) {
    job.stop();
    activeJobs.delete(key);
    console.log(`[Scheduler] Unregistered ${type} schedule ${scheduleId}`);
  }
}

Refresh Schedules

Reload all schedules from database:

POST /api/schedules/refresh

Execution Tracking

All scheduled tasks create execution records:

interface ScheduleExecution {
  id: number;
  scheduleType: 'container_update' | 'git_stack_sync' | 'system_cleanup';
  scheduleId: number;
  environmentId: number | null;
  entityName: string;
  triggeredBy: 'cron' | 'manual' | 'webhook';
  status: 'running' | 'success' | 'failed' | 'skipped';
  startedAt: string;
  completedAt: string | null;
  duration: number | null;
  logs: string;
  errorMessage: string | null;
  details: any;
}

View Execution History

GET /api/schedules/executions?scheduleType=container_update&limit=50

Response:

{
  "executions": [
    {
      "id": 123,
      "scheduleType": "container_update",
      "entityName": "nginx",
      "status": "success",
      "triggeredBy": "cron",
      "startedAt": "2024-03-04T03:00:00Z",
      "completedAt": "2024-03-04T03:02:15Z",
      "duration": 135000,
      "logs": "[2024-03-04T03:00:00Z] Checking container: nginx\n..."
    }
  ]
}

Manual Triggers

Trigger any scheduled task manually:

Trigger Container Update

POST /api/auto-update/settings/{id}/trigger

Trigger Git Stack Sync

POST /api/git/stacks/{id}/sync

Trigger System Job

POST /api/schedules/system/{jobId}/trigger

Available system jobs:

• schedule-cleanup
• event-cleanup
• volume-helper-cleanup

Best Practices

Cron Expression Tips

1. Avoid peak hours for intensive operations

   "0 3 * * *"  # Good: 3 AM
   "0 14 * * *" # Bad: 2 PM during work hours
   

2. Distribute load across the hour

   # Instead of all at the top of the hour
   "0 0 * * *"   # All jobs at :00
   
   # Spread them out
   "0 2 * * *"   # Schedule cleanup at 2 AM
   "0 3 * * *"   # Container updates at 3 AM
   "0 4 * * *"   # Git syncs at 4 AM
   

3. Consider timezones for multi-region deployments

   // US East Coast production
   { "timezone": "America/New_York", "cron": "0 2 * * *" }
   
   // European production
   { "timezone": "Europe/London", "cron": "0 2 * * *" }
   

Update Strategy

// Conservative: Daily updates with critical vulnerability blocking
{
  "cronExpression": "0 3 * * *",
  "vulnerabilityCriteria": "critical_high"
}

// Aggressive: Frequent updates, no blocking
{
  "cronExpression": "0 */6 * * *",
  "vulnerabilityCriteria": "never"
}

// Paranoid: Only update if new image is better
{
  "cronExpression": "0 3 * * 0",  // Weekly
  "vulnerabilityCriteria": "more_than_current"
}

Troubleshooting

Schedule Not Running

1. Check if scheduler is running:

   GET /api/health
   

2. Verify cron expression:

   POST /api/schedules/validate
   {"cronExpression": "0 3 * * *"}
   

3. Check execution logs:

   GET /api/schedules/executions?scheduleId={id}
   

Timezone Issues

// Get next run time in specific timezone
GET /api/schedules/next-run?cron=0 3 * * *&timezone=America/New_York

Response:
{
  "nextRun": "2024-03-05T03:00:00-05:00",
  "localTime": "2024-03-05T08:00:00Z"
}

Performance Issues

If too many schedules are impacting performance:

1. Consolidate schedules: Group similar tasks
2. Increase intervals: Use longer cron periods
3. Distribute timing: Spread jobs across different times
4. Monitor execution times: Check logs for slow operations

API Reference

# List all schedules
GET /api/schedules

# Get schedule details
GET /api/schedules/{type}/{id}

# Create/update schedule
POST /api/auto-update/settings
POST /api/git/stacks

# Trigger manually
POST /api/schedules/{type}/{id}/trigger

# Get execution history
GET /api/schedules/executions

# Validate cron expression
POST /api/schedules/validate

# Get next run time
GET /api/schedules/next-run

# Refresh all schedules
POST /api/schedules/refresh