Optimize OpenClaw
untuk Workflow Skala Besar

Pendahuluan

Ketika workflow scale up, OpenClaw yang berjalan smooth dengan 10-20 tasks per hari suddenly jadi bottleneck kalau handle 1000+ tasks. Performance optimization bukan optional—ini necessity.

Artikel ini cover optimization techniques yang proven untuk scale OpenClaw ke production workload: dari caching strategy, parallel processing, queue management, database tuning, sampai horizontal scaling.

Performance Tuning Basics

1. Node.js Configuration

# .env
NODE_ENV=production
NODE_OPTIONS=--max-old-space-size=4096

# Increase event loop pool
UV_THREADPOOL_SIZE=16

# Disable source maps in production
NODE_DISABLE_SOURCEMAPS=1

2. PM2 Cluster Mode

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'openclaw',
    script: './dist/index.js',
    instances: 'max', // atau angka spesifik
    exec_mode: 'cluster',
    max_memory_restart: '2G',
    node_args: '--max-old-space-size=4096',
    env_production: {
      NODE_ENV: 'production',
      PORT: 3000
    }
  }]
}

pm2 start ecosystem.config.js --env production
pm2 save

3. Connection Pooling

// config/database.json
{
  "postgres": {
    "max": 20,
    "min": 5,
    "idle": 10000,
    "acquire": 30000,
    "evict": 1000
  },
  "redis": {
    "maxRetriesPerRequest": 3,
    "enableReadyCheck": true,
    "maxConnections": 50,
    "minIdleConnections": 10
  }
}

Caching Strategy

Caching adalah low-hanging fruit untuk optimization.

1. Redis Cache Configuration

{
  "cache": {
    "enabled": true,
    "driver": "redis",
    "ttl": 3600,
    "layers": {
      "task_results": {
        "ttl": 86400,
        "strategy": "cache-aside"
      },
      "api_responses": {
        "ttl": 300,
        "strategy": "write-through"
      },
      "user_sessions": {
        "ttl": 7200,
        "strategy": "write-behind"
      }
    }
  }
}

2. Task dengan Cache

{
  "name": "analyze-logs-cached",
  "cache": {
    "enabled": true,
    "key": "logs:analysis:{{date}}",
    "ttl": 3600
  },
  "actions": [
    {
      "type": "shell",
      "command": "grep 'ERROR' /var/log/app.log | wc -l",
      "output": "error_count"
    },
    {
      "type": "ai-analyze",
      "prompt": "Analyze error trends",
      "input": "{{error_count}}"
    }
  ]
}

3. Cache Warming

{
  "name": "cache-warmer",
  "schedule": "0 6 * * *",
  "actions": [
    {
      "type": "http",
      "method": "GET",
      "url": "https://api.example.com/popular-data",
      "cache": {
        "key": "popular:data",
        "ttl": 86400
      }
    }
  ]
}

Parallel Processing

1. Parallel Actions

{
  "name": "multi-backup-parallel",
  "actions": [
    {
      "type": "parallel",
      "maxConcurrent": 5,
      "tasks": [
        {
          "type": "shell",
          "command": "mysqldump -u root db1 > /backup/db1.sql"
        },
        {
          "type": "shell",
          "command": "tar -czf /backup/uploads.tar.gz /var/www/uploads"
        },
        {
          "type": "http",
          "method": "POST",
          "url": "https://api.backup.com/sync"
        },
        {
          "type": "shell",
          "command": "rclone sync /backup remote:backup"
        }
      ]
    }
  ]
}

2. Batch Processing

{
  "name": "process-large-dataset",
  "input": {
    "items": "..." // array of 10000 items
  },
  "actions": [
    {
      "type": "batch",
      "items": "{{items}}",
      "batchSize": 100,
      "maxConcurrent": 10,
      "action": {
        "type": "http",
        "method": "POST",
        "url": "https://api.example.com/process",
        "body": {
          "item": "{{item}}"
        }
      }
    }
  ]
}

Queue Management

Queue system untuk handle burst traffic dan prevent overload.

1. Task Queue Configuration

{
  "queue": {
    "driver": "redis",
    "concurrency": 10,
    "maxQueueSize": 10000,
    "priorities": {
      "high": 3,
      "normal": 2,
      "low": 1
    },
    "retries": {
      "maxAttempts": 3,
      "backoff": "exponential",
      "initialDelay": 1000
    }
  }
}

2. Priority Queue

{
  "name": "critical-backup",
  "priority": "high",
  "actions": [...]
}

{
  "name": "routine-cleanup",
  "priority": "low",
  "actions": [...]
}

3. Rate Limiting

{
  "rateLimits": {
    "api_calls": {
      "window": 60,
      "max": 100
    },
    "ai_requests": {
      "window": 60,
      "max": 20
    },
    "database_writes": {
      "window": 1,
      "max": 50
    }
  }
}

Database Optimization

1. Query Optimization

# Add indexes untuk frequent queries
CREATE INDEX idx_tasks_status ON tasks(status);
CREATE INDEX idx_tasks_created ON tasks(created_at);
CREATE INDEX idx_logs_timestamp ON logs(timestamp);

# Composite index untuk complex queries
CREATE INDEX idx_tasks_status_created ON tasks(status, created_at);

2. Bulk Operations

{
  "name": "bulk-insert-logs",
  "actions": [
    {
      "type": "database",
      "driver": "postgresql",
      "operation": "bulkInsert",
      "table": "logs",
      "batchSize": 1000,
      "data": "{{log_entries}}"
    }
  ]
}

3. Read Replicas

{
  "database": {
    "master": {
      "host": "db-master.example.com",
      "port": 5432
    },
    "replicas": [
      {
        "host": "db-replica-1.example.com",
        "weight": 1
      },
      {
        "host": "db-replica-2.example.com",
        "weight": 1
      }
    ],
    "readStrategy": "round-robin"
  }
}

Resource Management

1. Memory Management

{
  "resources": {
    "memory": {
      "limit": "2GB",
      "warningThreshold": 0.8,
      "actions": {
        "onWarning": "throttle",
        "onLimit": "reject"
      }
    },
    "cpu": {
      "limit": 80,
      "warningThreshold": 70
    }
  }
}

2. Connection Limits

{
  "limits": {
    "maxConcurrentTasks": 50,
    "maxConcurrentApiCalls": 20,
    "maxConcurrentDbConnections": 30,
    "maxQueuedTasks": 1000
  }
}

Load Balancing

1. Nginx Load Balancer

upstream openclaw_backend {
    least_conn;
    
    server 127.0.0.1:3000 weight=3 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 weight=3 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3002 weight=2 max_fails=3 fail_timeout=30s;
    
    keepalive 64;
}

server {
    listen 80;
    server_name openclaw.example.com;
    
    location / {
        proxy_pass http://openclaw_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}

2. Task Distribution

{
  "distribution": {
    "strategy": "least-busy",
    "workers": [
      {
        "id": "worker-1",
        "capacity": 10
      },
      {
        "id": "worker-2",
        "capacity": 10
      },
      {
        "id": "worker-3",
        "capacity": 5
      }
    ]
  }
}

Scaling Strategies

1. Vertical Scaling

Scale up resources di single server:

• Upgrade RAM: 8GB → 16GB → 32GB
• More CPU cores: 4 cores → 8 cores
• Faster storage: HDD → SSD → NVMe
• Increase network bandwidth

2. Horizontal Scaling

Add more servers:

# Deploy multiple instances
pm2 start ecosystem.config.js -i 4

# Atau separate servers dengan Redis shared queue
# Server 1: Worker
pm2 start dist/worker.js --name worker-1

# Server 2: Worker
pm2 start dist/worker.js --name worker-2

# Server 3: API + Scheduler
pm2 start dist/api.js --name api
pm2 start dist/scheduler.js --name scheduler

3. Auto-scaling (Advanced)

{
  "autoscaling": {
    "enabled": true,
    "minInstances": 2,
    "maxInstances": 10,
    "metrics": {
      "cpu": {
        "target": 70,
        "scaleUp": 80,
        "scaleDown": 30
      },
      "queueLength": {
        "target": 100,
        "scaleUp": 200,
        "scaleDown": 20
      }
    },
    "cooldown": {
      "scaleUp": 60,
      "scaleDown": 300
    }
  }
}

Benchmarking & Profiling

1. Load Testing

# Install k6
curl https://github.com/grafana/k6/releases/download/v0.45.0/k6-v0.45.0-linux-amd64.tar.gz -L | tar xvz
sudo mv k6-v0.45.0-linux-amd64/k6 /usr/local/bin/

# Load test
k6 run load-test.js

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 }, // ramp up
    { duration: '5m', target: 100 }, // steady
    { duration: '2m', target: 0 },   // ramp down
  ],
};

export default function () {
  const res = http.post('http://localhost:3000/api/tasks/run', JSON.stringify({
    taskId: 'test-task'
  }), {
    headers: { 'Content-Type': 'application/json' },
  });
  
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  
  sleep(1);
}

2. Performance Profiling

# CPU profiling dengan clinic.js
npm install -g clinic
clinic doctor -- node dist/index.js

# Flame graph
clinic flame -- node dist/index.js

# Bubbleprof (async operations)
clinic bubbleprof -- node dist/index.js

3. Benchmark Results Tracking

{
  "name": "daily-benchmark",
  "schedule": "0 2 * * *",
  "actions": [
    {
      "type": "shell",
      "command": "k6 run --out json=benchmark.json load-test.js",
      "output": "results"
    },
    {
      "type": "database",
      "query": "INSERT INTO benchmarks (date, metrics) VALUES (NOW(), $1)",
      "params": ["{{results}}"]
    },
    {
      "type": "ai-analyze",
      "prompt": "Compare today's benchmark with historical data and identify performance regressions",
      "output": "analysis"
    },
    {
      "type": "notification",
      "message": "📊 Daily Benchmark:\n{{analysis.summary}}"
    }
  ]
}

VPS untuk Scale Besar

FAQ

Apakah caching always improve performance?

Tidak selalu. Cache add overhead (memory, network latency ke Redis). Hanya cache data yang frequently accessed dan expensive to compute. Measure sebelum dan sesudah.

Vertical vs horizontal scaling, mana yang lebih baik?

Vertical scaling lebih simple (no distributed system complexity), tapi ada limit. Horizontal scaling lebih scalable tapi lebih complex. Start vertical, scale horizontal kalau sudah hit limit.

Berapa concurrency yang optimal untuk task execution?

Tergantung task type. CPU-intensive: concurrency = CPU cores. I/O-intensive: concurrency = 2-4x CPU cores. Benchmark untuk find sweet spot.

How to handle traffic spikes?

Implement queue system, rate limiting, dan load balancing. Queue absorb spikes, worker process gradually. Auto-scaling kalau budget allow.

Kesimpulan

Optimization adalah incremental process. Prioritize berdasarkan impact:

1. Quick wins: Enable caching, connection pooling, cluster mode
2. Medium effort: Database indexing, query optimization, parallel processing
3. Long term: Load balancing, horizontal scaling, auto-scaling

👉 Langkah Selanjutnya

Performance yang optimal = Happy users + Lower costs.