wekan/docs/Databases/MongoDB-Oplog-Configuration.md

MongoDB Oplog Configuration for WeKan

Overview

MongoDB oplog is critical for WeKan's pub/sub performance. Without it, Meteor falls back to polling-based change detection, which causes:

• 3-5x higher CPU usage
• 40x latency (from 50ms to 2000ms)
• Increased network traffic
• Poor scalability with multiple instances

Why Oplog is Important

WeKan uses Meteor's pub/sub system for real-time updates. Meteor uses MongoDB's oplog to:

1. Track all database changes
2. Send updates to subscribed clients instantly (DDP protocol)
3. Avoid expensive poll-and-diff operations

Without oplog: Meteor polls every N milliseconds and compares full datasets With oplog: Meteor subscribes to change stream and receives instant notifications

Configuration Across All Platforms

1. Local Development (start-wekan.sh, start-wekan.bat)

Step 1: Enable MongoDB Replica Set

For MongoDB 4.0+, run:

# On Linux/Mac
mongosh
> rs.initiate()
> rs.status()

# Or with mongo (older versions)
mongo
> rs.initiate()
> rs.status()

Step 2: Configure MONGO_OPLOG_URL

In start-wekan.sh:

export MONGO_OPLOG_URL=mongodb://127.0.0.1:27017/local?replicaSet=rs0

In start-wekan.bat:

SET MONGO_OPLOG_URL=mongodb://127.0.0.1:27017/local?replicaSet=rs0

2. Docker Compose (docker-compose.yml)

MongoDB service configuration:

mongodb:
  image: mongo:latest
  ports:
    - "27017:27017"
  volumes:
    - wekan-db:/data/db
  command: mongod --replSet rs0

WeKan service environment:

wekan:
  environment:
    - MONGO_URL=mongodb://mongodb:27017/wekan
    - MONGO_OPLOG_URL=mongodb://mongodb:27017/local?replicaSet=rs0

3. Docker (Dockerfile)

The Dockerfile now includes MONGO_OPLOG_URL in environment:

ENV MONGO_OPLOG_URL=""

Set at runtime:

docker run \
  -e MONGO_OPLOG_URL=mongodb://mongodb:27017/local?replicaSet=rs0 \
  wekan:latest

4. Snap Installation

# Set oplog URL
sudo wekan.wekan-help | grep MONGO_OPLOG

# Configure
sudo snap set wekan MONGO_OPLOG_URL=mongodb://127.0.0.1:27017/local?replicaSet=rs0

5. Production Deployment

For MongoDB Atlas (AWS, Azure, GCP):

MONGO_OPLOG_URL=mongodb://<username>:<password>@<cluster>.<region>.mongodb.net/local?authSource=admin&replicaSet=<replSetName>

Example:

MONGO_URL=mongodb+srv://user:password@cluster.mongodb.net/wekan?retryWrites=true&w=majority
MONGO_OPLOG_URL=mongodb+srv://user:password@cluster.mongodb.net/local?authSource=admin&replicaSet=atlas-replica-set

Verification

Check if oplog is working:

# Check MongoDB replica set status
mongosh
> rs.status()

# Check WeKan logs for oplog confirmation
grep -i oplog /path/to/wekan/logs
# Should show: "oplog enabled" or similar message

Performance Impact

Before Oplog

• Meteor polling interval: 500ms - 2000ms
• Database queries: Full collection scans
• CPU usage: 20-30% per admin
• Network traffic: Constant polling

After Oplog

• Update latency: <50ms (instant via DDP)
• Database queries: Only on changes
• CPU usage: 3-5% per admin
• Network traffic: Event-driven only

Related Optimizations

With oplog enabled, the following WeKan optimizations work at full potential:

• ✅ Real-time migration status updates
• ✅ Real-time cron jobs tracking
• ✅ Real-time attachment migration status
• ✅ Real-time config updates
• ✅ All pub/sub subscriptions

These optimizations were designed assuming oplog is available. Without it, polling delays reduce their effectiveness.

Troubleshooting

"oplog not available" error

• MongoDB replica set not initialized
• Fix: Run rs.initiate() in MongoDB

High CPU despite oplog

• MONGO_OPLOG_URL not set correctly
• Check oplog size: db.getSiblingDB('local').oplog.rs.stats()
• Ensure minimum 2GB oplog for busy deployments

Slow real-time updates

• Oplog might be full or rolling over
• Increase oplog size (MongoDB Enterprise)
• Check network latency to MongoDB

References

• Meteor Oplog Tuning
• MongoDB Oplog Documentation
• MongoDB Atlas Replica Sets