Continuous background service for processing Keycloak sync queue and maintaining eventual consistency.
ObjKeycloakSyncService is a long-running background service that continuously monitors and processes the Keycloak sync queue. When Keycloak becomes available after an outage, it automatically syncs all queued changes (password changes, user updates, etc.) to maintain eventual consistency.
When Keycloak is unavailable:
sys_keycloak_sync_queueWhen Keycloak recovers:
# Run in foreground
python factory.core/ObjKeycloakSyncService.py start
# Or run as systemd service (recommended for production)
sudo systemctl start keycloak-sync
# Show current Keycloak and queue status
python factory.core/ObjKeycloakSyncService.py status
Output:
Keycloak Resilience Status
======================================================================
Connection:
Available: True
Circuit State: CLOSED
Mode: NORMAL
Failure Count: 0
Timestamps:
Last Success: 2026-02-07T10:30:45
Last Health Check: 2026-02-07T10:31:00
Queue Metrics:
Pending Syncs: 5
Cached Tokens: 12
Configuration:
Failure Threshold: 3
Cooldown: 60s
Grace Period: 24h
Offline Mode: True
======================================================================
# Test Keycloak connection and database tables
python factory.core/ObjKeycloakSyncService.py test
Output:
Testing Keycloak resilience configuration...
1. Configuration:
failure_threshold: 3
cooldown_seconds: 60
grace_period_hours: 24
offline_mode_allowed: True
health_check_cache_seconds: 30
2. Availability Test:
Keycloak available: True
Circuit state: closed
3. Database Tables:
Token cache entries: 12
Sync queue entries: 15
Pending syncs: 5
Test complete.
# Process sync queue once (useful for manual intervention)
python factory.core/ObjKeycloakSyncService.py sync-now
Output:
Processing sync queue...
Processed 5 sync items
# Clean up expired tokens and completed syncs
python factory.core/ObjKeycloakSyncService.py cleanup
Output:
Running cleanup...
Removed 8 expired tokens (>48h)
Removed 23 completed syncs (>7d)
Cleanup complete.
When running start, the service provides continuous feedback:
[2026-02-07 10:00:00] Keycloak Sync Service starting...
Version: 1.0.0
Sync interval: 60s
Batch size: 100
Cleanup interval: 3600s
Current Status:
Keycloak: Available
Circuit: CLOSED
Mode: NORMAL
Pending syncs: 5
Cached tokens: 12
Last success: 2026-02-07T09:58:32
[2026-02-07 10:01:00] Processed 5 sync items (total: 5)
[2026-02-07 10:02:00] Keycloak unavailable, skipping sync cycle
[2026-02-07 10:03:00] Keycloak unavailable, skipping sync cycle
[2026-02-07 10:04:00] Processed 3 sync items (total: 8)
[2026-02-07 11:00:00] Running periodic cleanup...
Removed 2 expired tokens
Removed 10 completed syncs
Service configuration in factory.core/ObjKeycloakSyncService.py:
# Configurable parameters
self._sync_interval_seconds = 60 # Check every 60 seconds
self._cleanup_interval_seconds = 3600 # Cleanup every hour
self._batch_size = 100 # Process max 100 items per cycle
To adjust:
ObjKeycloakSyncService.py__init__ methodRecommended Settings:
| Environment | Sync Interval | Batch Size | Cleanup Interval |
|---|---|---|---|
| Development | 30s | 50 | 1800s (30m) |
| Production | 60s | 100 | 3600s (1h) |
| High-Volume | 30s | 200 | 7200s (2h) |
For production, run as systemd service for automatic restart and logging.
/etc/systemd/system/keycloak-sync.service:
[Unit]
Description=Keycloak Sync Service
After=network.target mariadb.service
[Service]
Type=simple
User=axion
Group=axion
WorkingDirectory=/home/axion/Axion
Environment="AXION_PACKAGE=homechoice"
ExecStart=/home/axion/Axion/dev-env/bin/python \
factory.core/ObjKeycloakSyncService.py start
Restart=always
RestartSec=10
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=keycloak-sync
[Install]
WantedBy=multi-user.target
# Reload systemd
sudo systemctl daemon-reload
# Enable service (start on boot)
sudo systemctl enable keycloak-sync
# Start service
sudo systemctl start keycloak-sync
# Check status
sudo systemctl status keycloak-sync
# View logs
sudo journalctl -u keycloak-sync -f
┌─────────────────────────────────────────────────────────┐
│ SYNC SERVICE LIFECYCLE │
└─────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────┐
│ Service Starts │
│ - Load configuration │
│ - Initialize client │
│ - Print status │
└────────────┬───────────┘
│
▼
┌────────────────────────┐
│ Main Loop (60s cycle) │
└────────────┬───────────┘
│
┌───────────────┴───────────────┐
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Check Keycloak │ │ Periodic Cleanup │
│ Availability │ │ (every 1 hour) │
└────────┬─────────┘ └────────┬─────────┘
│ │
├─ Available ─────────┐ ▼
│ │ ┌──────────────┐
└─ Unavailable ───────┤ │ Clean tokens │
│ │ Clean syncs │
│ └──────────────┘
│
▼
┌─────────────────────┐
│ Process Sync Queue │
│ - Get pending items │
│ - Process up to 100 │
│ - Mark completed │
└──────────┬──────────┘
│
▼
┌─────────────┐
│ Sleep 60s │
└──────┬──────┘
│
└────► Loop
The service processes 4 types of sync operations:
# Queued when password changed during Keycloak outage
{
'sync_type': 'password_change',
'sync_data': {'new_password': '~encrypted_pass'},
'priority': 1
}
Processing: Updates user password in Keycloak using admin API.
# Queued when new user created during outage
{
'sync_type': 'user_create',
'sync_data': {
'username': 'john.doe',
'email': 'john@example.com',
'name': 'John Doe'
},
'priority': 3
}
Processing: Creates user in Keycloak realm.
# Queued when user profile updated
{
'sync_type': 'user_update',
'sync_data': {
'email': 'newemail@example.com',
'name': 'John Doe Updated'
},
'priority': 5
}
Processing: Updates user attributes in Keycloak.
# Queued when user deleted/deactivated
{
'sync_type': 'user_delete',
'sync_data': {'reason': 'User requested deletion'},
'priority': 10
}
Processing: Deletes or disables user in Keycloak.
failed after 5 attempts-- Retry failed syncs (reset to pending)
UPDATE sys_keycloak_sync_queue
SET Status = 'pending',
Attempts = 0,
LastError = NULL
WHERE Status = 'failed'
AND ProcessedAt < DATE_SUB(NOW(), INTERVAL 1 HOUR)
AND Attempts < 5;
-- View failed syncs
SELECT User, Package, SyncType, LastError, Attempts
FROM sys_keycloak_sync_queue
WHERE Status = 'failed'
ORDER BY QueuedAt DESC;
Frequency: Every 1 hour
Threshold: Remove tokens older than 48 hours
tokens_deleted = client.cleanup_expired_tokens(hours=48)
Why: Cached tokens beyond grace period are no longer useful and waste space.
Frequency: Every 1 hour
Threshold: Remove completed syncs older than 7 days
syncs_deleted = client.cleanup_completed_syncs(days=7)
Why: Completed syncs kept for audit trail but cleaned up after 7 days.
# View service logs
sudo journalctl -u keycloak-sync -f
# Count syncs processed today
sudo journalctl -u keycloak-sync --since today | grep "Processed" | wc -l
# Find errors
sudo journalctl -u keycloak-sync --since today | grep -i error
Set up alerts for:
| Condition | Threshold | Action |
|---|---|---|
| Pending syncs | > 100 | Investigate Keycloak availability |
| Failed syncs | > 10 | Review error logs |
| Circuit OPEN | > 5 minutes | Check Keycloak service |
| Service down | Any | Restart service |
# Check if already running
ps aux | grep ObjKeycloakSyncService
# Check database connectivity
python factory.core/ObjData.py preflight
# Check Keycloak config
python factory.core/ObjKeycloakSyncService.py test
# Check Keycloak availability
python factory.core/ObjKeycloakSyncService.py status
# Manually process queue
python factory.core/ObjKeycloakSyncService.py sync-now
# Check for failed syncs
mysql -u root -p axion -e "SELECT * FROM sys_keycloak_sync_queue WHERE Status='failed'"
# Reduce batch size in ObjKeycloakSyncService.py
self._batch_size = 50 # Instead of 100
# Increase cleanup frequency
self._cleanup_interval_seconds = 1800 # 30 minutes instead of 1 hour
ObjUser.py queues syncs when Keycloak unavailable:
from ObjKeycloakResilient import ObjKeycloakResilient
class User(ObjData):
def __init__(self):
super().__init__()
self._keycloak = ObjKeycloakResilient()
def ChangePassword(self, username, old_password, new_password, package):
# Update local password first
self._update_local_password(username, new_password)
# Queue sync if Keycloak unavailable
if not self._keycloak.is_available():
self._keycloak.queue_sync(
username, package, 'password_change',
{'new_password': new_password}, priority=1
)
return {'success': True, 'mode': 'queued'}
# Otherwise sync immediately
# ...
Add to ObjMonitor.py:
def collect_keycloak_sync_metrics(self):
"""Collect Keycloak sync service metrics."""
from ObjKeycloakResilient import ObjKeycloakResilient
client = ObjKeycloakResilient()
status = client.get_status()
metrics = {
'keycloak_available': status['available'],
'circuit_state': status['circuit_state'],
'pending_syncs': status.get('pending_syncs', 0),
'cached_tokens': status.get('cached_tokens', 0)
}
self.log_metric('keycloak_resilience', metrics)
Ensure indexes exist for performance:
-- Check indexes on sync queue
SHOW INDEX FROM sys_keycloak_sync_queue;
-- Should have indexes on:
-- - Status, QueuedAt (for getting pending items)
-- - Priority, QueuedAt (for priority ordering)
-- - User, Package (for user-specific queries)
factory.core/ObjKeycloakResilient.py - Resilient Keycloak clientfactory.core/ObjKeycloak.py - Base Keycloak clientfactory.web/ObjUser.py - User managementlocal.processing/schema/package.sync/tables/sys_keycloak_sync_queue.yaml