Fault-tolerant Keycloak authentication with automatic fallback and eventual consistency.
ObjKeycloakResilient provides a resilient wrapper around Keycloak authentication that ensures web users can continue working even when Keycloak is unreachable. It implements the circuit breaker pattern, token caching, and a sync queue for eventual consistency.
Key Principle: Local authentication (sys_user) is PRIMARY, Keycloak is an ENHANCEMENT for SSO and centralized management.
┌──────────────────────────────────────────────────────────┐
│ AUTHENTICATION LAYERS │
├──────────────────────────────────────────────────────────┤
│ │
│ Layer 1: LOCAL AUTH (sys_user) ──────► ALWAYS WORKS │
│ ↓ PRIMARY authentication │
│ ↓ Never depends on external services │
│ │
│ Layer 2: KEYCLOAK ENHANCEMENT ───────► BEST EFFORT │
│ ↓ SSO across applications │
│ ↓ Centralized user management │
│ │
│ Layer 3: TOKEN CACHE ─────────────────► GRACE PERIOD │
│ ↓ Cached Keycloak tokens │
│ ↓ Valid for 24 hours │
│ │
│ Layer 4: SYNC QUEUE ──────────────────► EVENTUAL SYNC │
│ ↓ Queue password changes │
│ ↓ Sync when Keycloak available │
│ │
└──────────────────────────────────────────────────────────┘
Automatically detects when Keycloak is unavailable and stops trying:
from ObjKeycloakResilient import ObjKeycloakResilient
client = ObjKeycloakResilient()
# Circuit breaker automatically manages availability
if client.is_available():
token = client.authenticate(username, password, package)
else:
# Use cached token or local auth only
cached = client.get_cached_token(username, package)
Caches successful Keycloak tokens for offline use:
# Token automatically cached on successful auth
token = client.authenticate("john.doe", "password", "homechoice")
# Later, when Keycloak is down
cached_token = client.get_cached_token("john.doe", "homechoice")
if cached_token:
# Can still authenticate user with cached token
print("Using cached token from:", cached_token['cached_at'])
Queues changes when Keycloak is unavailable for later synchronization:
# Queue password change when Keycloak is down
client.queue_sync(
username="john.doe",
package="homechoice",
sync_type="password_change",
sync_data={"new_password": "new_secure_pass"},
priority=1 # High priority
)
# Background service will sync when Keycloak available
Regular health checks with caching to avoid hammering Keycloak:
is_available()# Get detailed status
status = client.get_status()
print(f"Available: {status['available']}")
print(f"Circuit: {status['circuit_state']}")
print(f"Mode: {status['mode']}") # normal, degraded, offline
print(f"Pending syncs: {status['pending_syncs']}")
print(f"Cached tokens: {status['cached_tokens']}")
Configuration in config.yaml:
keycloak:
# ... existing Keycloak config ...
# Resilience configuration
resilience:
enabled: true
failure_threshold: 3 # Open circuit after 3 failures
cooldown_seconds: 60 # Wait 60s before retry
token_grace_period_hours: 24 # Cache tokens 24h
allow_offline_mode: true # Allow local auth when down
health_check_cache_seconds: 30 # Cache health check
High Availability (Conservative):
resilience:
failure_threshold: 1 # Fast fallback
cooldown_seconds: 30 # Quick retry
token_grace_period_hours: 48 # Long grace
Balanced (Recommended):
resilience:
failure_threshold: 3
cooldown_seconds: 60
token_grace_period_hours: 24
Keycloak-First (SSO Priority):
resilience:
failure_threshold: 5 # Try harder
cooldown_seconds: 120 # Longer cooldown
token_grace_period_hours: 12 # Shorter grace
Stores cached Keycloak tokens:
CREATE TABLE sys_keycloak_token_cache (
User varchar(150) NOT NULL,
Package varchar(100) NOT NULL,
AccessToken text NOT NULL,
RefreshToken text NULL,
ExpiresAt datetime NOT NULL,
CachedAt datetime NOT NULL,
LastUsed datetime NULL,
UseCount int NOT NULL DEFAULT 0,
PRIMARY KEY (User, Package)
);
Queues sync operations:
CREATE TABLE sys_keycloak_sync_queue (
SyncId char(36) NOT NULL PRIMARY KEY,
User varchar(150) NOT NULL,
Package varchar(100) NOT NULL,
SyncType varchar(50) NOT NULL,
SyncData text NULL,
QueuedAt datetime NOT NULL,
ProcessedAt datetime NULL,
Status varchar(20) NOT NULL DEFAULT 'pending',
Attempts int NOT NULL DEFAULT 0,
LastError text NULL,
Priority int NOT NULL DEFAULT 5
);
from ObjKeycloakResilient import ObjKeycloakResilient
client = ObjKeycloakResilient()
# Authenticate (works even if Keycloak down via cache)
token = client.authenticate("john.doe", "password", "homechoice")
if token:
if token.get('cached'):
print("Using cached token (Keycloak unavailable)")
else:
print("Authenticated with Keycloak")
else:
print("Authentication failed")
# When Keycloak is down, queue the change
if not client.is_available():
client.queue_sync(
username="john.doe",
package="homechoice",
sync_type="password_change",
sync_data={"new_password": "new_pass"},
priority=1
)
print("Password change queued for later sync")
# Manually process pending syncs
if client.is_available():
processed = client.process_sync_queue(batch_size=100)
print(f"Processed {processed} sync items")
# Remove expired tokens (>48 hours)
deleted_tokens = client.cleanup_expired_tokens(hours=48)
print(f"Removed {deleted_tokens} expired tokens")
# Remove completed syncs (>7 days)
deleted_syncs = client.cleanup_completed_syncs(days=7)
print(f"Removed {deleted_syncs} completed syncs")
Run the background service to continuously process sync queue:
# Start sync service
python factory.core/ObjKeycloakSyncService.py start
# Check status
python factory.core/ObjKeycloakSyncService.py status
# Test configuration
python factory.core/ObjKeycloakSyncService.py test
# Process queue now (one cycle)
python factory.core/ObjKeycloakSyncService.py sync-now
# Run cleanup
python factory.core/ObjKeycloakSyncService.py cleanup
[2026-02-07 10:00:00] Keycloak Sync Service starting...
Version: 1.0.0
Sync interval: 60s
Batch size: 100
Cleanup interval: 3600s
Current Status:
Keycloak: Available
Circuit: CLOSED
Mode: NORMAL
Pending syncs: 5
Cached tokens: 12
[2026-02-07 10:01:00] Processed 5 sync items (total: 5)
[2026-02-07 11:00:00] Running periodic cleanup...
Removed 3 expired tokens
Removed 10 completed syncs
| State | Keycloak | Circuit | Token Cache | User Experience |
|---|---|---|---|---|
| NORMAL | ✅ Online | CLOSED | Fresh | Full SSO, immediate sync |
| DEGRADED | ⚠️ Offline | OPEN | Valid | SSO with cached tokens |
| OFFLINE | ❌ Offline | OPEN | Expired | Local auth only |
In ObjUser.py, use ObjKeycloakResilient for authentication:
from ObjKeycloakResilient import ObjKeycloakResilient
class User(ObjData):
def __init__(self):
super().__init__()
self._keycloak = ObjKeycloakResilient()
def Login(self, username, password, package):
# TIER 1: Local authentication (PRIMARY)
local_auth = self._authenticate_local(username, password, package)
if not local_auth:
return {'success': False, 'reason': 'invalid_credentials'}
# TIER 2: Keycloak enhancement (OPTIONAL)
auth_mode = 'local'
keycloak_token = None
if self._keycloak.is_available():
keycloak_token = self._keycloak.authenticate(
username, password, package
)
if keycloak_token:
auth_mode = 'keycloak'
else:
# Try cached token
keycloak_token = self._keycloak.get_cached_token(
username, package
)
if keycloak_token:
auth_mode = 'cached'
# TIER 3: Create session (always succeeds)
session_id = self._create_session(username, package, auth_mode)
return {
'success': True,
'session_id': session_id,
'auth_mode': auth_mode,
'keycloak_status': 'online' if self._keycloak.is_available()
else 'offline'
}
def ChangePassword(self, username, old_password, new_password, package):
# Update local password (ALWAYS)
self._update_local_password(username, new_password)
# Update Keycloak (BEST EFFORT)
if self._keycloak.is_available():
try:
self._keycloak.authenticate(username, new_password, package)
mode = 'synchronized'
except Exception:
# Queue for later
self._keycloak.queue_sync(
username, package, 'password_change',
{'new_password': new_password}, priority=1
)
mode = 'queued'
else:
# Queue for later
self._keycloak.queue_sync(
username, package, 'password_change',
{'new_password': new_password}, priority=1
)
mode = 'queued'
return {'success': True, 'mode': mode}
status = client.get_status()
metrics = {
'keycloak_available': status['available'],
'circuit_state': status['circuit_state'],
'active_sessions': 142,
'cached_tokens': status['cached_tokens'],
'pending_syncs': status['pending_syncs'],
'auth_modes_last_hour': {
'keycloak': 95,
'local': 12,
'cached': 8
}
}
Test the resilient client:
# Run tests
python factory.core/ObjKeycloakResilient.py
# Output shows current status
Keycloak Resilience Status:
============================================================
available: True
circuit_state: closed
failure_count: 0
mode: normal
pending_syncs: 0
cached_tokens: 0
config:
failure_threshold: 3
cooldown_seconds: 60
grace_period_hours: 24
offline_mode_allowed: True
============================================================
factory.core/ObjKeycloak.py - Base Keycloak clientfactory.core/ObjKeycloakSyncService.py - Background sync servicefactory.web/ObjUser.py - User management with dual authenticationlocal.processing/schema/package.sync/tables/sys_keycloak_token_cache.yamllocal.processing/schema/package.sync/tables/sys_keycloak_sync_queue.yamlresource.notes/objuser_keycloak_resilience_design.md - Detailed designresource.notes/objuser_keycloak_quick_implementation.md - Implementation guideresource.notes/KEYCLOAK_INTEGRATION_SUMMARY.md - High-level overview