Real-time terminal dashboard for monitoring the Axion scheduler service using the Rich library.
The Scheduler TUI provides a comprehensive, real-time view of the scheduler's operation including:
Current Status:
Lifetime Statistics:
Configuration:
Shows currently executing tasks:
Displays up to 10 most recent running tasks.
Shows tasks waiting for execution:
Displays up to 8 queued tasks.
Shows chronological event history:
Displays last 8 events.
# Start with TUI (default workers = CPU count)
python factory.core/ObjScheduler.py service --tui
# Or use short flag
python factory.core/ObjScheduler.py service -t
# Specify number of workers
python factory.core/ObjScheduler.py service --tui --workers 4
# Short flags
python factory.core/ObjScheduler.py service -t -w 4
# Run without TUI (logs to stdout)
python factory.core/ObjScheduler.py service
# With custom worker count
python factory.core/ObjScheduler.py service --workers 8
Press Ctrl+C to gracefully shutdown the scheduler and TUI.
┌──────────────────┐
│ Worker Process │
│ │
│ Task Execution │
└────────┬─────────┘
│
│ Put event
▼
┌──────────────┐
│ Event Queue │
└──────┬───────┘
│
│ Get event
▼
┌─────────────┐
│ Main Process│
│ │
│ Process TUI │
│ Events │
└──────┬──────┘
│
│ record_event()
▼
┌─────────┐
│ TUI │
│Dashboard│
└─────────┘
Using multiprocessing.Manager():
running_tasks: Dict[GUID, TaskInfo] - Currently executing tasksqueued_tasks: Dict[GUID, QueueInfo] - Queued taskstask_queue: Queue - Tasks waiting for workersevent_queue: Queue - Events for TUI displayclass SchedulerTUI:
def __init__(
self,
running_tasks: Dict,
queued_tasks: Dict,
num_workers: int
):
"""Initialize scheduler TUI dashboard."""
def start(self) -> None:
"""Start the TUI in live mode."""
def stop(self) -> None:
"""Stop the TUI and cleanup."""
def log(self, message: str, level: str = "INFO") -> None:
"""
Add a log message.
Args:
message: Log message content
level: INFO, WARN, ERROR, SUCCESS
"""
def set_status(self, message: str) -> None:
"""Update status message in footer."""
def record_event(
self,
event_type: str,
guid: str,
message: str,
details: Optional[Dict] = None
) -> None:
"""
Record a task event.
Args:
event_type: START, COMPLETE, FAILED, RETRY
guid: Task GUID
message: Event message
details: Additional event information
"""
START: Task execution began
{
"type": "START",
"guid": "ABC123",
"message": "Task started",
"details": {"package": "homechoice"}
}
COMPLETE: Task finished successfully
{
"type": "COMPLETE",
"guid": "ABC123",
"message": "Task completed successfully",
"details": {"package": "homechoice"}
}
RETRY: Task failed, will retry
{
"type": "RETRY",
"guid": "ABC123",
"message": "Retry 2/3 after 4s",
"details": {
"retry_count": 2,
"backoff_seconds": 4,
"error": "Connection timeout"
}
}
FAILED: Task failed after max retries
{
"type": "FAILED",
"guid": "ABC123",
"message": "Failed after 3 retries",
"details": {"error": "Database connection failed"}
}
Error: TUI dependencies not available
Solution: Install Rich library
pip install rich
Problem: Garbled output or overlapping text
Solutions:
Problem: TUI updates slowly or freezes
Solutions:
Problem: Ctrl+C doesn't stop the service
Solution:
kill %1pkill -f ObjScheduler$ python factory.core/ObjScheduler.py service --tui --workers 4
⚙️ Axion Scheduler Dashboard | Workers: 4 | Uptime: 00:15:32
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┌─────────────────────────────────┐┌──────────────────────────────────────┐
│ 📊 Statistics ││ 🔄 Running Tasks (2) │
│ ││┌──────────┬─────────┬──────┬─────────┐
│ 📊 CURRENT STATUS ││GUID │Package │Start │Duration │
│ ────────────────────────────────││ABC12345 │core │14:20 │00:00:15 │
│ 🟢 Running Tasks 2 ││DEF67890 │homech.. │14:19 │00:01:23 │
│ ⏳ Queued Tasks 3 │└──────────┴─────────┴──────┴─────────┘
│ 👷 Active Workers 4 │
│ │┌──────────────────────────────────────┐
│ 📈 LIFETIME STATS ││ ⏳ Queued Tasks (3) │
│ ────────────────────────────────││┌──────────┬───────┬────────┬────────┐
│ ✅ Completed 1,234││GUID │Retries│Queued │Wait │
│ ❌ Failed 12││GHI11111 │0/3 │14:21 │00:00:05│
│ 🔄 Retries 45││JKL22222 │1/3 │14:20 │00:00:42│
│ 📊 Success Rate 99.0%││MNO33333 │0/3 │14:21 │00:00:03│
│ │└──────────┴───────┴────────┴────────┘
│ ⚙️ CONFIGURATION │
│ ────────────────────────────────│┌──────────────────────────────────────┐
│ Max Retries 3 ││ 📜 Recent Events │
│ Backoff Base 2s ││┌──────┬────────┬──────────┬─────────┐
│ Poll Interval 10s ││Time │Event │GUID │Message │
└─────────────────────────────────┘│14:21 │✅ DONE │ABC12345 │Completed│
││14:20 │🔄 RETRY│JKL22222 │Retry 2/3│
││14:20 │▶️ START│MNO33333 │Started │
│└──────┴────────┴──────────┴─────────┘
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Status: Tick at 14:21:15 - Queued: 3, Running: 2
[14:21:10] [INFO] Task ABC12345 completed successfully
[14:21:12] [WARN] Task JKL22222 retry 2/3 after 4s
[14:21:15] [INFO] Scheduler tick
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━