ObjMonitor is the Axion platform's system monitoring engine. It collects
hardware, software, network, security, and application-level metrics from
the host machine and persists them to a dedicated monitor.axion database
schema. Collected data feeds the TUI dashboard, daily email reports, MQTT
status publishing, and the ObjAlert threshold engine.
The module contains two classes:
Monitor -- the main collector class; inherits from ObjData forObjMonitorTUI -- a Rich-based live terminal dashboard thatObjData.ObjData
-> ObjMonitorPingMixin (extend.monitor/)
-> ObjMonitorLoginMixin (extend.monitor/)
-> ObjMonitorCommandMixin (extend.monitor/)
-> ObjMonitorSecurityMixin (extend.monitor/)
-> ObjMonitorCollectorMixin (extend.monitor/)
-> ObjMonitorPersistMixin (extend.monitor/)
-> ObjMonitorReportMixin (extend.monitor/)
-> Monitor (factory.core/ObjMonitor.py)
extend.monitor/)| Module | Lines | Methods | Purpose |
|---|---|---|---|
ObjMonitorPing |
1,359 | 17 | Service connectivity checks |
ObjMonitorCollector |
1,785 | 31 | System data collection |
ObjMonitorReport |
1,234 | 10 | HTML health report + alerts |
ObjMonitorPersist |
1,085 | 17 | Database write layer |
ObjMonitorSecurity |
978 | 24 | Security scanning + backups |
ObjMonitorCommand |
651 | 13 | Shell command auditing |
ObjMonitorLogin |
498 | 11 | SSH/login audit trail |
Shared class attributes (_WEEKDAYS, _AUTH_LOG_PATHS,
_SUDO_CMD_RE, _HISTORY_TS_RE, _APT_UPGRADABLE_RE)
are defined on the Monitor base class and available to
all mixins via MRO.
Monitor(db) bootstraps the monitoring schema on first instantiation
(class-level _schema_ensured flag), resolves the local IP and gateway,
calls collect_system() to populate OS attributes, and creates the host
record in the database.
Pass db=None for lightweight/test construction that skips DB and system
bootstrapping.
| Attribute | Description |
|---|---|
_Host |
Lowercase hostname |
_Ip |
Local IP address |
_Globalipgateway |
Default gateway IP |
_Os, _Kernel, _Distribution |
OS identification |
_Hostname |
Raw hostname from uname |
_Gitbranch, _Gitpulldate, _Gitpulltime |
Repository state |
_Cpucores |
Logical CPU count |
_Virtualization |
Virtualisation environment (kvm, wsl, etc.) |
iteration |
Collection cycle counter |
Runs the full set of quick-turnaround collectors. Designed to execute every
few minutes via the micro-service loop. Publishes an MQTT status message
and evaluates alert thresholds at the end.
Collectors called: collect_system, collect_drive, collect_ports,
collect_https, collect_dns, collect_isp, collect_files,
collect_ssh_logins, collect_command_activity, collect_cron_events,
collect_disk_latency, collect_network_connections, collect_open_fds,
collect_swap_pressure, collect_log_anomalies,
collect_supervisor_services, collect_ssh_sessions, collect_sensors,
collect_network_throughput, collect_docker_containers,
collect_connections, collect_wireguard_peers, collect_axion_health,
collect_cpu_hogs, collect_memory_trend, collect_disk_space_trend,
collect_ufw_status, collect_dns_health, collect_ollama.
Runs expensive or infrequent collectors, evaluates thresholds, and sends
the daily email report.
Collectors called: collect_endpoint, collect_pending_updates,
collect_security_scan, collect_apt_changelog, collect_crontab_diff.
A zero-side-effect snapshot returning CPU, memory, and disk percentages.
Used by the TUI for sub-second refresh without blocking.
| Method | Description |
|---|---|
collect_system() |
OS, hostname, kernel, boot time, uptime, distribution, virtualisation, IP. Creates/updates the host DB record and syncs LAN tables. |
collect_os() |
CPU usage, core counts, memory stats, and supervisor service status. |
collect_git() |
Current branch, last pull date/time from .git/FETCH_HEAD. |
collect_sensors() |
Temperature and fan readings via psutil.sensors_temperatures() and psutil.sensors_fans(). Persists to internet_sensor. |
collect_memory_trend() |
RAM/swap usage and OOM kill count from dmesg. |
collect_cpu_hogs() |
Top 10 processes by CPU usage. |
collect_disk_space_trend() |
Per-partition usage with growth estimation (excludes snap, tmpfs, devtmpfs). |
collect_disk_latency() |
Per-device I/O latency via iostat -dx. Persists to internet_disk_latency. |
collect_drive() |
Disk partition usage and I/O counters via psutil. |
| Method | Description |
|---|---|
collect_connections() |
Pings all configured services (databases, RabbitMQ, MongoDB, Redis, InfluxDB, MQTT, SMTP, Keycloak, IMAP, Cloudflare, Docker Registry, Asterisk, AWS S3) and returns a unified status dict. Persists to internet_service. |
collect_network_throughput() |
Per-interface bytes/packets/errors via psutil. |
collect_network_connections() |
Connection state counts (ESTABLISHED, CLOSE_WAIT, etc.) via ObjSecurityLocal. |
collect_ports() |
Open port scan via ObjInternet.Netprobe. |
collect_https() |
SSL certificate details (issuer, serial, validity) for the configured domain. |
collect_dns() |
MX records, SOA records, and nameserver IP for the configured domain. |
collect_dns_health() |
Resolution tests against google.com, github.com, technocore.co.za with latency. |
collect_isp() |
ISP speed test (download, upload, ping) via speedtest-cli. |
collect_wireguard_peers() |
WireGuard peer status from wg show all dump. |
| Method | Description |
|---|---|
collect_supervisor_services() |
Supervisor service status with PID and uptime. Persists to internet_service. |
collect_docker_containers() |
Docker container status and resource usage (CPU%, memory). Persists to internet_service. |
collect_axion_health() |
Running Serve* Python processes with uptime, threads, and restart detection. |
collect_axion_processes() |
Basic PID/memory listing for all Serve* processes. |
collect_endpoint() |
Probes inbound webhook endpoints locally and remotely. Runs in daily mode. |
collect_ollama() |
Ollama model list, running models, and GPU status via nvidia-smi. |
| Method | Description |
|---|---|
collect_security_scan() |
Full security audit: local hardening checks via ObjSecurityLocal (fail2ban, SSH keys, world-writable files, sshd_config, unattended-upgrades) plus remote audit via ObjSecurityAudit (port scan, SSL/TLS, HTTP headers, SSH algorithms, DNS SPF/DKIM/DMARC, database exposure). Persists to internet_security_finding. |
collect_ufw_status() |
UFW firewall active/inactive, rule count, recent block count. |
collect_open_fds() |
System-wide file descriptor usage. Delegates to ObjSecurityLocal. |
collect_swap_pressure() |
Swap usage and page-in/page-out rates. Delegates to ObjSecurityLocal. |
collect_log_anomalies() |
Syslog anomaly scanning. Delegates to ObjSecurityLocal. |
| Method | Description |
|---|---|
collect_ssh_logins() |
Parse auth.log or last for SSH login events. Deduplicates across cycles. Persists via record_login_event(). |
collect_ssh_sessions() |
Active SSH sessions via who. Persists to internet_session. |
collect_command_activity() |
Sudo commands, process accounting (lastcomm), and shell history. Persists via record_command(). |
collect_cron_events() |
Cron executions from syslog. Persists to internet_cron_event. |
collect_apt_changelog() |
Recent dpkg install/upgrade/remove entries from the last 7 days. Persists to internet_package_change. |
collect_crontab_diff() |
Compares current crontabs against the last scan, reports added/removed entries. |
collect_pending_updates() |
Pending apt package updates with security flag. Persists to internet_pending_update. |
| Method | Description |
|---|---|
collect_backup_health() |
Checks def_Backup + stage_backup tables and scans local.documents/backup/ filesystem. Classifies each as HEALTHY, STALE, CRITICAL, FAILED, MISSING, or SUSPECT. |
get_backup_errors() |
Recent backup error rows from stage_backup. |
| Method | Description |
|---|---|
collect_files() |
Tracks size and modification date of files listed in config.yaml under files.file0..file24. Logs size changes to internet_hostfilelog. |
collect_from_proxmox() |
Connects to Proxmox API endpoint for VM host data collection. |
All ping_* methods follow the same pattern: read credentials from
config.yaml, attempt a connection with timeout, and return a dict
with reachable, response_time_ms, error, and service-specific keys.
| Method | Service | Config Section |
|---|---|---|
ping_database() |
MySQL/MariaDB | database |
ping_databases() |
Primary + local DBs | database |
ping_rabbitmq() |
RabbitMQ | rabbitmq |
ping_mongodb() |
MongoDB | mongo |
ping_chromadb() |
ChromaDB | chromadb |
ping_redis() |
Redis | redis |
ping_influxdb() |
InfluxDB | influxdb |
ping_mqtt() |
MQTT (Mosquitto) | mqtt |
ping_smtp() |
SMTP | smtp |
ping_imap() |
IMAP | imap |
ping_keycloak() |
Keycloak OIDC | keycloak |
ping_asterisk() |
Asterisk SIP | asterisk |
ping_docker_registry() |
Docker Registry | docker |
ping_cloudflare() |
Cloudflare DNS (1.1.1.1:53) | N/A |
ping_aws_s3() |
AWS S3 | aws |
check_monitor_thresholds() evaluates collected data and fires alerts
via raise_alert() (which delegates to ObjAlert.alert_track()):
| Alert Name | Condition | Severity |
|---|---|---|
MONITOR_FD_CRITICAL |
FD usage > 90% | CRITICAL |
MONITOR_FD_HIGH |
FD usage > 80% | HIGH |
MONITOR_CLOSE_WAIT |
CLOSE_WAIT connections > 100 | HIGH |
MONITOR_CERT_EXPIRY |
SSL cert expires within 7 days | CRITICAL |
MONITOR_{category} |
Critical log anomaly | CRITICAL |
MONITOR_BACKUP_CRITICAL |
No backup for > 168 hours | CRITICAL |
MONITOR_BACKUP_FAILED |
Last backup status = ERROR | HIGH |
_publish_status_mqtt() runs at the end of every fast/collect cycle.
It publishes a compact JSON summary to topic monitor/{hostname}
containing CPU, memory, swap, disk, connection status, Docker state,
Axion services, alerts, and sensor readings.
send_monitor_report() builds a branded HTML email via ObjTemplate
and sends it through ObjWebMail. The report includes:
ObjAI.prompt() using the local Ollama model.Recipients are looked up from def_host.NotifyEmail. If
def_host.NotifyMqttTopic is set, the full context dict is also
published to that MQTT topic.
All tables live in the monitor.axion schema. Table DDL is defined in
ObjMonitor.yaml under database.schema and auto-created by
_ensure_monitor_schema() on first instantiation.
| Table | Purpose |
|---|---|
Internet_Host |
One row per monitored host. OS, IP, version, ISP details. |
Internet_Hostnames |
Hostname-to-client mapping. |
internet_hostlog |
Time-series CPU/memory samples per host. |
internet_hostisplog |
Time-series ISP speed samples per host. |
Internet_HostFile |
Tracked file sizes and dates. |
Internet_HostFileLog |
File size change history. |
| Table | Purpose |
|---|---|
internet_login_event |
Login/logout events (web, SSH, federated, API). |
internet_command_log |
Command activity (sudo, acct, history). |
internet_cron_event |
Cron job executions with success/fail. |
| Table | Purpose |
|---|---|
internet_pending_update |
Pending apt package updates per host. |
internet_disk_latency |
Per-device I/O latency samples. |
internet_security_finding |
Security scan findings with severity. |
internet_service |
Service status (supervisor, docker, connection, axion, endpoint). |
internet_service_history |
Service state change log. |
internet_session |
Active SSH sessions. |
internet_sensor |
Temperature and fan sensor readings. |
internet_throughput |
Per-interface network I/O counters. |
internet_package_change |
Apt install/upgrade/remove changelog. |
get_uptimerobot_uptime(start, end) queries the UptimeRobot API for
uptime ratios across a date range. Config keys: uptimerobot.api_key
and uptimerobot.monitor_ids. Returns per-monitor uptime percentages,
estimated downtime minutes, and whether the 99.9% SLA target is met.
All commands via typer. For production use, prefer ServeMonitor.py.
| Command | Description |
|---|---|
tui |
Launch the live Rich TUI dashboard (default) |
preflight |
Run full ObjPreflight suite + Proxmox check |
collect <mode> |
Run a collection pass: fast, complete, or daily |
cpu-count |
Display logical and physical CPU counts |
sink |
Start the MQTT collector sink |
process |
List running Python processes |
log |
Scan the access log file |
service |
Run as a micro-service daemon |
check-virt |
Print virtualisation environment |
status |
Print current system status as JSON |
python factory.core/ObjMonitor.py preflight
python factory.core/ObjMonitor.py collect fast
python factory.core/ObjMonitor.py collect daily
python factory.core/ObjMonitor.py tui
python factory.core/ObjMonitor.py status
The recommended production entry point. Starts the micro-service daemon
by default, with subcommands mirroring the ObjMonitor CLI.
python ServeMonitor.py # micro-service daemon
python ServeMonitor.py tui # live TUI dashboard
python ServeMonitor.py preflight # full preflight checks
python ServeMonitor.py collect fast
| Section | Key(s) | Used By |
|---|---|---|
database |
primaryip, primarydb, primaryuser, primarypassword, localip, etc. |
ping_database() |
rabbitmq |
host, port, username, password |
ping_rabbitmq() |
mongo |
host, port, username, password, data |
ping_mongodb() |
chromadb |
host, port |
ping_chromadb() |
redis |
host, port, password |
ping_redis() |
influxdb |
host, port, token, url |
ping_influxdb() |
mqtt |
host, port |
ping_mqtt() |
smtp |
server, port, sender |
ping_smtp(), send_monitor_report() |
imap |
server, port |
ping_imap() |
keycloak |
server, realm |
ping_keycloak() |
asterisk |
host, port |
ping_asterisk() |
docker |
registryhost, registryport |
ping_docker_registry() |
aws |
region |
ping_aws_s3() |
proxmox |
url, user, pass |
collect_from_proxmox(), check_environment() |
uptimerobot |
api_key, monitor_ids |
get_uptimerobot_uptime() |
files |
monitor, file0..file24 |
collect_files() |
{package} |
domain |
collect_https(), collect_dns() |
webhook |
port |
collect_endpoint() |
palette |
name |
TUI theme selection |
| Module | Purpose |
|---|---|
ObjData |
Database access (parent class) |
ObjInternet / ObjInet |
Network probing, LAN table persistence |
ObjHost |
Host and container management |
ObjAlert |
Alert tracking and escalation |
ObjTemplate |
Email HTML branding |
ObjAI |
AI summary generation for daily report |
ObjNotify |
Notification delivery (via ObjAlert) |
ObjWebMail |
Email sending |
ObjMqtt |
MQTT status publishing |
ObjPreflight |
Preflight check suite |
ObjSecurityLocal |
Local security hardening checks |
ObjSecurityAudit |
External vulnerability audit |
psutil, netifaces, cryptography, proxmoxer, speedtest-cli,
dnspython, pika, pymongo, chromadb, redis, influxdb-client,
requests, rich, typer.
ObjMonitorTUI provides a real-time terminal dashboard using Rich Live.
It refreshes fast metrics (CPU, memory, disk) every second and full
connectivity/container/process data every
TuiConstants.MONITOR_FULL_INTERVAL seconds.
Panels: Axion logo, system gauges, connectivity status, Docker
containers, supervisor services, Axion processes, recent logins, and
a heartbeat indicator.
Linux-only features (auth.log, iostat, apt, fail2ban, sshd_config,
lastcomm, wg, ufw) gracefully skip via FileNotFoundError handling.
Core psutil-based collectors work cross-platform.
cythonize -3 -a -i ObjMonitor.py
Compiling /home/axion/projects/axion/factory.core/ObjMonitor.py because it changed..[1/1] Cythonizing /home/axion/projects/axion/factory.core/ObjMonitor.py
| # | Fix | Before | After |
|---|---|---|---|
| 1 | body += → body_parts.append() |
O(n²) report HTML | O(n) with join() |
| 2 | Cache get_package() |
22 calls per cycle | _get_cached_package() |
| 3 | Timeout constants | 23 hardcoded 5 |
MonitorConstants.SOCKET/SUBPROCESS_TIMEOUT_SECS |
| 4 | Health score thresholds | Magic numbers 80/50 |
MonitorConstants.HEALTH_SCORE_HEALTHY/WARNING |
| 5 | Inline SQL → YAML | 2 remaining queries | Zero inline SQL |
| 6 | rows += → list.append() |
37 occurrences | List + join() |
| 7 | Non-blocking CPU sampling | interval=0.2 (200ms) |
interval=None (0ms) |
| 8 | Cache config lookups | 20+ _ini_get per cycle |
Cached in dict |
| 9 | Cache get_queries() |
54 calls per cycle | _get_queries_cached() |
| 10 | 7 mixins extracted | 9,084 lines | 1,849 lines core |
Module reduced from 9,084 → 1,849 lines (80% reduction).
Updated : 2026-04-13