A generic webhook deployment probe system for the Axion stack. It reads
def_webhook_deployment, fires real HTTP calls per entry using a recent
payload from the source DB, validates the response, and writes results back.
Built to cover LIVE and UAT environments, all packages, with TCP-level ping
before each HTTP probe.
factory.core/
extend.probe/
ObjHookProbeMixin.py — Mixin extracted from ObjHook:
record_metric / get_metrics
_mask_sensitive_headers / safe_debug
_payload_size
log_webhook_request / log_webhook_response
ObjWebhookProbe.py — Main probe object + Typer CLI
class ObjServiceApi(ObjApi.ObjApi)
ObjWebhookAlert.py — Post-probe alert dispatcher + Typer CLI
Reads probe results, fires via MQTT / log
ObjHook.py was updated:
(ObjHookProbeMixin, ObjApi.ObjApi)extend.probe added to sys.pathdef_webhook_deploymentLives on both:
13.245.52.247 hcapi.live (credentials in config.yaml)10.0.10.20 hcapi.live (credentials in config.yaml)CREATE TABLE def_webhook_deployment (
id INT AUTO_INCREMENT PRIMARY KEY,
WebhookCode VARCHAR(50) NOT NULL,
Package VARCHAR(50) NOT NULL DEFAULT 'homechoice',
Environment VARCHAR(10) NOT NULL DEFAULT 'LIVE',
DeployedUrl VARCHAR(255) NOT NULL,
AuthType VARCHAR(20) NOT NULL DEFAULT 'NONE', -- BEARER | NONE
AuthValue VARCHAR(255),
PayloadSql TEXT,
PayloadType VARCHAR(20) NOT NULL DEFAULT 'JSON', -- JSON | PYDICT
PayloadDb VARCHAR(100), -- host for payload fetch (NULL = default DB)
ExpectedFields VARCHAR(500),
ExpectedHttpCode INT NOT NULL DEFAULT 200,
Active CHAR(1) NOT NULL DEFAULT 'Y',
Notes TEXT,
LastTestedAt DATETIME,
LastTestHttpCode INT,
LastTestStatus VARCHAR(20),
LastTestDurationMs INT,
LastTestNote TEXT,
Version VARCHAR(10) NOT NULL DEFAULT '1.0'
);
| id | Code | Package | Env | Auth | PayloadDb |
|---|---|---|---|---|---|
| 1 | HCSCORE | homechoice | LIVE | BEARER | 13.245.52.247 |
| 2 | HCSCORECR | homechoice | LIVE | NONE | 13.245.52.247 |
| 3 | HCSCOREAO | homechoice | LIVE | NONE | 13.245.52.247 |
| 4 | HCSCOREWA | homechoice | LIVE | NONE | 13.245.52.247 |
| 5 | HCSCORE | homechoice | UAT | BEARER | NULL |
| 6 | HCSCORECR | homechoice | UAT | NONE | NULL |
Auth token (HCSCORE): stored in def_webhook_deployment.AuthValue
PayloadSql patterns:
SELECT Payload FROM bloom_hcscore WHERE Payload IS NOT NULL AND Payload != '' ORDER BY CreateTime DESC LIMIT 1SELECT bloom_payload FROM bloom_hcscorecr WHERE bloom_payload IS NOT NULL AND bloom_payload != '' ORDER BY CreateTime DESC LIMIT 1SELECT bloom_payload FROM bloom_hcscoreao WHERE bloom_payload IS NOT NULL ORDER BY CreateTime DESC LIMIT 1SELECT bloom_payload FROM bloom_hcscorewa WHERE bloom_payload IS NOT NULL ORDER BY CreateTime DESC LIMIT 1ExpectedFields per webhook:
Score,ScoreCategory,MaxTerm,CL_Segment,BureauAO_Guid,CreditLimit,MaxTerm,Bureau,CL_SegmentAO_Guid,AO_Score,AO_ScoreDateAO_Guid,CreditLimit,ClientMessageThree steps per deployment row:
TCP Ping — socket.create_connection(host, port, timeout=5)
ERROR: Unreachable: …Payload Fetch — runs PayloadSql via:
pymysql to PayloadDb host if set (needed when probe runssql_get_value() if PayloadDb is NULLast.literal_eval per PayloadTypeNone → SKIP: No payload foundHTTP Probe — requests.post(DeployedUrl, json=payload, headers=…, timeout=30)
uuid4 injected into guid/guidnumber field to bypass dedupExpectedHttpCodeExpectedFields present and non-null in JSON responsePASS | FAIL | ERRORResult dict per row:
{
'id': int,
'WebhookCode': str,
'Package': str,
'Environment': str,
'DeployedUrl': str,
'status': 'PASS' | 'FAIL' | 'ERROR' | 'SKIP',
'http_code': int | None,
'duration_ms': int,
'note': str, # TCP ping RTT prepended on PASS
'response': dict | None,
}
Results are written back to def_webhook_deployment.LastTest* columns.
Python env: /home/vheyn/workspace/projects/axion/dev-env/bin/python3
Working directory: /home/vheyn/workspace/projects/axion
cd /home/vheyn/workspace/projects/axion
# Probe LIVE (current package = homechoice by default)
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py probe --env LIVE
# Probe all environments
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py probe
# Probe specific webhook
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py probe --env LIVE --webhook HCSCORE
# Probe different package
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py probe --package homechoice
# List all configured deployments
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py list
# Show last stored results (no HTTP calls)
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py results --env LIVE
# Enable / disable a deployment by id
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py enable 5
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py disable 5
# Raw JSON output
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookProbe.py probe --env LIVE --json
Last test result (13 Apr 2026): 4/4 PASS, LIVE, all webhooks, ~3–9s each.
When --package is not specified, probe() defaults to self._Package
(resolved from config.yaml → current package context).
On this machine the active config reads HOMECHOICE / AO / DEV.
Override with --package homechoice if the resolved package doesn't match.
When running from Hyperion (or any non-production machine), the default DB
connection (10.0.10.20) does not have the bloom_* tables. PayloadDb
stores the host where the payload SQL should actually run.
For LIVE rows: PayloadDb = 13.245.52.247
For UAT rows: PayloadDb = NULL (UAT DB accessible from the framework connection)
Credentials are resolved from config.yaml per
webhook code, with fallback to a default section:
probe:
databases:
hcscore:
host: 13.245.52.247
user: root
password: secret
database: hcapi.live
hcscorecr:
host: 13.245.52.247
user: root
password: secret
database: hcapi.live
default:
user: axion
password: secret
database: hcapi.live
No hardcoded credentials in source code.
# Probe then alert on failures
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookAlert.py check --env LIVE
# Summary alert (one message instead of one per failure)
dev-env/bin/python3 factory.core/extend.probe/ObjWebhookAlert.py check --env LIVE --summary
Alert channels configured via config.yaml:
webhooks:
alerting:
enabled: true
channels: mqtt,log
MQTT publishes to topic webhook/alerts/{WebhookCode}.
mqtt.technocore.co.za:8886 (TLS)TLSV1_ALERT_INTERNAL_ERROR. Likely IP allowlist on the broker.log_webhook_request / log_webhook_response / MQTT alertObjWebhookProbe — scheduler / workflow:
Param1 = environment (LIVE / UAT / blank = all)
Param2 = WebhookCode (blank = all)
Param3 = Package (blank = current package)
ObjWebhookAlert — scheduler / workflow:
Param1 = environment
Param2 = WebhookCode
# Probes then alerts in a single run_workflow_direct call
ObjWebhookProbe into ObjScheduler for automated periodic runs