Date: 2026-02-08
Status: ✅ Completed
Version: 1.0.0
Phase 2 of webhook enhancements focuses on reliability, developer productivity, and operational tools: credential migration automation, retry logic for transient failures, and local testing environment.
Effort: LOW | Impact: MEDIUM | ROI: ⭐⭐⭐
Automated tool to migrate webhook credentials from database to Infisical with safety features.
Benefits:
Implementation:
resource.bin/migrate_webhooks_to_infisical.py (800+ lines)Features:
Usage:
# Dry run (no changes, safe to test)
python resource.bin/migrate_webhooks_to_infisical.py migrate --dry-run
# Migrate all outbound webhooks
python resource.bin/migrate_webhooks_to_infisical.py migrate
# Migrate specific webhook
python resource.bin/migrate_webhooks_to_infisical.py migrate --webhook STRIPE_PAYMENT
# Migrate by pattern
python resource.bin/migrate_webhooks_to_infisical.py migrate --pattern "STRIPE_*"
# Migrate inbound webhooks
python resource.bin/migrate_webhooks_to_infisical.py migrate --direction IN
# Verify migration status
python resource.bin/migrate_webhooks_to_infisical.py verify
# Check migration status for all webhooks
python resource.bin/migrate_webhooks_to_infisical.py status
# List backups
python resource.bin/migrate_webhooks_to_infisical.py list-backups
# Rollback from backup
python resource.bin/migrate_webhooks_to_infisical.py rollback backup_OUT_2026-02-08_143000.json
Output Examples:
Webhook Credential Migration Tool
Found 25 webhook(s) to migrate
Backup created: data.config/webhook_backups/backup_OUT_2026-02-08_143000.json
Migrating webhooks...
[STRIPE_PAYMENT] Migrated successfully
[GITHUB_WEBHOOK] Skipped - already in Infisical
[LEGACY_HOOK] Migrated successfully
Migration Summary:
Total: 25
Migrated: 20
Skipped: 3
Failed: 2
Verifying migration...
[STRIPE_PAYMENT] Verified successfully
[LEGACY_HOOK] Verified successfully
All migrations verified successfully!
Migration Status Output:
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Webhook Code ┃ DB Creds ┃ Infisical ┃ Status ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ STRIPE_PAYMENT │ user,key │ ✓ │ Migrated │
│ GITHUB_WEBHOOK │ user,key │ ✓ │ Migrated │
│ LEGACY_HOOK │ user,pass│ ✗ │ Pending │
└─────────────────┴──────────┴───────────┴───────────┘
Safety Features:
Effort: LOW | Impact: HIGH | ROI: ⭐⭐⭐
Automatic retry mechanism for webhook failures with exponential backoff delays.
Benefits:
Implementation:
factory.core/ObjHook.py (+200 lines)_execute_with_retry()_execute_post_request(), _execute_get_request(), _execute_put_request()Retry Behavior:
Retries ON:
NO Retries on:
Configuration:
# config.yaml
webhooks:
retry:
max_retries: 3 # Maximum retry attempts (default: 3)
base_delay_seconds: 2.0 # Base delay for exponential backoff (default: 2.0)
Exponential Backoff Pattern:
Attempt 1: Immediate
Attempt 2: Wait 2 seconds (base_delay * 2^0)
Attempt 3: Wait 4 seconds (base_delay * 2^1)
Attempt 4: Wait 8 seconds (base_delay * 2^2)
Example Scenarios:
Scenario 1: Server Error (503) - Retries
[08:10:00] POST request to https://api.example.com
[08:10:01] Server error 503, retrying in 2s (attempt 1/3)
[08:10:03] POST request to https://api.example.com
[08:10:04] Server error 503, retrying in 4s (attempt 2/3)
[08:10:08] POST request to https://api.example.com
[08:10:09] Response: 200 OK ✓
Scenario 2: Timeout - Retries
[08:10:00] POST request to https://api.example.com
[08:10:10] Network error: Timeout, retrying in 2s (attempt 1/3)
[08:10:12] POST request to https://api.example.com
[08:10:13] Response: 200 OK ✓
Scenario 3: Auth Error (401) - No Retry
[08:10:00] POST request to https://api.example.com
[08:10:01] Response: 401 Unauthorized
[08:10:01] No retry - client error
Code Example:
# In factory.core/ObjHook.py
def _execute_with_retry(
self,
http_method: str,
base_url: str,
max_retries: int = 3,
base_delay: float = 2.0,
**request_kwargs
):
"""
Execute HTTP request with exponential backoff retry logic.
Retries on: 5xx errors, timeouts, connection errors
No retry on: 4xx client errors
"""
for attempt in range(max_retries + 1):
try:
response = requests.{method}(base_url, **request_kwargs)
# Retry on 5xx errors
if response.status_code >= 500:
if attempt < max_retries:
delay = base_delay * (2 ** attempt)
time.sleep(delay)
continue
return response
except (Timeout, ConnectionError) as e:
if attempt < max_retries:
delay = base_delay * (2 ** attempt)
time.sleep(delay)
else:
raise
Logging Output:
[08:10:00] POST request to https://api.stripe.com (max_retries=3)
[08:10:05] Server error 502, retrying in 2.0s (attempt 1/3)
[08:10:07] Server error 502, retrying in 4.0s (attempt 2/3)
[08:10:11] Response: 200 OK
Effort: LOW | Impact: MEDIUM | ROI: ⭐⭐⭐
Local HTTP server for testing webhooks without hitting real endpoints.
Benefits:
Implementation:
resource.test/webhook_sandbox.py (500+ lines)Features:
Usage:
# Start sandbox server (default port 8888)
python resource.test/webhook_sandbox.py start
# Start on custom port
python resource.test/webhook_sandbox.py start --port 8080
# Inject 20% error rate (test retry logic)
python resource.test/webhook_sandbox.py start --error-rate 0.2
# Simulate slow endpoint (3 second delay)
python resource.test/webhook_sandbox.py start --delay 3.0
# Combine error injection and delay
python resource.test/webhook_sandbox.py start --error-rate 0.1 --delay 1.5
# List recorded requests from last session
python resource.test/webhook_sandbox.py list-requests
# Replay specific request
python resource.test/webhook_sandbox.py replay POST_1707389523456
# Replay to different endpoint
python resource.test/webhook_sandbox.py replay POST_1707389523456 --url https://real-api.com/webhook
# Configure custom endpoint response
python resource.test/webhook_sandbox.py configure-endpoint /webhook/test --status 201 --file response.json
Console Output:
Webhook Sandbox Started
Server: http://localhost:8888
Error rate: 0.0%
Delay: 0.0s
Available endpoints:
/webhook/* - Any webhook path
/test - Test endpoint
/api/* - API simulation
Press Ctrl+C to stop
POST /webhook/payment
Request ID: POST_1707389523456
Auth: Bearer sk_test_12345...
Body: {
"amount": 1000,
"currency": "USD"
}
Response: 200 OK
Request Recording Output:
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Request ID ┃ Timestamp ┃ Method ┃ Path ┃ Body Size ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ POST_1707389523456│ 2026-02-08 14:30:00│ POST │ /webhook/pay │ 1024 │
│ GET_1707389530123 │ 2026-02-08 14:31:00│ GET │ /api/status │ 0 │
└───────────────────┴────────────────────┴────────┴───────────────┴───────────┘
Testing Workflow:
Start Sandbox:
python resource.test/webhook_sandbox.py start --port 8888
Configure Webhook to Use Sandbox:
# In database or config
UPDATE def_webhook
SET Baseurl = 'http://localhost:8888/webhook/test'
WHERE Webhookcode = 'TEST_WEBHOOK'
Trigger Webhook:
from ObjHook import ObjHook
hook = ObjHook()
hook.read('TEST_WEBHOOK')
result = hook.call('DONE')
Inspect Request - Check sandbox console output
Test Error Scenarios:
# Restart with error injection
python resource.test/webhook_sandbox.py start --error-rate 0.5
Replay to Real Endpoint (after testing):
python resource.test/webhook_sandbox.py replay POST_1707389523456 \
--url https://api.stripe.com/v1/charges
Integration with ObjHook:
# Testing webhook locally
hook = ObjHook()
hook.read('STRIPE_PAYMENT')
# Override URL for testing
hook._Baseurl = 'http://localhost:8888/webhook/stripe'
# Call webhook (will go to sandbox)
result = hook.call('DONE')
# Check sandbox console for request details
Custom Response Example:
// response.json
{
"id": "ch_test_123",
"status": "succeeded",
"amount": 1000
}
python resource.test/webhook_sandbox.py configure-endpoint \
/webhook/payment --status 201 --file response.json
Total: 15 tests, all passing ✅
Test Categories:
Retry Logic (8 tests)
Retry Configuration (2 tests)
Edge Cases (3 tests)
Sandbox Integration (2 tests)
Run Tests:
# Run Phase 2 tests
pytest resource.test/pytests/factory.core/test_ObjHook_phase2.py -v
# Expected output: 15/15 passing
Before:
After:
Performance Characteristics:
| Scenario | Before | After | Improvement |
|---|---|---|---|
| 503 error (1 retry) | Failure | Success in 2s | 100% success |
| Timeout (2 retries) | Failure | Success in 6s | 100% success |
| 401 error | Failure | Fail immediately | No change (correct) |
| Success (no retry) | 100ms | 100ms | No overhead |
Delay Pattern (base_delay=2s):
# Webhook Retry Configuration
webhooks:
retry:
max_retries: 3 # Maximum retry attempts
base_delay_seconds: 2.0 # Base delay for exponential backoff
# Phase 1 features (still active)
alerting:
enabled: true
failure_rate_threshold: 10.0
auth_failure_threshold: 5
latency_threshold_seconds: 5.0
window_minutes: 5
channels: slack,mqtt,log
structured_logging:
enabled: true
mask_sensitive: true
mqtt_enabled: true
# 1. Review current status
python resource.bin/migrate_webhooks_to_infisical.py status
# 2. Dry run to see what would be migrated
python resource.bin/migrate_webhooks_to_infisical.py migrate --dry-run
# 3. Migrate all outbound webhooks
python resource.bin/migrate_webhooks_to_infisical.py migrate
# 4. Verify migration
python resource.bin/migrate_webhooks_to_infisical.py verify
# 5. Repeat for inbound webhooks
python resource.bin/migrate_webhooks_to_infisical.py migrate --direction IN
# 6. Check backups (in case rollback needed)
python resource.bin/migrate_webhooks_to_infisical.py list-backups
Retry logic is automatically enabled for all webhooks. No changes needed!
To customize retry behavior:
# config.yaml
webhooks:
retry:
max_retries: 5 # Increase retries
base_delay_seconds: 3.0 # Longer delays
# 1. Start sandbox
python resource.test/webhook_sandbox.py start --port 8888
# 2. Update test webhook to use sandbox
# (temporarily change Baseurl in database or code)
# 3. Test webhook
python -c "from ObjHook import ObjHook; h = ObjHook(); h.read('TEST_WEBHOOK'); h.call('DONE')"
# 4. Test retry logic
python resource.test/webhook_sandbox.py start --error-rate 0.5
# 5. View recorded requests
python resource.test/webhook_sandbox.py list-requests
Log Messages:
[08:10:00] POST request to https://api.example.com (max_retries=3)
[08:10:05] Server error 502, retrying in 2.0s (attempt 1/3)
[08:10:07] Server error 502, retrying in 4.0s (attempt 2/3)
[08:10:11] Response: 200 OK
MQTT Topics:
webhook/requests - Includes retry attempt countwebhook/responses - Includes final result after retriesMetrics to Track:
Backup Files:
data.config/webhook_backups/backup_{direction}_{timestamp}.jsonMigration Status Query:
# Check which webhooks are in Infisical
python resource.bin/migrate_webhooks_to_infisical.py status
resource.bin/migrate_webhooks_to_infisical.py (800 lines)
resource.test/webhook_sandbox.py (500 lines)
resource.test/pytests/factory.core/test_ObjHook_phase2.py (400 lines)
_execute_with_retry() method_execute_post_request() with retry logic_execute_get_request() with retry logic_execute_put_request() with retry logicRecommended Phase 3 implementations:
See resource.notes/WEBHOOK_ENHANCEMENTS_SUGGESTIONS.md for complete roadmap.
Problem: Retries not working
Solution: Check webhooks.retry configuration in config.yaml
Problem: Migration fails
Solution: Verify Infisical is configured and accessible
Problem: Sandbox not starting
Solution: Check if port is already in use, try different port
data.config/webhook_backups/Implementation Date: 2026-02-08
Author: Claude Code
Status: ✅ Production Ready
Test Coverage: 15/15 tests passing
Next Phase: Phase 3 (Circuit Breaker, Credential Rotation, Dashboard)