¶ (c) Technocore - All Rights Reserved.
NOTICE: All information contained herein is, and remains the property of TechnoCore.
The intellectual and technical concepts contained herein are proprietary to TechnoCore and dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from TechnoCore.
¶ ObjDataKey
Multi-backend key-value store manager for Axion. Supports Redis, RocksDB, and DragonflyDB as configurable backends for caching and key-value storage.
¶ Overview
ObjDataKey provides a unified interface for caching and key-value storage across three backends:
- Redis - Network-based distributed in-memory cache (default)
- DragonflyDB - Redis-compatible modern alternative with improved performance
- RocksDB - Embedded persistent key-value database for single-node deployments
Backend selection is configured via config.yaml under the keystore.type parameter.
¶ Class Overview
ObjDataKey inherits from ObjData.ObjData, providing full access to database connectivity, configuration management, and logging capabilities. Key responsibilities include:
- Backend-agnostic initialization based on configuration
- Unified caching interface (
set_cache,get_cache) - Health monitoring (
has_keystore) - Backward compatibility with
ObjDataRedis
¶ Configuration
¶ Backend Selection
Set the backend type in config.yaml:
keystore:
type: redis # Options: redis, dragonfly, rocksdb
¶ Redis Configuration
keystore:
type: redis
redis:
ip: localhost # Empty string defaults to localhost or Docker IP
port: 6379 # Redis server port
user: "" # Optional authentication
password: "" # Optional authentication
ssl: false # Enable SSL/TLS for remote connections
Docker environment detection: If ip is empty and DOCKER_AXION environment variable is set, defaults to 172.17.0.1.
¶ DragonflyDB Configuration
keystore:
type: dragonfly
dragonfly:
ip: localhost
port: 6379 # DragonflyDB default port
user: ""
password: ""
ssl: false
Note: DragonflyDB is Redis-protocol compatible, so the same redis Python library is used.
¶ RocksDB Configuration
keystore:
type: rocksdb
rocksdb:
path: local.documents/rocksdb # Relative or absolute path
create_if_missing: true # Create database if it doesn't exist
ttl_cleanup_interval: 300 # Seconds between expired key cleanup
Note: RocksDB is an embedded database (no server required). Requires python-rocksdb package.
¶ Cache TTL Configuration
system:
cachettl: 300 # Default TTL in seconds (5 minutes)
¶ Backward Compatibility
Old redis.* configuration keys are still supported but deprecated:
redis:
activate: yes
redisip: localhost # Deprecated: use keystore.redis.ip
redisport: 6379 # Deprecated: use keystore.redis.port
¶ Architecture
¶ Class Hierarchy
ObjData.ObjData (parent)
└── ObjDataKey (multi-backend keystore)
Inheritance Benefits:
- Access to database operations via
ObjDataparent - Configuration management via
get_ini_value() - Logging via
self.debug() - Full Axion framework integration
¶ Backend Selection Flow
1. __init__() called
2. Read keystore.type from config
3. Default to "redis" if empty
4. Call backend initializer:
├── _init_redis() → Connect to Redis server
├── _init_dragonfly() → Connect to DragonflyDB server
└── _init_rocksdb() → Open embedded RocksDB
5. Set compatibility aliases (redis_conn, redisConn)
6. Ready for cache operations
¶ Constants
DEF_VERSION = "8.0"- Module versionDO_DEBUG = False- Enable debug loggingDO_KEYSTORE = True- Global keystore enable/disable flagREDIS_DOCKER_IP = "172.17.0.1"- Default Docker bridge IPREDIS_HOST = "localhost"- Default Redis hostREDIS_PORT = "6379"- Default Redis portHAS_ROCKSDB- Boolean flag indicating if python-rocksdb is available
¶ Methods
¶ __init__()
Initializes the key-value store based on keystore.type configuration.
Backend Selection Logic:
- Read
keystore.typefromconfig.yaml - Default to "redis" if not specified
- Initialize appropriate backend (
_init_redis(),_init_dragonfly(), or_init_rocksdb()) - Set compatibility aliases (
redis_conn,redisConn)
Attributes Set:
self.KeystoreType(str) - Active backend typeself.keystore_conn- Connection object or 0 if failedself.redis_conn- Backward compatibility aliasself.redisConn- Backward compatibility alias for ObjData
Example:
from ObjDataKey import ObjDataKey
cache = ObjDataKey() # Auto-connects based on config
print(cache.KeystoreType) # "redis", "dragonfly", or "rocksdb"
¶ connect_keystore(**kwargs)
Universal connection method for Redis and DragonflyDB backends.
Parameters:
backend(str): Backend type override ("redis", "dragonfly", "rocksdb")ip(str): Server IP address (Redis/Dragonfly only)port(str): Server portuser(str): Username for authenticationpassword(str): Password for authentication
Returns: Connection object or 0 on failure
Note: RocksDB is initialized in _init_rocksdb() as it's embedded.
¶ set_cache(name, store_data, ttl=-1)
Stores data in the key-value store with optional TTL.
Parameters:
name(str): Cache keystore_data(str|dict|list): Data to store (auto-serialized to JSON)ttl(int): Time-to-live in seconds (-1 = use default fromsystem.cachettl)
Behavior:
- Auto-encodes data as JSON
- TTL defaults to
system.cachettlconfig value (minimum 5 seconds) - Backend-specific implementation:
- Redis/Dragonfly: Native TTL via
SETEXcommand - RocksDB: TTL emulated via metadata wrapper
- Redis/Dragonfly: Native TTL via
Example:
cache.set_cache("user:123", {"name": "Alice", "email": "alice@example.com"}, ttl=600)
¶ get_cache(name)
Retrieves data from the key-value store.
Parameters:
name(str): Cache key
Returns: Deserialized data (original type) or None if not found/expired
Behavior:
- Auto-deserializes JSON to original type
- Returns
Nonefor missing/expired keys - RocksDB: Automatically deletes expired keys on read
Example:
user_data = cache.get_cache("user:123")
print(user_data) # {'name': 'Alice', 'email': 'alice@example.com'}
¶ has_keystore()
Health check for key-value store connection.
Returns: bool - True if connected and healthy
Backend Checks:
- Redis/Dragonfly: Executes
PINGcommand - RocksDB: Verifies connection object exists
Example:
if cache.has_keystore():
print("Cache is healthy")
else:
print("Cache connection failed")
¶ has_redis() (Deprecated)
Backward compatibility alias for has_keystore().
¶ connect_redis() (Deprecated)
Backward compatibility alias for connect_keystore() with backend="redis".
¶ Backend-Specific Private Methods
¶ Redis/DragonflyDB Methods
_init_redis()
- Reads configuration from
keystore.redis.*or legacyredis.*keys - Detects Docker environment via
DOCKER_AXIONenvironment variable - Calls
connect_keystore()with Redis parameters - Sets
self.RedisIpandself.RedisPortattributes
_init_dragonfly()
- Reads configuration from
keystore.dragonfly.* - Sets
self.DragonflyIpandself.DragonflyPortattributes - Calls
connect_keystore()with DragonflyDB parameters
_set_cache_redis(name, data, ttl)
- Executes Redis
SETcommand withEX(TTL) parameter - Data is pre-serialized JSON string
- Catches and logs exceptions via
self.debug()
_get_cache_redis(name)
- Executes Redis
GETcommand - Returns raw JSON string or None
- Catches and logs exceptions
_set_cache_dragonfly() / _get_cache_dragonfly()
- Delegate to Redis methods (protocol-compatible)
¶ RocksDB Methods
_init_rocksdb()
- Checks for
python-rocksdbavailability viaHAS_ROCKSDBflag - Reads database path from
keystore.rocksdb.path - Converts relative paths to absolute
- Creates
rocksdb.Options()with configurable settings - Opens database via
rocksdb.DB(path, options) - Starts background cleanup thread
- Sets
self.RocksDbPathattribute
_set_cache_rocksdb(name, data, ttl)
- Wraps data in metadata:
{"data": <json>, "expires_at": <timestamp>} - Calculates expiration time:
current_time + ttl - Stores as UTF-8 encoded bytes
- TTL of -1 or 0 = never expires (
expires_at: -1)
_get_cache_rocksdb(name)
- Retrieves UTF-8 encoded bytes
- Deserializes metadata JSON
- Checks
expires_atagainst current time - Deletes key if expired (lazy deletion)
- Returns original data JSON string
_start_rocksdb_cleanup_thread(interval)
- Creates daemon thread for background cleanup
- Thread runs infinite loop with
time.sleep(interval) - Calls
_rocksdb_cleanup_expired()periodically - Catches and logs thread exceptions
_rocksdb_cleanup_expired()
- Iterates all keys via
keystore_conn.iteritems() - Deserializes metadata for each key
- Compares
expires_atwith current time - Deletes expired keys
- Logs deletion count if > 0
¶ Backend Implementation Details
¶ Redis Backend
Connection:
- Uses
redis-pylibrary - Supports SSL for remote connections
- Connection pooling handled automatically
TTL: Native support via SETEX command
Persistence: Optional (configured at Redis server level)
¶ DragonflyDB Backend
Connection:
- Uses same
redis-pylibrary (protocol-compatible) - Configuration under
keystore.dragonfly.*
TTL: Native support (Redis-compatible)
Persistence: Built-in (always persisted to disk)
¶ RocksDB Backend
Connection:
- Embedded database (no server)
- Opens database at configured path
- Single-process access only (file locking)
TTL Emulation:
- Values stored as:
{"data": <json>, "expires_at": <unix_timestamp>} - Expiration checked on retrieval
- Background cleanup thread removes expired keys periodically
Storage Format:
{
"data": "{\"user\": \"alice\"}",
"expires_at": 1703001234
}
Cleanup Thread:
- Runs as daemon thread
- Interval configured via
ttl_cleanup_interval(default: 300s) - Scans all keys and deletes expired entries
¶ Command-Line Interface
¶ check
Verify keystore connectivity and display configuration.
Usage:
python factory.core/ObjDataKey.py check
Example Output:
Reading folder /home/vheyn/projects/axion for config YAML (Objects)
Current keystore type: redis
Keystore connection is active.
¶ info
Display keystore configuration details.
Usage:
python factory.core/ObjDataKey.py info
Example Output:
Keystore Type: redis
Redis IP: localhost
Redis Port: 6379
¶ Migration from ObjDataRedis
¶ Code Changes
Old:
from ObjDataRedis import ObjDataRedis
redis_mgr = ObjDataRedis()
redis_mgr.connect_redis(ip="localhost", port="6379")
redis_mgr.set_cache("key", data, ttl=300)
value = redis_mgr.get_cache("key")
New:
from ObjDataKey import ObjDataKey
key_mgr = ObjDataKey()
# Connection happens automatically in __init__
key_mgr.set_cache("key", data, ttl=300)
value = key_mgr.get_cache("key")
¶ Configuration Migration
Old (config.yaml):
redis:
activate: yes
redisip: localhost
redisport: 6379
New (config.yaml):
keystore:
type: redis
redis:
ip: localhost
port: 6379
Note: Old configuration keys still work but show deprecation warnings.
¶ Import Changes
Replace all imports:
# Old
from ObjDataRedis import ObjDataRedis
# New
from ObjDataKey import ObjDataKey
Update class references:
# Old
obj = ObjDataRedis()
# New
obj = ObjDataKey()
¶ Backend Comparison
| Feature | Redis | DragonflyDB | RocksDB |
|---|---|---|---|
| Deployment | Network server | Network server | Embedded library |
| Protocol | Redis | Redis-compatible | Native |
| TTL Support | Native | Native | Emulated (metadata) |
| Clustering | Yes | Yes | No |
| Persistence | Optional | Yes (always) | Yes (always) |
| Performance | Excellent (in-memory) | Excellent+ (in-memory) | Very good (disk-based) |
| Memory Usage | High (all in RAM) | Medium (smart caching) | Low (mostly disk) |
| Best For | Distributed cache, high-speed reads | Modern Redis replacement | Single-node persistent cache |
| Dependencies | Redis server | DragonflyDB server | python-rocksdb library |
¶ Usage Examples
¶ Basic Usage
from ObjDataKey import ObjDataKey
# Initialize (auto-connects based on config)
cache = ObjDataKey()
# Store data
cache.set_cache("session:abc123", {"user_id": 42, "role": "admin"}, ttl=3600)
# Retrieve data
session = cache.get_cache("session:abc123")
print(session) # {'user_id': 42, 'role': 'admin'}
# Check connection
if cache.has_keystore():
print("Cache is operational")
¶ Using RocksDB for Local Development
Config (config.yaml):
keystore:
type: rocksdb
rocksdb:
path: local.documents/rocksdb
create_if_missing: true
Code:
from ObjDataKey import ObjDataKey
cache = ObjDataKey()
# RocksDB is now embedded - no server needed!
cache.set_cache("dev_data", {"status": "testing"}, ttl=60)
¶ Storing Complex Data
cache = ObjDataKey()
# Store nested structures
report_data = {
"title": "Q4 Sales Report",
"metrics": {
"revenue": 125000,
"growth": 15.3
},
"regions": ["North", "South", "East"]
}
cache.set_cache("reports:q4_2024", report_data, ttl=86400) # 1 day
# Retrieve maintains structure
report = cache.get_cache("reports:q4_2024")
print(report["metrics"]["revenue"]) # 125000
¶ DragonflyDB for High Performance
Config:
keystore:
type: dragonfly
dragonfly:
ip: dragonfly.internal.network
port: 6379
Code:
cache = ObjDataKey()
# Uses DragonflyDB backend (faster than Redis for many workloads)
for i in range(10000):
cache.set_cache(f"item:{i}", {"index": i}, ttl=300)
¶ Troubleshooting
¶ Connection Failures
Redis/DragonflyDB:
KeystoreError: Redis connection failed: [Errno 111] Connection refused
Solutions:
- Verify server is running:
redis-cli pingordocker ps - Check IP/port in
config.yaml - Verify firewall rules allow connection
- Check authentication credentials
RocksDB:
RocksDBError: IO error: lock /path/to/rocksdb/LOCK: Resource temporarily unavailable
Solutions:
- Another process has the database open
- Check for zombie processes:
ps aux | grep python - Remove lock file (only if certain no other process is using it)
- Ensure only one process accesses RocksDB instance
¶ TTL Not Working
Symptoms: Cached data persists longer than expected
RocksDB-specific:
- Check cleanup thread is running (look for log: "Started RocksDB cleanup thread")
- Adjust
ttl_cleanup_intervalin config for more frequent cleanup - Verify system time is correct
Redis/Dragonfly:
- Verify TTL is set:
redis-cli TTL <key> - Check Redis eviction policy (should allow TTL expiration)
¶ Missing Dependencies
RocksDB backend requested but python-rocksdb not installed
Solution:
source dev-env/bin/activate
uv pip install python-rocksdb
¶ Performance Issues
Redis/Dragonfly:
- Check network latency
- Monitor connection pool usage
- Consider read replicas for read-heavy workloads
RocksDB:
- Reduce
ttl_cleanup_intervalif cleanup impacts performance - Consider periodic compaction:
db.compact_range() - Monitor disk I/O
¶ Performance Considerations
¶ Redis/Dragonfly
- Network latency directly impacts performance
- In-memory storage provides fast reads/writes
- Horizontal scaling via clustering
- Connection pooling recommended for high traffic
¶ RocksDB
- No network overhead (embedded)
- Fast reads for recently accessed keys (cached in memory)
- TTL cleanup may cause periodic performance spikes
- Periodic compaction recommended for optimal performance
- Disk I/O is the primary bottleneck
¶ Security Considerations
- Passwords: Use strong passwords for Redis/Dragonfly
- SSL: Enable SSL for remote Redis/Dragonfly connections
- File Permissions: Secure RocksDB database directory (chmod 700)
- Encryption: Data is JSON-encoded but not encrypted at rest
- Access Control: Use firewall rules to restrict network access
¶ Version History
- v8.0: Initial multi-backend implementation
- Added support for Redis, DragonflyDB, and RocksDB
- Refactored from
ObjDataRedistoObjDataKey - Implemented TTL emulation for RocksDB
- Added backward compatibility layer
¶ Environment Variables
¶ DOCKER_AXION
When set (any value), indicates the application is running in a Docker container.
Effect: If keystore.redis.ip is empty, defaults to 172.17.0.1 (Docker bridge IP) instead of localhost.
Example:
export DOCKER_AXION=1
python factory.core/ObjDataKey.py check
¶ AXION_PACKAGE
Specifies the active package/environment (e.g., "homechoice", "test", "production").
Effect: Used by configuration system to load package-specific settings.
¶ Advanced Usage Patterns
¶ Disabling Keystore Globally
Set DO_KEYSTORE = False in the module to disable all keystore operations:
import factory.core.ObjDataKey as ObjDataKeyModule
ObjDataKeyModule.DO_KEYSTORE = False
# All cache operations will now no-op
cache = ObjDataKey()
cache.has_keystore() # Returns False
¶ Multiple Backend Instances
You can create multiple instances with different backends (not recommended for RocksDB due to file locking):
# Redis cache for sessions
redis_cache = ObjDataKey() # Uses config default
# Manually override for specific instance (advanced)
from ObjData import ObjData
custom_cache = ObjDataKey()
custom_cache.KeystoreType = "rocksdb"
custom_cache._init_rocksdb()
Warning: Overriding KeystoreType after initialization may cause inconsistent state.
¶ Integration with ObjData Parent
Classes inheriting from ObjData automatically get keystore access:
from ObjData import ObjData
class MyService(ObjData):
def __init__(self):
super().__init__()
# keystore_conn is available via parent
def cache_report(self, report_id, data):
# Use inherited cache methods
self.set_cache(f"report:{report_id}", data, ttl=3600)
def get_cached_report(self, report_id):
return self.get_cache(f"report:{report_id}")
Note: The ObjData parent class uses redisConn attribute, which is aliased to keystore_conn for compatibility.
¶ Namespacing Cache Keys
Use consistent key prefixes to organize cached data:
cache = ObjDataKey()
# Namespace patterns
cache.set_cache("user:session:abc123", session_data)
cache.set_cache("api:response:endpoint_name", api_response)
cache.set_cache("report:daily:2024-01-15", report_data)
cache.set_cache("temp:processing:job_42", temp_data)
# Easy to delete all keys in namespace (Redis/Dragonfly)
# redis-cli KEYS "user:session:*" | xargs redis-cli DEL
¶ Conditional Caching
Cache only when keystore is available:
cache = ObjDataKey()
def get_expensive_data():
# Try cache first
if cache.has_keystore():
cached = cache.get_cache("expensive_computation")
if cached:
return cached
# Compute if not cached
result = expensive_computation()
# Cache if keystore available
if cache.has_keystore():
cache.set_cache("expensive_computation", result, ttl=3600)
return result
¶ TTL Strategy Patterns
Short TTL for frequently changing data:
cache.set_cache("stock_price:AAPL", price, ttl=60) # 1 minute
Long TTL for static data:
cache.set_cache("user_profile:123", profile, ttl=86400) # 24 hours
Session-based TTL:
# Match session timeout
session_ttl = 3600 # 1 hour
cache.set_cache(f"session:{session_id}", session_data, ttl=session_ttl)
Permanent cache (RocksDB only, no expiration):
# Only works with RocksDB backend
# Redis/Dragonfly minimum is 5 seconds
cache.set_cache("permanent_config", config, ttl=-1)
¶ Deployment Considerations
¶ Docker Deployment
When deploying in Docker containers:
-
Redis/DragonflyDB: Use service names as hostnames
keystore: type: redis redis: ip: redis-service # Docker service name port: 6379 -
RocksDB: Mount persistent volume
volumes: - ./data/rocksdb:/app/local.documents/rocksdb -
Set environment variable:
ENV DOCKER_AXION=1
¶ High Availability
Redis: Use Redis Sentinel or Redis Cluster for HA
# Configure multiple Redis nodes (requires custom connection logic)
keystore:
type: redis
redis:
nodes:
- host: redis-1.internal
port: 6379
- host: redis-2.internal
port: 6379
DragonflyDB: Use replication (primary-replica setup)
RocksDB: Not suitable for HA (single-process only). Use Redis/Dragonfly for distributed deployments.
¶ Monitoring and Metrics
Key metrics to monitor:
- Cache hit rate
- Connection failures
- TTL expiration patterns
- Storage size (RocksDB)
- Memory usage (Redis/Dragonfly)
Example monitoring:
import time
class MonitoredCache(ObjDataKey):
def __init__(self):
super().__init__()
self.hits = 0
self.misses = 0
def get_cache(self, name):
start = time.time()
result = super().get_cache(name)
duration = time.time() - start
if result:
self.hits += 1
else:
self.misses += 1
# Log metrics
self.debug(f"Cache GET {name}: {duration:.4f}s, hit_rate={self.hit_rate():.2%}")
return result
def hit_rate(self):
total = self.hits + self.misses
return self.hits / total if total > 0 else 0
¶ Best Practices
-
Always check keystore health before critical operations
if not cache.has_keystore(): # Fallback logic -
Use appropriate TTLs - Don't cache forever, don't expire too soon
-
Namespace your keys - Use prefixes like
user:,session:,api: -
Handle cache misses gracefully - Always have fallback logic
-
Monitor cache performance - Track hit rates and response times
-
Choose backend appropriately:
- Redis: Distributed systems, high concurrency
- DragonflyDB: Drop-in Redis replacement, better performance
- RocksDB: Single-node, persistent cache, lower memory
-
Serialize complex objects - Use JSON, pickle, or msgpack
-
Avoid storing secrets - Don't cache passwords, tokens, or API keys
-
Set reasonable TTLs - Balance freshness vs. performance
-
Use connection pooling (Redis/Dragonfly) for high-traffic applications
¶ Limitations
¶ RocksDB Limitations
- Single-process access: Only one process can open the database
- No network access: Cannot share across machines
- TTL not precise: Cleanup runs periodically, not immediately at expiration
- No native TTL: Implemented via metadata wrapper
¶ Redis/DragonflyDB Limitations
- Network dependency: Requires network connection to server
- Memory-bound: Limited by available RAM
- Minimum TTL: 5 seconds (enforced by code)
¶ General Limitations
- No encryption at rest: Data stored as plain JSON
- No access control: No user-level permissions
- No transactions: Each operation is atomic but not grouped
- No query support: Key-value only, no complex queries
¶ Test Coverage
The ObjDataKey module has test coverage for configuration and initialization.
¶ Test Summary
| Test Suite | Tests | Type | Purpose |
|---|---|---|---|
test_ObjDataKey.py |
5 | Unit | Keystore type configuration, has_keystore checks |
| Total | 5 |
¶ Running Tests
# Run all ObjDataKey tests
pytest resource.test/pytests/factory.core/test_ObjDataKey.py -v
¶ Test Features
- Configuration Testing: Validates keystore type settings from config.yaml
- Connection Mocking: Tests use mocked Redis/RocksDB connections
- Fast Execution: All tests complete in <1 second
¶ See Also
ObjData.md- Parent class documentationresource.notes/howto/caching.md- Caching strategiesfactory.core/Objects.py- Global configuration object- Redis Documentation
- DragonflyDB Documentation
- RocksDB Wiki
- Redis Best Practices
- Python redis-py Documentation