¶ (c) Technocore - All Rights Reserved.
NOTICE: All information contained herein is, and remains
the property of TechnoCore.
The intellectual and technical concepts contained
herein are proprietary to TechnoCore and dissemination of this information or reproduction of this material
is strictly forbidden unless prior written permission is obtained
from TechnoCore.
¶ ObjData
ObjData is a foundational and versatile class within the Axion framework, designed to serve as a comprehensive data management and operations hub. It provides a high-level abstraction layer for interacting with various database systems, handling configuration, managing system resources, and executing data-related tasks.
¶ Class Overview
ObjData inherits from multiple mixins in extend.data/,
each handling a specific domain. Its primary goal is to
standardize and simplify data access and manipulation,
regardless of the underlying data source.
¶ Inheritance Chain
Objects.Object
-> ObjDataSql (extend.data/) — SQL execution layer
-> ObjDataDDL (extend.data/) — Schema/DDL operations
-> ObjDataMongo (extend.data/) — MongoDB operations
-> ObjData (factory.core/) — Core data layer
+ ObjDataEventMixin — Event logging (MQTT/InfluxDB)
+ ObjDataRemoteMixin — Remote DB connections
+ ObjDataBrandingMixin — Palette, logo, signatures
+ ObjDataCacheMixin — Redis/keystore cache
+ ObjDataQueryExecMixin — Named query executors
¶ Module Layout (extend.data/)
| Module | Lines | Purpose |
|---|---|---|
ObjDataSql |
1,539 | SQL execution, pagination, tracking |
ObjDataDDL |
797 | Schema ops, has_table, create_tables_from_yaml |
ObjDataEvent |
645 | log_event, MQTT, InfluxDB connectors |
ObjDataRemote |
572 | remote_connect, credential lookup |
ObjDataBranding |
386 | get_palette, get_logo, get_signature, get_mermaid_theme, get_print_palette |
ObjDataMongo |
364 | MongoDB connect/insert/pipeline |
ObjDataQueryExec |
232 | query_get_value, query_get_dict, etc. |
ObjDataCache |
142 | has_redis, set_cache, get_cache |
¶ Core Functionalities:
- Multi-Database Interface: Provides a unified API for both relational (MySQL, MariaDB, MSSQL, SQLite) and NoSQL (MongoDB, InfluxDB) databases.
- Connection Management: Manages remote database connections with thread-local storage caching and connection pooling via
ObjConnectionPoolfor production-ready, thread-safe connection handling. - Thread-Safe Operations: Uses thread-local storage for remote connection caching to prevent race conditions in multi-threaded environments.
- Configuration Handling: Acts as a gateway to the application's global configuration, providing methods to retrieve package, deployment, role, and other system settings.
- Schema Management: Includes methods for creating, altering, and querying the database schema (tables, columns, etc.) through its
ObjDataDDLheritage. - Query Abstraction: Offers a powerful query interface that can identify and execute different types of queries (SQL, Mongo, HTTPS, etc.).
- Event Logging: Integrates with MQTT and InfluxDB for robust event and metric logging.
- Preflight Checks: Includes a comprehensive suite of preflight checks to ensure system readiness.
- CLI Endpoints: Exposes several command-line functions for diagnostics and administrative tasks.
¶ Key Methods
¶ Initialization
¶ __init__(self, db=0, use_tui=False)
The constructor initializes the ObjData object. It establishes a database connection, sets up server context, and can optionally initialize a Terminal User Interface (ObjTUI) for interactive scripts.
¶ Configuration and System Info
get_ini_value(context, name): Retrieves a configuration value from the globalconfig.yaml.get_package()/get_deployment()/get_role(): Cached methods to get essential application context.is_simulation()/is_virtual_service(): Checks if the system is running in simulation or virtual service mode.get_package_info(package_name): Retrieves package-specific information from thedef_packagetable.
¶ AI Model Selection (proxies to ObjAI)
Thin static proxies so any ObjData subclass (which is most of
the codebase) can call self.get_model*() without importing
ObjAI directly. Each delegates to the matching static on
ObjAI, which resolves a hardware-appropriate model name from
config.yaml under ai.model.<category>.<tier> with fallback to
AiConstants constants.
get_model()— generation / chat model.get_model_vision()— vision (multimodal) model; may
return""when vision is disabled on the active tier.get_model_embedding()— embedding model for RAG / ChromaDB.
See factory.core/ObjAI.md for tier thresholds, the config
sample, and the full resolution algorithm.
¶ Branding Helpers (from ObjDataBrandingMixin)
In addition to get_palette(), get_logo(), and get_signature(),
two methods support themed rendering in reports:
get_mermaid_theme() -> str— Returns"dark"or"neutral"
based on the active palette background luminance. Delegates to
ObjPalette.get_mermaid_theme(). Used by documentation reports
to auto-select a readable Mermaid diagram theme.get_print_palette() -> dict— Loads the PRINT palette
(PaletteName='PRINT',Package='ALL') for PDF/print output.
Falls back to hardcoded light defaults if no PRINT palette exists.
Delegates toObjPalette.get_print_palette().
¶ Database and Schema Operations (from ObjDataDDL)
has_table(table_name): Checks if a table exists.has_column(table, column): Checks if a column exists in a table.add_column(table, column, type): Adds a column to a table.create_tables_from_yaml(schema_name): Creates database tables from a YAML schema file.
¶ SQL Data Retrieval (from ObjDataSql)
sql_execute(sql): Executes a given SQL statement.sql_get_value(sql): Retrieves a single scalar value.sql_get_dict(sql): Retrieves a single row as a dictionary.sql_get_dictionary_list(sql): Retrieves multiple rows as a list of dictionaries.sql_get_dataframe(sql): Returns a pandas DataFrame from a SQL query.
¶ MongoDB Operations (from ObjDataMongo)
mongo_connect(): Establishes a connection to the MongoDB server.mongo_insert(collection, content): Inserts a document into a collection.mongo_pipeline(collection, pipeline): Executes a MongoDB aggregation pipeline.
¶ Query Abstraction
query_identity(query): Identifies the type of a query string (e.g.,SQL,MONGO,HTTPS).query_get_value(query)/query_get_dict(query): Executes a query string and returns the result in the desired format, automatically handling the underlying database type.
¶ Logging
log_event(...): Logs an event to the configured logging backends (MQTT, InfluxDB).note_check(testname, result): Logs the pass/fail status of a preflight check with a visual indicator (✅/❌).
¶ Preflight Checks
preflight()/check_preflight(): Runs a comprehensive suite of checks to validate the system's configuration, filesystem, dependencies, and external service connectivity. When running interactively (TTY), also sets the terminal window title to identify the service, package, deployment, and host.
¶ Service Connectivity Helpers
_ping_service_tcp(host, port, timeout): Lightweight TCP reachability check usingsocket.create_connection. ReturnsTrueif a connection can be opened. Used to probe ChromaDB and MSSQL._ping_http_service(host, port, path, timeout): HTTP reachability check usingurllib.request. ReturnsTrueif the endpoint returns any HTTP response (any status code). Used to probe Infisical's/api/statusendpoint.
¶ Preflight Auto-Start
When a service is unreachable on localhost or 127.0.0.1, preflight automatically
launches the corresponding Docker start script from resource.bin/ and polls until
the service becomes available, using note_progress() for an in-place countdown display.
| Service | Config gate | Start script | Poll timeout | Check method |
|---|---|---|---|---|
| MQTT | mqtt.host == localhost |
start_mqtt.sh |
60 s / 3 s | log_mqtt_event("TEST") |
| MongoDB | mongo.host == localhost |
start_mongo.sh |
60 s / 3 s | mongo_connect() |
| InfluxDB | influxdb.host == localhost |
start_influxdb.sh |
60 s / 3 s | connect_flux() |
| RabbitMQ | rabbitmq.host == localhost |
start_rabbitmq.sh |
60 s / 3 s | mq.connect_rabbit() |
| ChromaDB | chromadb.host configured |
start_chromadb.sh |
60 s / 3 s | TCP :8000 |
| Infisical | infisical.enabled = true |
start_infisical.sh |
120 s / 5 s | HTTP /api/status |
| MSSQL | database.type = mssql |
start_mssql.sh |
90 s / 5 s | TCP :1433 |
Connector resets on retry:
- MQTT:
MQTT_CONNECTOR = None— forceslog_mqtt_eventto reconnect - InfluxDB:
INFLUX_CONNECTOR = 0— forcesconnect_fluxto reconnect - MongoDB:
ObjDataMongo.MONGO_CONNECTOR = 0andMONGO_HOST = None - RabbitMQ / ChromaDB / Infisical / MSSQL: stateless checks, no reset needed
config.yaml entries required:
base:
chromadb:
host: localhost
port: 8000 # matches resource.config/chromadb.yaml
infisical:
enabled: false # set to true to enable Infisical check
host: https://app.infisical.com
port: 8080
¶ Command-Line Interface (CLI)
ObjData.py provides several CLI commands for system diagnostics and administration, accessible via typer.
Usage: dev-env/bin/python3 factory.core/ObjData.py [COMMAND]
¶ Commands
-
preflight
Runs all preflight checks and reports the system's readiness status. On an
interactive TTY, sets the terminal window title to⚡ Axion — preflight [package/deployment] host. -
create-tables <schema_name>
Creates database tables from a specified YAML schema file. -
comparetables <table1> <table2>
Compares the schema and data of two tables to check for differences. -
collation
Checks and displays the default character set and collation of the database.
¶ Connection Pooling
ObjData integrates with ObjConnectionPool to provide efficient, thread-safe database connection management for remote database operations.
¶ Thread-Local Connection Caching
Version 8.0 introduced thread-local storage for remote connection caching to improve thread safety:
# Thread-local storage prevents race conditions
_thread_local_data = threading.local()
def get_remote_connections_cache() -> dict:
"""Returns a thread-local dictionary for caching remote connections."""
if not hasattr(_thread_local_data, "remote_connections"):
_thread_local_data.remote_connections = {}
return _thread_local_data.remote_connections
¶ Benefits
- Thread Safety: Each thread maintains its own connection cache
- Connection Reuse: Reduces overhead of creating new connections
- Automatic Pooling: Integration with
ObjConnectionPoolprovides validation, recycling, and overflow handling - Production Ready: Handles corrupted connections and automatic retry logic
¶ Usage with Remote Connections
from ObjData import ObjData
# Create ObjData instance
data = ObjData()
# Connect to remote database (uses connection pool internally)
remote_conn = data.db_connect_remote("secondary")
# Use connection
cursor = remote_conn.cursor()
cursor.execute("SELECT * FROM remote_table")
results = cursor.fetchall()
cursor.close()
For detailed information on connection pooling, see ObjConnectionPool.md.
¶ Remote Database Connections
The remote_connect() method provides connectivity to external databases, enabling cross-database operations like data transfer, remote alerting, and distributed reporting.
¶ Current Credential Storage
Remote connection credentials are currently stored in two locations:
-
YAML-based (Modern):
config.yamlunderdatabase.connectionsdatabase: connections: - name: reference-mssql type: mssql ip: mariadb.netbird.cloud db: referencedb.axion user: sa password: encrypted_value -
Table-based (Legacy):
Def_RemoteConnectionsdatabase table- Managed by
ObjConnection.py - May be deprecated in future versions
- Managed by
¶ Migration to Infisical
MIGRATION PLAN: Remote database connection credentials will be migrated to Infisical secret management as services are updated.
¶ Benefits
- Enhanced Security: Credentials stored in dedicated secret management system
- Centralized Management: Single source of truth for all secrets
- Audit Trail: Track access and changes to connection credentials
- Environment Isolation: Separate credentials per environment (dev, uat, prod)
- Rotation Support: Easier credential rotation and updates
¶ Implementation
- Migration will be gradual, service-by-service
- The
remote_connect()method will be updated to support Infisical lookups - Backward compatibility maintained during transition period
- Connection naming convention:
DATABASE_<name>_<property>(e.g.,DATABASE_REFERENCE_HOST,DATABASE_REFERENCE_USER)
For detailed information on remote connection usage across the codebase, see resource.notes/RemoteConnectionsUsage.md.
¶ Cache TTL from Companion YAML
¶ get_cache_ttl(default=0) -> int
Reads the cache.ttl value from the module's companion YAML file
and returns it as an integer. If the key is missing or the YAML has
no cache section, returns default.
Used by WebServer and WebPageReport to determine the Redis cache
duration for a report's rendered HTML. The module YAML TTL is
compared with the def_report table TTL (_Server_ttl); the
larger value wins.
¶ Companion YAML Format
Add a cache section to any module's companion YAML:
cache:
ttl: 300
| Key | Type | Description |
|---|---|---|
cache.ttl |
int | Redis cache duration in seconds. 0 = no caching. |
¶ TTL Resolution Order
When a report is served, the effective TTL is:
max(module_yaml_cache_ttl, def_report_server_ttl, REPORT_TTL_DEFAULT)
This ensures that a module author can enforce a minimum cache
duration via YAML, while operators can raise it further via the
def_report database table.
¶ Testing
The test cases for this object are located in resource.test/pytests/factory.core/test_objData.py. All tests passed successfully on 2025-10-02.
¶ Related Test Suites
test_ObjConnectionPool.py- Comprehensive connection pool testing (22 tests, all passed on 2025-12-23)
¶ Recent Updates (2026-03-10)
DEF_VERSIONadvanced to8.1.sql_get_dictionary_listnow consistently returnslist[dict]and logs fetch failures instead of returning mixed types on cursor errors.
¶ Performance Optimisations (2026-04-13)
| # | Fix | Before | After |
|---|---|---|---|
| 1 | Duplicate global declarations removed | 2 duplicates | Clean |
| 2 | copy.deepcopy() → copy.copy() |
Deep copy on YAML config | Shallow copy |
| 3 | String += → list.append() + join() |
O(n²) in SQL builders | O(n) |
| 4 | Mutable default keys=[] |
6 methods | keys=None with guard |
| 5 | Python 2 from builtins removed |
Dead import | Removed |
| 6 | is_my_sql() cached locally |
4 calls per sql_set_dict |
1 call |
| 7 | Dead code removed | BuildSchema, get_form, form_layout_factory, has_x_display, 4 more |
Deleted |
| 8 | MQTT/InfluxDB recovery | Stuck at -1 forever |
60s retry timer |
| 9 | seaborn/matplotlib moved to local import |
Module-level | Function-level |
| 10 | Bare except blocks | Silent pass |
Logged with self.debug() |
| 11 | 5 mixins extracted to extend.data/ |
4,487 lines | 2,703 lines core |
| 12 | ObjDataDDL + ObjDataMongo + ObjDataSql moved |
factory.core/ |
extend.data/ |
| 13 | ObjDataConfig deleted |
Dead code (886 lines) | Removed |
Updated: 2026-04-13