NOTICE: All information contained herein is, and remains
the property of TechnoCore Automate.
Import coordinator for the dataimport service. Dynamically loads a
format-specific parser from factory.import/ and drives the full
lifecycle: file detection → parse → block commit → post-SQL →
workflow trigger → metrics.
Lightweight in-process accumulator for session-level statistics
(total imports, rows, elapsed time, error rate). Shared across all
ObjDataImport instances via the class-level _metrics attribute.
record_import(rows, elapsed, success) — accumulate one import.get_stats() → dict — current aggregates.print_stats() — formatted console output.watchdog.FileSystemEventHandler mixin that puts matched files onto an
internal queue when a create/modify/move event fires. Driven by
ObjDataImport.watch().
Main orchestrator. Inherits from ObjSupervisor.Supervisor.
Class-level attributes
| Attribute | Default | Purpose |
|---|---|---|
_Cleanupcounter |
0 |
Tracks imports since last table cleanup |
_Cleanupinterval |
100 |
How often cleanup runs (every N imports) |
_metrics |
ImportMetrics() |
Shared session metrics |
_Normalizationsizethreshold |
50 MB |
Files larger than this skip line-ending normalisation |
Instance attributes declared in __init__
| Attribute | Default | Notes |
|---|---|---|
_Dataimportcode |
"" |
Populated by sql_read_object from def_dataimport |
_Encoding |
"" |
Passed to the format parser |
_Directory |
"" |
Source directory |
_Delimiter |
"," |
Passed to CSV-like parsers |
_Filename |
"" |
File glob / path |
_Sheetnumber |
1 |
Sheet index for Excel formats |
_Useloaddatainfile |
False |
Enables LOAD DATA INFILE (requires MySQL perms) |
_Watchconfigcache |
None |
Cached watch-config result |
_Watchconfigcachetime |
None |
Timestamp of last cache fill |
_Cachettl |
300 |
Watch-config cache TTL in seconds |
Key methods
process(guid, data_import_code) — top-level entry point; readsdef_dataimport, iterates over files in the configured directories,import_file for each match.import_file(file_name, directory) — handles one file end-to-end:factory_object(filename) — scans factory.import/ forObjDataImport{ext}.py/.so/.pyd, imports it, instantiatesObjImportApi. Passes _Encoding, _Delimiter, and otherdef_dataimport.commit_block() — bulk-inserts the current in-memory block intocommit_data(columns) — writes parsed rows into the temporary buildimport_file(...) now follows anOverwrite=Y,CREATE TABLE ... AS SELECT from the build table,INSERT IGNORE whencleanup_old_import_tables() — drops temp import tables older than_Cleanupinterval imports.write_metrics_to_db(...) — persists per-import telemetry todef_dataimport_metric.detect_anomalies(import_code, sensitivity, days_baseline) — Z-score_track_import(guid, filename) — records audit traildata_import_track.pre_sql() / post_sql() — execute optional SQL hooks defined indef_dataimport.factory.import/ObjDataImport{ext}.py with classObjImportApi(ObjData.ObjData) implementing:prep_file, open_file, close_file, column_list, next_row"EOF" when exhausted).| Enum | Source | Used for |
|---|---|---|
YesNo.YES / NO |
ObjEnum |
ImportSuccess and NormalizationSkipped flags in def_dataimport_metric; _Overwrite check |
ImportAnomalyType |
ObjEnum |
"type" key in anomaly dicts (PERFORMANCE_DEGRADATION, SLOW_EXECUTION, MEMORY_INCREASE) |
ImportAnomalySeverity |
ObjEnum |
"severity" key in anomaly dicts (HIGH, MEDIUM, LOW) |
| Table | Purpose |
|---|---|
def_dataimport |
Import job definitions |
def_dataimport_metric |
Per-import telemetry (rows, throughput, memory) |
data_import_track |
Import audit trail (who, when, what, from where) |
stage_dataimport |
Queued import requests |
Every import is automatically recorded in
data_import_track via _track_import(). This runs
after both successful and failed imports.
| Column | Source |
|---|---|
ImportGuid |
Import session UUID |
Package |
Active package |
Module |
ObjDataImport |
ImportedWhen |
Current timestamp |
ImportedBy |
ACL user (via get_acl()) |
ImportNotes |
{DataimportCode}: {filename} (+ optional description suffix) |
ImportfromRemote |
FTP/SFTP remote source |
Query recent imports:
SELECT ImportGuid, ImportedWhen, ImportedBy,
ImportNotes, ImportfromRemote
FROM data_import_track
WHERE Package = 'homechoice'
ORDER BY ImportedWhen DESC
LIMIT 20
# Run a specific import once
python factory.core/ObjDataImport.py direct MY_IMPORT_CODE
# Start the continuous file-watch service
python factory.core/ObjDataImport.py watch
# Interactive mode
python factory.core/ObjDataImport.py interactive
# Create/migrate tables
python factory.core/ObjDataImport.py create-tables
Attributes on def_dataimport rows are mapped directly onto the parser
instance after factory_object() loads it. Any column whose name
matches a parser attribute will be forwarded automatically.
Common attributes:
| Column | Parser | Purpose |
|---|---|---|
Delimiter |
CSV | Field separator (auto-detected if blank) |
Quotechar |
CSV | Quote character (default ") |
Encoding |
all | File encoding (default utf-8) |
Sheetnumber |
XLSX / ODS | 1-based sheet index |
Importtype |
all | Override file-type detection |
field_specs |
FWF | (name, start, end) column layout |