Source: TechnoCore System Update presentation
Last updated: 2026-03-29
|
Real-time / Live |
Batch / Bulk |
| Relational/Row |
MySQL, MariaDB |
|
| Column store |
ClickHouse |
Parquet |
| Directed graph |
GraphX |
|
| Schema free |
Hadoop HDFS |
|
- Primary database for Axion platform
- Row-oriented storage — optimised for transactional workloads (OLTP)
- Real-time read/write, ACID compliant
- Used across all client packages (HomeChoice, FullHouse, TLC, etc.)
¶ Row vs Column Storage
Row-oriented systems store data as complete records:
001:10,Smith,Joe,40000;
002:12,Jones,Mary,50000;
003:11,Johnson,Cathy,44000;
004:22,Jones,Bob,55000;
Column-oriented systems store data by column:
10:001,12:002,11:003,22:004;
Smith:001,Jones:002,Johnson:003,Jones:004;
Joe:001,Mary:002,Cathy:003,Bob:004;
40000:001,50000:002,44000:003,55000:004;
Column storage enables compression via record shredding and assembly algorithm — can transform into:
...,Smith:001;Jones:002,004;Johnson:003;...
¶ 2. Column Store
- Compressed, efficient columnar data representation
- Uses Map/Reduce to compress an entire schema into a linear structure
- Metadata stored separately from data
- Can be split over multiple files
- Layout: logical table → row layout (sequential) vs column layout (grouped by column → encoded chunks)
- Open source column-oriented DBMS
- Real-time generation of analytical data reports using SQL queries
- Complements MariaDB — not a replacement
- MUCH faster than relational DBs for analytical workloads (aggregations, scans)
- Multicore processing — scales horizontally
- In relational systems, composite objects are fragmented across many relations, requiring joins to gather all parts
- Directed graphs are the best representation where there are transitive or recursive relationships embedded in the data
- Most important graph algorithm: PageRank — measures importance of each vertex in a graph, assuming an edge from u to v represents an endorsement of v's importance by u
- Unifies ETL, exploratory analysis, and iterative graph computation within a single system
- View the same data as both graphs and collections, transform and join graphs
- Big library of graph functions
- Implemented in Python — far more direct execution code than a 4GL like SQL
- Uses Property Graph model with Vertex Table (id → properties) and Edge Table (srcId → dstId → property)
- Multiple data sources with internal structure
- No inherent schema relationship in the data
- Traditionally use Map/Reduce to extract meaning without a predefined ETL
- HDFS Architecture: Namenode (metadata) → Datanodes (blocks) with replication across racks
- Filesystems to store distributed data across multiple data servers
- Map takes a set of data and converts it into another set of data, where individual elements are broken down into tuples
- Reducer processes data from the mapper, producing a new set of output stored in HDFS
- Map and Reduce functions run in parallel across sections of the dataset — processing limited only by number of nodes
- Since HDFS distributes data across servers, there is very little network traffic
| Technology |
Axion Usage |
Status |
| MariaDB |
Primary database, all packages |
Active — production |
| MongoDB |
Document store, workflow queues |
Active — production |
| InfluxDB |
Time-series metrics, IoT |
Active — production |
| ChromaDB |
Vector embeddings, RAG |
Active — ObjAiRag |
| ClickHouse |
Analytical workloads |
Evaluated — future |
| Parquet |
Batch data export/import |
Supported via ObjDataExport |
| GraphX |
Relationship analysis |
Future consideration |
| HDFS/Hadoop |
Distributed processing |
Not needed at current scale |
| TiDB |
Cloud MySQL-compatible |
Active — serverless |
| Neon |
Cloud PostgreSQL |
Active — serverless |