These are notes relating to the SwitchX Data Insights Platform (DIP) and New Development Database (NDD) machine. This server runs processes controlling both of these projects.
- Built by, and Primary Maintenance: Owen
- For help/assistance: Owen, Ryan
- Likely Client: Cody (SwitchX)
- ServeReport9400: For serving front end.
- ServeWebHook9500: For serving WebHooks.
- ObjChannelmail_dispatcher & ObjChannelmail_worker: Reads stage_channel and sends emails to users either welcoming them to the platform or password change requests.
- ObjService_worker & ObjService_dispatcher: Workers for monitoring stage_service and running services for the New Development Database project.
The Data Insights Platform is a front-end Visualisation and User Management tool. The heart of this project is an iFrame of a PowerBI report, used by SwitchX and some of their parters (Ooba, MortgageMax, etc.). The
¶ Key Files and Features
- factory.report/package.core/ObjReportPowerBIEmbed.py
- This report type controls the iFrame for the embedded PowerBI report. This requires credentials, found in config.yaml.
- local.documents/package.switchx/geojson
- This is the storage location for geojson files that are required for the PowerBI report. Files correspond to an entry in def_document.
- Ryan will occasionally need a file updated. Replace the file in the above directory with the new one from Ryan so that the report can be updated.
The New Development Database project involves an asynchronous WebHook where we receive Package Letters from SwitchX. We then have a service to send those documents to Google's Document AI service (name on Google Cloud is SwitchX Data Capture) and receive data on new property developments across South Africa. There is then another WebHook which is used by SwitchX users to fetch the data. This project involves all Back-End processes.
¶ Key features and Files
- def_webhook => WebHookCode = "PDFOCR"
- This inbound WebHook receives a package letter from SwitchX in the form of a Base64 string. The string either represents a PDF file or a Tiff Image. We also receive some metadata relating to that document (this metadata can be handed to SwitchX in case we need them to query a document they have sent through).
- factory.service/package.core/ObjServiceGoogleOcr.py
- This is a synchronous service that sends Package Letters (as a Base64 String) to Google's Document AI service. Once the data is returned, it is stored in data_documentfile_entities, along with meta data that we receive from SwitchX. This service also queues the NewDevelopmentDB service
- factory.service/package.core/ObjServiceNewDevelopmentDB.py
- This Service processes data from data_documentfile_entities for a single transaction. During processing, the data is formatted, and stored to a series of tables labelled ndd_{tablename}. We store the data according to SX's database design.
- factory.webhook/ObjHookDevelopments.py, def_webhook.WebHookCode = "DEVELOPMENTS"
- This is the WebHook used by SwitchX and their partners to query the data in the ndd_{tablename} tables. When this WebHook is called, the caller must identify themselves with an AuthGroup (located in the request header), which corresponds to an ORG_LEVEL_ID (see ORG_LEVEL_HIERARCHY table). The WebHook functions in two ways:
- Broad Search: The user can send a request with a search string (in the body of the payload). We then search through developments in ndd_prop_development and apply a fuzzy match to the development names and try to return matching developments with some details and a Development ID.
- Narrow Search: The user can send a request with the development ID in the URL. In this case, we return more detailed results relating to the specifically requested development.
- Issue: Processes spawn too many stagnant connections to the database, which then stops accepting new connections and the processes are no longer able to access the DB. Causes fatal errors and unusability.
- Remedy:
supervisorctl restart all => run command to reset processes. Will close all stagnant connections. Is currently an issue due to DB also being used by Prism project.
- Issue: Machine storage has filled up from processing package letters. Machine bricks because there is no storage left. Documents are stored in
local.documents/admin/axion/temp. Issue may initially present as a failed MongoDB connection.
- Remedy:
- Currently, there is a cron job running weekly to clear out storage in the above documents.
- If the cronjob fails, the following command can be run to clear out the storage.
sudo rm /home/admin/axion/local.documents/temp/*
- Issue: Users will sometimes forget their passwords and request a password reset (for which there is already a form). This will generally be in the form of Cody reaching out.
- Remedy:
- Generate a random string as a password (can be a password generator), then:
SELECT CONCAT("~", SHA2("RANDOM_STRING", 224)).
- In sys_user, find the user and paste the encrypted string in
password field.
- Send user new password.