¶ Makro architecture
Our back office involves a lot of disparate systems, database systems and
flows of data, and the macro architecture of the business seems to be in a bit
of flux. If we leave it to emerge spontaneously from our independent choices
it will make adding new components much more complex.
Most of these concepts have already been locked in but some questions have
resurfaced. In this context we need to review how we specify how we
interoperate going forward:
¶ Data transfer
¶ We need to pin down how we transfer data between systems as we move on from a REST only design.
I'll quickly review what we have in place.
¶ REST - Representational State Transfer
- REST APIs is used to act as a bridge between client and server. The client requests data from the server, and the server sends back a response. A micro_service can act as either a client or server
- With the REST API, you request and await a response. It is also done on demand.
- With REST APIs, you can store data in the database on the server.
- The REST API is unidirectional, i.e., you can only send or receive a response at a time.
- You are typically expecting a response back when using REST APIs.
- API technologies like REST would require multiple HTTP calls to access data from multiple sources.
¶ Data flow
Modulr callback JSON
- JSON stands for JavaScript Object Notation
- JSON is a lightweight format for storing and transporting data
- JSON is often used when data is sent from a server to a web page
- JSON is "self-describing" and easy to understand
¶ XML vs JSON
- JSON is a data interchange format and only
provides a data encoding specification. - XML is a language to specify custom markup languages,
and provides a lot more than data interchange.
¶ AVRO specification for a Modulr callback
{
"name": "ModlrHook",
"type": "record",
"namespace": "intelect.modlr.avro",
"fields": [
{
"name": "Type",
"type": "string"
},
{
"name": "Payee",
"type": {
"name": "Payee",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "Address",
"type": {
"name": "AddressFields",
"type": "record",
"fields": []
}
},
{
"name": "Identifier",
"type": {
"name": "Identifier",
"type": "record",
"fields": [
{
"name": "Type",
"type": "string"
},
{
"name": "SortCode",
"type": "string"
},
{
"name": "AccountNumber",
"type": "string"
}
]
}
}
]
}
},
{
"name": "Payer",
"type": {
"name": "Payer",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "Address",
"type": {
"name": "Address",
"type": "record",
"fields": []
}
},
{
"name": "Identifier",
"type": {
"name": "IdentifierFields",
"type": "record",
"fields": [
{
"name": "Type",
"type": "string"
},
{
"name": "SortCode",
"type": "string"
},
{
"name": "AccountNumber",
"type": "string"
}
]
}
}
]
}
},
{
"name": "Amount",
"type": "string"
},
{
"name": "EventId",
"type": "string"
},
{
"name": "Currency",
"type": "string"
},
{
"name": "DateTime",
"type": "string",
"logicalType": "date"
},
{
"name": "AccountId",
"type": "string"
},
{
"name": "EventName",
"type": "string"
},
{
"name": "EventTime",
"type": "string",
"logicalType": "date"
},
{
"name": "PaymentAppliedTime",
"type": "string",
"logicalType": "date"
},
{
"name": "PayerName",
"type": "string"
},
{
"name": "PaymentId",
"type": "string"
},
{
"name": "SchemeInfo",
"type": {
"name": "SchemeInfo",
"type": "record",
"fields": [
{
"name": "Id",
"type": "string"
}
]
}
},
{
"name": "TransactionId",
"type": "string"
},
{
"name": "PaymentReference",
"type": "string"
},
{
"name": "AccountExternalRef",
"type": "string"
},
{
"name": "OriginatedOverseas",
"type": "boolean"
},
{
"name": "OriginatingPayment",
"type": {
"name": "OriginatingPayment",
"type": "record",
"fields": [
{
"name": "Institution",
"type": {
"name": "Institution",
"type": "record",
"fields": [
{
"name": "Id",
"type": "string"
}
]
}
}
]
}
}
]
}
¶ AVRO
AVRO is used to formally specify data packets' structure and the data within. This is in contrast with the informal data packages in JSON.
- It has a direct mapping to and from JSON
- It has a very compact format. The bulk of JSON, repeating every field name with every single record, is what makes JSON inefficient for high-volume usage.
- It is very fast.
- It has great bindings for a wide variety of programming languages including Java and Python
- It has a rich, extensible schema language defined in pure JSON
- It has the best notion of compatibility for evolving your data over time.
- Untagged data: Since the schema is present when data is read, considerably less type information need to be encoded with data, resulting in a smaller serialization size.
- Avro makes sense for structured interchanges, GraphQL fall outside its scope.
¶ Modulr call callback sample
{
"Type": "PI_FAST",
"Payee": {
"Name": "Harry Potter",
"Address": {},
"Identifier": {
"Type": "SCAN",
"SortCode": "040010",
"AccountNumber": "00001457"
}
},
"Payer": {
"Name": "Hermione Granger",
"Address": {},
"Identifier": {
"Type": "SCAN",
"SortCode": "203002",
"AccountNumber": "00004588"
}
},
"Amount": "6.00",
"EventId": "7a1b81bc-5d5c-4045-9099-66b6ea841969",
"Currency": "GBP",
"DateTime": "2020-01-01T16:38:06+0000",
"AccountId": "A120C8D3",
"EventName": "PAYIN",
"EventTime": "2020-01-01T16:38:06+0000",
"PaymentAppliedTime": "2020-01-01T16:38:06+0000",
"PayerName": "Hermione Granger",
"PaymentId": "P12000MWF8",
"SchemeInfo": {
"Id": "FP00009O144093771020200101826070123"
},
"TransactionId": "T12000N5TJ",
"PaymentReference": "Payment from Hermione Granger: For herbs",
"AccountExternalRef": "HOGWARTS ACADEMY",
"OriginatedOverseas": false,
"OriginatingPayment": {
"Institution": {
"Id": "070436"
}
}
}
¶ GraphQL
- In a REST architecture, the client makes an HTTP request and data is sent as an HTTP response, while in GraphQL, the client requests data with queries.
- It’s also worth noting that in REST, the structure of the request object is defined on the server. In GraphQL, you define the object on the client
- The main benefit of using GraphQL is the ability to send a query that specifies only the information you need and receive exactly that. However, you can achieve this same effect using REST, by passing the name of the fields you want to use in the URL, then implementing the parsing and returning logic yourself.
- REST can do much of what GraphQL does It’s important to keep in mind that GraphQL is an alternative to REST for developing APIs, not a replacement.
¶ Salesforce
- Salesforce message bus is based on Kafka, at least conceptially.
- The Salesforce Connector integrates Salesforce.com with Kafka and includes
both a sink connector and a source connector. - The Salesforce source connector provides the capability to capture changes from
Salesforce.com via Salesforce Streaming API PushTopics or Salesforce Enterprise Messaging
Platform Events. - PushTopics provide the ability to subscribe to create, update, delete, and undelete events
related to Salesforce Objects (SObjects). - Subscribing to sales force events might make our layer design simpler.
https://docs.confluent.io/kafka-connectors/salesforce/current/overview.html