Search…
🏃♀️ Get started
🏄🏻♀️ Integration Tutorials
🏗️ Creating streams
🔧 Editing streams
🎛️ Monitoring streams
⚙️ Administration
Go-live best practice
The aim of this guide is to provide best practices when making a stream in the current Control Panel set up. Currently we don't have access to the Stream Wizard and validation in all sections of the Control Panel is being investigated, but for now we would like to provide a guide to the best way to create a stream with with the environment that we currently have.
The purpose of this is to reduce the risks of hiccups like we have seen recently when onboarding and launching new clients and maintaining current clients with as little interruption to service as possible so that we can help them develop trust in the system.
File name formatting - FTP destination file
Description: This section is created to avoid the risk of overriding a destination file, if two files with the same name format are processed at the same time.
Best Practice: When creating a file name always use a contextual prefix and use ##EVENTID## to ensure the naming convention is unique, this can also be combined with PPK values.
Example:
1
filenameformat = "Order_##EVENTID##.xml"
Copied!
Track entity in the core
Description: Every time we sync an entity we should store the entity in the core. This is how we identify if an entity already exists and the system defines if an event should be aborted. The purpose is to avoid duplications of orders.
Best Practice: Always make sure there is a tracked PPK, for examples on how to setup tracking have a look at creating new transformations.
To avoid duplications in the control panel also make sure to check if the entity i.e order already exists. This can be done using the below settings:
1
"--abort_check": {
2
"*ppk*": "id",
3
"*post_format*": [
4
{
5
"key_lookup": {
6
"*match*": "s_id",
7
"*pluck*": "d_id",
8
"*on_match*": "abort",
9
"*on_fail*": "ppk"
10
}
11
}
12
]
13
},
Copied!
Source Filters
Description: When retrieving entities from Shopify, filters should be added to source settings to ensure we are getting the correct entities.
Best Practice: When creating a stream that will be getting orders from Shopify ensure to add the correct filters to get the right data. Important: note that the schedule delta should never be zero (e.g. the time filter should never match the schedule) and that if we make the window for picking up too short we risk missing orders. Another thing to consider is that if the window is too long for example over 3 hours or even days, for a busy store this will cause an unnecessary amount of aborted events.
1
filter_created_at_min = "CURRENT-2 hour"
2
filter_created_at_max = "CURRENT"
3
filter_financial_status = "paid, partially_refunded"
4
filter_fulfillment_status = "unfulfilled"
5
filter_status = "open"
Copied!
Archive files after processing
Description: We need to ensure that the client and HiCo/ET always have visibility of files after they have been removed from FTP. The reason for this is if there is an issue with a file and we need to access to figure out what went wrong.
Best Practice: Ensure we have the file backed up by moving the file to an archive folder. To do this add the following setting:
1
move_remote_file = "archive"
Copied!
It is important to make sure the 'archive' folder exists on the clients FTP. At the moment our system does not have the capability to create a new directory and even so we might not always have to correct permissions to create a new folder. Therefore this will need to be discussed with the client.
Manual peer-to-peer validation
At the moment we don't have automatic validation nor the capability of creating drafts in place for adding settings to streams, sources and destinations. As best practice therefore when you have created or updated a stream, please have someone else look at it and sign off on it just to double check that there are no errors. Checklist for reviewing a stream:
- Stream, Source and destination settings (checking filters and ensuring no spelling mistakes)
- Looking over the transformation file to make sure that there is no obvious errors