# Create your first integration Automating the streaming of data between a Source and Destination system using HighCohesion is done by creating a Stream. ![The anatomy of a stream](/files/-MNdHhqmuIBiLH5VHiCV) {% hint style="info" %} A system here refers to any type of source and destination, be it an a specific 3rd party application, or something more generic like a FTP location, other file storage types or a communication endpoint like email or Slack. {% endhint %} **A Stream is created by combining three key components:**\ 1\. **The Source function** - The Source Function decides what data should be pulled from the Source system and when. You will need a System Key in order to pull the data from your Source system.\ 2\. **The Transformation** - The Transformation is responsible for transforming the data of the Source system to match the data of the Destination System.\ 3\. **The Destination** - The Destination Function decides what data should be pulled from the Destination system and when. You will need a System Key in order to post the data to your Destination system. ## Use case based Stream setup guide **In the next 5 steps we will walk through a use case based guide of how to integrate and automate the flow of data between Shopify & a 3rd party logistics company's FTP-location.** Unless you're a Developer using HighCohesion's SDK, all configuration for HighCohesion is done from the Control Panel. # Step 1 - Creating your Source & Destination System keys ### Overview Gaining secure access to the systems that you need to integrate is central to building a working solution on HighCohesion. Our **Keys** system allows you to securely add access credentials in to the HighCohesion system via the Control Panel. ### Security Security is always the main concern when it comes to system access credentials. This is why we have taken extra precautions when it comes to the transfer and storage of your **Keys** in our system. Our system uses: * **SSL encryption** between the browser and the server ensuring that the data transmitted is encrypted. * **At rest encryption** in the secure storage facility where the keys are stored. * Method of **Least Privilege Permissions** in our access levels to the key storage area, meaning only the correct access is granted to use the credentials. {% hint style="info" %} Quite often the easiest way in to the system is via your user account in the control panel. Please ensure that you activate the [multi factor authentication](https://app.gitbook.com/@highcohesion/s/simplify-data/~/drafts/-MNgusOFB42VKDX8G_k6/getting-started/creating-an-account/login-and-security/two-factor-authentication-2fa-setup) option on your account as soon as it is setup to increase the security on your login. {% endhint %} ### Add a new key To add a new set of credentials in to your HighCohesion account, please follow these steps: 1. From the control panel, go to the [**Keys**](https://control.highcohesion.com/resources/keys) section. 2. At the top right hand side of the page, press the button that says **Create Key**. 3. This will then load the new Key form. On this form complete the following options: 1. **System** - This is a drop down list of the systems available in the HighCohesion system. Please select from this list the system that the credentials are required for. When you select a system, a *Key Template* will automatically load in to the credentials box below, this will give you a template for the credentials that you need to add. 2. **Title** - Give the Key a title that is relative to the credentials, for example: "*Live UK Shopify*" or "*GBP Staging Store*" etc. 3. **Credentials** - This box will contain a pre-set JSON credentials template. You will need to complete the template by adding in the correct data to the right hand values in the template. {% hint style="info" %} For this use case you will find a Guide on how to attain the correct Data for your Source system (Shopify) [here](https://app.gitbook.com/@highcohesion/s/simplify-data/~/drafts/-MNgusOFB42VKDX8G_k6/getting-started/creating-your-first-stream/step-1-creating-your-source-and-destination-system-keys/shopify-key) and your Destination System (FTP) [here](https://app.gitbook.com/@highcohesion/s/simplify-data/~/drafts/-MNgusOFB42VKDX8G_k6/getting-started/creating-your-first-stream/step-1-creating-your-source-and-destination-system-keys/ftp-key) {% endhint %} 4\. Once the credentials have been added, press the save button at the bottom left of the page.\ 5\. The new credentials will now be securely saved and available to use in the system. # Shopify Key Guide to setting up a Shopify private app key Shopify private apps are a quick and easy way to setup a connection between HighCohesion and your Shopify system. Private apps can be added to both Shopify and Shopify+ subscriptions, both follow the same guide below: ### In Shopify The first steps are to be completed from within your Shopify admin area. 1. From your Shopify admin, go to [**Apps**](https://www.shopify.com/admin/apps). 2. Scroll to the bottom of the page and click **Manage private apps**. 3. Click **Create a new private app**. 4. In the **App details** section enter the following details: 1. In the name field, enter: **High Cohesion** 2. In the contact email address, enter: **** 5. In the **Admin API** section, select the areas of your store that you want the to access via the HighCohesion system. Please use logic when selecting the options in the app, for example if you only want to extract order data from Shopify, only select **Read** on the orders API option etc. \ \ Four permissions are selected by default: **Read products and collections**, **Read and modify customers**, **Read and modify checkouts**, and **Read blogs and articles**. 6. When you're done, click **Save**. 7. The API key, secret and password for your app are now displayed on screen. {% hint style="warning" %} Treat the API key and password like you would any other password, because whoever has access to these credentials has full API access to the store. After you have the required credentials, the private app can access your store's data. {% endhint %} ### In the Control Panel 1. From the Control Panel, navigate to the menu item "Keys" 2. On the "Keys" page, click the button "Create Key" 3. Enter the System the Key is related to (in this example Shopify) 4. When selecting the system the Content box will be directly loaded with a template of which data is required. 5. Copy and paste the correct data directly from Shopify 6. Click "Create Key" Congrats, your Key has been created! # FTP Key How to gather data for FTP and sFTP keys. One of the most popular connections is FTP or sFTP, a method normally used for exchanging files between servers. This is often used where companies require CSV, XML or JSON files to be deposited or collected from specified servers at certain times. Adding a FTP connection to HighCohesion is simple, you will require the following details about the FTP server that is being connected to: * **Host / Address**\ This can be either a fully qualified domain name e.g. ftp.highcohesion.com or an IP address e.g. 192.168.2.1 * **Username**\ The username required to connect. * **Password**\ The password required to connect. * **Port**\ The port required to connect, please note that the default port for FTP is 21 and the default port for sFTP is 22. * **Passive**\ This is a `true / false` option, allows a passive connection setting to be activated. * **Remote Path**\ This is an optional setting that allows a base path for your connection to be set. For example `/export/orders/` {% hint style="info" %} Please note that the Remote Path setting can be overridden and not appended to in Source/Destination and Stream settings. {% endhint %} # Step 2 - Select a Source Function ## **What is a Source Function?** The Source Function decides what data should be pulled from the Source system and when. ## Configure your Source Through the Control Panel you can easily access the page "Sources" in the left side column. This is the page where you will find, edit and create your Sources. If you want to configure a Source from an already existing System, you can do so by navigating to the "Create Source" button. This is how you should read and populate the Create Source page: * **Title** - Give the Source function a title that is relative to the source, for example: "*Shopify GET Orders*" or "*GBP Store Orders*" {% hint style="info" %} For this example, name it "Shopify GET Orders Test" {% endhint %} * **System** - This is a dropdown list of the Systems available in the HighCohesion system. You can see the full list of available systems [**here**](https://app.gitbook.com/@highcohesion/s/simplify-data/~/drafts/-MNi8z8epW9ji6t4XV2U/control-panel/setup/systems/system-library). When you select a System from the list, the page will automatically load the available functions in the Source Function box below. {% hint style="info" %} For this example, select Shopify {% endhint %} * **Function** - This is a dropdown list of the already available functions in the System you selected in step 2, in this case Shopify.. Please select from this list the functions that you would like to use to consume the data in for the Source configuration. {% hint style="info" %} For this example, select GetOrders {% endhint %} * **Key** - This is a dropdown list of the keys you have added to HighCohesion. Please select the key for the Source system as enabled in Step 1
* **Trigger** - The Trigger defines the logic for when the Source function should be triggered. You can choose from three pre-defined options: * Manual (e.g. the Source is triggered manually from the Control Panel). This option is well suited if the Stream is rarely used, for example Quarterly product imports or similar. * Scheduled (e.g. the Source is triggered by a scheduled Cron Job). This option is well suited for Inventory and Shipment updates. * Webhook (e.g. the Source is triggered in real time by a Source System defined event). This option is well suited for Systems that handles web event well, for example Shopify or other commerce solutions. {% hint style="info" %} For this example, choose scheduled and configure the cron job to run \ every second hour , 0 \*/2 \* \* \* {% endhint %} * **Settings** - The settings give you as a user flexibility to add additional configuration to the Source function if you have selected the triggers "Manual" or "Scheduled". All of the settings added to Source configurations should be entered in TOML format, this makes the settings easy to read, edit and update. For a guide on TOML format please see the link: . {% hint style="warning" %} When pulling data from our Source system using the triggers "Manual" or "Scheduled" we want to ensure we are only pulling the data we are interested in. For this use case we are pulling Orders from Shopify to a 3rd party logistics company's FTP location. The function used to pull the data is "Shopify:GetOrders". If we would use the function without settings we would pull **all Orders stored in Shopify** to the FTP, which is not what we are looking to do. Instead we should consider what filters we should apply to ensure that the 3rd party logistics company only get the information they need, in this case newly placed orders created less than 2 hours ago that are paid or partially paid and unfulfilled. For this example, use the the settings below: {% endhint %} ``` filter_created_at_min = "CURRENT-2 hour" filter_created_at_max = "CURRENT" filter_financial_status = "paid, partially_refunded" filter_fulfillment_status = "unfulfilled" filter_status = "open" ``` # Step 3 - Select a Destination Function ## **What is a Destination Function?** The Destination Function decides what data should be posted to the Destination system and when. ## Configure your Destination Function Through the Control Panel you can easily access the page "Destinations" in the left side column. When entering the Destination page you will see a list of all Public Destinations, as well as your Private ones. If you want to configure a Destination, you can do so by clicking the "Create Destination" button. From the Create Destination page you should fill in the below fields. * **Title** - Give the Destination function a title that is relative to the Destination, for example: "*FTP POST Orders*" or "*GBP Orders to FTP*" {% hint style="info" %} For this example, choose FTP POST Orders Test {% endhint %} * **System** - this is your Destination system (e.g. the System you are posting data to) {% hint style="info" %} For this example, choose FTP {% endhint %} * **Function** - this is your Destination function (e.g. the configuration of what information you will post to the system) {% hint style="info" %} For this example, choose PostFiles {% endhint %} * **Key** - This is a dropdown list of the keys you have added to HighCohesion. Please select the key for the Destination system as enabled in Step 1
* **Trigger** - The Trigger defines the logic for when the Destination function should be triggered. You can choose from three pre-defined options: * **Aggregate** (e.g. the Destination is triggered when all events for a specific Job has been completed). This is well suited for the cases when you want all Events to get posted to your Destination system as one file * **Real time** (e.g. the Destination is triggered when the first Event for a specific Job has been completed). This is well used for Streams with Sources run on webhooks, to directly POST a file to the destination System. * **Scheduled** (e.g. the Destination is triggered based on your defined Cron Schedule). This is rarely used, but could be used for Destination systems that require that Files are POSTED at a specific time. {% hint style="info" %} For this example, please select Aggregate {% endhint %} * **Settings** - this is necessary to provide additional details such as the the FTP location paths, access rights etc. All of the settings added to Destination configurations should be entered in TOML format, this makes the settings easy to read, edit and update. For a guide on TOML format please see the link: . {% hint style="info" %} The below settings states that the destination system format is XML, defines the output file naming convention, instructs the system of where to look for files on the remote location and indicates the root XML element. **For this use case, please add the remote\_path to your test FTP location. For more FTP configurations, see** [**here**](https://app.gitbook.com/@highcohesion/s/simplify-data/control-panel/setup/systems/system-library/ftp) {% endhint %} ``` formatting = "xml" filename_format = "orders_##DATETIMEFORMAT__Ymd_H-i-s__##.xml" remote_path = "" root_attribute = "version=1.0" root_element = "root_order" ``` # Step 4 - Select a Transformation ## **What is a Transformation?** The Transformation is responsible for transforming the data of the Source system to match the data of the Destination System ## Configure your Transformation Through the Control Panel you can easily access the page "Transformations" in the left side column. When entering the Transformation page you will see a list of all Public Transformations, as well as your private ones. If you want to create a Transformation, you can do so by navigating to the Create Transformation button. From the Create Transformation page you should fill in the necessary data. * **Source -** Choose the Source you **created in Step 2** * **Destination** - Choose the Destination you **created in Step 3** * **Title** - Name the Transformation here we suggest to clearly explain what the transformation does {% hint style="info" %} For this example, name the transformation "Shopify to FTP Test" {% endhint %} * **Primary data type** - This field describes what type of data the transformation is transforming (e.g. order, product, inventory etc.) {% hint style="info" %} For this example, name it "order" {% endhint %} * **Content** - this is were the transformation logic will be stored for undertaking the transformation {% hint style="info" %} For this example we will use the below transformation. This transformation maps the Shopify payload-in JSON to match the XML format of the 3rd Party Logistics Company's FTP requirements {% endhint %} ``` { "interfaceDate": { "date": { "*ppk*": "created_at", "*post_format*": [ { "current_timestamp": true, "date_format": { "input": "%s", "output": "%Y%m%d" } } ] }, "time": { "*ppk*": "created_at", "*post_format*": [ { "current_timestamp": true, "date_format": { "input": "%s", "output": "%H%M%S" } } ] }, "GMTOffset": { "*static_value*": "GMT" } }, "orders": { "order": { "currency": { "*ppk*": "currency" }, "express_delivery": { "*static_value*": 0 }, "preferred_carrier": { "*static_value*": null }, "preferred_carrier_service": { "*static_value*": null }, "delivery_instructions": { "*static_value*": null }, "gift_message": { "*static_value*": null }, "tax_included": { "*ppk*": "taxes_included" }, "email": { "*ppk*": "email" }, "external_customer_reference": { "*static_value*": null }, "spectrum_customer_id": { "*static_value*": null }, "order_number": { "*ppk_tracked*": { "*ppk*": "name", "*tracked_field*": [ "name" ] } }, "--source_id": { "*ppk_tracked*": { "*ppk*": "id", "*tracked_field*": [ "s_id" ] } }, "source_code": { "*static_value*": "W111" }, "created_at": { "*ppk*": "created_at", "*post_fomrat*": [ { "date_format": { "input": "%Y-%m-%dT%H:%M:%S%z", "output": "%s" } } ] }, "sub_channel": { "*static_value*": "uk-website" }, "attributes": { "*static_value*": null }, "billing_address": { "external_reference": { "*ppk*": "billing_address.id" }, "title": { "*ppk*": "billing_address.title" }, "firstname": { "*ppk*": "billing_address.first_name" }, "lastname": { "*ppk*": "billing_address.last_name" }, "company": { "*ppk*": "billing_address.company" }, "street1": { "*ppk*": "billing_address.address1" }, "street2": { "*ppk*": "billing_address.address2" }, "city": { "*ppk*": "billing_address.city" }, "region": { "*ppk*": "billing_address.province" }, "postcode": { "*ppk*": "billing_address.zip" }, "country_id": { "*ppk*": "billing_address.country_code" }, "telephone": { "*ppk*": "billing_address.phone" } }, "shipping_address": { "external_reference": { "*ppk*": "shipping_address.id" }, "title": { "*ppk*": "shipping_address.title" }, "firstname": { "*ppk*": "shipping_address.first_name" }, "lastname": { "*ppk*": "shipping_address.last_name" }, "company": { "*ppk*": "shipping_address.company" }, "street1": { "*ppk*": "shipping_address.address1" }, "street2": { "*ppk*": "shipping_address.address2" }, "city": { "*ppk*": "shipping_address.city" }, "region": { "*ppk*": "shipping_address.province" }, "postcode": { "*ppk*": "shipping_address.zip" }, "country_id": { "*ppk*": "shipping_address.country_code" }, "telephone": { "*ppk*": "shipping_address.phone" } }, "shipping_method": { "*static_value*": null }, "items": { "*list*": "line_items", "*list_fields*": { "item": { "sku": { "*ppk*": "sku" }, "qty": { "*ppk*": "quantity" }, "unit_price": { "*ppk*": "price" }, "line_total": { "*ppk_math*": [ "price", "*", "quantity" ] }, "tax_total": { "*ppk*": "tax_lines.0.price" } } } }, "totals": { "subtotal": { "*ppk*": "total_price" }, "shipping_amount": { "*ppk*": "shipping_lines.0.price" }, "shipping_tax": { "*ppk*": "shipping_lines.0.tax_lines.0.price" }, "gift_wrap_amount": { "*static_value*": null }, "gift_wrap_tax": { "*static_value*": null }, "tax_amount": { "*ppk*": "total_tax" }, "grand_total_net": { "*ppk_math*": [ "total_price", "-", "total_tax" ] }, "grand_total_gross": { "*ppk*": "total_price" } }, "payments": { "payment": { "type": { "*static_value*": "CARD" }, "amount": { "*ppk*": "total_price" }, "token": { "*static_value*": null } } } } } } ``` # Step 5 - Put it all together to create a Stream ## **What is a Stream?** The Stream is the combination of all components necessary to automate the streaming of data between a Source and a Destination system. ## Create your Stream Through the Control Panel you can easily access the page "Streams" in the left side column. When entering the Streams page, click on the "Create Stream" button. * **Active / Test mode -** Select Active mode * **Title -** Name the Stream, we suggest to name it following the Naming convention: \[Data Type : Source System to Destination System] {% hint style="info" %} For this example, name the stream "Orders : Shopify to FTP" {% endhint %} * **Description -** Here you can input some free text field information about the Stream, to remind yourself what it's used for {% hint style="info" %} For this example, Orders from Shopify to 3rd party logistics company's FTP location \[test] {% endhint %} * **Source -** Select the Source you created in Step 1 from the dropdown list * **Transformation -** Select the Transformation you created in Step 3 from the dropdown list * **Destination -** Select the Destination you created in Step 2 from the dropdown list * **Data type -** Use a word to describe the data type (e.g. product, inventory) {% hint style="info" %} For this example, name the data type "order" {% endhint %} * **Settings -** This field describes what type of data the stream is transferring (e.g. order, product, inventory etc.) **Click "Create Stream". Congratulations, you have setup your first Stream!** # Step 6 - Run your stream ### Manual Stream execution The **HighCohesion** system allows for manual execution of a stream. To run a service manually, please follow these steps:‌ 1. From the control panel, go to the Streams section 2. Here you will see a list of streams that you have setup under your account, in this list find the stream that you would like to update. 3. On the far right hand side of the row, there is a small eye / view icon. 4. Press this icon to load the view screen. 5. On the view stream screen, look to the top right and you will see a dropdown entitled Select Actions and a small play button. 6. From the actions dropdown, select Manual Run. 7. Press the play button. 8. You will then get an options popup, enter any required additional settings in this box. 9. Press the pink Run Action button. 10. The stream will then be queued to run as soon as possible. ‌ ### Execution hierarchy A HighCohesion stream allows for data streaming to occur, however the stream is simple the configuration for for the the data stream. Once activated, either manually or via a schedule, the stream will lead to the creation of the following (*in order of hierarchy*).‌ * **Jobs** - A stream will create a job. When a stream is executed, it will create a Job. A manual job is set to run as soon as possible (in line with system capacity at the time) and a scheduled job is set to run at the next defined interval. * **Events** - A job creates between zero and many events. Events are created when the job finds relevant data in the source system. If no data is found, no events will be created. * **Entities** - Events create or update entities. Entities relate to data that is contained within an event. We recommend reading the information about [**Reporting**](https://app.gitbook.com/@highcohesion/s/control-panel/control-panel-functions/reporting) that will give further insight in to how this data is used by **HighCohesion** or can be used by you to review what is happening when the stream is executed.‌ # Go-live best practice The aim of this guide is to provide best practices when making a stream in the current Control Panel set up. Currently we don't have access to the Stream Wizard and validation in all sections of the Control Panel is being investigated, but for now we would like to provide a guide to the best way to create a stream with with the environment that we currently have. The purpose of this is to reduce the risks of hiccups like we have seen recently when onboarding and launching new clients and maintaining current clients with as little interruption to service as possible so that we can help them develop trust in the system. ### **File name formatting - FTP destination file** **Description:** This section is created to avoid the risk of overriding a destination file, if two files with the same name format are processed at the same time. **Best Practice:** When creating a file name always use a contextual prefix and use ##EVENTID## to ensure the naming convention is unique, this can also be combined with PPK values. **Example:** ``` filenameformat = "Order_##EVENTID##.xml" ``` ### **Track entity in the core** **Description:** Every time we sync an entity we should store the entity in the core. This is how we identify if an entity already exists and the system defines if an event should be aborted. The purpose is to avoid duplications of orders. **Best Practice:** Always make sure there is a tracked PPK, for examples on how to setup tracking have a look at [creating new transformations](https://app.gitbook.com/@highcohesion/s/users/transformations/new-transformations/example-2#add-in-tracking). To avoid duplications in the control panel also make sure to check if the entity i.e order already exists. This can be done using the below settings: ``` "--abort_check": { "*ppk*": "id", "*post_format*": [ { "key_lookup": { "*match*": "s_id", "*pluck*": "d_id", "*on_match*": "abort", "*on_fail*": "ppk" } } ] }, ``` ### Source Filters **Description:** When retrieving entities from Shopify, filters should be added to source settings to ensure we are getting the correct entities. **Best Practice:** When creating a stream that will be getting orders from Shopify ensure to add the correct filters to get the right data. Important: note that the schedule delta should never be zero (e.g. the time filter should never match the schedule) and that if we make the window for picking up too short we risk missing orders. Another thing to consider is that if the window is too long for example over 3 hours or even days, for a busy store this will cause an unnecessary amount of aborted events. ``` filter_created_at_min = "CURRENT-2 hour" filter_created_at_max = "CURRENT" filter_financial_status = "paid, partially_refunded" filter_fulfillment_status = "unfulfilled" filter_status = "open" ``` ### **Archive files after processing** **Description:** We need to ensure that the client and HiCo/ET always have visibility of files after they have been removed from FTP. The reason for this is if there is an issue with a file and we need to access to figure out what went wrong. **Best Practice:** Ensure we have the file backed up by moving the file to an archive folder. To do this add the following setting: ``` move_remote_file = "archive" ``` {% hint style="info" %} It is important to make sure the 'archive' folder exists on the clients FTP. At the moment our system does not have the capability to create a new directory and even so we might not always have to correct permissions to create a new folder. Therefore this will need to be discussed with the client. {% endhint %} ### Manual peer-to-peer validation At the moment we don't have automatic validation nor the capability of creating drafts in place for adding settings to streams, sources and destinations. As best practice therefore when you have created or updated a stream, please have someone else look at it and sign off on it just to double check that there are no errors. Checklist for reviewing a stream: * Stream, Source and destination settings (checking filters and ensuring no spelling mistakes) * Looking over the transformation file to make sure that there is no obvious errors # Shopify with Descartes Peoplevox Tutorial on how to setup a Shopify with Descartes Peoplevox integration via HighCohesion. This tutorial will guide you through how to setup the standard Shopify to Descartes Peoplevox integration via the HighCohesion system. The integration will contain four data streams: 1. **Products** - Shopify to Descartes Peoplevox 2. **Stock Level** - Descartes Peoplevox to Shopify 3. **Sales Orders** - Shopify to Descartes Peoplevox 4. **Shipments** - Descartes Peoplevox to Shopify The video walkthrough of this integration is shown below: {% embed url="" %} Peoplevox x Shopify setup video {% endembed %} ## Prerequisites This tutorial should be followed to create a **Shopify to Descartes Peoplevox** integration in just a few quick steps. These steps assume that you have the following accounts and information to hand: * **Shopify Store** - An active Shopify or Shopify Plus store. * **Descartes Peoplevox Account** - A Descartes Peoplevox account and have the login information to hand. * **HighCohesion Account** - A HighCohesion account. ([**get one here**](https://www.highcohesion.com/sign-up)) ## Guide All of the following steps should be taken in order. Unless specified the steps assume that you are logged in to the [**HighCohesion control panel**](https://control.hico.io/). ### 1. Keys The first step is to add the system account keys. The keys are what the HighCohesion system use to connect to the 3rd party system, for example API credentials or login details. In the control panel click on the [**Keys link**](https://control.hico.io/resources/keys) from the left navigation and then press the Add Key button located at the top right of the page. The next page will be the create key form, you will need to enter the following settings before pressing save. * **Title** - A title for the key your creating, e.g. "*Descartes Peoplevox Key*" * **System** - Select the system from the dropdown list, in this case "Descartes Peoplevox" * **Credentials** - Add the following details to the credentials section: * client\_id = Your Descartes Peoplevox Client ID * username = Your Descartes Peoplevox Username * password = Your Descartes Peoplevox Password * env = The Descartes Peoplevox environment, this is normally `wms` or could alternatively be `qac` if using the QA environment. Once you have pressed save on the first create key form you will be returned to the list of existing keys. You'll need to press the Create Key link once more to add the Shopify key. For the link to work you will need to activate a "Private App" connection to Shopify. To do this,[ follow this guide](/creating-streams/keys-1/key-library/shopify-private-apps-1). * **Title** - A title for the key your creating, e.g. "*Shopify Key*" * **System** - Select the system from the dropdown list, in this case "*Shopify*" * **Content** - Add the following details to the credentials section: * store = Your shopify URL, e.g. shopifystore.myshopify.com * api\_key = Copied from the private app screen in Shopify. * password = Copied from the private app screen in Shopify. * shared\_secret = Copied from the private app screen in Shopify. ### 2. Sources Our next step is to define the four sources that will be required for our four data streams. Navigate to the [**Sources link**](https://control.hico.io/resources/sources) from the left hand navigation in the HighCohesion control panel, then select the "Create Source" button (located at the top right side of the screen. The next page will load the source form, in that form complete the following information: * **Title** - Create a title for the source, normally "Shopify Get Sales Orders". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "GraphQL\GetOrders::fromShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Key". * **Trigger** - Select the "Manual" option. (*we will activate this later*) * **Settings** - Add the following setting so that the source will only collect orders created in the last 15 minutes that are marked as paid. ``` filter_created_at-gt = "CURRENT-15 minutes" filter_financial_status = "paid" ``` Repeat the above three more times with the follow changes: * **Title** - Create a title for the source, normally "Shopify Get Variants". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "GetVariants::fromShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Key". * **Trigger** - Select the "Manual" option. * **Settings** - `filter_updated_at_min = 'CURRENT-15 minutes'`
* **Title** - Create a title for the source, normally "Descartes Peoplevox Get Shipments". * **System** - Select "Descartes Peoplevox" from the dropdown. * **Function** - Select "ReceiveDespatchPackageDespatched::fromPvx()" from the dropdown. * **Key** - Select the name of the key defined in step one "Descartes Peoplevox Key". * **Trigger** - Select the "Dynamic Webhook" option. * **Settings** - `post_params = "despatch={Despatch.DespatchNumber}"` * **Title** - Create a title for the source, normally "Descartes Peoplevox Get Stock Level". * **System** - Select "Descartes Peoplevox" from the dropdown. * **Function** - Select "ReceiveAvailabilityChanges::fromPvx()" from the dropdown. * **Key** - Select the name of the key defined in step one "Descartes Peoplevox Key". * **Trigger** - Select the "Static Webhook" option. * **Settings** - `post_params = "sku={ItemCode}&newQty={Available}"` ### 3. Destinations Next up is the four destinations for each stream to send data. To add the destination, click on the [**Destinations link**](https://control.hico.io/resources/destinations) from the left hand nav and then press the "Create Destination" button at the top right of this page. Then enter the following details in to the form: * **Title** - Create a title for the destination, normally "Descartes Peoplevox Post Orders". * **System** - Select "Descartes Peoplevox" from the dropdown. * **Function** - Select "PostOrders::toPeoplevox()" from the dropdown. * **Key** - Select the name of the key defined in step one "Descartes Peoplevox Key". * **Trigger** - Select the "Real Time" option. * **Aggregate Events** - Tick this option * **Pagination** - Set to 50 * **Settings** - Leave empty. Repeat the above step three more times to create the remaining three destination configurations: * **Title** - Create a title for the destination, normally "Descartes Peoplevox Post Items". * **System** - Select "Descartes Peoplevox" from the dropdown. * **Function** - Select "PostItems::toPeoplevox()" from the dropdown. * **Key** - Select the name of the key defined in step one "Descartes Peoplevox Key". * **Trigger** - Select the "Real Time" option. * **Aggregate Events** - Tick this option * **Pagination** - Set to 50 * **Settings** - Leave empty.
* **Title** - Create a title for the destination, normally "Shopify Post Inventory Level". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "PostInventoryLevel::toShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Key". * **Trigger** - Select the "Real Time" option. * **Settings** - Leave empty.
* **Title** - Create a title for the destination, normally "Shopify Post Shipment". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "PostShipment::toShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Key". * **Trigger** - Select the "Real Time" option. * **Settings** - Leave empty. ### 4. Streams Step four in the process is to configure the four Streams. The stream in HighCohesion terms is the part that brings all of the settings together in to an active integration. To set this up, select the [**Streams**](https://control.hico.io/resources/streams) option from the left hand navigation. Then on the next page press the "Create Stream" button at the top right side of the page. On the next page, complete the following settings: * **Title** - Create a title for the stream: "\[SO-1001] Shopify x Descartes Peoplevox Sales Orders". * **Description** - Optional, e.g. "Shopify to Descartes Peoplevox for sales orders." * **Source** - From the dropdown, select the destination "Shopify Get Sales Orders", as created in step 2 above. * **Transformation** - from the dropdown select the public transformation file "Shopify x Descartes Peoplevox :: Sales Orders". * **Destination** - From the dropdown, select the destination "Descartes Peoplevox Post Orders", as created in step 3 above. * **Data Type** - For the data type enter: sales\_order * **Active** - Tick this option to activate the stream. Once the first stream has been created, repeat a further three times to create the remaining streams. * **Title** - Create a title for the stream, e.g "\[PR-2001] Shopify x Descartes Peoplevox Items". * **Description** - Optional, e.g. "Shopify to Descartes Peoplevox for items." * **Source** - From the dropdown, select the destination "Shopify Get Variants", as created in step 2 above. * **Transformation** - from the dropdown select the public transformation file "Shopify x Descartes Peoplevox :: Items". * **Destination** - From the dropdown, select the destination "Descartes Peoplevox Post Items", as created in step 3 above. * **Data Type** - For the data type enter: product * **Active** - Tick this option to activate the stream.
* **Title** - Create a title for the stream, e.g "\[IN-3001] Descartes Peoplevox x Shopify Inventory Level". * **Description** - Optional, e.g. "Descartes Peoplevox to Shopify for inventory levels." * **Source** - From the dropdown, select the destination "Descartes Peoplevox Get Stock Level", as created in step 2 above. * **Transformation** - from the dropdown select the public transformation file "Descartes Peoplevox x Shopify :: Sales Orders". * **Destination** - From the dropdown, select the destination "Shopify Post Inventory Level", as created in step 3 above. * **Data Type** - For the data type enter: inventory * **Active** - Tick this option to activate the stream.
* **Title** - Create a title for the stream, e.g "\[SH-4001] Descartes Peoplevox x Shopify Shipments". * **Description** - Optional, e.g. "Descartes Peoplevox to Shopify for shipments." * **Source** - From the dropdown, select the destination "Descartes Peoplevox Get Shipments", as created in step 2 above. * **Transformation** - from the dropdown select the public transformation file "Descartes Peoplevox x Shopify :: Shipments". * **Destination** - From the dropdown, select the destination "Shopify Post Shipments", as created in step 3 above. * **Data Type** - For the data type enter: shipment * **Active** - Tick this option to activate the stream. ### 5. Install the Peoplevox Webhooks You can install the Peoplevox webhooks directly in the HighCohesion control panel. These will need to be installed to allow the Inventory and Shipment streams to work. This can be done using the following steps: 1. Go to the edit page of the Inventory stream 2. Tick the install webhook button 3. Save the stream 4. Repeat the process for the Shipments stream ### 6. Activation Now that all four streams have been setup, we can start to activate them. We would normally recommend activating the streams in the following order: 1. **\[PR-2001] Shopify x Descartes Peoplevox Items**
2. **\[IN-3001] Descartes Peoplevox x Shopify Inventory Level**
3. **\[SO-1001] Shopify x Descartes Peoplevox Sales Orders**
4. **\[SH-4001] Descartes Peoplevox x Shopify Shipments**
### 7. Smoke tests Once the streams are activated a smoke test can take place. 1. **\[PR-2001] Shopify x Descartes Peoplevox Items**\ This stream can be tested by logging in to the Shopify admin area, then creating a new product. Then in the HighCohesion control panel, navigate to the Items stream page. In the actions drop down, select Manual Trigger and press the play button. This will schedule a new job to be run. Once you are happy that the product has successfully been sent to Peoplevox, the source should be updated to be on a schedule. This can be done by going to the edit page of the Items source and changing the Trigger from Manual to Scheduled. The Trigger Options field will then appear which accepts a cron expression. We recommend that you use `*/15 * * * *` 2. **\[IN-3001] Descartes Peoplevox x Shopify Inventory Level**\ Once the new product has appeared in Descartes Peoplevox, our next smoke test would be to add stock for the items, thus allowing us to see if the stock level is pushed back correctly to Shopify. To do this you will first need to create a Purchase Order for the product in Descartes Peoplevox. Once the Purchase Order has been created, use the Peoplevox Android App to create a Goods Receipt. This goods receipt will add the selected number in stock. \ \ Given that the stock updates are pushed to HighCohesion via notification message we must now wait for Descartes Peoplevox to send that data to us. Once the data is sent you will see the message appear in the events list.
3. **\[SO-1001] Shopify x Descartes Peoplevox Sales Orders**
This stream can be tested by logging in to the Shopify admin area, then creating a new order. Then in the HighCohesion control panel, navigate to the Orders stream page. In the actions drop down, select Manual Trigger and press the play button. This will schedule a new job to be run. Once you are happy that the order has successfully been sent to Peoplevox, the source should be updated to be on a schedule. This can be done by going to the edit page of the Items source and changing the Trigger from Manual to Scheduled. The Trigger Options field will then appear which accepts a cron expression. We recommend that you use `*/15 * * * *`
4. **\[SH-4001] Descartes Peoplevox x Shopify Shipments**
Once the new order has appeared in Descartes Peoplevox, our next smoke test would be to add ship the order, thus allowing us to see if the shipment is pushed back correctly to Shopify. To do this you will first need to create an action for the order in Descartes Peoplevox. Once the action has been created, use the Peoplevox Android App to create a Shipment. Then despatch the shipment in the web client. \ \ Given that the shipment updates are pushed to HighCohesion via notification message we must now wait for Descartes Peoplevox to send that data to us. Once the data is sent you will see the message appear in the events list. ### Summary Congratulations, you are now setup and ready to enjoy a working **Shopify x Descartes Peoplevox** **integration**. ### Optional Extra **Transformation Customisations**\ The most common customisations can be handled in our simple transformation logic. Common customisations include adding custom attributes or tag values in to data flowing between the systems. Follow **this link** to our guide specific to Descartes Peoplevox transformation file editing. **Returns Stream** Additionally, you can set up a stream to sync your returns data from Descartes Peoplevox into Shopify. You can achieve this by setting up an additional source, destination and stream with the following settings: Source: * **Title** - Create a title for the source, normally "Peoplevox Returns". * **System** - Select "Peoplevox" from the dropdown. * **Function** - Select "ReceiveReturns::fromPvx()" from the dropdown. * **Key** - Select the name of the key defined in step one "Peoplevox Key". * **Settings -** post\_params='salesOrderNumber={SalesOrder.SalesOrderNumber}\&itemCode={ReturnItems.ItemType.ItemCode}\&quantityReturned={ReturnItems.QuantityReturned}\&comments={ReturnItems.Comments}\&description={ReturnItems.ReturnsCondition.Description}\&isReusable={ReturnItems.ReturnsCondition.IsReusable}\&returnReason={ReturnItems.ReturnReason.Description}' Destination: * **Title** - Create a title for the destination, normally "Shopify Returns". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "PostRefund::toShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Key". * **Settings -** Leave blank Stream: * **Title** - Create a title for the stream: "\[RE-5001] Descartes Peoplevox x Shopify Returns". * **Description** - Optional, e.g. "Descartes Peoplevox to Shopify returns." * **Source** - From the dropdown, select the source "Peoplevox Returns" * **Transformation** - from the dropdown select the public transformation file "Descartes Peoplevox x Shopify :: Returns". * **Destination** - From the dropdown, select the destination "Shopify Post Returns" * **Data Type** - For the data type enter: return * **Active** - Tick this option to activate the stream. You will then need to install the Webhook in Peoplevox using the same process as described earlier in this guide ## Common Questions **Can I customise the attributes in the transformation files specific to my store?** \ Yes! Follow this link to our guide about how to customise transformation files with specific examples for Descartes Peoplevox. # Shopify with FTP Shopify & FTP integrations are some of the most popular and easy to customise integrations. The simplicity of FTP or sFTP connections make them popular with 3PL warehouse system. The generic FTP functions contained inside of HighCohesion allow you to customise file pickup and drop off on to any FTP or sFTP server. The most common implementations here result in CSV or XML files on the FTP location being imported or exported to Shopify. # Shopify with FTP Export Shopify Sales Orders to FTP as XML or CSV files. This tutorial will guide you through a simple Sales Order export from a Shopify system that results in a CSV file placed on an FTP server. ## Prerequisites This tutorial should be followed to create a **Shopify to FTP** integration in just a few quick steps. These steps assume that you have the following accounts and information to hand: * **Shopify Store** - An active Shopify or Shopify Plus store. * **FTP or sFTP Account** - A working FTP or sFTP server with the login information to hand. * **HighCohesion Account** - A HighCohesion account. ([**get one here**](https://www.highcohesion.com/sign-up)) ## Guide All of the following steps should be taken in order. Unless specified the steps assume that you are logged in to the [**HighCohesion control panel**](https://control.hico.io/). ### 1. Keys The first step is to add the system account keys. The keys are what the HighCohesion system use to connect to the 3rd party system, for example API credentials or login details. In the control panel click on the [**Keys link**](https://control.hico.io/resources/keys) from the left navigation and then press the Add Key button located at the top right of the page. The next page will be the create key form, you will need to enter the following settings before pressing save. * **Title** - A title for the key your creating, e.g. "*FTP Key*" * **System** - Select the system from the dropdown list, in this case "FTP" * **Content** - Add the following details to the credentials section: * host = Your FTP server URL * username = Your FTP server username * password = Your FTP server password * port = normally either `21` for FTP or `22` for sFTP Once you have pressed save on the first create key form you will be returned to the list of existing keys. You'll need to press the Create Key link once more to add the Shopify key. For the link to work you will need to activate a "Private App" connection to Shopify. To do this,[ follow this guide](/creating-streams/keys-1/key-library/shopify-private-apps-1). * **Title** - A title for the key your creating, e.g. "*Shopify Key*" * **System** - Select the system from the dropdown list, in this case "*Shopify*" * **Content** - Add the following details to the credentials section: * store = Your shopify URL, e.g. shopifystore.myshopify.com * api\_key = Copied from the private app screen in Shopify. * password = Copied from the private app screen in Shopify. * shared\_secret = Copied from the private app screen in Shopify. ### **2. Source** A source is where the data will be coming from. To create a source log in to [Control Panel](https://control.highcohesion.com/login) and select sources from the right side panel and select Create Source. To create a source you will need the following information below. * **Title** - Enter a title for the source, e.g. Get Shopify Orders * **System** - Select **Shopify** from the systems list. * **Function** - Select function from dropdown in this case `GraphQL\GetOrders::fromShopify()` * **Key** - Select the Shopify key that was created in step 1 above. * **Trigger** - Choose an option from the drop down list, either Manual, Scheduled or Web hook. For the purpose of the tutorial we are going to select "Manual", however after the smoke test in step 6 you may want to change this to scheduled. * **Settings** - Add the following setting so that the source will only collect orders created in the last 15 minutes that are marked as paid. ``` filter_created_at-gt = "CURRENT-15 minutes" filter_financial_status = "paid" ``` For the scenario in this tutorial we are going to get sales order information from Shopify, however this could be any other information, such as Products, Customers, Transactions etc, these would just require a different Function / Settings. {% hint style="info" %} Trigger options for source includes web hooks, manual and scheduled. If you choose scheduled you will need to set a scheduled time in the trigger option field. For scheduling options checkout this site {% endhint %} ### 3. Destination The destination is where the data will be streamed to after it has been transformed. To create a destination log in to [Control Panel](https://control.highcohesion.com/login) and select Destinations from the right side panel and select Create Destination. To create a Destination you will need the following information: * **Title** - Enter a title for the destination, e.g FTP Orders CSV * **System** - Select **FTP** from the systems list. * **Function** - Select function from dropdown in this example `PostFiles::toFtp()` * **Key** - Select the FTP key that was created in step 1 above. * **Trigger** - From the dropdown list select "Real Time". * **Settings** - Add the following setting so that the destination function will create a CSV file. These settings will also determine the name of the CSV file. For more information on what settings are available, check [this link to FTP System Settings](/creating-streams/systems/system-library/ftp-integration-functions#settings). ``` formatting = "csv" csv_file_headers = true filename_padding = 8 filename_format = "ORDERS##INCREMENT##.csv" remote_path = "/3pl_orders" ``` ### 4. Transformation The transformation file is required to translate data between the source system format and destination system format. To create a transformation file log in to [Control Panel](https://control.highcohesion.com/login) and select [Transformations](https://control.hico.io/resources/private-transformations) from the right side panel and select Create Private Transformation. The transformation creation screen gives the following inputs: * **Title** - A meaningful title e.g "Shopify x FTP CSV :: Sales Order Data" * **Source** - Choose the source system from the dropdown list, **Shopify** in this case. * **Destination** - Choose the destination system from the dropdown list, **FTP** in this case. * **Primary Data Type** - The data type will be what data is being streamed e.g order, product, payment etc... in this example the data type will be order. * **Content** - This will contain the mapping file which translate between the source and destination, for details on how to create a transformation file please check [this link](/creating-streams/transformations/writing-a-transformation/example-1-order-file). {% hint style="info" %} Information on how to create transformation files can be found in this [Example Tutorial](/creating-streams/transformations/writing-a-transformation/example-1-order-file), additionally all transformation functions are details on the [Writing a Transformation](/creating-streams/transformations/writing-a-transformation) page. {% endhint %} ### 5. Stream Step five in the process is to configure the our data stream. The stream in HighCohesion terms is the part that brings all of the settings together in to an active integration. To set this up, select the [**Streams**](https://control.hico.io/resources/streams) option from the left hand navigation. Then on the next page press the "Create Stream" button at the top right side of the page. On the next page, complete the following settings: * **Title** - Create a title for the stream: "\[SO-1001] Shopify x FTP CSV for Sales Orders". * **Description** - Optional, e.g. "Shopify to FTP for sales orders." * **Source** - From the dropdown, select the destination "Get Shopify Orders", as created in step 2 above. * **Transformation** - from the dropdown select the private transformation file that was created in step 4 above. * **Destination** - From the dropdown, select the destination "FTP Orders CSV", as created in step 3 above. * **Data Type** - For the data type enter: sales\_order * **Active** - Tick this option to activate the stream. ### 6. Smoke test Depending on which source function was chosen for the source in your stream will depend on how you can smoke test the integration. In the example above we used GetSalesOrders::fromShopify(). Therefore in order to test this we would create a new Sales Order in the Shopify system. The sales order should then be collected by the integration, visible in the **HighCohesion** control panel ([Events List](https://control.hico.io/resources/events)) and visible as a file on the FTP server. ### Summary Congratulations, you are now setup and ready to enjoy a working **Shopify x FTP integration**. ## Common Questions **Can I customise the attributes in the transformation files specific to my store?** \ Yes! Follow this link to our guide about how to customise transformation files with specific examples. **What about the Shipment file coming back to my Shopify system?**\ The shipment file being read back in to Shopify can be completed in part two of this tutorial. ([Click Here](/integration-tutorials/shopify-with-ftp/shopify-x-ftp-part-2)) # Shopify with FTP (Part 2) Taking the shipment data back from FTP to Shopify. In our first [Shopify to FTP tutorial](/integration-tutorials/shopify-with-ftp/shopify-orders-to-ftp) (part one) we looked at moving data from Shopify out to an FTP server as a CSV file. In this second part we will look at moving data back from the FTP location to update Shopify. To follow on from part one, in this part we will move shipment data back from the FTP to complete the sales order loop, ensuring that orders in Shopify are kept up-to-date with what the 3PL has shipped. ## Prerequisites This tutorial should be followed to create a data stream back from the FTP location to Shopify. These steps assume that you have the following accounts and information to hand: * **Completion of Part 1** - Please complete part one of this tutorial [here](/integration-tutorials/shopify-with-ftp/shopify-orders-to-ftp). ## Guide All of the following steps should be taken in order. Unless specified the steps assume that you are logged in to the [**HighCohesion control panel**](https://control.hico.io/). ### 1. Keys In part one of this tutorial two keys were created, these should be in your **HighCohesion** account as: * FTP Key * Shopify Key No additional keys are required for this second part. ### **2. Source** A source is where the data will be coming from. To create a source log in to [Control Panel](https://control.highcohesion.com/login) and select sources from the right side panel and select Create Source. To create a source you will need the following information below. * **Title** - Enter a title for the source, e.g. Get FTP Shipments * **System** - Select **FTP** from the systems list. * **Function** - Select function from dropdown in this case `GetFiles::fromFtp()` * **Key** - Select the FTP key that was created in step 1 above. * **Trigger** - Choose an option from the drop down list, either Manual, Scheduled or Web hook. For the purpose of the tutorial we are going to select "Manual", however after the smoke test in step 6 you may want to change this to scheduled. * **Settings** - Add the following setting so that the source will only collect files that start with the prefix `SHP_` and are located in the folder `/outbound/shipments` on the FTP. The setting `archive` will also move the file to an archive folder on the FTP once collected. ``` file_prefix = "SHP_" file_type = "csv" remote_path = "/outbound/shipments" move_remote_file = "archive" ``` For the scenario in this tutorial we are going to get a shipments CSV file from our dummy 3PL FTP. However this same process could be used for any other information coming from FTP, such as Products, Customers, Discounts etc, these would just require a different Function / Setting on the destination side. {% hint style="info" %} Trigger options for source includes web hooks, manual and scheduled. If you choose scheduled you will need to set a scheduled time in the trigger option field. For scheduling options checkout this site {% endhint %} ### 3. Destination The destination is where the data will be streamed to after it has been transformed. To create a destination log in to [Control Panel](https://control.highcohesion.com/login) and select Destinations from the right side panel and select Create Destination. To create a Destination you will need the following information: * **Title** - Enter a title for the destination, e.g Shopify Post Shipments * **System** - Select **Shopify** from the systems list. * **Function** - Select function from dropdown in this example `PostShipments::toShopify()` * **Key** - Select the Shopify key that was created in step 1 above. * **Trigger** - From the dropdown list select "Real Time". * **Settings** - Leave empty, no additional settings are required. ### 4. Transformation The transformation file is required to translate data between the source system format and destination system format. To create a transformation file log in to [Control Panel](https://control.highcohesion.com/login) and select [Transformations](https://control.hico.io/resources/private-transformations) from the right side panel and select Create Private Transformation. The transformation creation screen gives the following inputs: * **Title** - A meaningful title e.g "Shopify x FTP CSV :: Sales Order Data" * **Source** - Choose the source system from the dropdown list, **Shopify** in this case. * **Destination** - Choose the destination system from the dropdown list, **FTP** in this case. * **Primary Data Type** - The data type will be what data is being streamed e.g order, product, payment etc... in this example the data type will be order. * **Content** - This will contain the mapping file which translate between the source and destination, for details on how to create a transformation file please check [this link](/creating-streams/transformations/writing-a-transformation/example-1-order-file). For this example, please paste in the transformation below: {% code title="example\_shipments\_transformation.json" %} ```javascript { "fulfillment": { "order_number": { "*ppk_tracked*": { "*ppk*": "OrderNumber", "*tracked_field*": [ "s_id", "name" ] } }, "location_id": { "*stream_setting*": "location_id" }, "tracking_number": { "*ppk*": "CourierTrackingCode" }, "notify_customer": { "*static_value*": true } } } ``` {% endcode %} The transformation file logic above is the simplest form of shipment data to be sent to shopify and has been built to work with the example smoke test CSV file shown below in part 6a. This transformation maps the order number and tracking number from the smoke test CSV file in to the Shopify fulfilment format. {% hint style="info" %} Information on how to create transformation files can be found in this [Example Tutorial](/creating-streams/transformations/writing-a-transformation/example-1-order-file), additionally all transformation functions are details on the [Writing a Transformation](/creating-streams/transformations/writing-a-transformation) page. {% endhint %} ### 5. Stream Step five in the process is to configure the our data stream. The stream in HighCohesion terms is the part that brings all of the settings together in to an active integration. To set this up, select the [**Streams**](https://control.hico.io/resources/streams) option from the left hand navigation. Then on the next page press the "Create Stream" button at the top right side of the page. On the next page, complete the following settings: * **Title** - Create a title for the stream: "\[SH-1201] FTP CSV x Shopify for Shipments". * **Description** - Optional, e.g. "FTP to Shopify for shipments." * **Source** - From the dropdown, select the destination "Get FTP Shipments", as created in step 2 above. * **Transformation** - from the dropdown select the private transformation file that was created in step 4 above. * **Destination** - From the dropdown, select the destination "Shopify Post Shipments", as created in step 3 above. * **Data Type** - For the data type enter: `shipment` * **Active** - Tick this option to activate the stream. * **Settings** - We will need to add in a stream setting for the shopify location ID (shown below), replace this with the relevant location ID from your [Shopify](/creating-streams/systems/system-library/shopify-integration-functions) store. ```javascript location_id = 1234567890 ``` ### 6. Smoke test In order to test this data stream you will need to place a Shipments file in the `/outbound/shipments/received` folder on the FTP. There is an examples CSV file attached below. Once you have uploaded this file to the FTP, goto the Streams list, press the view link on the `[SH-1201] FTP CSV x Shopify for Shipments` stream. Then from the stream view page, click on the Manual Run option from the "Actions" dropdown (top right of page). This action will create a Job that will collect the file from the FTP and then create a Shipment event. The event will be visible on the Events list page. If all has gone correctly, the event in **HighCohesion** will be marked as `ok` and the order in Shopify will be marked as Fulfilled. #### 6.a Example File Copy the data below in to a text editor and save as a CSV file. This will then create your example shipments file that can be uploaded to your FTP for the smoke test. Please remember to change the order numbers in the first column to match the test order number(s) in your [Shopify](/creating-streams/systems/system-library/shopify-integration-functions) store. ```javascript OrderNumber,CourierTrackingCode,ShipmentDate "#1001","ABC23424001","29/06/2021" "#1002","VEC23424433","29/06/2021" "#1003","HTR23475441","29/06/2021" ``` ### Summary Congratulations, you are now setup and ready to enjoy a working **3PL shipments integration for Shopify**. ## Common Questions **Can I customise the attributes in the transformation files specific to my store?** \ Yes! Follow this link to our guide about how to customise transformation files with specific examples. # Shopify with ReBOUND Tutorial on how to setup a Shopify with ReBOUND integration via HighCohesion. This tutorial will guide you through how to setup a Shopify to ReBOUND returns integration via the HighCohesion system. The video walkthrough of this integration is shown below: ## Prerequisites This tutorial should be followed to create a **Shopify to ReBOUND** integration in just a few quick steps. These steps assume that you have the following accounts and information to hand: * **Shopify Store** - An active Shopify or Shopify Plus store. * **ReBOUND Account** - A ReBOUND account and have the login username and API key to hand. * **HighCohesion Account** - A HighCohesion account. ([**get one here**](https://www.highcohesion.com/sign-up)) ## Guide All of the following steps should be taken in order. Unless specified the steps assume that you are logged in to the [**HighCohesion control panel**](https://control.hico.io/). ### 1. Keys The first step is to add the system account keys. The keys are what the HighCohesion system use to connect to the 3rd party system, for example API credentials or login details. In the control panel click on the [**Keys link**](https://control.hico.io/resources/keys) from the left navigation and then press the Add Key button located at the top right of the page. The next page will be the create key form, you will need to enter the following settings before pressing save. * **Title** - A title for the key your creating, e.g. "*ReBOUND Test Key*" * **System** - Select the system from the dropdown list, in this case "*Rebound*" * **Content** - Add the following details to the credentials section: * login = Your login/username for your account. * api\_key = Your api key for your account. Once you have pressed save on the first create key form you will be returned to the list of existing keys. You'll need to press the Create Key link once more to add the Shopify key. As this integration will use Shopify notifications, there is no need to add any specific API credentials. Simply add the following details in to the create key form: * **Title** - A title for the key your creating, e.g. "*Shopify Test Key*" * **System** - Select the system from the dropdown list, in this case "*Shopify*" * **Content** - Leave all inputs blank, scroll down and press save. ### 2. Source Our next step is to define the source. Navigate to the [**Sources link**](https://control.hico.io/resources/sources) from the left hand navigation in the HighCohesion control panel, then select the "Create Source" button (located at the top right side of the screen. The next page will load the source form, in that form complete the following information: * **Title** - Create a title for the source, normally "Shopify Orders web hook". * **System** - Select "Shopify" from the dropdown. * **Function** - Select "GetOrders::fromShopify()" from the dropdown. * **Key** - Select the name of the key defined in step one "Shopify Test Key". * **Trigger** - Select the "Static web hook" option. * **Settings** - Leave empty. ### 3. Destination Next up is the destination for the data to be sent to. To add the destination, click on the [**Destinations link**](https://control.hico.io/resources/destinations) from the left hand nav and then press the "Create Destination" button at the top right of the page. Then enter the following details in to the form: * **Title** - Create a title for the destination, normally "ReBOUND Orders". * **System** - Select "ReBOUND" from the dropdown. * **Function** - Select "PostOrders::toRebound()" from the dropdown. * **Key** - Select the name of the key defined in step one "ReBOUND Test Key". * **Trigger** - Select the "Real Time" option. * **Settings** - Leave empty. ### 4. Stream Step four in the process is to configure the Stream. The stream in HighCohesion terms is the part that brings all of the settings together in to an active integration. To set this up, select the [**Streams**](https://control.hico.io/resources/streams) option from the left hand navigation. Then on the next page press the "Create Stream" button at the top right side of the page. On the next page, complete the following settings: * **Title** - Create a title for the stream, e.g "Shopify x ReBOUND Orders". * **Description** - Optional, e.g. "Shopify x ReBOUND Orders" * **Source** - From the dropdown, select the destination "Shopify Orders web hook", as created in step 2 above. * **Transformation** - from the dropdown select the public transformation file "Shopify x ReBOUND :: Sales Orders". * **Destination** - From the dropdown, select the destination "ReBOUND Orders", as created in step 3 above. * **Data Type** - For the data type enter: sales\_order * **Active** - Tick this option to activate the stream. ### 5. Shopify Notification The final step before testing is to add the web hook notification in the Shopify admin area. To do this follow these steps inside of the Shopify & HighCohesion control panels: 1. In the HighCohesion control panel, click on the [**Streams**](https://control.hico.io/resources/streams) link from the left hand navigation. Then click on the View link for the stream created in Step 4 above. 2. On the stream view page, you will see the web hook notification URL that is unique to your stream. Copy this URL as we will use it in the next step. 3. In the Shopify admin area, click on the **Settings** link (bottom left side of the page) 4. From the settings screen, select the **Notifications** option. 5. Scroll to the bottom of the page to find the Webhook section, here press the Create Webhook button. 6. In the popup box, enter the following settings: * **Event** - Select Order Create from the dropdown list. * **Format** - Leave as default: JSON. * **URL** - Paste in the URL that was copied from the stream view page above. * **Version** - Select the (Latest) version. ### 6. Testing The stream has been setup with a Shopify web hook notification to trigger. The testing process therefore is as simple as creating an order in the Shopify system. This can be created in either the admin area or the front end of the website. Once the order is created, Shopify will send the order data to the HighCohesion system via a notification system. HighCohesion will then convert the data in to ReBOUND format and pass the data on to the ReBOUND orders API. Remember that all orders will be trackable in the HighCohesion control panel as events. This allows you to see the events as they're created and also track the success or failure rates through to the ReBOUND system. ### Summary Congratulations, you are now setup and ready to enjoy a working **Shopify x ReBOUND integration**. ## Common Questions **Can I have a two streams to cover my ReBOUND test and production accounts?** \ Yes! Simply repeat the steps above in the HighCohesion control panel to setup two different streams with different keys. For example you Could have two different ReBOUND keys, one for production and one for test and then two different streams+destinations for each. **There are no events appearing in the HighCohesion control panel?**\ The most common cause of no events appearing in the HighCohesion control panel is that the stream URL is not correct in the Shopify web hook URL setting. To resolve, double check that the URL has been copied and pasted correctly. **How can I stop the events or turn off the integration?**\ The quickest way to stop the integration is to delete the notification web hook in the Shopify control panel. This will stop any data from being sent to HighCohesion and subsequently ReBOUND. # WooCommerce WooCommerce Integration Tutorials The following tutorials are available for integrating WooCommerce with various 3rd party systems via the HighCohesion platform: * [WooCommerce x FTP Integration](/integration-tutorials/woocommerce-integration-tutorials/woocommerce-ftp-integration) {% hint style="info" %} For a list of functions that are available in HighCohesion for WooCommerce [click here](/creating-streams/systems/system-library/woocommerce-integration-functions). {% endhint %} # WooCommerce with FTP How to integration WooCommerce with FTP for product data. # NetSuite ## Setting up a key Firstly the company and account number are the first part of the NetSuite URL for that client. For example if the URL is then the company and account number would be 123456-sb1. Next within NetSuite, go to Setup -> Integration -> Manage Integrations -> New. Give the integration a name, ensure that Token-based authentication, TBA: Issuetoken Endpoint and User Credentials are selected. Everything else should not be. Click on Save.
A consumer key and consumer secret will then appear in the client credentials section. Add these to the consumer key and consumer secret within the control panel key. These only appear once. Next go back to the homepage by clicking on the house icon. Scroll to the bottom of the page and click on "Manage Access Tokens". Click New My Access Token. Select the Application that we just created and press save.
The Token ID and Token secret will then show up which can be added to the key. All done. # Intro Control panel setup menu The setup menu in the control panel gives you the ability to create, view, edit and remove all aspects your HighCohesion stream configuration. The sections below walk you through each section in further detail: * [Systems](/creating-streams/systems) * [Keys](/creating-streams/keys-1) * [Sources](/creating-streams/sources) * [Transformations](/creating-streams/transformations) * [Destinations](/creating-streams/destinations) * [Streams](/creating-streams/streams) * [Data uploads](/creating-streams/data-uploads) * [Lookup table](/creating-streams/lookup-table) # Streams Your integration data stream. In HighCohesion the stream configuration contains all the information for our system to run the integration. Streams are made up of three main components: ### Source Contains the configuration for which data [source](/creating-streams/sources) to use. ### Transformation Contains the configuration for which [transformation](/creating-streams/transformations) file to use. ### Destination Contains the configuration to send the data to the [destination](/creating-streams/destinations) system. ### Data Type The data type used on the stream should be relative to entity contained within. For consistency we recommend that you use singular, lowercase words with no spaces. Underscores are allowed. Examples of this are: * Sales Orders = `sales_order` or `order` * Products = `product` * Refunds = `refund` * Stock Trades = `stock_trade` * Shipments = `shipment` or `fulfilment` * Purchase Orders = `purchase_order` * Goods Receipts = `goods_receipt` or `grn` ### Settings The settings is for adding any additional settings or config for the stream. Settings in this box should follow the basic TOML formatting. ()
# Web hook installation There are some systems that support the web hook installation directly from the control panel. For example, Peoplevox. To install the web hook automatically on these systems, follow these steps: 1. Add a new key to access to Peoplevox system. Add the credentials that are needed to install the web hook. 2. Create a source and select the Static or Dynamic trigger option. 3. Create a new stream as normal. 4. After the stream has been created, edit the stream and select the field **Install Web hook** and press save. If the system is supported, you should see some configuration settings like source web hook configuration id in the Web hook Credentials field. If the source system is not supported, you will see an alert message. # Web hook support ### Generic Notifications Any system regardless of if HighCohesion support it natively or not can send notifications to your HighCohesion account. By creating a stream that has a "Static Web hook" type of trigger, you will get a secure web hook URL shown on the stream view page. This URL can then be used in any other 3rd party system to send notifications data to HighCohesion. Once the data is received by HighCohesion it will be converted into an Event and processed as normal to any destination function using any transformation. ### System Specifics The following systems are currently supported for web hook style notifications: * [DEAR Systems](/creating-streams/streams/web-hook-supported-systems/dear-systems) * [Descartes Peoplevox](/creating-streams/streams/web-hook-supported-systems/peoplevox) * [Shopify](/creating-streams/streams/web-hook-supported-systems/shopify) # DEAR Systems Notifications from DEAR Systems. # Descartes Peoplevox Peoplevox credentials for installing a web hook from the control panel # Shopify Shopify Web hooks # Systems The functions for the systems that HighCohesion connects to. ### Overview **HighCohesion** has been built conceptually to work universally with any system. The microservice architecture helps to decouple processes and make it easier to extend. **HighCohesion** has an ever expanding list of source and functions, these functions are categorised by system, meaning the system to which the function is related to. ### System Functions List Systems exist in the system with a list of available functions. These functions are often categorised as get, post and notification services. Each function represents a functional piece of code within the SDF codebase. The system functions list has to be updated before a new function is available to use within the system. # System library The below Systems are already available within the HighCohesion platform. For more information on each System and their configurations, select the System you want to explore further in the left side menu. * **eCommerce** * Shopify & Shopify Plus * Magento * Magento 2 * **ERP** * Cegid Y2 * Microsoft Finance & Operations Dynamics 365 * Netsuite (basic) * **3PL** * Whistl * Parcelab * **File transfer & storage** * FTP / SFTP * Amazon S3 * **Inventory / warehouse management** * DearSystems * Peoplevox * **Misc** * RestAPI - Data access * Email - Communication * ReBOUND - Returns * NetTickIT - Ticketing * SageOne - Accounting # DEAR Systems Functions and settings available for the DEAR Systems endpoint ## Functions The following list of functions are available in the DEAR Systems system: ### GetFulfillment ### GetProduct ### GetProductAvailability ### GetPurchase ### GetPurchaseList ### GetPurchaseOrder ### GetPurchaseOrderSkus ### GetSaleList ### PostInventoryStockLevels ### PostItemArrivals ### PostProduct ### PostPurchaseOrder ### PostSaleOrder ### PostSaleUpdate ### PostShipment ### PostShipmentUpdate ### PostSupplier {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for DEAR Systems integrations. We have implemented the filters by prefix. You need to add the prefix of the function that you are using for the filter that you want to apply. For example, if you are using the function Purchase, the filter settings will look like: ``` Purchase_Status="PROCESSING" ``` # Email Functions and settings available for the Email endpoint ## Functions The following list of functions are available in the Email system: ### GetEmail ### SendEmail {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for Email integrations. # FTP Functions and settings available for the FTP endpoint ## Functions The following list of functions are available in the FTP system: ### getFiles ### postFiles {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for FTP type integrations. ### array\_key tbc. ### array\_merging Used when combining multiple events together in the destination functions. This settings allows the specification of a top level key that will be merged together. * **Type**: Text * **Required**: No * **Default**: N/A **Applicable functions:** * PostFiles **Example:** ```yaml array_merging = "products" ``` ### edi\_format edi\_line\_identifier edi\_group\_terminator edi\_force\_multiples edi\_multi\_line ### event\_grouping tbc ### file\_far Ability to find and replace in a file after its generation. This feature can be helpful if the use of protected key names is required. * **Type**: Text * **Required**: No * **Default**: N/A **Applicable functions:** * PostFiles **Example:** ```yaml file_far = "find_first|replace_first,find_second|replace_second" ``` ### filename\_format Defines the output file naming convention * **Type**: Text * **Required**: No * **Default**: ##DATETIME##.##EXTENSION## **Applicable functions:** * PostFiles **Example:** ```yaml filename_format = "Payout_Report_##DATETIME##.xml" ``` **Dynamic field values:** | Setting | Output | Example | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ | | ##JOBID## | Injects the current job id used in the data stream by HighCohesion. | 1b503450-89ea-11ea-903e-6ef3fac04b7e | | ##DATETIME## | Injects the current date and time in the format: YYYY-MM-DD-HH-mm-SS | 2020-06-01-11-00-00 | | ##DATETIMEEXACT## | Injects the current date and time with milliseconds in the format: YYYY-MM-DD-HH-mm-SS-zzz | 2020-06-01-11-00-00-000 | | ##DATETIMEFORMAT\_\_XXX\_\_## |

Inject a custom date time format. Where **XXX** is replaced with the required formatting. Example being: ##DATETIMEFORMAT\_\_Ymd\_\_##

See: for PHP formatting examples.

| 20200601 | | ##TIMESTAMP## | Injects the current Unix timestamp. | 1588149986 | | ##EXTENSION## | Injects the extension as per the [formatting](broken://pages/-M64-oiaJDnE7g9nuSzj#formatting) setting. | xml | | ##INCREMENT## | Injects an incremented number based on the stream ID. Checks the HighCohesion internal increment pointer for the stream in use. Each time that the stream is triggered the pointer will increment by one. | 42 | | ##PK\_\_XXX\_\_## | Injects the value of a specified payload key where XXX is replaced with a payload key. For example ##PK\_\_order\_number\_\_## | GB101231 | ### formatting Defines the input or output file formatting. * **Type**: List * **Required**: No * **Default**: xml **Applicable functions:** * GetFiles * PostFiles **Example:** ```yaml formatting = "csv" ``` **List values:** | Setting | Output | | ------- | ------------------- | | xml | XML file formatter | | json | JSON file formatter | | csv | CSV file formatter | | xls | XLS file formatter | ### group\_on Allows the grouping of lines from an input file based on a specific column. * **Type**: Text * **Required**: No * **Default**: N/A * **Relative to**: CSV and XLS file imports. **Applicable functions:** * GetFiles **Example:**\ The example below would group lines of a CSV file together based on the order\_number column. The is helpful when receiving CSV files from systems such as warehouse management systems that can send multiple lines in one file. ```yaml group_on = "order_number" ``` ### increment\_in\_file Ability to inject the data stream increment reference into the payload\_out data, before file generation. For example if the destination needs unique file number reference in the file. The value should be the path to add the increment to in the output data. * **Type**: Text * **Required**: No * **Default**: false **Applicable functions:** * PostFiles **Example:** ```yaml increment_in_file = "header.reference.number" ``` ### index\_array Allows the breaking up in to individual events based on a key that should be present in the input file data. * **Type**: Text * **Required**: No * **Default**: N/A * **Relative to**: XML file imports. **Applicable functions:** * GetFiles **Example:**\ ... ```yaml index_array = "Products" ``` ### line\_index\_array ### remote\_path Allows definition of the remote path. Use this setting to instruct the data stream where to look for files on the remote location. * **Type**: Text * **Required**: No * **Default**: / **Applicable functions:** * GetFiles * PostFiles **Example:** ```yaml remote_path = "/orders/uk_website/" ``` ### root\_element Allows the specification of the root XML element. * **Type**: Text * **Required**: No * **Default**: data * **Relative to**: XML file exports. **Applicable functions:** * PostFiles **Example:** ```yaml root_element = "Orders" ``` ### skip\_incrementer tbc. ### skip\_path\_check Defines if the system should skip the remote path check. This is especially helpful for some sFTP destinations where the remote list functionality is not available. * **Type**: Boolean * **Required**: No * **Default**: false **Applicable functions:** * PostFiles **Example:** ```yaml skip_path_check = true ``` # Descartes Peoplevox Functions and settings available for the Descartes Peoplevox endpoint ## Tutorials Integration setup tutorials: * [**Peoplevox Shopify Integration**](/integration-tutorials/shopify-x-descartes-peoplevox) ## Authentication The Descartes Peoplevox system handles authentication in the following format: {% code title="example\_peoplevox\_key.json" %} ``` { "client_id": "", "username": "", "password": "", "env": "wms" } ``` {% endcode %} These details are provided to you by Descartes Peoplevox. ## Functions The following list of functions are available in the Descartes Peoplevox system: ### GetData ### GetReportData ### PostCustomer ### PostItem ### PostPurchaseOrder ### PostRMA ### PostSalesOrder ### ReceiveAvailabilityChanges ### ReceiveDespatchPackageDespatched ### ReceiveGoodsReceived ### ReceiveReturns {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} # Magento 1.x Functions and settings available for the Magento 1.x endpoint ## Functions The following list of functions are available in the Magento 1.x system: ### GetCustomers ### GetOrders ### GetProducts {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for Magento 1.x integrations. # Magento 2.x Functions and settings available for the Magento 2.x endpoint ## Functions The following list of functions are available in the Magento 2.x system: ### GetCustomers ### GetOrders ### GetProducts {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for Magento 2.x integrations. # ParcelLab Functions and settings available for the ParcelLab endpoint ## Functions The following list of functions are available in the ParcelLab system: ### PostShipment ### PostPresage {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for ParcelLab integrations. # ReBOUND Functions and settings available for the ReBOUND endpoint ## Functions The following list of functions are available in the ReBOUND system: ### GetReturns Ability to get return requests (RMA) from ReBOUND. ### PostOrder Ability to send new order records in to the ReBOUND system. {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for ReBOUND integrations. ### returns\_end\_date ### standard\_return\_days ### refund\_days # Shopify Functions and settings available for the Shopify endpoint ## Tutorials Integration setup tutorials: * [**Shopify to FTP integration**](/integration-tutorials/shopify-with-ftp/shopify-orders-to-ftp) * [**Shopify to Peoplevox integration**](/integration-tutorials/shopify-x-descartes-peoplevox) * [**Shopify to ReBOUND integration**](/integration-tutorials/shopify-x-rebound) ## Functions The following list of functions are available in the Shopify system: ### GetCustomers ### GetFulfilments ### GetOrders ### GetPayoutTransactions ### GetProducts ### GetVariants ### PostBlogs ### PostCustomers ### PostDiscountCode ### PostDiscountCodeBatch ### PostInventoryAdjustment ### PostInventoryLevel ### PostMeta ### PostOrders ### PostPriceRule ### PostPriceRuleAndDiscountCode ### PostProducts ### PostRefund ### PostShipments ### PostVariant {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for Shopify integrations. # WooCommerce Functions and settings available for the WooCommerce endpoint ## Tutorials [WooCommerce integration setup tutorials](/integration-tutorials/woocommerce-integration-tutorials): * [WooCommerce to FTP integration](/integration-tutorials/woocommerce-integration-tutorials/woocommerce-ftp-integration) ## Functions The following list of functions are available in the **WooCommerce** system: ### GetCoupons Function for retrieving coupon information from **WooCommerce** via the REST API. For more details on the schema that is returned from WooCommerce and filters that are available please see the documentation at: This function is has the following filters that are commonly used: | Filter | Example | Description | | ------- | -------------------------------------- | --------------------------------------------------------------------------------- | | `after` | `filter_after = "CURRENT -15 minutes"` | Used to filter coupons that have been created/updated in the previous 15 minutes. | ### GetCustomers Function for retrieving customer data from **WooCommerce** via the REST API. For more details on the schema that is returned from WooCommerce and filters that are available please see the documentation at: This function has the following filters that are commonly used: | Filter | Example | Description | | ------- | -------------------------------------- | ----------------------------------------------------------------------------------- | | `after` | `filter_after = "CURRENT -15 minutes"` | Used to filter customers that have been created/updated in the previous 15 minutes. | ### GetOrders Function for retrieving sales orders from WooCommerce via the REST API. This function is has the following filters that are commonly used. Please also review the filters in the official WooCommerce () documentation which can also be used in the same `filter_` prefixed way in the HighCohesion control panel. | Filter | Example | Description | | ------- | -------------------------------------- | ------------------------------------------------------------------------ | | `after` | `filter_after = "CURRENT -15 minutes"` | Used to filter orders that have been created in the previous 15 minutes. | {% hint style="info" %} To reduce bandwidth and speed up your integration, consider using **WooCommerce** web hooks that work natively with the **HighCohesion** system. {% endhint %} ### GetProducts Function for retrieving products from **WooCommerce** via the REST API. For more details on the schema that is returned from WooCommerce and filters that are available please see the documentation at: This function is has the following settings available: | Filter | Example | Description | | ------- | -------------------------------------- | -------------------------------------------------------------------------- | | `after` | `filter_after = "CURRENT -15 minutes"` | Used to filter products that have been created in the previous 15 minutes. | ### PostCustomer Allows the creation and update of customer records in the WooCommerce system. For details on the WooCommerce customer schema, please visit: ### PostProduct Allows the creation and update of products inside of the WooCommerce system. Public transformation files are available in the HighCohesion control panel. For details on the WooCommerce product schema, please visit: The following settings are available for this function: | Setting | Values | Description | | ---------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `allow_product_search` | `true` / `false` | If set to `true` the function will allow the lookup of existing product when the `id` has not been set in the payload. As default the system will attempt to match the `slug` | | `search_key` | Valid search key from the WooCommerce product API. | Defaults to `slug`, can optionally be set to `sku` or `attribute_term` | ### PostOrder Allows the creation and update of orders inside of the WooCommerce system. For details on the WooCommerce customer schema, please visit: | Setting | Values | Description | | -------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------- | | `allow_order_update` | `true` / `false` | If set to `true` the function will allow the update of existing orders when the `id` has been set in the payload out. | ### PostShipment Allows the creation of shipment tracking records against sales orders inside of the WooCommerce system. For details on the WooCommerce customer schema, please visit: {% hint style="info" %} Shipments require the Order ID to successfully post. However the HighCohesion system will accept either an `order_id` field or an `order_number` field in the payload out. If you provide the order number field, the function will attempt to lookup the order id before it posts to shipment information. {% endhint %} There are no additional settings for this function. {% hint style="warning" %} Functions lists are expanding weekly, this list was accurate at the date of publishing. {% endhint %} ## Settings The following settings below are optional for WooCommerce integrations. ## Web hooks **WooCommerce** web hooks work natively with the HighCohesion system and can be easily setup via the WooCommerce settings area in Wordpress. Follow the steps below to setup the most common Sales Orders web hook, from WooCommerce to HighCohesion: * Login to your Wordpress system and click on the WooCommerce > Settings link from the left hand navigation. * Once on the settings page, click on the "*Advanced*" tab at the top of the page. * Next click on the "*Webhooks*" link at the top, just below the tabs. ![The WooCommerce web hooks screen.](/files/-MeOsxtT30S57sMR9iQK) * Now the web hooks page is shown, click the purple button that says "*Create new web hook*". * On the new web hook form, enter the following information: * **Name** - Give the web hook notification a name, e.g. `Sales orders to HighCohesion`. * **Status** - Set to `Active` * **Topic** - Select the top that you want to be notified about, in this examples "Order Create" * **Delivery URL** - Set this to your web hook URL for your stream, (see [how to create one here](/creating-streams/streams/web-hook-installation)) * **Secret** - Leave blank * **API Version** - Select the version required, most select V3. ![Example web hook form in WooCommerce](/files/-MeOvClQpImA0Ykv_l-Y) * Once complete, now press save and the WooCommerce system will be setup to send HighCohesion notifications when new records are created. {% hint style="info" %} For a guide on how to generate your HighCohesion web hook URL, please [click on this link](/creating-streams/streams/web-hook-installation). {% endhint %} # Keys Securely stored access credentials for your systems. ### Overview Gaining secure access to the systems that you need to integrate is central to building a working solution on **HighCohesion**. Our **Keys** system allows you to securely add access credentials in to the **HighCohesion** system via the Control Panel. ### Security Security is always the main concern when it comes to system access credentials. This is why we have taken extra precautions when it comes to the transfer and storage of your **Keys** in our system. Our system uses: * **SSL encryption** between the browser and the server ensuring that the data transmitted is encrypted. * **At rest encryption** in the secure storage facility where the keys are stored. * Method of **Least Privilege Permissions** in our access levels to the key storage area, meaning only the correct access is granted to use the credentials. {% hint style="info" %} Quite often the easiest way in to the system is via your user account in the control panel. Please ensure that you activate the [multi factor authentication](/administration/login-and-security/two-factor-authentication-2fa-setup) option on your account as soon as it is setup to increase the security on your login. {% endhint %} ### Add a new key To add a new set of credentials in to your High Cohesion account, please follow these steps: 1. From the control panel, go to the [**Keys**](https://control.highcohesion.com/resources/keys) section. 2. At the top right hand side of the page, press the button that says **Create Key**. 3. This will then load the new Key form. On this form complete the following options: 1. **System** - This is a drawn down list of the systems available in the High Cohesion system. Please select from this list the system that the credentials are required for. When you select a system, a *Key Template* will automatically load in to the credentials box below, this will give you a template for the credentials that you need to add. 2. **Title** - Give the Key a title that is relative to the credentials, for example: "*Live UK Shopify*" or "*GBP Staging Store*" etc. 3. **Credentials** - Once a system is selected the input boxes relative to the system will be shown in the credentials section. 4. Once the credentials have been added, press the save button at the bottom left of the page. 5. The new credentials will now be securely saved and available to use in the system. ### Updating a key If passwords or other credentials change over time, it is essential that you update the key in the High Cohesion control panel swiftly. This is easy and securely done by following the steps below: 1. From the control panel, go to the [**Keys**](https://control.highcohesion.com/resources/keys) section. 2. Here you will see a list of Keys that you have setup under your account, in this list find the Key that you would like to update. 3. On the far right hand side of the row, there is a small pen and paper icon. 4. Press this icon to load the edit screen, the credentials will be available for editing. Please note that details are hashed to provide security. Hashed values will no be changed upon save. 5. Once updated, press the save button at the bottom left of the page. 6. The updated credentials will now be securely saved and active in the system. ### Deleting a key A key can be securely deleted from the High Cohesion control panel when it is no longer required. To do this follow these steps: 1. From the control panel, go to the [**Keys**](https://control.highcohesion.com/resources/keys) section. 2. Here you will see a list of Keys that you have setup under your account, in this list find the Key that you would like to remove. 3. On the far right hand side of the row, there is a small Bin icon. 4. Press this to remove the credentials from the system. {% hint style="info" %} A key that is in use can not be deleted, it will need to be removed from any active [Source](/creating-streams/sources) or [Destination](/creating-streams/destinations) systems before the system will allow you to delete it. {% endhint %} # Key library Different systems have different key formats, for example some systems use simple username and password where as other systems use more complex token based authentication. Details of different systems can be found below: * [Peoplevox](/creating-streams/systems/system-library/peoplevox-integration-functions#authentication) Additional information available: * [Shopify](/creating-streams/keys-1/key-library/shopify-private-apps-1) * [Netsuite](/creating-streams/keys-1/key-library/netsuite) * [FTP](/creating-streams/keys-1/key-library/ftp-and-sftp) * [oAuth](/creating-streams/keys-1/key-library/oauth) # Shopify private apps Guide to setting up a Shopify private app keys Shopify private apps are a quick and easy way to setup a connection between High Cohesion and your Shopify system. Private apps can be added to both Shopify and Shopify+ subscriptions, both follow the same guide below: ### In Shopify The first steps are to be completed from within your Shopify admin area. 1. From your Shopify admin, go to [**Apps**](https://www.shopify.com/admin/apps). 2. Scroll to the bottom of the page and click **Manage private apps**. 3. Click **Create a new private app**. 4. In the **App details** section enter the following details: 1. In the name field, enter: **High Cohesion** 2. In the contact email address, enter: **** 5. In the **Admin API** section, select the areas of your store that you want the to access via the High Cohesion system. Please use logic when selecting the options in the app, for example if you only want to extract order data from Shopify, only select **Read** on the orders API option etc. \ \ Four permissions are selected by default: **Read products and collections**, **Read and modify customers**, **Read and modify checkouts**, and **Read blogs and articles**. 6. When you're done, click **Save**. 7. The API key, secret and password for your app are now displayed on screen. {% hint style="warning" %} Treat the API key and password like you would any other password, because whoever has access to these credentials has full API access to the store. After you have the required credentials, the private app can access your store's data. {% endhint %} ### In High Cohesion Once you have generated the private app credentials through the steps above, follow the guide on [adding a new key](broken://pages/-LzkxfZA_INw3d4hGgL7#add-a-new-key), selecting Shopify from the system list. # Netsuite Details on adding Netsuite credentials. ### About this task Before you configure the **Netsuite Connector** in **HighCohesion**, there are several prerequisites that need to be completed in **Netsuite** in order to prepare your account for integration. Let's take a look at the configuration in NetSuite: * Install the HighCohesion Netsuite Bundle from the Netsutie marketplace. * If not already assigned, assign your NetSuite integration user account the [Highcohesion Integration role.](https://system.netsuite.com/app/help/helpcenter.nl?fid=section_N897798.html) You must have a dedicated NetSuite integration user (employee) account that has a full access role in order to connect with Highcohesion. * [Set the time zone](https://system.netsuite.com/app/help/helpcenter.nl?fid=section_3775069733.html) for the integration user account to ???. If you cannot set this preference, it is because the Company General Preferences are blocking it. To remove this restriction, go to **Setup** > **Organisation** > **General Preferences** and then make sure that **Time Zone** is checked under the **Overriding Preferences** section. * Make sure the following **NetSuite SuiteCloud** [features are enabled](https://system.netsuite.com/app/help/helpcenter.nl?fid=chapter_N232138.html): Client SuiteScript, Server SuiteScript, SuiteScript Server Pages, SuiteTalk Web Services, SuiteSignOn, and Token-Based Authentication. After NetSuite is properly set up and the NetSuite Connector is configured in **HighCohesion**, you will need to edit the NetSuite Connector settings before data exchanges will occur. If you plan to integrate SuiteCommerce Advanced with **HighCohesion** apps, you will need to [complete additional configuration steps in NetSuite](https://app.gitbook.com/s/-Lzkg5CaFPWC2d3HNSxq/control-panel/setup/keys-1/key-library/netsuite????) after the **NetSuite Connector** has been configured. See [Configure NetSuite App Integrations](https://help.bronto.com/bmp/task/t_bmp_connector_netsuite_app_integrations.html) for the full details about SuiteCommerce Advanced and **HighCohesion** app integration. ### Procedure 1. Complete all of the **NetSuite** configuration prerequisites listed above. 2. Go to **Settings** > **Integrations** > **Connector**. 3. Click **Add Connection**. 4. Click **NetSuite**. 5. Enter the account ID for your NetSuite account in the **Account ID** box. To find the Account ID in Netsuite, go to **Setup** > **Integration** > **Web Services Preferences**. 6. Enter the NetSuite account username in the **Username** box. This is the NetSuite login username for the dedicated NetSuite integration user account that you set up for the connector. 7. Enter the NetSuite account password in the **Password** box. This is the NetSuite login password for the dedicated NetSuite integration user account that you set up for the connector. 8. Enter the **Token ID** You need to create a new token in NetSuite using NetSuite's two-factor authentication. To create a token: 1. In NetSuite, go to **Setup** > **Users and Roles** > **Access Tokens** > **New**. 2. For Application Name, select **Internal NS Application HIGHCOHESION** 3. Find the user you want to use. 4. Under Role, select **HighCohesion Integration**. 5. Click **Save**. You will see the Token ID and the Token Secret. They will only be shown one time so be sure to save this information. 9. Enter the **Token Secret**. Two-factor authentication will be complete after you enter your Token ID and Token Secret in **HighCohesion**. 10. Select the type of NetSuite account from the **Instance Type** list: This should reflect whether you NetSuite account is a **Production** or **Sandbox** account. If you're not sure, you probably have a production account. See NetSuite's Sandbox help topic for more information about Sandbox accounts. 11. Click **Login**. The connector uses the provided credentials to establish the connection between **HighCohesion** and NetSuite and you are returned to **HighCohesion** Connectors page. You should see the NetSuite Connector you just configured in the list of connectors on this page. 12. Continue to the appropriate configuration topics to enable data exchange: * Click [here](https://help.bronto.com/bmp/task/t_bmp_connector_netsuite_contacts_configure.html) to learn how to set up contact data exchanges * Click [here](https://help.bronto.com/bmp/task/t_bmp_connector_netsuite_orders_configure.html) to learn how to set up order data imports * Click here to learn how to set up app integrations # FTP and sFTP How to gather data for FTP and sFTP keys. One of the most popular connections is FTP or sFTP, a method normally used for exchanging files between servers. This is often used where companies require CSV, XML or JSON files to be deposited or collected from specified servers at certain times. Adding a FTP connection to High Cohesion is simple, you will require the following details about the FTP server that is being connected to: * **Host / Address**\ This can be either a fully qualified domain name e.g. ftp.highcohesion.com or an IP address e.g. 192.168.2.1 * **Username**\ The username required to connect. * **Password**\ The password required to connect. * **Port**\ The port required to connect, please note that the default port for FTP is 21 and the default port for sFTP is 22. * **Passive**\ This is a `true / false` option, allows a passive connection setting to be activated. * **Remote Path**\ This is an optional setting that allows a base path for your connection to be set. For example `/export/orders/` {% hint style="info" %} Please note that the Remote Path setting can be overridden and not appended to in Source/Destination and Stream settings. {% endhint %} Once you have gathered the details above you can add the credentials to High Cohesion by following the guide on [adding a new key](/creating-streams/keys-1#add-a-new-key), selecting FTP from the system list. # oAuth # Sources The source of data in your integration data stream. ### Overview The Source is the configuration that our stream needs to get the information from the Source System. The primary fields are ![](/files/-Mbaz5YSowMZLMYkNIXI) ### Source List View The default page that is shown when clicking on the "Sources" link from the left navigation is the source list page. On this page you can view all of the existing sources of data that you have setup in HighCohesion. Towards the right hand side of each existing row are actions for viewing, editing and deleting existing entries. {% hint style="info" %} Please note that a source cannot be deleted it if is attached to a stream, the source will need to be removed from the stream or stream deleted first. {% endhint %} ### Creating a Source 1. From the control panel, go to the **Source** section. 2. At the top right hand side of the page, press the button that says **Create Source**. 3. This will then load the new Source form. On this form complete the following options:
1. **Title** - Give the Key a title that is relative to the source, for example: "*Shopify GET Orders*" or "*GBP Store Orders*" etc.
2. **System** - This is a drown down list of the systems available in the High Cohesion system. Please select from this list the system that the credentials are for. When you select a system from this list, the page will automatically load available functions in the Source Function box below.
3. **Source Function** - This is a dropdown list of the functions available in the System that you have selected in step 2. Please select from this list the functions that you would like to consume the data in for this source config.
4. **Key** - The key field is a dropdown list of the keys that have already been setup in your account. Please select the key that is relative to the system and environment that you want to connect to.
5. **Trigger** - There are four options available here to control how the source is triggered while the data stream is in use: 1. *Manual* - The source is triggered when the user creates a new job manually. 2. *Scheduled* - The source is triggered on a cron job. We use cron expressions to configure this option. 3. *Dynamic Web hook* - The dynamic web hook configuration creates a new endpoint so that the external source system can send the data directly to it. This type of web hook will send the data directly to the source function. 4. *Static Web hook* - The static web hook configuration will create a new endpoint so that the external source system can send the data directly to it. This type of web hook will send the data directly to the core.
6. **Trigger Options** - This is an options box for the trigger above, please follow the guidelines below depending on the trigger selected above: 1. *Scheduled* - Enter a valid CRON expression for when the source trigger should be ran. See here for help generating a valid CRON expression:
7. **Settings** - This box is for adding any additional settings or config for the source. Settings in this box should follow the basic TOML formatting. ()
4. Once the settings have been added, press the save button at the bottom left of the page. 5. The new source will now be securely saved and available to use in the system. ### Deleting a Source A source can be deleted from either the list page by clicking on the bin icon, or from the single view page, also by the bin icon. {% hint style="info" %} Please note that a source cannot be deleted it if is attached to a stream, the source will need to be removed from the stream or stream deleted first. {% endhint %} # Transformations Transforming data from source for to destination format. A transformation is a file that defines the translation of mappings between two systems. Many common transformations already exist as **Public** files. You can browse and use the many **Public** level transformation files that have already been created in the Control Panel. Most **HighCohesion** users can setup data streams in minutes using one of the standard format transformation documents. These existing transformations are explained further in the [Transformation library](https://app.gitbook.com/@highcohesion/s/public/control-panel/setup/transformations/transformation-library) documentation. Our Control Panel not only enables you to use existing transformations - you can also create new, completely custom transformations to meet your requirements. When creating a data stream between a source and a destination, you will almost always require some kind of transformation to take place on the data between the systems. HighCohesion have developed a simple and effective style of JSON formatting (*referred to as a transformation file*) that allows anyone to utilise powerful transformation functions, entirely from the Control Panel. These types of transformations are explained further in the [writing a custom transformation](/creating-streams/transformations/writing-a-transformation) or the [example sales order transformation](/creating-streams/transformations/writing-a-transformation/example-1-order-file) documentation. For further reading on transformations, please see: * [Public transformations](/creating-streams/transformations/new-transformations) * [Private transformations](/creating-streams/transformations/private-transformations) * [Customising a Public Transformation](/creating-streams/transformations/customising-a-public-transformation) * [Write a custom transformation](/creating-streams/transformations/writing-a-transformation) * [Example sales order transformation](/creating-streams/transformations/writing-a-transformation/example-1-order-file) ### Public or Private? With transformation files, there are two types of visibility: * **Public** - Any user of the **HighCohesion** platform (inside and outside of your organisation) can see and use this transformation file. Public transformation files can only be edited by **HighCohesion** once live. * **Private** - Only members of your organisation can view, use, edit or delete private transformation files. When creating a new transformation file, you are able to specify if the file should be private to your business. If you create a new transformation file and select **Public**, a member of the **HighCohesion** team will review the file before it is made live to other users. # Public Transformations Transformations files that are available to anyone using HighCohesion. Public transformation files are available for any organisation in the HighCohesion platform to use. They normally represent the minimal viable fields required for an integration to function between two systems. Public transformation files will not contain any customisations but are most popular when looking to get an integration up and running quickly. Guides on what standard transformation are available in HighCohesion are covered in the [system library](/creating-streams/systems/system-library). {% hint style="info" %} If you need to customise the fields in your integration, please see [Private Transformation](/creating-streams/transformations/private-transformations) and the help guide: [Customising Public Transformations](/creating-streams/transformations/customising-a-public-transformation). {% endhint %} # Private Transformations Transformations files that are private to your organisation. Private transformation files are only visible to your organisation. These transformation are often bespoke or contain custom attribute mapping that are relative to you. Private transformations are the most common way in **HighCohesion** of customising data flows between different systems. ### Creating To create a new private transformation file, follow the steps below: 1. From the control panel, click on the [Transformations link](https://control.hico.io/resources/private-transformations/new) from the left hand navigation. 2. At the top right hand side of the page, press the button that says [**Create Transformation**](https://control.hico.io/resources/private-transformations/new). 3. This will then load the new transformation file form. On this form complete the following options: 1. **Source** - This is a drown down list of the systems available in the **HighCohesion** system. Please select from this list the source system for the data. 2. **Destination** - This is a drown down list of the systems available in the **HighCohesion** system. Please select from this list the destination system for the data. 3. **Title** - Give the Transformation file a title that is relative, for example: "*Magento x Shopify Orders*" or "*Netsuite to Shopify Customer*" etc. 4. **Data Type** - Enter the data type for the typical entity in the transformation, e.g sales\_order or customer 5. **Content** - This box is for adding the actual content of the transformation file. The content must be a valid JSON document (). For more in depth information about the HighCohesion JSON format and functions available within it, please see the [transformation developer guide](/creating-streams/transformations/writing-a-transformation) or the example[ sales order documentation](/creating-streams/transformations/writing-a-transformation/example-1-order-file). 4. Once the settings have been added, press the **Save** button at the bottom left of the page. 5. The new private transformation file will now be securely saved and available for your organisation to use in the HighCohesion system. ### Viewing Viewing a list of existing transformation files in your account can be seen by clicking on the Transformations link in the left hand navigation. Once this page has loaded you will see a table list of existing private transformation in your account. To view a single private transformation file then click on the view icon on the right hand side of the row. ### Editing To edit a private transformation file, from the list view, click on the small pen icon on the right side to load the edit form. The editing form in the control panel has the same options as shown in the "Creating" step above. ### Deleting To delete a private transformation file, from the list view, click on the small bin icon on the right side. This will trigger the deletion of the file. {% hint style="info" %} Please note that a transformation file that is associated with a stream cannot be deleted. Please remove from the stream config before deleting. {% endhint %} ### Promoting to Public If you are a developer and you have created a transformation file that could benefit other users you may want to promote this file to be [Public](/creating-streams/transformations/new-transformations). To do this, please email with the name of the organisation and the name of the transformation file to promote. The team will then review the request and promote to public if relative. Once public, your transformation file will be available to use by any organisation in the HighCohesion platform. # Customising a Public Transformation How to customise existing public transformation files. So far we've spoken about the concepts of [Public Transformation](/creating-streams/transformations/new-transformations) files and [Private Transformation](/creating-streams/transformations/private-transformations) files in the **HighCohesion** system. One of these files will form a critical part of your integration, wether you are able to use a standard public transformation file, or if you are using a customised private transformation. Customising existing Public Transformation files is the most popular option when it comes to updating a stream to working with your requirements. This process basically involves copying the public transformation in to a new private transformation and then customising. Please note writing a completely new or bespoke transformation file is detailed in [this page](/creating-streams/transformations/writing-a-transformation). ### Process The following steps are relative to the HighCohesion control panel: 1. Click on the [Transformations](https://control.hico.io/resources/transformations) link from the left hand menu 2. Click on the "[Public Transformations](https://control.hico.io/resources/private-transformations)" button at the top right hand side of the page. 3. Locate the existing transformation that you would like to customise and then click on the View link on the right side. 4. Copy (`ctrl + c` or `cmd + c`) all of the JSON transformation data that is shown in the "Content" box. 5. Go back to the [Transformations](https://control.hico.io/resources/transformations) link from the left hand menu. 6. Click on the pink "Create Private Transformation" in the top right side of the page. 7. This then shows the new transformation form, in that form complete the following data: 1. **Title** - Enter a relative title for the transformation. 2. **Source** - Select the destination system from the dropdown list. 3. **Destination** - Select the destination system from the dropdown list. 4. **Data Type** - Enter the main data type for the transformation, e.g order or shipment or customer etc 5. **Content** - Paste (`ctrl + v` or `cmd + v`) in the transformation data that was copied in step 4 above. Then either save and customise later, or customise before saving. 8. Once saved you will be able to use your new private transformation with your own customisation in it. {% hint style="info" %} For a full tutorial on how transformation logic is built in the HighCohesion system, [follow this link](/creating-streams/transformations/writing-a-transformation/example-1-order-file). {% endhint %} # Write a custom transformation ## Field Types Field types in the transformation files are defined in JSON key names that start and end with a `*` character. For example using `"*static_value*"` in the transformation document will indicate which type of field to use for the output data. The examples below will show each of the different field definitions and an example of how they can be used. ### Static Value Used for specifying a static value in the transformation file. Using the example below, the output data will contain a field called `"status"` and that field will contain the value of "Released". {% tabs %} {% tab title="static\_string\_example.json" %} ```javascript "status": { "*static_value*": "Released" } ``` {% endtab %} {% endtabs %} Static values can be any string type required, for example above, the output will be a string. The two examples below show how a integer or boolean value could be used: {% tabs %} {% tab title="static\_integer\_example.json" %} ```javascript "location_id": { "*static_value*": 1432526 } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="static\_boolean\_example.json" %} ```javascript "active": { "*static_value*": true } ``` {% endtab %} {% endtabs %} ### Producer Payload Key Using the `*ppk*` acronym in the transformation document will allow the lookup of a value based on the key provided. We defined the phrase "*Producer Payload Key*" as being the path to the data in the payload produced by the source system. The path is relative to the list that the transformation is currently looping, and hierarchy can be indicated by used a full stop. Two examples below show a path on the current level, and one that is nested. {% tabs %} {% tab title="basic\_ppk\_example.json" %} ```javascript "date": { "*ppk*": "created_at" } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="nested\_ppk\_example.json" %} ```javascript "email": { "*ppk*": "customer.email" } ``` {% endtab %} {% endtabs %} To help explain the hierarchy difference in the two example above, let's look at the example payload in from the source system below (`dummy_payload_in.json`). In the two examples above in order to map the correct data we reference two fields from the dummy payload below. The first example references a key on the top level `created_at` and maps it to a field called `date` in the output. The second example references a nested key inside of the customer block, therefore to get the nested data we have to give the full path: `customer.email`, to see a working example of this check out [Simple Example 1](broken://pages/-M4mhSRvBpxxcMkgKGtq). {% code title="dummy\_payload\_in.json" %} ```javascript { "channel": "website", "created_at": "2020-01-01 13:43:23" "customer": { "email": "joe.blogs@highcohesion.com" "firstname": "Joe", "lastname": "Blogs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } } } ``` {% endcode %} In this third example shown below, we can see how how to join two `PPK`'s together where it is necessary e.g to join first name and last name to get the full name. {% tabs %} {% tab title="ppk\_join\_example.json" %} ```javascript "full_name": { "*ppk*": [ "first_name", "last_name", ] } ``` {% endtab %} {% endtabs %} {% hint style="info" %} For more example on how the mappings files work, check out [Simple Example 1](broken://pages/-M4mhSRvBpxxcMkgKGtq) and [Simple Example 2](broken://pages/-M29Tj3wk2o2Ra4-sfmQ). {% endhint %} ### Producer Payload Key Tracked Using the `*ppk_tracked*` option in the transformation file allows the tracking of data between two systems. This normally takes the shape of tracking ID's from the source and the destination system, tying them together with a `name` field. The name field normally contains a value that is the same in both systems. Inside of the `*ppk_tracked*` field, there is the a standard `*ppk*` block with an additional `*tracked_fields*` option. This option details how the core system should track the data. The options here are: * **s\_id** - Source ID * **s\_pid** - Source Parent ID * **d\_id** - Destination ID * **d\_pid** - Destination Parent ID * **name** - A field that can contain a reference that should be the same in both the source and destination systems (e.g order number) {% code title="ppk\_tracked\_example.json" %} ```javascript "order_number": { "*ppk_tracked*": { "*ppk*": ".order", "*tracked_field*": [ "s_id", "name" ] } } ``` {% endcode %} With the addition of the key\_lookup post formatting function the core system will be able to check in our database to see if we have seen the order (in the example) before. If the system has seen it before it then aborts the transformation. For more details on `key_lookup`, [click here](broken://pages/-LzmK6Ri_4aCxXJz0e5B#key_lookup). {% code title="ppk\_tracked\_with\_deduplication\_example.json" %} ```javascript "order_number": { "*ppk_tracked*": { "*ppk*": ".order", "*tracked_field*": [ "s_id", "name" ], "*post_format*": [ { "key_lookup": { "*match*": "s_id", "*pluck*": "d_id", "*on_match*": "abort", "*on_fail*": "ppk" } } ] } } ``` {% endcode %} Alternatively, if for example this was a product record, the key\_lookup function can return the destination ID. The destination can then use a Update method rather than a create. {% code title="ppk\_tracked\_with\_existing\_id\_lookup\_example.json" %} ```javascript "id": { "*ppk_tracked*": { "*ppk*": "sku", "*tracked_field*": [ "s_id", "name" ], "*post_format*": [ { "key_lookup": { "*match*": "s_id", "*pluck*": "d_id", "*on_fail*": "empty" } } ] } } ``` {% endcode %} ```json "id": { "*ppk_tracked*": { "*ppk*": "sku", "*tracked_field*": [ "s_id", "name" ], "*post_format*": [ { "key_lookup": { "*match*": "s_id", "*pluck*": "d_id", "*on_fail*": "abort", "*abort_messsage*": "Destination product id not found. Make sure that this product is created." } } ] } } ``` ### PPK Math The ability to add, subtract, multiply or divide two ppk values. {% code title="ppk\_math\_example.json" %} ```javascript "unit_tax": { "*ppk_math*": [ "tax_lines.0.price", "/", "quantity" ] }, ``` {% endcode %} ### Stream Setting When creating a stream in the control panel optional setting parameters can easily be added. This allows for single transformation files to be used across multiple streams when they only require small changes in certain fields. For example (*shown below*) where orders could have a different "warehouse\_id" setting that is defined in the control panel via settings: ![](/files/-LdEKIQcN71XMvAtmQvj) The `*stream_setting*` option in the mapping file example shown below will attempt a lookup based on the defined JSON value. Seen below is an JSON example of how the setting from the setting from the screenshot above could be pulled in to the transformation: ```javascript "warehouse": { "*stream_setting*": "warehouse_id" } ``` The result of using the above code/settings would be that the output data would contain then value`ROTH001` with a key of `warehouse`. ### If Statement Used to execute a simple if statement on one or multiple input data variables. The example below shows changing the output of the `line1` line based on if the input data has a company name set: {% tabs %} {% tab title="inline\_if\_example.json" %} ```javascript "Haulier code":{ "*ppk*": "shipping_lines.0.title", "*post_format*":[ { "if":{ "value": "DPD Express Next Working Day Delivery", "expression": "==", "then": [{ "static_value": "DPD ND" }], "else": [{ "static_value": "DPD 2DAY" }] } } ] }, ``` {% endtab %} {% endtabs %} 
"Haulier code":{
  "*ppk*": "shipping_lines.0.title",
  "*post_format*":[
    {
      "if":{
        "value": "DPD Express Next Working Day Delivery",
        "expression": "==",
        "then": [{
        	"static_value": "DPD ND"
        }],
        "else": "abort",
        "abort_message": "The shipphing method is not found!"
      }
    }
  ]
}
### Value Lookup ... {% tabs %} {% tab title="vlookup\_example.json" %} ```javascript "id": { "*vlookup*": "line_items in orders" } ``` {% endtab %} {% endtabs %} ### Counter Used as an iterative counter from within the data. For example a line counter on sales order item lines. Shown below is an example of a line counter: {% tabs %} {% tab title="iterive\_counter\_example.json" %} ```javascript "PositionNo": { "*counter*": "line_item" } ``` {% endtab %} {% endtabs %} ### List & List Fields See lists definition below. ## Lists Lists will occur in most occurrences of data transformation. When it comes to the transformation process a list is used as a definition when to loop through a repeating list of input data. For example when receiving a batch of 50 orders, the transformation process will need loop through each of the orders individually in order to create the output data. Using a list definition in the transformation file is simple. The example below shows how in an input file for move order data between Shopify and FTP we define an input list in order to create an output list. ```javascript //... "*list*": "orders", "*list_fields*": { "order_number": { "*ppk*": "name" } //... ``` From the example above you can see that lists have two requirements: 1. `*list*` - The list definition first defines the path to the input data where the list is available. In the example above, where we are receiving data from Shopify, the Order data comes in the format of an array, which is keyed by the word "orders". 2. `*list_fields*` - The next definition indicates the start of the output field list. This is the point at which the input data (in example from Shopify) will start to be looped over. In the list\_fields, each field will then be created in an output array with the relative values to the input data. {% hint style="info" %} It should be noted that while inside a list, the input parameter key are relative to the list in which it is looping. For example in the raw input data, the path to the order name from Shopify would have been `orders.0.name`, but in the example above you can see that inside the list, it can simply be referenced as `name`. {% endhint %} Additional list formatting options: `*list_field*` - Instead of `*list_fields*`, the singular `*list_field*` can be used to create a list of items in the output that does not have a key. for example: ```javascript //... "tracking_numbers": { "*list*": "line_items.line_item", "*list_field*": { "*ppk*": "courier_tracking_code" } }, //... ``` `*list_count*` - Can be used to get the total count of items in a list. Example: ```javascript //... "total_lines": { "*list_count*": "line_items" }, //... ``` ## Pre-Formatting Pre-formatting functions can be used in the transformation file to alter data before it is retrieved from the transformation process. Pre-formatting needs to be added at the beginning of the transformation file to take the effect of the function before start processing the rest of the payload\_in ### check\_array Check the element array structure, if this does not exist, it will create a new structure and update the payload in. If the array exists, it will return the original payload. ``` PAYLOAD_IN.JSON { "shippingAddress": { "firstName": "Jonas", "lastName": "Munk", } } TRANSFORMATION.JSON { "*pre_format*": [ { "array_check": { "check_array_element": "shippingAddress.0", "take_array_element": "shippingAddress", "new_array_key": "shippingAddress" } } ], "addresses": { "*list*": "shippingAddress", "*list_fields*": { "first_name": { "*ppk*": "firstName" } } } } PAYLOAD_OUT.JSON { "addresses": [ { "first_name": "Jonas" } ] } ``` If you want to check a multi level array, then you can check the array key and move the result to the root level. If the array is multilevel, you need to move the result to the root level, **the function won't update multilevel arrays** ``` PAYLOAD_IN.JSON { "shippingAddress": { "node": { "addresses" : { "phone": "12345678780" } } } } TRANSFORMATION.JSON { "*pre_format*": [ { "array_check": { "check_array_element": "shippingAddress.node.addresses.0", "take_array_element": "shippingAddress.node.addresses", "new_array_key": "shippingAddress" } } ], "addresses": { "*list*": "shippingAddress", "*list_fields*": { "first_name": { "*ppk*": "firstName" } } } } PAYLOAD_OUT.JSON { "addresses": [ { "first_name": "Jonas" } ] } ``` ### group\_on The group on function can be used to group together blocks of data based on certain key within the input data. This is especially useful if data originates from flat files such as CSV. Defining a key to group the data on, such as a `parent_id` can result in output data that is multi-dimensional and more fitting to the destination system requirements. ``` For example, the following array will be transformed as ``` ```json PAYLOAD_IN.JSON [ { "barcode":"1234", "qty":1 }, { "barcode":"1234", "qty":1 } ] TRANSFORMATION.JSON { "*pre_format*": [ { "group_on": { "key": "barcode", "index": "products" } } ], "products": { "*list*": "priceList", "*list_fields*": { "barcode": { "*ppk*": "group.0.barcode" }, "qty": { "*ppk*": "group.0.qty" } } } } PAYLOAD_OUT.JSON { "products": [ { "barcode":"123", "qty":1 }, { "barcode":"1234", "qty":5 } ] } ``` ### unset\_key We can use this function to find an element inside an array, compare it with a value and remove the full item where the field was found. You can use different expressions, for example, `==, <>, >=, <=, <, >, contains, does not contain, starts with, ends with.`\ \ In the example below, we want to remove the items that contains an email with the value . ```json { "*pre_format*": [ { "unset_key": { "key": "product.line_items", "field": "email", "expression": "==", "value": "test@test.com" } } ] } ``` ## Post Formatting Post formatter functions can be used in the transformation file to alter data after it has been retrieved from the input data. Post formatting functions can be added on to fields as shown in the following example code: ```javascript "option_values": { "*ppk*": "product.options.0.values", "*post_format*": [ { "squash": "," } ] } ``` In the example above we can see that the output value (the key on the left side) of `option_values` has been mapped to the input variable (the key on the right side) of `product.options.0.values`. In the case above, we are expecting the input variable to be in the format of an array, therefore we apply the post format function of `squash`, then pass in the option ",". The net result of this post formatting will be that the input array will be squashed down to a single string, using a "," to separate the different options (if found). Post formatting options can be daisy chained together, the functions will be executed in the order in which they are entered in to the transformation document. (*example below in the `add_key_to_list` definition*). {% hint style="info" %} When daisy chaining functions together, if the functions are incompatible with each other, it may result in a transformation level error. {% endhint %} ### absolute Ability to convert a negative int or float value in to a positive. {% tabs %} {% tab title="absolute\_example.json" %} ```javascript "discount": { "*ppk*": "total_discount", "*post_format*": [ { "absolute": true } ] } ``` {% endtab %} {% endtabs %} ### add\_check\_digit This function converts an EAN 13 digit code from 12 digits code. For example, you can convert this code 111456749112 into 1114567491122 {% tabs %} {% tab title="add\_check\_digit.json" %} ```javascript "sku": { "*ppk*": "product.sku", "*post_format*": [ { "add_check_digit": true } ] } ``` {% endtab %} {% endtabs %} ### add\_key\_to\_list The add key to list function will normally be used in conjunction with the expand function above. This function will loop a list and prefix each element with a defined key. The example below builds on the `expand` example below, where by it adds in a key of `src` to each of the image elements in the array. {% tabs %} {% tab title="add\_key\_to\_list\_example.json" %} ```javascript "images": { "*ppk*": "group.0.images", "*post_format*": [ { "expand": "," }, { "add_key_to_list": "src" } ] } ``` {% endtab %} {% endtabs %} ### stream\_setting Allows the retrieval of a setting for from stream config file. This includes source and destination settings. This option is typically only used in conjunction with the if statement. {% tabs %} {% tab title="stream\_setting\_example.json" %} ```javascript "language_iso3": { "*ppk*": "note_attributes.0.value", "*post_format*": [ { "if": { "expression": "in", "value": [ "de", "fr" ], "then": [ { "ppk": "note_attributes.0.value" } ], "else": [ { "stream_setting": "language" } ] } } ] } ``` {% endtab %} {% endtabs %} ### stream\_setting\_lookup Allows the retrieval of a setting value. For now, there is only one option, which is to pass the output as the dictionary key. In the future, we can add more options to get the values. {% tabs %} {% tab title="stream\_setting\_example.json" %} ```javascript "ShippingCode": { "*ppk*": ".shippingLine.code", "*post_format*": [ { "stream_setting_lookup": { "get_key_from": "output" } }, { "get_value_from_dic": { "key": "code" } } ] } ``` {% endtab %} {% endtabs %} ### calculate This function allows for the mathematical calculation of a given string, normally used in conjunction with `ppk_replace` when a calculated figure is required from the input data. {% tabs %} {% tab title="calculate\_example.json" %} ```javascript "tax": { "*static_value*": "total - shipping - subtotal" "*post_format*": [ { "ppk_replace": { "total": "grand_total", "shipping": "shipping_amount", "subtotal": "subtotal" }, "calculate": true } ] } ``` {% endtab %} {% endtabs %} ### camel\_case This functions allows a string to be converted into a camel case string. For example it would change United Kingdom to unitedKingdom. ```json "country": { "*ppk*": "country", "*post_format*": [{ "camel_case": true }] } ``` ### camel\_case\_to\_sentence This allows for a camel case string to be converted to sentence case. For example it would change camelCaseString to Camel case string ```json "ShippingAddressCountry": { "*ppk*": "country", "*post_format*": [ { "camel_case_to_sentence": true } ] }, ``` ### cast This function allows for field values to be cast to a specific string type. The example below will cast the quantity output to an integer. {% tabs %} {% tab title="cast\_example.json" %} ```javascript "quantity": { "*ppk*": "qty_ordered" "*post_format*": [ { "cast": "int" } ] } ``` {% endtab %} {% endtabs %} ### country\_code\_convert (to be released in Jan 2020) Different systems can often hand country codes in different formats. This function allows the simple translation of one format to another. Shown in the example below is a conversion from a source system giving the country code in ISO2 format and a destination system requirement of ISO3 formatting: {% tabs %} {% tab title="contry\_code\_convert\_example.json" %} ```javascript "country": { "*ppk*": "billing_address.country" "*post_format*": [ { "iso2_to_iso3": true } ] } ``` {% endtab %} {% endtabs %} Our system allows for the following conversion options: * ISO2 (ISO 3166-1 alpha-2) * ISO3 (ISO 3166-1 alpha-3) * ISO# (ISO 3166-1 numeric) * NAME (Official state name) Example converting a country name to a ISO2 code {% tabs %} {% tab title="name\_to\_iso2\_example.json" %} ```javascript "country": { "*ppk*": "billing_address.country" "*post_format*": [ { "name_to_iso2": true } ] } ``` {% endtab %} {% endtabs %} Example converting a ISO2 code to a ISO3 code {% tabs %} {% tab title="iso2\_to\_iso3\_example.json" %} ```javascript "country": { "*ppk*": "billing_address.country" "*post_format*": [ { "iso2_to_iso3": true } ] } ``` {% endtab %} {% endtabs %} {% hint style="info" %} More information on country code formatting can be found at: {% endhint %} ### col\_lookup Ability to pluck data from the data storage table. {% tabs %} {% tab title="col\_lookup\_example.json" %} ```javascript "variant_id": { "*ppk*": "sku" "*post_format*": [ { "col_lookup": { "*match*": "name", "*pluck*": "d_id", "*data_type*": "product", "*key_type*": "source", "*on_fail*": "empty" } } ] } ``` {% endtab %} {% endtabs %} {% code title="col\_lookup\_example\_abort.json" %} ```json "variant_id": { "*ppk*": "sku" "*post_format*": [ { "col_lookup": { "*match*": "name", "*pluck*": "d_id", "*data_type*": "product", "*key_type*": "source", "*on_match*": "abort", "*abort_message*": "Product has already been sent" } } ] } ``` {% endcode %} ### current\_timestamp Gets the current Unix timestamp. This can then be formatted in by date\_format. {% tabs %} {% tab title="current\_timestamp.json" %} ```javascript "current_timestamp": { "*static_value*": "" "*post_format*": [ { "current_timestamp": true } ] } ``` {% endtab %} {% endtabs %} additional example with formatting: {% tabs %} {% tab title="current\_timestamp\_formatted.json" %} ```javascript "current_date": { "*ppk*": "na", "*post_format*": [ { "current_timestamp": true, "date_format": { "input": "%s", "output": "%Y%m%d" } } ] } ``` {% endtab %} {% endtabs %} ### date\_format The date post formatter function can be used to transform date variables between different systems where the destination date time format may be different from the source. You must specify the input and the output formats. timedelta option allows you to increase or decrease the date field. {% tabs %} {% tab title="date\_example.json" %} ```javascript "order_date": { "*ppk*": "created_at", "*post_format*": [ { "date_format": { "input": "%Y-%m-%dT%H:%M:%S%z", "output": "%Y%m%d%H%M%S%w", "timedelta": 300 // OPTIONAL } } ] } ``` {% endtab %} {% endtabs %} {% hint style="info" %} For guidance on the formatting string options please refer to the Python documentation for datetime at: {% endhint %} You can also convert the timezone of a datetime string by adding the input and output timezone. For example: {% tabs %} {% tab title="First Tab" %} ```json "order_date": { "*ppk*": "createdAt", "*post_format*": [ { "date_format": { "input": "%Y-%m-%dT%H:%M:%SZ", "input_timezone": "utc", "output": "%Y-%m-%dT%H:%M:%S", "output_timezone": "Europe/London" } } ] } ``` {% endtab %} {% tab title="Second Tab" %} {% endtab %} {% endtabs %} ### do\_nothing Commonly only used in conjunction with the if function. This will no just return the value as is. {% tabs %} {% tab title="do\_nothing\_example.json" %} ```javascript TBC ``` {% endtab %} {% endtabs %} ### expand Expand is the opposite function to the squash function shown above. It will allow a given string to be expanded in to an array based on a separator. The example below shows expanding an input string based on a comma to create an array. {% tabs %} {% tab title="expand\_example.json" %} ```javascript "images": { "*ppk*": "group.0.images", "*post_format*": [ { "expand": "," } ] } ``` {% endtab %} {% endtabs %} The code above is expecting to receive a comma separated list of image URLs (from the input key of `group.0.images`), which it will then split based on a comma. ### fallback The ability to fallback to another value if the value of the `ppk` is empty. {% tabs %} {% tab title="fallback\_example.json" %} ```javascript "first_name": { "*ppk*": "firstname", "*post_format*": [ { "fallback": "addresses.billing_address.firstname" } ] } ``` {% endtab %} {% endtabs %} ### format\_numeric\_str This function formats a string or numeric value into a formatted string value. To find more information about the format options see ```javascript "price": { "*ppk*": "price", "*post_format*": [ { "format_numeric_str": "This is the new price {0:.2f}" } ] } ``` The result will be Input: ```javascript { "price": 12.0222222 } ``` Output: ```javascript { "price": "This is the new price 12.02" } ``` ### get\_increment This function returns an incremented value. ```javascript "order_number": { "*static_value*": "0", "*post_format*": [ { "get_increment": { "universal_key": "random_uuid" // this could be the stream id } } ] }, ``` ### get\_lead\_zero Fill the output with zeros at the left of the string. ```javascript "order_number": { "*ppk*": "id", // 52645 "*post_format*": [ { "get_lead_zero": 15 } ] }, The output will be a string like 000000000052645 ``` ### get\_value\_from\_dic Similar to the default get\_*value function. This allows you to apply the same logic in the post\_formatting array*. For example, you can get a dict from the stream settings then get a specific value by key. ```javascript "ShippingCode": { "*ppk*": ".shippingLine.code", "*post_format*": [ { "stream_setting_lookup": { "get_key_from": "output" } }, { "get_value_from_dic": { "key": "code" } } ] }, ``` ### list\_fallback This function is used to fallback to a value relative to the list that is currently being iterated over. {% tabs %} {% tab title="list\_fallback\_example.json" %} ```javascript "price": { "*ppk*": "parent_item.price", "*post_format*": [ { "list_fallback": "price" } ] } ``` {% endtab %} {% endtabs %} ### max\_length This function allows you to define the maximum length of a given field in the transformation document. This is helpful if the destination system has a limit on the number of characters allowed in a field. The example below shows how a destination system that has a hard limit of 36 characters on the address\_line\_one field can be accommodated. {% tabs %} {% tab title="uppercase\_example.json" %} ```javascript "address_line_one": { "*ppk*": "billing_address.address1", "*post_format*": [ { "max_length": 36 } ] } ``` {% endtab %} {% endtabs %} {% hint style="info" %} Consideration should be given when limiting the length of fields in the transformation document. Building on the example above, if your source system has allowed user input of more than 36 characters in the address line one field, the input will simply be truncated when reaching the destination system. This could cause issues if data is truncated. {% endhint %} ### lowercase This function allows will change all of the characters in the given field to be lowercase. {% tabs %} {% tab title="lowercase\_example.json" %} ```javascript "first_name": { "*ppk*": "billing_address.firstname", "*post_format*": [ { "lowercase": true } ] } ``` {% endtab %} {% endtabs %} ### length\_value Returns the length value of a given field. {% tabs %} {% tab title="length\_value\_example.json" %} ```javascript "description_length": { "*ppk*": "description", "*post_format*": [ { "length_value": true } ] } // input { "description": "test" } // output { "description_length": 4 } ``` {% endtab %} {% endtabs %} ### ltrim Trim the given character from the left side of a value. {% tabs %} {% tab title="suffix\_example.json" %} ```javascript "description": { "*ppk*": "desc", "*post_format*": [ { "ltrim": " " } ] } ``` {% endtab %} {% endtabs %} ### negative Ability to convert a int or float value in to a negative. {% tabs %} {% tab title="absolute\_example.json" %} ```javascript "discount": { "*ppk*": "total_discount", "*post_format*": [ { "negative": true } ``` {% endtab %} {% endtabs %} ### pad This function allows create padding to the right or left. Where "side", "length" and "char" definable. "side" - is the field which defines padding side, "length" - is the field which defines padding length and "char" - is the field which defines padding symbol. {% code title="pad\_example.json" %} ```javascript "test_order_number_padded": { "*ppk*": "name", "*post_format*": [ { "pad": { "side": "right", "length": 40, "char": "0" } } ] }, ``` {% endcode %} ### phone\_numbers\_only The same as `numbers_only`, but allows for the `+` symbol in addition to the numbers 0 to 9. {% code title="phone\_numbers\_only.json" %} ```javascript "phone": { "*ppk*": "telephone", "*post_format*": [ { "phone_numbers_only": true } ] } ``` {% endcode %} ### ppk\_replace This function allows replacing of words in a string (this could be a static\_value or a ppk value) with value from another ppk. {% tabs %} {% tab title="ppk\_replace\_example.json" %} ```javascript "tax": { "*static_value*": "total - shipping - subtotal" "*post_format*": [ { "ppk_replace": { "total": "grand_total", "shipping": "shipping_amount", "subtotal": "subtotal" } } ] } ``` {% endtab %} {% endtabs %} ### math This function allows using simple math functionality inline in mapping file done. Where operator" and "value" definable. "operator" - is the mathematical operator, "value" - is the value for mathematical operations. {% code title="math\_example.json" %} ```javascript "first_name": { "*ppk*": "billing.price", "*post_format*": [ { "math": { "operator": "*" "value": 1.25 } } ] } ``` {% endcode %} In the example above, output\_value will be multiplied on . ### math\_round This function allows to round a value to X decimal values. {% code title="math\_example.json" %} ```javascript "price": { "*ppk*": "billing.price", "*post_format*": [ { "math_round": 2 } ] } ``` {% endcode %} 
{
 "price": 10.33
}
### numbers\_only Ability to strip out all other characters from a string that are not 1 2 3 4 5 6 7 8 or 9. {% code title="numbers\_only\_example.json" %} ```javascript "phone": { "*ppk*": "telephone", "*post_format*": [ { "numbers_only": true } ] } ``` {% endcode %} ### if The "If" function can cause 3 type of result. Where "expression", "output" and "then" - definable.\ "expression" - is the comparison operator, "output" - comparison value and "then" - result action. Current list of expression operations: * in * contains * does not contain * starts with * ends with * \> * < * \=> * <= * <> The first is running of the indicated functions {% code title="if\_value\_equals\_string\_format\_example.json" %} ```javascript "first_name": { "*ppk*": "billing_address.first_name", "*post_format*": [ { "if": { "expression": "==", "value": "Andy", "then": [ { "uppercase": true } ] } } ] } ``` {% endcode %} In the example above, if current value is equal to "output" then all functions from "then" block will be run. The second result enable you to delete a block with the given type of if. For example: {% code title="if\_value\_equals\_string\_remove\_block\_example.json" %} ```javascript "first_name": { "*ppk*": "billing_address.first_name", "*post_format*": [ { "if": { "expression": "==", "value": "Andy", "then": "drop_block" } } ] } ``` {% endcode %} In the example above, if current value is equal to "output" then "first\_name" field will be deleted form result object. The third result enables to to halt or abort an event on certain parameters being. {% code title="if\_value\_equals\_string\_abort\_example.json" %} ```javascript "first_name": { "*ppk*": "billing_address.first_name", "*post_format*": [ { "if": { "expression": "==", "value": "Andy", "then": "abort", "abort_message": "this name is not valid" } } ] } ``` {% endcode %} In the example above, if current value is equal to "output" then the event will be aborted in the transformation process based on the output\_value. ```json "first_name": { "*ppk*": "billing_address.first_name", "*post_format*": [ { "if": { "expression": "==", "value": "Andy", "then": "error" "error_message": "This name is not valid" } } ] } ``` In the example above, we are going to fail the event when the logic is true Another example of how if can be used is with an else condition as shown below: {% code title="if\_else\_example.json" %} ```javascript "published_at": { "*ppk*": "status", "*post_format*": [ { "if": { "expression": "==", "value": "1", "then": [ { "ppk": "created_at" } ], "else": [ { "static_value": "" } ] } } ] } ``` {% endcode %} ### remove\_first\_characters Removes the first X characters from a string. The below would return String ``` { "test": { "*static_value*": "Test String" "*post_format*": [{ "remove_first_characters": 5 }] } ``` ### remove\_last\_characters Removes the last X characters from a string. The below would return Test ``` { "test": { "*static_value*": "Test String" "*post_format*": [{ "remove_last_characters": 6 }] } ``` ### static\_fallback Ability to fallback to a static value if the `ppk` is empty, the value of `static_value` will be used in the output data instead of blank. {% tabs %} {% tab title="static\_fallback\_example.json" %} ```javascript "ship_by": { "*ppk*": "shipping_title", "*post_format*": [ { "static_fallback": "Standard Shipping" } ] } ``` {% endtab %} {% endtabs %} ### split\_and\_take This function allow a string to be broken and partially returned. In the example below, it is splitting the street value from the source system (*Magento in this case*) and splitting on a `new line`. In the example, `address1` takes the first element, and `address2` takes the second element. \**Note*: If the input string doesn't contain the `on` value, it will only return the input value in part `0`. {% code title="split\_and\_take\_example.json" %} ```javascript "address1": { "*ppk*": "street", "*post_format*": [ { "split_and_take": { "on": "\n", "take": 0 } } ] }, "address2": { "*ppk*": "street", "*post_format*": [ { "split_and_take": { "on": "\n", "take": 1 } } ] } ``` {% endcode %} ### sum\_list\_key This function allows to calculate the sum of the items in an array. {% code title="" %} ```javascript "total_price": { "*static_value*": 0, "*post_format*": [ { "sum_list_key": { "list": "LineItem", "key": "OrderLine.PurchasePrice" } } ] } ``` {% endcode %} ### sum\_list\_key1\_by\_key2 This function allows to calculate the sum of the items in an array multiplying them by a second item in the array. {% code title="" %} ```javascript "total_price": { "*static_value*": 0, "*post_format*": [ { "sum_list_key1_by_key2": { "list": "LineItem", "key_1": "OrderLine.PurchasePrice", "key_2": "OrderLine.OrderQty" } } ] } ``` {% endcode %} ### table\_lookup This function allows to lookup a value based on an input. Possibly a JSON representation of a table etc. Where "*match*" - is the comparison operator, "*table*" - is the simple key x value table, "*fallback*" - is the optional field, which will be assigned to "output\_value" if "match" statement is false. This will give the event transformer a basic ability of a lookup table, stored in the JSON, see the example below: {% code title="table\_lookup\_example.json" %} ```javascript "currency_test": { "*ppk*": "currency", "*post_format*": [ { "table_lookup": { "*match*": "==", "*table*": { "GBP": "GB Pounds", "USD": "US Dollar", "EUR": "Euro", "JPY": "Japanese Yen" } } } ] } ``` {% endcode %} In the example above, the output\_value is mapped to currency, which in the case of our example will give GBP. The table\_lookup is a simple key x value table which should use the output value to == exact match in this case to convert the output\_value to ‘GB Pounds’. {% code title="table\_lookup\_with\_fallback\_example.json" %} ```javascript "currency_test": { "*ppk*": "currency", "*post_format*": [ { "table_lookup": { "*match*": "==", "*fallback*": "GB Pounds", "*table*": { "USD": "US Dollar", "EUR": "Euro", "JPY": "Japanese Yen" } } } ] } ``` {% endcode %} To use the dynamic table lookup as defined in the control panel area: [Lookup Tables](https://staging.control.hico.io/resources/lookup-tables/) you can pass in the UUID of the lookup table (created, viewed and editable in the control panel) in the transformation file, like in the example shown below: {% code title="table\_lookup\_with\_fallback\_example.json" %} ```javascript "currency_test": { "*ppk*": "currency", "*post_format*": [ { "table_lookup": { "*match*": "==", "*fallback*": "GB Pounds", "*table*": "f670f2c0-4521-11eb-4c83-r98fbd0c4463" } } ] } ``` {% endcode %} ### title\_case This will make each new word in a sentence be upper case. For example "A great sentence for example" would be converted to "A Great Sentence For Example" ```json "ShippingAddressCountry": { "*ppk*": "shippingAddress.country", "*post_format*": [ { "title_case": true } ] }, ``` ### key\_lookup This is a powerful function that allows the lookup of data stored in the HighCohesion entity table. This function allows to give the user the ability to switch the mapping file all together, e.g. Stop and Start again! with a new transformation ID. {% code title="key\_lookup\_example.json" %} ```javascript "id": { "*ppk*": "ref", "*post_format*": [ { "key_lookup": { "*match*": "s_id", "*pluck*": "d_id", "*on_fail*": "empty", "*on_match*": "transformation_switch", "*transformation_id*": "ef755f7a-bdcc-11e9-8692-366c3ae2703e" } } ] } ``` {% endcode %} ### list\_find This function allows to lookup a value by the key in nested JSON data, e.g. if the output\_value is a JSON string, decode it and lookup the desired value. Where "match", "field" and "return" - definable. "match" - is the name of key, "field"- is the value of the key and "return" - is the name of getting value. {% code title="list\_find\_example.json" %} ```javascript "order_lang_from_tags_test": { "*ppk*": "order_options_tags", "*post_format*": [ { "list_find": { "match": "name", "field": "lang_tag", "return": "value" } } ] } ``` {% endcode %} ### json This function allows to lookup a value by the key in nested JSON data, e.g. if the output\_value is a JSON string, decode it and lookup the desired value. {% code title="json\_example.json" %} ```javascript "order_lang_from_json_test": { "*ppk*": "order_options_json", "*post_format*": [ { "json": { "key": "value" } } ] } ``` {% endcode %} ### serial This function allows ability to lookup a value in serialized data, e.g. if the output\_value is serialized data, function deserialize it and lookup the desired value. {% code title="serial\_example.json" %} ```javascript "order_lang_from_serial_test": { "*ppk*": "order_options_serial", "*post_format*": [ { "serial": { "key": "value" } } ] } ``` {% endcode %} ### prefix This function allows for simple prefixing to to done with any field in the transformation process. The example shown below will add a prefix of `UKWEB` to the start of the `order_number` field. {% tabs %} {% tab title="prefix\_example.json" %} ```javascript "order_number": { "*ppk*": "name", "*post_format*": [ { "prefix": "UKWEB" } ] } ``` {% endtab %} {% endtabs %} ### random\_value This function returns a random value. There are three options: uuid, integer or string. {% tabs %} {% tab title="random\_value\_uuid\_example.json" %} 
"order_number": {
  "*static_value*": "",
  "*post_format*": [
    {
      "random_value": {
        "type": "uuid"
      }
    }
  ]
}

// Result
{
  "order_number": "49c7d4fb-7000-4a8d-bfe4-10b83362e895"
}
{% endtab %} {% endtabs %} {% tabs %} {% tab title="random\_value\_string\_example.json" %} 
"order_number": {
  "*static_value*": "",
  "*post_format*": [
    {
      "random_value": {
        "type": "string",
        "length": 10
      }
    }
  ]
}

// Result
{
  "order_number": "dekalstlsa"
}
{% endtab %} {% endtabs %} {% tabs %} {% tab title="random\_value\_integer\_example.json" %} 
"order_number": {
  "*static_value*": "",
  "*post_format*": [
    {
      "random_value": {
        "type": "integer",
        "length": 20
      }
    }
  ]
}

// Result
{
  "order_number": 2134567845623579764
}
{% endtab %} {% endtabs %} ### replace This function allows for text based find and replace to to done with any field in the transformation process. The example shown below will replace any occurrence of the string `#UK` with the new string of `ONLINE`. {% tabs %} {% tab title="replace\_example.json" %} ```javascript "order_number": { "*ppk*": "name", "*post_format*": [ { "replace": { "find": "#UK", "replace": "ONLINE" } } ] } ``` {% endtab %} {% endtabs %} ### regex This function allows for regex-based validation to be applied to any field during the transformation process. The example shown ensures that any value for `commodityCode` must match the pattern of one or more digits `[0-9]+` **Note:** Avoid using `\` in your regex patterns to prevent issues with escaping. {% tabs %} {% tab title="replace\_example.json" %} ```json "commodityCode": { "*ppk*": "commodity_code", "*post_format*": [ { "regex": { "pattern": "[0-9]+" } } ] } ``` {% endtab %} {% endtabs %} ### rtrim Trim the given character from the right side of a value. {% tabs %} {% tab title="suffix\_example.json" %} ```javascript "description": { "*ppk*": "desc", "*post_format*": [ { "rtrim": " " } ] } ``` {% endtab %} {% endtabs %} ### squash Squash is used to merge an array of data down in to a single string. It uses the Python function of `.join()` to join the elements of the array and is similar to the `implode()` function in PHP. The example shown below gives an example of how an input array (being `product.options.0.values`) is converted to a comma separated list of the values. ```javascript "option_values": { "*ppk*": "product.options.0.values", "*post_format*": [ { "squash": "," } ] } ``` The value shown in speech marks after the `squash` key is the join character. For example if this was changed to a "|" pipe, the product options would be merged together with the pipe symbol instead of a comma. ### suffix This function allows for simple suffixing to to done with any field in the transformation process. The example shown below will add a prefix of `PAYPAL` to the end of the `order_number` field. {% tabs %} {% tab title="suffix\_example.json" %} ```javascript "order_number": { "*ppk*": "name", "*post_format*": [ { "suffix": "PAYPAL" } ] } ``` {% endtab %} {% endtabs %} ### trim Trim the given character from the start or end of a value. {% tabs %} {% tab title="suffix\_example.json" %} ```javascript "description": { "*ppk*": "desc", "*post_format*": [ { "trim": " " } ] } ``` {% endtab %} {% endtabs %} ### uppercase This function allows will change all of the characters in the given field to be uppercase. {% tabs %} {% tab title="uppercase\_example.json" %} ```javascript "first_name": { "*ppk*": "billing_address.firstname", "*post_format*": [ { "uppercase": true } ] } ``` {% endtab %} {% endtabs %} ## List Post Formatting List post formatting allows list modifications to be made after the data has been formatted. This is common when building flat data for CSV or XML files where source data has been multi-dimensional. ### flat\_array Transforms a multidimensional array into a single dimensional array 
For example, if your list result is like
{
    "list": [[1,2,3], [4,5,6]]
}

After applying this post format list function, the result will look
{
    "list": [1,2,3,4,5,6]
}

This is useful when you need to extract sub items from an array.

For example:

{
 "Lines": {
        "*list*": "Order.TransformLines",
        "*list_fields*": {
            "*list*": "BOMs",
            "*list_fields*": {
                "productcode": {
                    "*ppk*": "SKU"
                }
            }
        },
        "*list_post_format*": [
            {
                "flat_array": true
            }
        ]
    }
}
### merge\_down ## Message Data ### message\_data You can grab information from the event message and replace it in the given field ``` "stream_name": { "*message_data*": "stream.spec.title" } ``` # Transformation Library ### Transformation Library The transformation library is accessible via the HighCohesion control panel. The link for this section is via the [**Transformations**](https://control.highcohesion.com/resources/transformations) link in the left side menu. Once the next page has loaded, you will see a table list of all of the transformation files available in your account. This list will include all **Public** transformation files as well as any **Private** transformation files that you have created in your account. The list of transformation files can be quickly filtered by using the filter dropdown at the top right side of the list. Using filters such as Source or Destination can quickly identify if there is already a transformation file available for your data stream. # Example 1 - Sales Order Building an example sales order transformation file from scratch. We have created a simple example transformation file below. This file shows the basic concepts of a transformation. The data has two states, payload in (the data that we receive from a source system) and payload out (the converted output data we send to a destination system). We are able to convert the data between the two formats based on a single transformation file. All three are shown below: {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer_email": "joe.bloggs@highcohesion.com", "customer_name": "Joe Bloggs" } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "order_number": { "*ppk*": "order" }, "order_email": { "*ppk*": "customer_email" }, "full_name": { "*ppk*": "customer_name" } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "full_name": "Joe Bloggs" } ``` {% endtab %} {% endtabs %} ‌So now we know how the most basic of transformations can be built in the HighCohesion format. In the next section we build out more examples to help build your understanding. Throughout the following examples the `payload_in.json` will remain the same, however the `payload_out.json` will change depending on the changes to the transformation file.‌ ## Scenario ‌In the example above, we had just mapped three basic fields. For the following sections, please imagine that aim is to get the data in to a format so that a warehouse system can read the a CSV file (generated after payload out is sent to the destination) and contain enough information to ship the customers order.‌ ### Start with the basics ‌Lets start with just the basic order information, pretty simple we're just going to map three fields to start with: {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer": { "email": "joe.bloggs@highcohesion.com", "firstname": "Joe", "lastname": "Bloggs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } }, "address": { "street": "43 Test Street,\nTest town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } "order_items": [ { "sku": "ABF3439", "price_each": 9.5, "quantity": 2 }, { "sku": "GHNT0001", "price_each": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "order_number": { "*ppk*": "order" }, "order_email": { "*ppk*": "customer_email" }, "ship_service": { "*ppk*": "shipping_type" } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day"" } ``` {% endtab %} {% endtabs %} ### ‌Adding in the address Next we will add in the address fields: {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer": { "email": "joe.bloggs@highcohesion.com", "firstname": "Joe", "lastname": "Bloggs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } }, "address": { "street": "43 Test Street,\nTest town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } "order_items": [ { "sku": "ABF3439", "price_each": 9.5, "quantity": 2 }, { "sku": "GHNT0001", "price_each": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% tab title="Transformation" %} ```coffeescript { "order_number": { "*ppk*": "order" }, "order_email": { "*ppk*": "customer_email" }, "ship_service": { "*ppk*": "shipping_type" }, "address_line_1": { "*ppk*": "address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 0 } } ] }, "address_line_2": { "*ppk*": "address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 1 } } ] }, "city": { "*ppk*": "address.city" }, "post_code": { "*ppk*": "address.post_code" }, "country": { "*ppk*": "address.country" }, "phone": { "*ppk*": "address.phone" } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } ``` {% endtab %} {% endtabs %} Note that we've added in a new concept here of post formatting functions. These are instructions to the core system to apply a function to the field before returning it to the payload\_out.json. In the example above, our payload\_in.json contained the users street fields combined in one field. Therefore we added in a `*post_format*` block to instruct the core system to format the output value, in this case we used the `split_and_take` function.‌ ### Add in the customer ‌Next we will add in some of the basic customer information: {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer": { "email": "joe.bloggs@highcohesion.com", "firstname": "Joe", "lastname": "Bloggs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } }, "address": { "street" "43 Test Street,\nTest town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } "order_items": [ { "sku": "ABF3439", "price_each": 9.5, "quantity": 2 }, { "sku": "GHNT0001", "price_each": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "order_number": { "*ppk*": "order" }, "order_email": { "*ppk*": "customer_email" }, "ship_service": { "*ppk*": "shipping_type" }, "address_line_1": { "*ppk*": "address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 0 } } ] }, "address_line_2": { "*ppk*": "address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 1 } } ] }, "city": { "*ppk*": "address.city" }, "post_code": { "*ppk*": "address.post_code" }, "country": { "*ppk*": "address.country" }, "phone": { "*ppk*": "address.phone" }, "customer_name": { "*ppk*": [ "customer.firstname", "customer.lastname" ] }, "allows_marketing": { "*ppk*": "customer.allows_marketing" } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439", "customer_name": "Joe Bloggs", "allows_marketing": true } ``` {% endtab %} {% endtabs %} Note that we've added in a new concept here using multiple values in the `*ppk*` field. This can be seen on the `customer_name` field in the transformation, which is a join for `firstname` and `lastname` from the payload in. The core system grab each of these separate values and join the together with a space.‌ ### Add in the items ‌Getting slightly harder now. We need to add in the items, however in our payload\_in.json there are multiple items. In order to get all of the data, we will use the [List](https://app.gitbook.com/@highcohesion/s/control-panel/control-panel-functions/integration-setup/transformations/developer-guide#list-and-list-fields) functionality: {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer": { "email": "joe.bloggs@highcohesion.com", "firstname": "Joe", "lastname": "Bloggs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } }, "address": { "street" "43 Test Street,\nTest town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } "order_items": [ { "sku": "ABF3439", "price_each": 9.5, "quantity": 2 }, { "sku": "GHNT0001", "price_each": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "*list*": "order_items", "*list_fields*": { "order_number": { "*ppk*": ".order" }, "order_email": { "*ppk*": ".customer_email" }, "ship_service": { "*ppk*": ".shipping_type" }, "address_line_1": { "*ppk*": ".address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 0 } } ] }, "address_line_2": { "*ppk*": ".address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 1 } } ] }, "city": { "*ppk*": ".address.city" }, "post_code": { "*ppk*": ".address.post_code" }, "country": { "*ppk*": ".address.country" }, "phone": { "*ppk*": ".address.phone" }, "customer_name": { "*ppk*": [ ".customer.firstname", ".customer.lastname" ] }, "allows_marketing": { "*ppk*": ".customer.allows_marketing" } } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { [ { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439", "customer_name": "Joe Bloggs", "allows_marketing": true, "sku": "ABF3439", "price": 9.5, "quantity": 2 }, { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439", "customer_name": "Joe Bloggs", "allows_marketing": true, "sku": "GHNT0001", "price": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% endtabs %} Here we introduced the concept of lists. Lists occur in payload where the are multiples of the same object block. In our example of an order, the order has two different line items in it. The `*list*` and `*list_fields*` keys appear at the start of the list. Please also note the `.` that has appeared at the start of the existing `*ppk*` values. This denotes that the key path is absolute. Remember that when you are inside a list the keys will be relative to the data list that we're looping through. To access data outside of the list item we use the absolute path. Further detail on lists can be found [here](/creating-streams/transformations/writing-a-transformation#list-and-list-fields).‌ ### Add in tracking ‌In this part we will add in some tracking fields so that we can use HighCohesions inbuilt de-duplication options to help protect against any duplicate orders being shipped. {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sub_total": 48.5, "shipping": 4.95, "grand_total": 53.45, "shipping_type": "Next Day", "created_at": "2020-01-01 13:43:23", "customer": { "email": "joe.bloggs@highcohesion.com", "firstname": "Joe", "lastname": "Bloggs", "marketing": { "allow": true, "last_updated": "2020-01-01 13:43:23" } }, "address": { "street" "43 Test Street,\nTest town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439" } "order_items": [ { "sku": "ABF3439", "price_each": 9.5, "quantity": 2 }, { "sku": "GHNT0001", "price_each": 29.5, "quantity": 1 } ] } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "--source_id": { "*ppk_tracked*": { "*ppk*": "order", "*tracked_field*": [ "s_id", "name" ] } }, "*list*": "order_items", "*list_fields*": { "order_number": { "*ppk*": ".order" }, "order_email": { "*ppk*": ".customer_email" }, "ship_service": { "*ppk*": ".shipping_type" }, "address_line_1": { "*ppk*": ".address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 0 } } ] }, "address_line_2": { "*ppk*": ".address.street" "*post_format*": [ { "split_and_take": { "on": "\n", "take": 1 } } ] }, "city": { "*ppk*": ".address.city" }, "post_code": { "*ppk*": ".address.post_code" }, "country": { "*ppk*": ".address.country" }, "phone": { "*ppk*": ".address.phone" }, "customer_name": { "*ppk*": [ ".customer.firstname", ".customer.lastname" ] }, "allows_marketing": { "*ppk*": ".customer.allows_marketing" } } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript [ { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439", "customer_name": "Joe Bloggs", "allows_marketing": true, "sku": "ABF3439", "price": 9.5, "quantity": 2 }, { "order_number": "#GB123042", "order_email": "joe.bloggs@highcohesion.com", "ship_service": "Next Day", "address_line_1": "43 Test Street,", "address_line_2": "Test town", "city": "Testercity", "post_code": "TE5 7ER", "country": "United Kingdom", "phone": "+442043939439", "customer_name": "Joe Bloggs", "allows_marketing": true, "sku": "GHNT0001", "price": 29.5, "quantity": 1 } ] ``` {% endtab %} {% endtabs %} Here we can see that we have modified the order\_number field as this contains a unique reference for the order. The modifications include two parts: 1. Wrapping in `*ppk_tracked*`, this denotes to the core system that this is a tracked field. Inside of the `*ppk_tracked*` field, there is the a standard `*ppk*` block with an additional `*tracked_fields*` option. This option details how the core system should track the data. The options here are: 1. **s\_id** - Source ID 2. **s\_pid** - Source Parent ID 3. **d\_id** - Destination ID 4. **d\_pid** - Destination Parent ID 5. **name** - A field that can contain a reference that should be the same in both the source and destination systems (e.g order number) 2. Addition of the `key_lookup` post formatting function. This function essentially checks in the database to see if we have seen the order (in the example) before. If the system has seen it before it then aborts the transformation.
3. By using a double negative at the start of the tracking field `--source_id`, indicates that it is not to be included in the payload\_out.json. ### The end result The above payload out with CSV settings in the destination function would result in the following CSV file being created: ``` "order_number","order_email","ship_service","address_line_1","address_line_2","city","post_code","country","phone","customer_name","allows_marketing","sku","price","quantity" "#GB123042","joe.bloggs@highcohesion.com","Next Day","43 Test Street,","Test town","Testercity","TE5 7ER","United Kingdom","+442043939439","Joe Bloggs","True","ABF3439","9.5","2" "#GB123042","joe.bloggs@highcohesion.com","Next Day","43 Test Street,","Test town","Testercity","TE5 7ER","United Kingdom","+442043939439","Joe Bloggs","True","GHNT0001","29.5","1" ``` ​ # Example 2 - Shipment Building an example order shipment transformation file from scratch. ## Scenario In our first example we created a sales order transformation file that could be used in the example of taking sales order data from a ecommerce system and sending it to a 3PL provider as a **CSV file**. Building on this, in this second example we're going to have a look at building a custom Shipment transformation file. This scenario would be for shipment data flowing back to the ecommerce system from the 3PL provider. In this example we're going to move data from a CSV file on FTP to the Shopify fulfilment REST API. ### Starting with the destination schema To begin with lets take a look at the fields that **Shopify** needs to create a fulfilment against a Sales Order, the example below being taken from {% tabs %} {% tab title="Destination schema example" %} ```javascript { "fulfillment": { "location_id": 905684977, "tracking_number": "123456789010", "tracking_company": "fed ex", "tracking_url": "https://www.new-fedex-tracking.com/?number=123456789010", "line_items": [ { "id": 466157049 }, { "id": 518995019 }, { "id": 703073504 } ] } } ``` {% endtab %} {% endtabs %} ### Reviewing the input data Next lets take a look at the example CSV file that has been provided from the 3PL partner: {% tabs %} {% tab title="CSV input data example" %} ``` "order","shipment_date","courier","tracking_code","tracking_url" "#GB123042","2021-07-01","DPD","DPD1234567890-AA","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-AA" "#GB123043","2021-07-01","DPD","DPD1234567891-BB","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567891-BB" "#GB123044","2021-07-01","DPD","DPD1234567892-CC","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567892-CC" ``` {% endtab %} {% endtabs %} In this typical CSV file that has been provided by a 3PL partner, we can see the basic fields required to keep the ecommerce system up to date with the shipment information. ### Creating the transformation Above, we have taken a look at the required destination format (payload out) and the input data from the 3PL (payload in). The next step is to bring this all together in our transformation file, mapping the input data to create the output data. The example tabs below show the data as received by **HighCohesion**, the transformation and finally the payload out that would be sent to **Shopify**. {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "shipment_date": "2021-07-01", "courier": "DPD", "tracking_code": "DPD1234567890-EE", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "fulfillment": { "location_id": { "*static_value*": 12345678900 }, "tracking_number": { "*ppk*": "tracking_code" }, "tracking_company": { "*ppk*": "courier" }, "tracking_url": { "*ppk*": "tracking_url" } } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "fulfillment": { "location_id": 12345678900, "tracking_number": "DPD1234567890-EE", "tracking_company": "DPD", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" } } ``` {% endtab %} {% endtabs %} The transformation file ends up being quite simple, just four fields are required to ship the sales order in Shopify. Three of these fields are mapped using the `*ppk*` option, meaning that the transformation logic will use a value from the payload in. The fourth value `location_id` is populated using a `*static_value*` which in the example above enters the value 12345678900 in to the payload out. {% hint style="info" %} **No Products?** In the example above there are no line items covered in the transformation file. This is due (in this example) to the 3PL file not containing the line level detail. In the **HighCohesion** system, if there are no line items, then all lines will be fulfilled automatically. {% endhint %} ### Example with line items Building on the example above, some 3PL partners offer good line level detail which can be mapped over to Shopify to create partial or split shipments. In the example below we look at how the source information has changed and how the transformation file is changed to handle it: {% tabs %} {% tab title="CSV input data with line items example" %} ``` "order","sku","qty","shipment_date","courier","tracking_code","tracking_url" "#GB123042","ABC001",3,"2021-07-01","DPD","DPD1234567890-AA","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-AA" "#GB123042","CEG032",4"2021-07-01","DPD","DPD1234567890-AA","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-AA" "#GB123042","AGJF22",1"2021-07-01","DPD","DPD1234567890-AA","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-AA" "#GB123043","BBC901",1"2021-07-01","DPD","DPD1234567891-BB","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567891-BB" "#GB123043","BBC902",1"2021-07-01","DPD","DPD1234567891-BB","https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567891-BB" ``` {% endtab %} {% endtabs %} The lines of the CSV will be grouped together by the common piece of data, in this case the order column that contains the order number will be used to group the data. {% tabs %} {% tab title="Payload In" %} ```javascript { "order": "#GB123042", "sku": "ABC001", "qty": 3, "shipment_date": "2021-07-01", "courier": "DPD", "tracking_code": "DPD1234567890-EE", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" "line_items": [ { "order": "#GB123042", "sku": "ABC001", "qty": 3, "shipment_date": "2021-07-01", "courier": "DPD", "tracking_code": "DPD1234567890-EE", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" }, { "order": "#GB123042", "sku": "CEG032", "qty": 4, "shipment_date": "2021-07-01", "courier": "DPD", "tracking_code": "DPD1234567890-EE", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" }, { "order": "#GB123042", "sku": "AGJF22", "qty": 1, "shipment_date": "2021-07-01", "courier": "DPD", "tracking_code": "DPD1234567890-EE", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" } ] } ``` {% endtab %} {% tab title="Transformation" %} ```javascript { "fulfillment": { "location_id": { "*static_value*": 12345678900 }, "tracking_number": { "*ppk*": "tracking_code" }, "tracking_company": { "*ppk*": "courier" }, "tracking_url": { "*ppk*": "tracking_url" }, "line_items": { "*list*": "line_items", "*list_fields*": { "sku": { "*ppk*": "sku" }, "quantity": { "*ppk*": "qty" } } } } } ``` {% endtab %} {% tab title="Payload Out" %} ```javascript { "fulfillment": { "location_id": 12345678900, "tracking_number": "DPD1234567890-EE", "tracking_company": "DPD", "tracking_url": "https://www.dpd.co.uk/apps/tracking/?reference=DPD1234567890-EE" "line_items": [ { "sku": "ABC001", "quantity": 3 }, { "sku": "CEG032", "quantity": 4 }, { "sku": "AGJF22", "quantity": 1 }, ] } } ``` {% endtab %} {% endtabs %} The transformation above adds in a `*list*` and `*list_fields*` block in order to create the array of line items that is sent to Shopify. {% hint style="info" %} **SKU vs ID**\ In the example above we use the SKU code in the line items as that is the data that is available from the 3PL partner shipment file. The HighCohesion system will automatically match SKU code to the line item ID in **Shopify** for fulfilment to work correctly. {% endhint %} {% hint style="info" %} For a tutorial example for [FTP to Shopify integration with CSV files](/integration-tutorials/shopify-with-ftp/shopify-x-ftp-part-2), check [this link](/integration-tutorials/shopify-with-ftp/shopify-x-ftp-part-2). {% endhint %} # Destinations The destination of data in your integration data stream. ### Overview The destination is the configuration that our stream needs to send the information to the destination System. The primary fields are ![](/files/-Mbaz5YSowMZLMYkNIXI) ### Destination List View The default page that is shown when clicking on the "Destinations" link from the left navigation is the destination list page. On this page you can view all of the existing destinations of data that you have setup in HighCohesion. Towards the right hand side of each existing row are actions for viewing, editing and deleting existing entries. {% hint style="info" %} Please note that a destination cannot be deleted it if is attached to a stream, the destination will need to be removed from the stream first or stream deleted all together. {% endhint %} ### Creating a Destination 1. From the control panel, go to the **Destination** section. 2. At the top right hand side of the page, press the button that says **Create Destination**. 3. This will then load the new Destination form. On this form complete the following options:
1. **Title** - Give the Key a title that is relative to the destination, for example: "*Shopify POST Shipments*" or "*GBP Store Shipments*" etc.
2. **System** - This is a drown down list of the systems available in the High Cohesion system. Please select from this list the system that the credentials are for. When you select a system from this list, the page will automatically load available functions in the Destination Function box below.
3. **Destination Function** - This is a dropdown list of the functions available in the System that you have selected in step 2. Please select from this list the functions that you would like to consume the data in for this destination config.
4. **Key** - The key field is a dropdown list of the keys that have already been setup in your account. Please select the key that is relative to the system and environment that you want to send data to.
5. **Trigger** - There are three options available here to control how the destination is triggered while the data stream is in use: 1. *Aggregate* - 2. *Real Time* - 3. *Scheduled* -
6. **Trigger Options** - This is an options box for the trigger above, please follow the guidelines below depending on the trigger selected above: 1. *Scheduled* - Enter a valid CRON expression for when the destination trigger should be ran. See here for help generating a valid CRON expression:
7. **Settings** - This box is for adding any additional settings or config for the destination. Settings in this box should follow the basic TOML formatting. ()
4. Once the settings have been added, press the save button at the bottom left of the page. 5. The new destination will now be securely saved and available to use in the system. ### Deleting a Destination A destination can be deleted from either the list page by clicking on the bin icon, or from the single view page, also by the bin icon. {% hint style="info" %} Please note that a destination cannot be deleted it if is attached to a stream, the destination will need to be removed from the stream first or stream deleted all together. {% endhint %} # Aggregated Events When opting to aggregate the events, the core will combine multiple events together rather than sending them to the destination individually. This is useful for systems such as file based ones as it allows you to combine multiple events into a single file. Additionally, you can choose a page size for the aggregated events. This allows control over how many events will be combined into one. For example if you set a page size of 10, and a job creates 50 events, then the aggregation will combine this into 5 events containing 10 of the original events each. {% hint style="info" %} Please be aware that when using aggregation, additional events are created that act as the "parent" event that combine all of the events that are to be aggregated. These additional events are charged for as normal. {% endhint %} # Data uploads Manual data uploading Awaiting information. # Lookup table Simple data table lookups. Awaiting information. # Updating source and destinations Controlling where the data comes from and where it goes. ## Updating A source or destination function can be easily tweaked and updated. Below are the most common changes and how to undertake them: ### Editing the Source or Destination trigger 1. Navigate to the Source or Destination list page 2. Click on the edit button for the Source or Destination you want to edit 3. Select trigger and edit the settings. If you want to add a webhook to a system which previously ran on a manual or schedule, please see **this guide** on how to enable Source webhooks. 4. Click "Update Source / Destination" {% hint style="info" %} All Jobs pending with the old schedule will be removed from the Job table, ensuring that Source or Destination edits are applied directly {% endhint %} ### Editing the Source or Destination settings 1. Navigate to the Source or Destination list page 2. Click on the edit button for the Source or Destination you want to edit 3. Edit the settings, the most common things to edit are: 1. Source filter settings 2. Destination system route path 3. Destination system file format 4. Click "Update Source / Destination" {% hint style="info" %} All Jobs pending with the old settings will be removed from the Job table, ensuring that Source or Destination edits are applied directly {% endhint %} ## Deleting A Source or Destination configuration can be securely deleted from the HighCohesion Control Panel when it is no longer required. To do this follow these steps: 1. From the control panel, go to the **Sources** or **Destinations** section. 2. Here you will see a list of sources or destinations that you have setup under your account, in this list find the configuration that you would like to remove. 3. On the far right hand side of the row, there is a small Bin icon. 4. Press this to remove the configuration from the system. {% hint style="info" %} A source or destination that is in use can not be deleted, it will need to be removed from any active Stream configurations before the system will allow you to delete it. {% endhint %} # Deleting a stream ### Deleting a stream A stream can be securely deleted from the **HighCohesion** control panel when it is no longer required. To do this follow these steps:‌ 1. From the control panel, go to the streams section. 2. Here you will see a list of stream that you have setup under your account, in this list find the Stream that you would like to remove. 3. On the far right hand side of the row, there is a small Bin icon. 4. Press this to remove the stream from the system. **Danger Zone** - Please note that deleted streams can **NOT** be recovered and will have to be recreated if needed again. # Job Statuses This page outlines the status structure for Jobs & Events ## Job statuses The Job is created by a Source and creates one or multiple Events. **A Job can exist in 5 different statuses:** * **PENDING** * When a Job is scheduled to run, but has not been kicked-off yet * **PROCESSING** * When a Job is in the process of creating Events * **FINISHED** * When a Job has successfully created Events and all Events have the status OK * **FINISHED WITH ISSUES** * When a Job has successfully created Events, but some of the Events have failed or seen issues * **FAILED** * When the Job has failed in the Source system, most commonly no Events have been created ![An overview of how the Job statuses correlates to one another](/files/-MSDh_3_sic9gyLhSWGT) # Event Statuses ## **Event** statuses An Event is created by a Job and is the vehicle that transfers a specific data type through a Stream between two systems, for example an Order file. **An Event can exist in 6 different statuses:** * **NEW** * The first time an Event is created * **TRANSFORMED** * When the Event data has been successfully transformed from the Source system data structure to fit the Destination system structure * **OK** * When an Event has been successfully transformed and posted to the Destination system * **ERROR** * When the Event errors during the transformation * **FAILED** * When the Event fails to be posted to the Destination system # Resending data ## # Replicate Job Create a new Job using the Job data for an existing Job. Powerful when a Job containing multiple Events has failed. ## When to use? The replicate Job feature is a commonly used error resolution feature. Replicating a Job allows you to edit the Job data of an existing Job. This feature is commonly used when a Job has failed, common reasons for Job failure are: * No access to Source system * Source system timeout * Exceeding worker memory limits ## How to use? ### Job replication Manual / Scheduled Source Enter the Job that you want to replicate ### Job replication Dynamic web hook ### Job replication Static web hook By entering the Event view page of an Event that has failed, you can easily click the replicate button. On clicking the button a modal window will be loaded with the complete payload-in for the specific Event. From the modal window you can edit the payload data directly based on your needs. When clicking on "Replicate Event" a new Event, with the changes to the payload will be sent directly to the Core of the system to be transformed. You can now go back to the Event list and monitor to ensure the newly created Event is processed OK. # Replicate Event Create a new Event using an existing payload-in. Powerful when you want to edit the payload-in due to faulty data entry in the Source system, such as faulty postcodes or invalid characters. ## When to use? The replicate event feature is one of the most commonly used error resolution features across the HighCohesion platform. Replicating an Event allows you to edit the payload-in of an existing Event. This feature is commonly used when an Event has failed in the Destination due to faulty data, common issues are: * Faulty postcodes * Faulty characters in text fields (e.g. space/text in phone number field) ## How to use? By entering the Event view page of an Event that has failed, you can easily click the replicate button. On clicking the button a modal window will be loaded with the complete payload-in for the specific Event. From the modal window you can edit the payload data directly based on your needs. When clicking on "Replicate Event" a new Event, with the changes to the payload will be sent directly to the Core of the system to be transformed. You can now go back to the Event list and monitor to ensure the newly created Event is processed OK. # Replicate Event without Event in Control Panel Replicate an Event that cannot be found directly from the Control Panel **Steps:** 1\. Login to your Shopify admin console 2\. Enter the Shopify order page for the order you are looking to reprocess 3\. Delete the last part of the URL. See below for example: **Before**:\ **After**:\ 4\. Add “.json” at the end of the URL. The page will reload and the order information will now be displayed as json directly in the browser. This json file is the payload-in for this specific order.\ See below for example:\ 5\. Copy the json from the browser 6\. Paste the json in a text editor and make the following two edits in the start and the end of the file: In the start, replace: ``` { "order": { ... ``` With: ``` { ... ``` In the end, remove: ``` } ``` 7\. Copy the updated json 8\. Enter the Control Panel (if you are an admin or reseller account, please ensure that you have selected the Organisation you are about to reprocess the event for) and navigate to the Event list page **9. Filter the event list with the Stream you want to replicate the event for (example Shopify x FTP (UK))** 10\. Enter the Event detail page for any of the previously processed Events that have been run within the stream you selected above 11\. In the upper right corner, click the "Replicate" button 12\. In the modal window. Delete the existing json (the payload-in for the existing event) and paste the json you copied in step 6 (e.g. your new payload-in) 13\. Click "Replicate" 14\. Enter the event list page and ensure that the new event has been processed correctly If you want to replicate multiple events, repeat step 1-14 # Reporting Data available from the control panel. Reporting elements in the control panel have been designed to give users the ability to find, diagnose and fix and issues in your data streams. Reporting has been segmented in to the following areas: * [Jobs](/monitoring-streams/reporting/jobs) * [Events](/monitoring-streams/reporting/events) * [Entities](/monitoring-streams/reporting/entities) ### Execution hierarchy A HighCohesion stream allows for data streaming to occur, however the stream is simple the configuration for for the the data stream. Once activated, either manually or via a schedule, the stream will lead to the creation of the following (*in order of hierarchy*).‌ * **Jobs** - A stream will create a job. When a stream is executed, it will create a Job. A manual job is set to run as soon as possible (in line with system capacity at the time) and a scheduled job is set to run at the next defined interval. * **Events** - A job creates between zero and many events. Events are created when the job finds relevant data in the source system. If no data is found, no events will be created. * **Entities** - Events create or update entities. Entities relate to data that is contained within an event. # Jobs Starting the process of moving data. ## Overview Jobs will be created by the system when there is need for a stream to be executed. The different options for creating a job could be: * **Manual trigger** - Triggered by a user in the control panel. * **Scheduled trigger** - Triggered on a schedule attached to the [source](/creating-streams/sources) by the system. * **Notification/web hook trigger** - Triggered by the HighCohesion system receiving notification (web hook) from a 3rd party system. Once a job has been created, it is possible to view the outcome of the job and any related log files in the control panel. ### Job List The job list page is shown when clicking on the "Jobs" link from the left hand navigation in the control panel. This table of jobs from your account are defaulted in the scheduled date/time execution order, with the latest jobs being listed at the top of the page. The jobs list can be filtered using the filter dropdown panel at the top right hand side on the table. ### Job View The job single view page allows you to view the logs and any related source files for the selected job. ## Functions ### Replicate Job The control panel allows you to replicate a previously run job. This can be done by pressing the "replicate job" button in the top right hand corner on the jobs page. You can edit some settings of the job before running it. The replication process varies slightly depending on the type of job being replicated. This is outlined below. #### Static and Dynamic Web Events When you click on replicate job on a static or dynamic web event job, a popup will appear that contains the original web event payload. You are then free to edit this payload with new information. When the job is replicated, this new payload will be used rather than the original. #### Manual and Scheduled Jobs When you click on replicate job on a manual or scheduled job, a popup will appear that contains the job specification from when the job ran. You are then free to edit the specification for the job. When the job is replicated, the new specification will be used rather than the original. # Events ## Overview Events are created by [jobs](/monitoring-streams/reporting/jobs) when relative data is found by the source function. ### Event List The event list page is shown when clicking on the "Events" link from the left hand navigation in the control panel. This table of events from your account are defaulted in the created at date/time order, with the latest events being listed at the top of the page. The events list can be filtered using the filter dropdown panel at the top right hand side on the table. ### Events list search & filter We have made it easy to find the Events you are looking for with the below search and filter options: **You can search in the main search bar by:** * Event name **You can filter by:** * Status `dropdown` * Stream `dropdown` * Source system `dropdown` * Destination system `dropdown` * Created at date range `calendar date range selector` * Destination ID `free text field` * Job ID `free text field` ![](/files/-MdRI2E1pWsYH8MKKW5A) ### Event View The event single view page allows you to view the logs and any related files for the selected event. ## Functions ### Replicate See [here](https://app.gitbook.com/@highcohesion/s/public/~/drafts/-MdR1c_QtvizdZRbgHGG/getting-started/testing-an-integration/error-resolution-wip/replicate-event) ### Re-drive See [here](https://app.gitbook.com/@highcohesion/s/public/~/drafts/-MdR1c_QtvizdZRbgHGG/getting-started/testing-an-integration/error-resolution-wip/redrive-event) ### Export as Excel You can easily select Events and export the Events list as an Excel file. The format of the file will follow the same structure as the Events list table. ![](/files/-MdRei5FrX_5N9AwPODO) # Entities Individual data entity tracking. ## Overview Within HighCohesion the concept of an Entity is a record of a specific data type, for example, an Order, Customer, or SKU. Entities come in handy to track the Entity Source and Destination identifiers for the system to apply rules to the Entity. For example, the system can identify if the Entity has already been received by the Destination system and abort a sync to avoid duplications. ### Entities List The Entities list page is shown when clicking on the "Entities" link from the left hand navigation in the control panel. This table of Entities from your account are defaulted in created at order, with the latest Entities being listed at the top of the page. The events list can be filtered using the filter dropdown panel at the top right hand side on the table. ### Entities search and filters We have made it easy to find the Entities you are looking for with the below search and filter options: **You can search in the main search bar by:** * Event name **You can filter by:** * Data type `dropdown` * Created at date range `calendar date range selector` * Source ID `free text field` * Destination ID `free text field` ![](/files/-MdRI9OSY5EoxTn2HW5C) ### Entities View The job single view page allows you to view the logs and any related source files for the selected job. ### Entities Edit Ability to edit the select entity ## Functions ### Export to Excel You can easily select Entities and export the Entities list as an Excel file. The format of the file will follow the same structure as the Entities list table. # Alerts This section specifies why the alerting service is important and how it is structured ## Why an alerting service? Errors often occur in integrations, it can be anything from faulty data entry in the Source system, to time out or formatting issues in the Destination system. When an error occurs it is of upmost importance to understand that something has happened, what happened and why it happened for an end-user to easily address the issue. The HighCohesion alerting service is design to support in all of the above four steps, in real time and in granular detail on a Job and Event basis. ## What is the alerting service and how is it structured? The HighCohesion alerting service is configurable to fit every organisations specific needs, and alerts on three alert types: #### JOB: * This alert notifies the client about a failed job. #### EVENT\_TRANSFORMED: * This alert notifies the client about a failed event in the transformation process (CORE) #### EVENT\_DESTINATION: * This alert notifies the client about a failed event in the destination (SDF) **The alerts can easily be configurable by:** * Deciding who shall receive the email alerts * Selecting which Streams from the Organisation that should be included * Configuring a flood control based on count & time, to ensure the end-user does not get spammed with emails in case a high traffic Stream sees errors * Opt-in for daily or weekly alert reports with a summary of the alert count per Stream for a specific alert type ## Use cases See below illustration for an example: **Alert configuration settings**: \ **flood\_type = streamID**\ **flood\_count = 20**\ **flood\_time = 15\_MIN** **Scenario**: \ Simple Street Supplies sees the highest trading volume of the year during Black Friday. When the amount of orders peak, their FTP times out resulting in 53 events with alert\_type = Error (failed in destination), the 53 Events with error are all created within 5 seconds. **Alerting service output:** ![](/files/-MUTQ3GloVtcpbwfzz-C) ## Email alert templates ### JOB\_ERROR ![](/files/-MUTPubqw5a71JxZ4oAt) ### EVENT\_DESTINATION ERROR ![](/files/-MUU0VnZ8Zs_8M4I20-j) ### EVENT\_TRANSFORMATION ERROR ![](/files/-MUU0bcbwl503hQwCypy) {% hint style="info" %} The above templates are examples of emails that are sent to the end-user for the first time. If the email is triggered after having been retained by the flood control feature, a paragraph will be added on the top of the email outlining how many Jobs / Events failed before the email was triggered (e.g. the number of the flood\_count). See example below: {% endhint %} ![](/files/-MUTPjXd36zgXm4IFk7c) ## Alert report The alert report feature provides a holistic overview of all the alerts that have been triggered over the last **day** or **week** for a specific alert type (e.g. JOB ERROR, EVENT\_TRANSFORMATION ERROR or EVENT*\_*DESTINATION ERRROR. This is especially helpful for organisations that are using the Flood Control feature, too easily see the exact number of alerts triggered, per stream and alert type. A DAILY report will be sent every day at 16:00 UST A WEEKLY REPORT will be sent every Friday at 16:00 UST ### How can I easily differentiate an alert email from a report email? The alert report will be sent from . The subject line will always start with "DAILY/WEEKLY REPORT". ### **ALERT REPORT TEMPLATE** ![](/files/-MUTPdvRBJoYC6NrGgme) # Login and security {% hint style="info" %} Please note that you will need to be granted an account before you are able to log in. New accounts for your organisation can be requested by emailing . {% endhint %} ## Login The Control Panel can be accessed via the following secure link: [https://control.hico.io](https://control.hico.io/). \ \ Once the page has loaded, you will be presented with a login form, the form allows you to enter your email address (associated with your account) and your password. Once entered press login and the system will attempt to authenticate your credentials. ## Forgotten password If you have forgotten your password, the system has the ability to regenerate it via an email authentication link. You will see a link on the login form (*url above*) titled "Forgotten Password". To reset your password, simply click on this link and enter your email address from your account on the next page. You should then receive an email within 10 minutes with a secure link for resetting your password. {% hint style="info" %} Remember to check SPAM mailboxes for the forgotten password email. {% endhint %} ## Two factor authentication As an added layer of security our system allows users to add two factor authentication to their logins. Adding two factor authentication (*often abbreviated to 2FA*) adds another step in the login process before you are authenticated in to the system. We recommend that you add two factor authentication as quickly as possible to your account. {% hint style="info" %} Our guide on how to setup two factor authentication is available via [this link](https://app.gitbook.com/@highcohesion/s/users/~/drafts/-MLlNbOT0P864EvDdHLp/getting-started/creating-an-account/login-and-security/two-factor-authentication-2fa-setup). {% endhint %} # Two factor authentication (2FA) setup ### Introduction At HighCohesion security is of upmost importance, which is why we enforce two factor authentication (2FA) on all Control Panel users. 2FA requires a user to enter two different types of information to access your account: 1. The email and password you have decided yourself 2. An automatically generated code by an external application, such as Google Authenticator or Authy Following the steps below you will be up and running in only a couple of minutes! ### Setup The following steps will guide you through setting up two factor authentication on your account login. The things that you'll need are listed below: 1. A HighCohesion account and valid login credentials 2. An authentication app on your computer or mobile phone. Popular free options for this are: 1. **Google Authenticator** (mobile only), available from: [Google Play Store](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2\&hl=en_GB) or [Apple App Store](https://apps.apple.com/us/app/google-authenticator/id388497605) 2. **Authy** (mobile and desktop): When creating a new User two factor authentication is automatically enforced. When logging into your account for the first time, you will complete the setup following these steps: 1. Log in to your **HighCohesion** account using your email and password 2. You will now be presented with a QR code (*see image below*) 3. Open your chosen authenticator app and select the "add new" option 4. Scan the QR-code displayed on the page after login. If you are unable to scan the QR-code, the code below the QR-code can be entered manually 5. Enter the code as generated by the authenticator app in the free text field 6. Click "Validate" 7. If you have entered the correct code you will be redirected to the Control Panel homepage 8. If you have entered the wrong code, you will be asked to re-enter the code as automatically generated by the Authenticator app {% hint style="warning" %} **Save the reset code somewhere safe! This is the absolute easiest way to quickly and safely reset your account should you delete the authenticator app configuration by mistake.** {% endhint %} ![](/files/-MZHM0HLFe0DuS-9vIZ-) ### Usage Once you have followed the steps above to setup two factor authentication on your account you will be asked for an authentication code on each login. When logging in, open the authenticator app and type in the 6 digit code. ### Lost your reset code If you have deleted the authentication app and lost your reset code, please send an email to . # Signup to HighCohesion without QR scan ![Enter your login credentials to the HighCohesion sign up page](/files/-MZIQZwjS_-tg5bzAEII) ![Either scan the QR code or enter the character code manually](/files/-MZIP8J5ql48Nx3vMy2w) ![On the Authy app add an account](/files/-MZIPBDVerWG6EBkCAaI) ![Enter the code as provided on the HighCohesion sign up page](/files/-MZIPEIkZ7noavNt5TkR) ![Choose a name and a logo for the account ](/files/-MZIPGkYHY503bTViyBs) ![The app will automatically start generating codes. Enter the code on the sign up page.](/files/-MZIPJCqB5iY_kZUVz3y) ![Welcome to HighCohesion. You have now signed-up correctly!](/files/-MZIPKzIfKljh9yvWVHp) # Creating an organisation How to set up an organisation as an invited partner As a Partner you can easily set up organisations for merchants that are being onboarded to HighCohesion. {% embed url="" %} Step-by-step how to video guide {% endembed %} ## Step-by-step guide As a Partner you can easily set up an organisation by simply following the below 4 steps: 1. Login to the HighCohesion Control Panel using your provided credentials 2. Navigate to the section "Organisations" in the left side menu 3. Click on "Create organisation" 4. Fill out the mandatory fields 5. Click on "Create organisation" Congratulations - you have created an organisation in HighCohesion. The next step from here is to create Users for this organisation and share the credentials with the merchant. You can follow a step-by-step guide on how to create a user [**here**](https://app.gitbook.com/@highcohesion/s/public/getting-started/creating-an-account/creating-a-user). {% hint style="info" %} Remember that you currently require an invite in order to get your initial access to the HighCohesion platform. When you have access as a Partner, you can then create Organisations yourself {% endhint %} # Billing Information Secure online payments. Automated and secure billing are at the core of the HighCohesion system. Our recurring billing provider [**ChargeBee**](https://www.chargebee.com/) () handles all sensitive data in an industry leading way. Your billing information must be entered in to the control panel if your service is to continue uninterrupted. The billing details screen is easily accessible by logging in to the Control Panel, then clicking on "Organisation" from the left navigation, then clicking on the edit (pen) icon next to your organisation name. From the edit screen you will see the button for managing your payment information: ![View / Update billing information graphic.](/files/-McOK9lHlKEKt4xBy0f3) Clicking on this button will load the billing overlay where you can see existing stored cards, add new or remove old ones. All billing information is stored on the ChargeBee server. # Creating a user This is a step-by-step guide of how to add a user to your organisation Being part of an organisation, and having the correct access rights you can easily add a new user using the HighCohesion Control Panel. {% embed url="" %} Step-by-step video tutorial of how to add a user {% endembed %} ## Step-by-step guide As a platform user with access rights higher than "Editor" you can easily add users to the organisation(s) you are overseeing by following the below steps: 1. Login to the organisation you want to add a user to 2. Navigate to the "Users" item in the left side menu 3. Click on "Create user" 4. Fill in the mandatory fields of the person you are creating the user for 5. You will be asked to provide what role you want to assign the user to, for more information about the HighCohesion roles, see [here](https://app.gitbook.com/@highcohesion/s/public/control-panel/settings/organisations/roles-and-permissions) 6. Click on "Create user" 7. When the user has been created you will have to share their email, the password you created and the link to the [Control Panel](https://control.hico.io/) 8. As the user enters the details on the Control Panel, they will be taken through our [login process](https://app.gitbook.com/@highcohesion/s/public/~/drafts/-MbfNtB4z8zQD0Lh-31S/getting-started/creating-an-account/login-and-security) and requested to sign up to 2FA (two factor authetnication). Congratulations, you have now created a user for the organisation you are overseeing! # Roles & Permissions This permission matrix outlines what platform tools each user role has access to and what actions can be carried out depending on the permissions of each user group.
Tools HiCo AdminOwnerResellerEditorView OnlyReports Only
Tools HiCo AdminOwnerResellerEditorView OnlyReports Only
Systems

View

Create

Edit

Remove

For own organisations:

View

For own organisations:

View

For own organisation:

View

For own organisation:

View

For own organisation:

No Access

Keys

View

Create

Edit

Remove

For own organisations:
View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

No Access

For own organisation:

No Access

For own organisation:

No Access

Sources

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

View

For own organisation:

No Access

Public Transformations

View

Create

Edit

Remove

For own organisations:

View

For own organisations:

View

For own organisation:

View

For own organisation:

View

For own organisation:

No Access

Transformations

View

Create

Edit

Remove

For own organisations: View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

View

For own organisation:

No Access

Streams

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

View

For own organisation:

No Access

Destinations

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

View

For own organisation:

No Access

Lookup Tables

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

View

For own organisation:

No Access

Data Uploads

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisation:

View

Create

Edit

For own organisation:

No Access

For own organisation:

No Access

Events

View

Replicate

For own organisations:

View

Replicate

For own organisations:

View

For own organisation:

View

For own organisation:

View

For own organisation:

View

Export CSV

Jobs

View

For own organisations:

View

For own organisations:

View

For own organisation:

View

For own organisation:

View

For own organisation:

View

Entities

View

For own organisations:

View

For own organisations:

View

For own organisation:

View

For own organisation:

View

For own organisation:

View

Organisation

View

Create

Edit

Remove

For own organisations:

View

Edit

For own organisations:

View

Create

Edit

For own organisation:

View

For own organisation:

View

For own organisation:

View

Users

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own account

View

Edit
View

For own account

Edit

For own account

Edit

BillingView ViewNo Access ?No AccessNo AccessNo Access
Referrals

View

For own organisations:

View

For own organisations:
View

No AccessNo AccessNo Access
Alerts

View

Create

Edit

Remove

For own organisations:

View

Create

Edit

Remove

For own organisations:
View

Create

Edit

For own organisation:

View

For own organisation:

No Access

For own organisation:

No Access

Update QB Token AccessAccessNo AccessNo AccessNo AccessNo Access
Action ListView ViewViewNo AccessNo AccessNo Access
Support Tickets

View

Create

Edit

View

Create

Edit

View

Create

Edit

View

Create

Edit

ViewNo Access
## **Role Definitions:** **HighCohesion Only**\ **HighCohesion Admin:** A HighCohesion Admin has access to all tools and aspects of the platform in order to assist companies in setting up streams and to test / resolve issues that may arise for companies.\ \ **System Integration Partners Only**\ **Reseller:** A Reseller has the same access rights as an owner, but can access multiple organisations. An example of a Reseller is a System Integrations Partner that manages multiple organisations' streams.\ \ **HighCohesion, System Integration Partners and\or Customers**\ **Owner:** An Owner is an admin of a specific organisation. The owner will have access to all tools and will be able to view, create, edit and delete resources belonging to their own organisation.\ \ **Editor:** An Editor can create and edit streams using a public transformation, but is not able to create private transformations. The Editor is also restricted from replicating & re-driving events.\ \ **View Only:** A View Only user is restricted to view platform tools and information. The role is able to pull reports and monitor the system, without risking editing settings in the live or staging environment. # Glossary of platform terminology We created this guide to help our new team members, then thought that everyone could benefit 🤯➡️🤓 **The following terms appear across the HighCohesion system and documentation. The glossary below should help give a definition of how each term is used by us. Most of the terms are HighCohesion specific, but some are marked as ‌*****"industry"***** - denoting that they are a commonly used term across technology companies worldwide.** ### API Key The API Key is the HighCohesion terminology for the credentials required for HighCohesion to access an external system, such as Shopify or an sFTP. ### Alert An email alert that is triggered by the HighCohesion alerting service. An alert is triggered at a Job or Event level in real time as an integration issue occurs. An HighCohesion alert contains all contextual information necessary to directly address the issue. ‌ ### **Bulk Upload** Description‌ ### Control Panel A web application for authoring all aspects of the the core system, creating streams and the triggers that automate the data between the different endpoints. The control panel is accessible via ​‌ ### Core Used to describe the proprietary system that has been built inside AWS using API Gateway, Lambda, DynamoDB, S3 and SQS. Our core system handles the transformation of data between two different systems in one process.‌ ### CRON schedule CRON (also CRON schedule / job) is a time-based scheduler that allows an end-user to schedule Jobs to run periodically at a fixed time, dates or intervals. At HighCohesion we use CRON jobs to schedule when a Source or Destination should be triggered. A CRON job is scheduled by using a CRON expression, for more informations on cron expressions see: ### CSV - *industry* A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. CSVs are commonly used across the eCommerce landscape today. ### Event type Description‌ ### **Destination**‌ Description‌ ### **Entity - industry** Description.‌ ### ETL - industry Extract, Transform, Load (ETL) three functions combined into a single, simple to deploy tool to pull data from one database and place it into another database.‌ * **Extract** is the process of digesting data from a database (or in High Cohesion - the payload). In this stage, the data is collected from specified sources. * **Transform** is the process of transforming the extracted data from its original form into the form it needs to be in to be placed into another database. Transformation occurs using Translation/Transformation Documents. * **Load** is the process of writing the data into the target database/system. ### **Event** An event is how we classify a specific entity of data e.g. an order, moving through a data stream. The event is generated at the source and is consumed by the destination.‌ ### **FTP - industry** File Transfer Protocol is a standard communication protocol used for the transfer of computer files from a server to a client on a computer network. sFTP (secure file transfer protocol), is a more secure way of interacting with an FTP. sFTPs/FTPs are a common way for eCommerce retailers to transfer data between an eCommerce shop and a legacy system. ### **Function** ### **ID** In the HighCohesion ecosystem an ID is a UUID consisting of 36 characters. The ID is used to tag all key parts of the platform such as Organisations, Users, Sources, Transformations, Destinations, Jobs, Event & Entities. ‌ ### **Integration** An integration is a connection between two systems that allows for the transfer of data. In the HighCohesion world, we call this a [stream](https://app.gitbook.com/@highcohesion/s/public/~/drafts/-MhOfHptr3o5i8XHhADY/help-and-support/glossary#stream). Although integrations can be very wide ranging in the type of data that is integrated between systems, in our platform it refers to specific data types e.g. the sending of new orders from a website to a WMS (Warehouse Management System). ### Job ### JSON - industry d ### Key Description.‌ ### **Key file** Description.‌ ### **Log - industry** ‌d ### **Log file - industry** #### ‌ ### Lookup Table Description‌ ### Message Queue - industry Message Queues are managed services designed to send, store and receive messages between software components at any volume, without the risk of losing messages or demanding other services be available.‌ ### oAuth - industry Open Authentication (oAuth) is an open standard for token-based authentication and authorisation on the Internet. Its primary function is to allow an end user’s account information to be used by third-party services without exposing the user’s password. oAuth acts as an intermediary on behalf of the end user, providing the service with an access token that authorises specific account information to be shared.‌ ### Organisation The HighCohesion account holder company name. Each organisation is assigned a unique ID, ensuring that all associated configuration, processes and data are accessible only to linked accounts.‌‌ ### Payload - industry The *payload* is the part of transmitted data that is the actual intended message.‌ ### Payload in The data as pulled from the Source system in JSON format before HighCohesion transformation logic has been added.‌ ### Payload out The data as posted to the Destination system in JSON format, after HighCohesion transformation logic has been added.‌ ### **Referrals**‌ Description‌ ### Reporting The explorer sections of the control panel allow the viewing of previous; job, event and entity data held in the HighCohesion system.‌ ### **Response Code** Response codes are HTTP response codes that are given by API’s in response to any request. A full list of response codes in general can be viewed at , however the codes in use by the core api currently are:‌ * **200** - *Status* = Ok * **400** - *Status* = Bad Requests (something was wrong with the data or the service, in this case, an error message is normally returned with the 400 response code to help debugging). * **401** - *Status* = Unauthorised (login details not working) * **500** - *Status* = Internal Server Error (something seriously went wrong) ### Serverless - industry Serverless computing is a type of cloud computing where the customer does not have to provision servers for the back-end code to run on, but accesses services as needed.‌ It forms part of a cloud-computing execution model in which the cloud provider (e.g. Amazon AWS, Google Cloud Platform) runs the server, and dynamically manages the allocation of machine resources.‌ Serverless architectures benefit from significantly reduced operational cost, complexity and engineering lead times.‌‌ ### **Source** Description.‌ ### **Status** ‌Statuses as defined within the HighCohesion ecosystem is the state in which a Job or Event exists at a specific point in time. The statuses give us a quick overview of how live integrations are performing and enables efficient error reporting. ### Stream A TOML configuration file that instructs what a Job does once invoked by a Trigger within the core HighCohesion system. A stream is the integration that moves data from one system or application to another. In HighCohesion a stream consists of a source function, a transformation and a destination function. The stream creates jobs that run to extract data from the source, transform it, and then load it into the destination function. Each entity of data created by the job is known as an 'event.' ### System A system is used as a term describing a collection of code that can facilitate the production or consumption of data (by acting as a Source or Destination). A system will normally be built around a singular 3rd party application, for example there is a "Shopify System" that exists to facilitate the import/export of data to and from the Shopify e-commerce platform.‌ HighCohesion has connections to and functions for many of the common eCommerce systems. ### Two-**f**actor authentication (2FA) - industry A security process in which the user provides two different authentication factors to verify themselves to better protect the user’s credentials and the resources the user can access.‌ ### Transformation A document held in JSON format that stores the mapping of fields. Used by the core system to translate data between multiple endpoints.‌ A public transformation is available to use for all users of HighCohesion. A private transformation is accessible only by a single organisation. ### **Trigger** The logic & schedule that defines **when** a Source or Destination should run. ### **User** An individual that has any type of access to the HighCohesion Control Panel. ### **UUID - industry** UUID, or Universally Unique Identifier, is a 128 bit code used to identify information in a computer system. At HighCohesion we associate each Organisation, User, Stream, Job, Event and Entity with a UUID. ### Visibility (Sharing) * Draft * Private * Public ### XML - industry Extensible Markup Language (**XML**) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. # Security standards ### Platform **Authentication** Passwords are encrypted using best-in-class encryption algorithms (up to 448-bit keys). Two-factor authentication (2FA) is enforced whenever a user accesses the Control Panel. Sessions are maintained for 10 hours to ensure a fresh login cache. ### **Data security** Integrations configuration data is store in AWS RDS instances and are only reachable from within our Private Network. Transaction Information is store in Mongo Atlas instances and are only reachable from within our Private Network. System Keys & Customer related data, such as order and product files are stored in private Amazon S3 buckets only reachable from within our Private Network. ### **Data Storage** We only store data required for running workflows and keep it only as long as needed. Where possible, all stored data can be deleted upon request. All data is encrypted during transfer across both internal and external networks. Below is a list of the types of data we store and our retention periods. **Personal Account Information** Personal details you provide when creating a HighCohesion account, such as your name and email address, are stored for as long as your account remains active. You can request account deletion at any time, and this data will be removed from our systems. **Personally Identifiable Information (PII)** We use a number of different tools to help us track usage of the product such as raw server logs and analytics tools (Google Analytics etc). These tools may receive personally identifiable information such as your computers IP address and in some cases your name and/or email address. Raw server logs are not stored for more than 14 days and, inline with GDPR you can request any PII that we store, to be removed from our systems and any sub processors we employ. **Authentication Data** To allow workflows to process data between different 3rd party services on your behalf, we will often require you to provide authentication to these 3rd party services in the form of usernames, passwords and access tokens. This sensitive authentication data is encrypted at REST in our databases, using strong 256 bit encryption, and will be removed on deletion of your HighCohesion account. All sensitive authentication data is obfuscated when passed through Stream execution state and logs. **Stream Data** When you run a Stream on the HighCohesion platform, we store stateful data as part of the execution process for post-execution logging, Root Cause Analysis (RCA), and error resolution. We retain detailed execution data and log information only as long as necessary. Stream log data—viewable by customers in the "Debug" view of each workflow—is stored for 14 days before deletion. Stream execution data, including payload in, payload out, and transformation files, is stored for 60 days before being erased. Designated HighCohesion and partner staff can access this data when needed for their role, such as assisting customers with their integrations. ### **Hosting & physical security** HighCohesion-provided services are all hosted in physically private secure data centers managed by AWS. Servers are procted by a Private Network. ### **Logging & monitoring** All user activity on the platform is logged to a central log server, with restricted access to HighCohesion Development & Operations teams. ### **Network security** The flow of customer related data, such as order and product data, is kept in our Private Network and is not accessible for outside parties. HighCohesion protocols to Get/Post external data follow external Source and Destination systems’ protocols, such as HTTPS and SFTP. Traffic over these protocols is encrypted and protected from interception by unauthorised 3rd parties. ### **User roles & permissions** Users of the HighCohesion platform do not have privileged access to the application, but must supply their application credentials to connect. Additionally, HighCohesion Admins can further restrict and revoke Users’ access to the system directly from the Admin Control Panel. HighCohesion uses three permission roles for individual organisations: Owners, Editors and View-only Users. An Owner is eligible to access all features necessary to configure and setup a Stream, such as System Keys, Source, Transformation and Destination functions. Additionally an Owner is eligible to create, edit and revoke users within their own Organisation. A View-only User is solely eligible to view system configurations and Event/Job reporting to safely access the Control Panel, without the risk of making unwanted edits to the live environment. To ensure data segregation, data for a particular Organisation is accessible only by a unique Organisation ID. Data is passed by first authenticating as the Organisation, thus giving access only to data related to that specific Organisation. ### Network security The flow of customer related data, such as order and product data, is kept in our Private Network and is not accessible for outside parties. HighCohesion protocols to Get/Post external data follow external Source and Destination systems’ protocols, such as HTTPS and SFTP. Traffic over these protocols is encrypted and protected from interception by unauthorised 3rd parties. ### User roles & permissions Users of the HighCohesion platform do not have privileged access to the application, but must supply their application credentials to connect. Additionally, HighCohesion Admins can further restrict and revoke Users’ access to the system directly from the Admin Control Panel. HighCohesion uses three permission roles for individual organisations: Owners, Editors and View-only Users. An Owner is eligible to access all features necessary to configure and setup a Stream, such as System Keys, Source, Transformation and Destination functions. Additionally an Owner is eligible to create, edit and revoke users within their own Organisation. A View-only User is solely eligible to view system configurations and Event/Job reporting to safely access the Control Panel, without the risk of making unwanted edits to the live environment. To ensure data segregation, data for a particular Organisation is accessible only by a unique Organisation ID. Data is passed by first authenticating as the Organisation, thus giving access only to data related to that specific Organisation. # Data retention policy This section outlines how HighCohesion stores data All Job related data as listed below is securely stored in privately hosted Amazon S3 buckets to support trouble shooting and error resolution before being permanently deleted after 60 days. * event.json * payload\_in.json * payload\_out.json * transformation.json * Any output XML / CSV file The Job & Event data will be stored in two separate operational databases for 60 days. After 60 days the data will be permanently removed from the operational databases and then obfuscated and stored in a database for Business Intelligence purposes. Platform logs will be stored for 14 days to support in troubleshooting and root cause analysis before being permanently removed from the platform. Non-personal entity data will be stored from that an Organisation is created until the organisation account is shut down. On account closure the organisation's entity data will be saved for an additional 60 days before being permanently removed. # Contact us We're here to help! We're developers too! and we're here to help. We have setup the following channels to help you get up and developing with HighCohesion: * **Email**\ Contact our support team via email at: [**support@highcohesion.com**](mailto:support@highcohesion.com) * **Documentation**\ You've already found it! This portal contains all of our end-user and developer specific documentation. Is something missing? Please let us know on [**support@highcohesion.com**](mailto:support@highcohesion.com)