Uploading data into the data foundation via SFTP
This is an note/information message.
This article details the steps to upload physical data files into the Pay Suite data foundation via SFTP. To upload data into the data foundation using the data ingestion API, refer to Uploading data into the data foundation via API. To upload CSV files via the application UI, refer to Uploading data into the data foundation.
Preparing the data
To successfully integrate the Pay Suite data foundation with the IAM service (user management service), all the records must be prepared in a predefined manner. The expected outcome is to have every worker in the data foundation synchronized as users at each data load.
To achieve this goal, you must prepare individual .csv files for each of the fourteen entities listed in Data integration principles.
For optional fields, if there is no data to be provided, then the field must be left empty. For example, in the Worker.csv file where the FirstName field is mandatory but the MiddleName field is optional, the MiddleName may be left empty.
This is an attention message.
The following formats are not accepted as empty:
NULL
null
““
For tenant clarity, only one root organization item and one admin user may exist. The format of the root item must be: Root Organization External ID: All Company.
It is possible to use another External ID for the organization item for the first load. However, once organizations are uploaded and consumed, if the customer wants to change the root organization External ID, the old one must be disabled or deleted.
In addition, the following principles govern any data upload into the beqomPay Suite data foundation:
Only full uploads are supported. This means that each time you add or update values, you must provide a complete list of values. Partial updates are not allowed to maintain data consistency and integrity.
After a successful upload, only the specified values from the uploaded files will be used for file integration.
Any record that does not match the uploaded list of values will trigger an error, ensuring that only valid data is integrated into the system.
This is a warning message.
All file names (entity files and enum files), as well as all field names within the files are case-sensitive. Using the incorrect case will result in a failure of the ingestion attempt. Make sure you respect the case outlined in the following documents:
Uploading the data
This is an attention message.
The data upload behavior (full data upload or Incremental data upload) that will be used for your upload, is configured in the UI.
To correctly upload the data .csv files, proceed as follows:
Upload your prepared standard .csv files into the SFTP server, in the input/data-foundation folder. You can place both the entity files and the enum files in the same location.
After execution of the upload, check the result of the upload on the output/data-foundation folder. For each csv upload, a sub-folder structure is created for the specific data ingestion according to the pattern output > data-foundation > year > month > day > {ingestion ID}. Within this folder path, an upload result file is generated in the following format: {data type name}.json. The following figure illustrates the folder structure where the ingestion result file is generated:
Any errors that might have occurred during the upload will be available here. For more information about data upload errors, refer to Data upload error handling.
This is an note/information message.
Before files can be uploaded on the SFTP, a request must be put in with beqom to setup the SFTP server and provide credentials.
Going further