Deleting data from the data foundation via API
The PaySuite platform enables you to "delete" data that was previously uploaded into the data foundation. Using this process, you can remove data to make sure it is no longer reflected or usable in the business areas of the platform.
While data is removed using the soft deletion feature, the data is not physically deleted from the platform database, using a specific flag (IsDeleted = true). This approach to data removal makes sure that the integrity of the data is preserved by maintaining referential relationships, leaves audit trails intact because the historical data remains accessible, and ensures compliance with data retention policies.
One of the additional advantages of the soft deletion method is that the records can be restored if required.
This is an attention message.
Records in the entities listed below cannot be deleted using the soft deletion method because they can be removed using other methods, specifically:
In the Worker entity, you can deactivate a worker by setting the field ProvisionActiveWorker for the relevant worker to false.
In the Organization entity, an organization can be deactivated by setting the field Active for the relevant organization to false.
In the CompensationElement entity, a comp element can be deactivated by setting the field CompensationElementIsActive for the relevant comp element to false.
Operating principles
To be able to perform a soft deletion of a record, you need to provide all the unique keys of the record to be delete; the value of the unique key cannot be NULL.
For referenceable entities (i.e. entities whose data can be inserted in other entities, such as CostCenter, LegalEntity or PerformanceRatingScales), the unique key is the value of the ExternalId field.
This is a warning message.
To maintain referencing integrity in the Data Foundation, when removing a record from a referenceable entity via API, you must make sure that you remove all the records referencing the record to be deleted before you can actually proceed. The system will not delete the cascading records, nor will it update the referencing field to NULL.
-
For non-referenceable entities, the behavior depends on the schema of the entity. In particular, non-referenceable entities can contain multiple unique keys. For instance:
The PaidCompensation entity contains both WorkerExternalId and CompensationElementExternalId as unique keys
The Demographics entity contains only WorkerExternalId
The GoalPlanGoalAssignment entity contains WorkerExternalId, GoalExternalId and GoalPlanExternalId
If the record contains effective dates, the effective date, or effective date range must also be mentioned in the deletion operation. This is due to the fact that, because of the effective dating, there can be multiple records for a single worker in the entity. There is no possibility to delete all effective dated records for a single worker within an entity simultaneously. Refer to the examples below for more details.
For enums, the unique key is the code of the record. Enums never have effective dating. Records can be deleted from all enums, except: Language, Country, Currency, Timezone, AmountSource and GoalUnit, which are predefine enums whose values are provided by beqom.
This is a miscellaneous message.
Refer to the Data Foundation map to see the unique keys of each entity.
(missing or bad snippet)
Examples
Example 1: deleting data from the Employment entity. This entity contains one unique key (WorkerExternalId) and is effective date (EffectiveDate). To delete records from this entity, the following information needs to be provided:
WorkerExternalId |
EffectiveDate |
|---|---|
| Worker1 | 01-01-2014 |
| Worker1 | 30-06-2025 |
| Worker 2 | 01-01-2026 |
Example 2: deleting records from the PaidCompensation entity. This entity contains two unique kays (WorkerExternalId and CompensationElementExternalId) and each records has an effective date range (EffectiveRangeStartDate and EffectiveRangeEndDate). To delete records from this entity, the following information needs to be provided:
WorkerExternalId |
CompensationElementExternalId |
EffectiveRangeStartDate |
EffectiveRangeEndDate |
|---|---|---|---|
| Worker1 | BaseSalary | 01-01-2024 | 31-12-2024 |
| Worker1 | Bonus | 01-01-2024 | 31-12-2024 |
| Worker1 | BaseSalary | 01-01-2025 | 31-12-2025 |
Deleting records using the API
Authenticating the use the API
The outbound API endpoints are secured using OAuth 2.0, and are protected by an access token. To obtain an access token, you must call the /connect/token endpoint with your unique client credentials, as illustrated in the following figure:
API authentication call
Where:
client_id: id of the client for which the token is requested; corresponds to the ClientName or CompanyName.
grant_type: constant value client_credentials.
client_secret: client secret; for more information, contact beqom.
scope: constant value integrationapi.
This is a tip message.
To obtain your API connection information and obtain your token, get in touch with our Support team.
Preparing the json request body
For soft deletion to work as expected, you must structure the body the of request as follows:
Identify the entity or enum in which the record to be deleted is located. Make sure you map out all dependencies (for more information about Data Foundation dependencies, refer to the Data Foundation map).
Write a json array containing the identifying properties of the records to be deleted for each data type, as illustrated in the example below:
[
{
"ExternalId": "L1"
},
{
"ExternalId": "L2"
}
]Request body example for the LegalEntity entity
[
{
"WorkerExternalId": "W1",
"EffectiveDate": "2024-01-01"
},
{
"WorkerExternalId": "W2",
"EffectiveDate": "2024-01-01"
}
]Request body example for the Employment entity
This is an attention message.
Only include in the request body the identifying properties and effective dating properties for the data type.
-
Call the following API endpoint: To soft-delete records from the Data Foundation, use the following API method:
POST /integrationapi/v1/data-foundation/data-type/{dataTypeName}/delete
Content-Type: application/json
If the call succeeded, you will see 202 Accepted with the following response body:
{
"ingestionId": "Guid",
"message": "String"
}This is an note/information message.
202 means that the deletion is being processed asynchronously. You can use the ingestion ID to track the progress.
This is a warning message.
This API endpoint only supports one data type per call. If you need to delete records in multiple entities/enums, you must make one call per entity/enum.
Validation rules
During deletion, the system performs the following checks
Required properties: the system checks that all the fields for the entity/record pair have been provided in the file. If not, an error indicating that a required property is missing, will be displayed.
Property value: the system checks that none of the required properties contains a null value. If that is the case, an error indicating that a required property has no value, will be displayed.
Incorrect data type: if the data type of the entered data does not match that of the field (for instance, a string was entered instead of a date), an error indicating the presence of a data type mismatch, will be displayed.
You can view the errors, if any, in the upload log file.
In addition to the validation performed on the data, the system applies the following limitations:
Only one process (data upload or deletion) can be performed at a time for a single entity/enum. If a process is already running, any new requests will fail with an "ingestion already running" error.
The maximum number of records that can be deleted per request is 60,000. If you need to delete more, you need to split the deleting into multiple requests.
Upload statuses
The following statuses are possible for a data deletion process:
Queued: the request was received and it is waiting to be processed.
Running: the request is currently being processed.
Completed: the request was successfully executed.
Failed: an error occurred during the processing of the request.
Error handling & troubleshooting
The following table contains the common errors you can encounter during the deletion process and how to solve them.
| Error | Cause | Solution |
|---|---|---|
| IngestionIsAlreadyRunning | Another upload or deletion process is already in progress for the entity/enum. | Wait for the current process to finish, and then try again. |
| RequestIsTooLarge | The request contains more than 60;000 unique keys. | Split the request in smaller batches. |
| RequiredPropertyNotFound | An identifying property (unique ID or date) is missing in the request. | Make sure that all identifying properties are properly filled in. |
| PropertyTypeMismatch | A field in the request contains a value of the incorrect type (for example a string instead of a date). | Make sure that the data types are correct in the request. For more information about each data type, see .Data foundation entities and Data foundation enums. |
| NoValidFilesToDelete | The name of the CSV file does not match any known file. | Make sure that the file name corresponds exactly to the name of the targeted entity/enum (for example: PaidCompensation.csv). Templates downloaded from the system have the correct name; make sure you don't change them. |
| FileUploadError | A problem occurred during the upload of the file. | Check file size limits, network connectivity, and file permissions. |