Deleting data from the Data Foundation
The PaySuite platform enables you to "delete" data that was previously uploaded into the data foundation. Using this process, you can remove data to make sure it is no longer reflected or usable in the business areas of the platform.
While data is removed using the soft deletion feature, the data is not physically deleted from the platform database, using a specific flag (IsDeleted = true). This approach to data removal makes sure that the integrity of the data is preserved by maintaining referential relationships, leaves audit trails intact because the historical data remains accessible, and ensures compliance with data retention policies.
One of the additional advantages of the soft deletion method is that the records can be restored if required.
This is an attention message.
Records in the entities listed below cannot be deleted using the soft deletion method because they can be removed using other methods, specifically:
In the Worker entity, you can deactivate a worker by setting the field ProvisionActiveWorker for the relevant worker to false.
In the Organization entity, an organization can be deactivated by setting the field Active for the relevant organization to false.
In the CompensationElement entity, a comp element can be deactivated by setting the field CompensationElementIsActive for the relevant comp element to false.
Operating principles
To be able to perform a soft deletion of a record, you need to provide all the unique keys of the record to be delete; the value of the unique key cannot be NULL.
-
For referenceable entities (i.e. entities whose data can be inserted in other entities, such as CostCenter, LegalEntity or PerformanceRatingScales), the unique key is the value of the ExternalId field. To maintain referencing integrity in the Data Foundation, when removing a record from a referenceable entity, you must make sure that you remove all the records referencing the record to be deleted before you can actually proceed. The system will not delete the cascading records, nor will it update the referencing field to NULL.
When performing a soft deletion by using the csv upload, the system handles dependency ordering, meaning that it will delete child records before deleting parent records, leaving no "orphan" records (i.e. without a reference).
-
For non-referenceable entities, the behavior depends on the schema of the entity. In particular, non-referenceable entities can contain multiple unique keys. For instance:
The PaidCompensation entity contains both WorkerExternalId and CompensationElementExternalId as unique keys
The Demographics entity contains only WorkerExternalId
The GoalPlanGoalAssignment entity contains WorkerExternalId, GoalExternalId and GoalPlanExternalId
If the record contains effective dates, the effective date, or effective date range must also be mentioned in the deletion operation. This is due to the fact that, because of the effective dating, there can be multiple records for a single worker in the entity. There is no possibility to delete all effective dated records for a single worker within an entity simultaneously. Refer to the examples below for more details.
For enums, the unique key is the code of the record. Enums never have effective dating.
This is a miscellaneous message.
Refer to the Data Foundation map to see the unique keys of each entity.
(missing or bad snippet)
Examples
Example 1: deleting data from the Employment entity. This entity contains one unique key (WorkerExternalId) and is effective date (EffectiveDate). To delete records from this entity, the following information needs to be provided:
WorkerExternalId |
EffectiveDate |
|---|---|
| Worker1 | 01-01-2014 |
| Worker1 | 30-06-2025 |
| Worker 2 | 01-01-2026 |
Example 2: deleting records from the PaidCompensation entity. This entity contains two unique kays (WorkerExternalId and CompensationElementExternalId) and each records has an effective date range (EffectiveRangeStartDate and EffectiveRangeEndDate). To delete records from this entity, the following information needs to be provided:
WorkerExternalId |
CompensationElementExternalId |
EffectiveRangeStartDate |
EffectiveRangeEndDate |
|---|---|---|---|
| Worker1 | BaseSalary | 01-01-2024 | 31-12-2024 |
| Worker1 | Bonus | 01-01-2024 | 31-12-2024 |
| Worker1 | BaseSalary | 01-01-2025 | 31-12-2025 |
Deleting records in Workbench
The process of deleting records in Workbench is performed under Data Management > Data Upload. It is very similar to the data upload process. You can download deletion templates directly from the system interface and populate them with the relevant information.
This is a tip message.
The templates contain, for each entity, a column for each of the fields that must be provided for the deletion, according to the principles outlined above.
To delete records from the application user interface, proceed as follows:
Open the PaySuite platform, and then navigate to Workbench > Data Management > Data Upload. The list of data upload/deletion operations is displayed.
Click Download template in the toolbar located above the operation list. A contextual window is opened. In this window, you will see two sections: Ingestion and Deletion (you need to scroll down to see deletion templates). A template is provided for all entities, enums and custom entities existing in the system.
In the Download template window, scroll down to the Deletion section and locate the entity/enum from which you want to delete records.
Click the download icon located on the right of the template name. The template file is downloaded on your machine.
Fill in the fields from the file. To help with this operation, you can use the Data Viewer page, where you will be able to find all the required information (see Data Viewer).
Once you have properly filled in your template, go back to the PaySuite platform and click Upload CSV. The File upload contextual window is opened.
In the window, click Upload or drag & drop your template file into the drop zone.
Click Start upload. The system starts processing your file and deleting the records.
Validation rules
During upload, the system performs the following checks
Required properties: the system checks that all the fields for the entity/record pair have been provided in the file. If not, an error indicating that a required property is missing, will be displayed.
Property value: the system checks that none of the required properties contains a null value. If that is the case, an error indicating that a required property has no value, will be displayed.
Incorrect data type: if the data type of the entered data does not match that of the field (for instance, a string was entered instead of a date), an error indicating the presence of a data type mismatch, will be displayed.
You can view the errors, if any, in the upload log file.
In addition to the validation performed on the data, the system applies the following limitations:
Only one process (data upload or deletion) can be performed at a time for a single entity/enum. If a process is already running, any new requests will fail with an "ingestion already running" error.
The maximum number of records that can be deleted per request is 60,000. If you need to delete more, you need to split the deleting into multiple requests.
Upload statuses
The following statuses are possible for a data upload/deletion process:
Queued: the request was received and it is waiting to be processed.
Running: the request is currently being processed.
Completed: the request was successfully executed.
Failed: an error occurred during the processing of the request.
Error handling & troubleshooting
The following table contains the common errors you can encounter during the deletion process and how to solve them.
| Error | Cause | Solution |
|---|---|---|
| IngestionIsAlreadyRunning | Another upload or deletion process is already in progress for the entity/enum. | Wait for the current process to finish, and then try again. |
| RequestIsTooLarge | The request contains more than 60;000 unique keys. | Split the request in smaller batches. |
| RequiredPropertyNotFound | An identifying property (unique ID or date) is missing in the request. | Make sure that all identifying properties are properly filled in. |
| PropertyTypeMismatch | A field in the request contains a value of the incorrect type (for example a string instead of a date). | Make sure that the data types are correct in the request. For more information about each data type, see .Data foundation entities and Data foundation enums. |
| NoValidFilesToDelete | The name of the CSV file does not match any known file. | Make sure that the file name corresponds exactly to the name of the targeted entity/enum (for example: PaidCompensation.csv). Templates downloaded from the system have the correct name; make sure you don't change them. |
| FileUploadError | A problem occurred during the upload of the file. | Check file size limits, network connectivity, and file permissions. |