10 Principles of Data Archive & Purge – Cleanup before a cloud migration

November 10, 2020

The success of an ERP/application hinges on the ability to keep relevant data for “as long as it’s useful” (i.e., for your business needs) and “Only as short a time as required” (i.e. according to laws and regulations).

Whether it is through organic growth, mergers, divestitures, or a combination of factors, most organizations face a growing mountain of information assets and obsolete code, much of which is no longer relevant to the business. The job of dealing with this data bloat and obsolete code is often postponed, with a catalyst event like a decision to migrate to cloud-based systems or a request to reduce IT storage costs driving action.

Here are few scenarios where these 10 Principles can be applied on a growing Oracle Database:

  1. The need to migrate to the cloud IaaS+Paas
  2. Database upgrade e.g upgrade to Oracle 12.2
  3. Migrating On-prem to new infrastructure or data center
  4. Improving existing ERP performance & reducing time for maintenance activities, such as patching, backup, recovery, and cloning
  5. Clean up before a Merger or Divestiture
  6. Implementation of an ILM strategy

 

Principle – 1: Data Archive and Purge management processes and tools will be managed by the IT Department

Rationale:

• Businesses need to have a consistent and repeatable process that all business areas would use to purge and archive transactional data.

• These processes and tools have close dependencies on reporting strategy and data integrity issues, thus the management processes and tools to achieve this will be overseen by a central group that is responsible for these areas.

Implications:

• Close communication with the business process owners will be required to ensure processes and tools meet the business requirement

• Purge and Archive processes must be well communicated in the standard business process format

Principle – 2: Data Retention policies will be created and maintained by the data owners

Rationale:

Data retention is defined by legal and legislative needs, internal audit needs, and functional business needs. The Data Owners are the ultimate decision-maker to understand these needs and address them and here is a ‘free strategy toolkit’ in faster decision making.

Implications:

• Data Retention policies will be documented by the Data Owners in standard business process format

• Data Owners will work with all relevant stakeholders to determine data retention policies for their area

• Need to work closely with the IT Department who provides management processes and tools to service purge and archive

• Processes will be updated and kept relevant moving forward

Principle – 3: ERP transactional integrity reports will be run on a regular basis

Rationale:

Business data must be accurate at all times. A purge and archive solution will not adversely affect this requirement.

Implications:

• Data Owners will identify ERP Integrity reports required to meet business needs. If these reports do not exist, they will be created

• IT Department will be responsible for scheduling and processing of required reports

• Verification of integrity reports will be required by all business units

Principle – 4: Only fully processed and closed transactions are to be archived

Rationale:

For software applications to be effective and stable, business and logic rules must be adhered to. Transactions that historically have been left unattended will need to be addressed to receive maximum benefits from a purge and archive process.

Implications:

• IT Department to be responsible to create policies and processes that guarantee no open transactions to be archived

• Standard policies will be designed to allow business units to identify and reconcile, within the appropriate application, old and open transactions

Principle – 5: Archive and Purge will follow best practices, standards, and procedures

Rationale:

Effective and efficient procedures, best practices, and standards will be established that fit our business needs and culture. Archive and purge may be expanded/reduced in scope based on what is best for the company. Learn 8 best practices for Archive and Purge here.

Implications:

• Standard operating procedures must be created and documented following standard business process standards

• Limited industry standards exist. Ours will be the best fit for our company

• IT Department will be responsible for the delivery and maintenance of these policies

• Policy Review will be required to facilitate continuous improvement

Principle – 6: A change management process shall govern all archived data

Rationale:

As an Archive and Purge solution evolves over time, a strong change management process is required. All successful implementations demand a thorough change management approach. Failure of such an approach invites a high probability of failure.

Implications:

• Detailed change management practices will be developed to govern change requests to existing production purge and archive solutions

• Change requests to existing purge and archive processes will be centralized and approved by business process owners

• Processes will be put in place to increase awareness that modifications to applications or database files may impact purge and archive processes and need to be taken into consideration as part of application development

Principle – 7: Master data will not be archived

Rationale:

Master data will not be included in the archive database. The archive database will contain transactional data only that has met pre-defined conditions. This will guarantee the requirement that master data have one source of truth and that master data will only reside in the production environment

Implications:

• A Master Data Management team will be created as part of the ERP Instance Consolidation Project

• Configuration for a “read-only” archive environment will be created to retrieve master data from the production environment

• Any reporting tools or processes using the archive database as their source must retrieve master data from the production environment or from a replicated copy of the production database

• IT Department to ensure no master data to be included in any archive and purge procedures

Principle – 8: Data Integrity is paramount and will not be compromised

Rationale:

The benchmark of any computerized system and an IT organization resides in data integrity. EBS requires that archive and purge processes be configured to obey rules and regulations pertaining to data integrity and not in any way violate these constraints.

Implications:

• IT Department will create and maintain robust and proven purge/archive processes that will not compromise data integrity.

• Archive and purge processes will be designed with a top drown approach. No data will be archived until all related transactions can be archived

• Restore/rollback processes must be created to allow a return to a pre-archived and purge status in the event that data corruption is encountered in an archive database represents a historical fact. It will not be altered unless corresponding master data has been altered and archive data needs to reflect this change for integrity.

Principle – 9: Archived Data will be secured as “read-only”

Rationale:

Data residing in an archive database represent historical facts. It will not be altered unless corresponding master data has been altered and archive data needs to reflect this change for integrity.

Implications:

• Security configuration on the archive database will be restricted to a “read-only” status with the exception of a change management profile

• Auditing reports will be created to flag any update attempts to this database. Any violations will be investigated

• IT Department to allow one profile to allow update authorization to the archive database. This user profile will execute the archive and purge processes and will be a secured profile

• Access to archived data via an Enterprise One application will be controlled and secured by the IT Department to ensure compliance

• Processes will be created to allow alteration of archive data to coincide with master data changes

Principle – 10: Transactional data will be created/maintained at an appropriate level of detail

Rationale:

Data in an ERP system must be complete and accurate, but not retained at a level of detail that causes concern to business units or IT Department.

Implications:

• Some business units may need to be flexible in regards to what data is kept where and at what level of detail

• When possible, G/L transactions interfaced from other ERP modules or bolt-on applications should be created at a summarized level

• When possible, batch posting processes should be set to create summarized offsetting entries

• When designing and implementing ERP modules, consideration and ramifications regarding data volumes must be included as part of the process.

Subscribe to our blog