Data migration

From Clinfowiki
Revision as of 22:50, 10 February 2015 by Annathehybrid (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Data Migration is the process of moving data from one source to another and typically occurs between a legacy system and a new target system.(1) The remainder of this discussion focuses on the concept of data migration in the context of migrating data between legacy and new electronic health record (EHR) systems.


As of 2013 it was estimated that more than 50 percent of healthcare providers were replacing their electronic health record (EHR) systems for various reasons including dissatisfaction, mergers, hospital acquisitions, and to obtain the latest software.(2) As part of this transition process one of the key factors that providers must consider is how they are going to migrate their legacy EHR data into the new target EHR system.(2) Many healthcare organizations have been collecting data since their first go-live and now have terabytes of data that require transfer as part of the migration process.(2) This data is often stored with multiple vendors that may use their own proprietary data formats(2) in turn creating lots of ways for potential errors to be made, including loosing data, data becoming inaccessible, losing the context/meaning of the data, and creating erroneous context/meanings.(3) These complexities in addition to the already inherent complexities of healthcare data has been shown to make the process of data migration one of the most challenging aspects in a new EHR adoption.(3) The Health Information Technology (HIT) Policy Committee (HITPC) who serves as the public advisory body on HIT for the National Coordinator for Health IT recently recognized the challenge of data migration between EHR systems.(3) At an information exchange workgroup meeting held in August of 2014, HITPC declared data migration as one of the two critical use cases that needs to be addressed in order to create an efficient HIT market and support safe and effective care.(3) As part of this discussion HITPC has recommended that the HIT standards committee develop standards that address provider data migration by stage 3 of Meaningful Use.(3)


The process of data migration is commonly found to be an ad-hoc highly variable process between providers.(3) For this reason this data migration process discussion consists of only a brief high-level overview of the process and will likely vary depending on the data migration project.

The data migration process traditionally includes the following three stages:(1)

  1. Extract
  2. Transform
  3. Load

The extraction phase is the beginning of the data migration process where the data in the legacy system is thoroughly reviewed to determine what needs to be extracted, tested to make sure it can be extracted, prepared for the conversion process, and lastly extracted to one or more locations. Each patient record often contains many different types of data that must be properly accounted for and as a result it is critical that any special handling for both extraction and data conversion be identified.(4) Additionally, often times included as part of this phase is the consolidation of the various heterogeneous data sources found in the legacy system.(1)

The transformation phase includes converting the data extracted into a format that is necessary for it to be read correctly by the new EHR. In some cases if the provider is just upgrading to a newer version of the same EHR then little transformation is required. However, when conversion of the data is required this often includes validating, cleansing, and mapping(1) the data between the future and legacy system through the development of a system crosswalk.(4) Often times this stage incudes developing a consensus on the appropriate available code standards (e.g. Health Level 7 Code Set) that can be used to be map the different data identifiers in both the legacy and future EHR systems to each other.(4)

Once the data is converted into the format necessary it may be loaded into the EHR through some form of manual or automated data entry (such as through scripts). For scripted data entry, typically a workflow is first created for inputting the transformed legacy data into the new EHR. This workflow is then used as a procedural guide to create custom scripts following the process outlined and can then be executed as defined.(4) While the data is being loaded it is critical that the accuracy and efficiency of the input process be monitored by some form of process evaluation.(4) Typically, this evaluation includes the use of a migration log and manual chart reviews.(4) The migration log is used to check for any skipped records and data corruption, and manual chart reviews are used to check for accuracy in both the legacy and target systems.(4)

Data Migration Strategies

The most common EHR data migration strategies include:(2)

  • One-time data conversion
  • Ongoing system-to-system interface
  • Vendor neutral archive

These strategies are often used in some combination depending on the needs of the provider. For example, typically a one-time data conversion is performed as a best practice to include the last 36 months of all patient data in the new EHR system.(2) Additionally, a one-time data conversion may be needed in some cases because a system-to-system interface cannot be used because of a lack of standard data outputs.(2) Once the one-time conversion is performed then the remaining data that is not needed for a one time conversion can then be made directly accessible by the way of a system-to-system interface.(2)

A system-to-system interface keeps the data in the old EHR system and then provides links to access this data in the new system.(2) However, the system-to-system interface has been found to have many disadvantages including the fact that the old EHR system must also now be maintained and supported to ensure the legacy data stays accessible.(2) Also, a system-to-system interface requires clinicians to switch between systems which in-turn often reduces efficiency and can significantly increase costs.(2) As a result of these disadvantages, providers are often choosing to take the vendor neutral archive (VNA) approach for the remaining data that is not inputted directly into the new EHR.(2)

The VNA approach includes extracting the legacy data and storing it in an online data repository.(2) Interfaces are then developed from the VNA to the new EHR system to ensure that the data can be accessed by the new EHR system.(2) This solution has been found to provide the benefits of support for all types of data, an easy to access single repository for historical data, vendor support for the lifecycle of the data, cost efficiencies (such as minimal staff and data conversion fees) and supports Health Insurance Portability and Accountability Act (HIPPA) compliant data retention and destruction policies.(2)

Lessons Learned

The data migration lessons learned are taken from the Minnesota’s PACE clinic EHR implementation project and includes the following: [1]

  • Document vendor responsibilities and get commitment in writing
  • Check resources and availability
  • Schedule time to plan
  • Ensure proper mapping is performed for a complete data transfer
  • Perform continuous testing during migration
  • Mitigate any communication barriers

From the lessons learned it was found that adequate vendor engagement from both the legacy and new EHR vendors is critical for a successful data migration project.[1] To assist with this engagement it is recommended that the providers get a signed agreement in place with the EHR vendors documenting their responsibilities in the data migration process. This should include the vendors resources, time, commitment, what they can and cannot do, the requirement of multiple test migrations, and agreement that all vendor errors will be corrected at no charge.[1] Next, it is important that all key resources are available during the data migration project, such as health information management, information technology personnel, providers, office managers, legal personnel, and legacy and new EHR vendor personnel.[1] When scheduling a data migration project it is critical that there is plenty of time for the planning phase so that each step can clearly be identified and then assigned to the appropriate resources responsible.[1] The result of the planning phase should be a project plan that has been reviewed by all parties involved and agreed upon.[1] Proper mapping is needed to align data elements between systems and it is recommended that all maps be made as detailed as possible.[1] Once completed, the maps should be reviewed with the entire transition team and ensure that the responsibilities of the vendor are fully understood.[1] For testing it is recommended that a minimum of three rounds of testing are performed using subsets of data that include unique situations.[1] Once the testing is completed vendor signoff should be obtained that the extracted and imported data will not be modified.[1] Lastly, one of the primary causes of many of the issues that often occur with data migration are related to communication barriers.[1] To help overcome these barriers it is recommended that a standard change log is kept and updated regularly to provide a central location and format to document any changes to the project.[1]


  1. 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 Noreen N. Data migration lessons learned: one facility relates hard-learned best practices after migrating to a new EHR. J AHIMA. 2013 Sep; 84(9):40-42.
  1. Thalheim B, Wang Q. Data migration: A theoretical perspective. Data & Knowledge Engineering. 2013 Sep;87:260–78.
  2. West S. Need versus cost: understanding EHR data migration options. J Med Pract Manage. 2013 Dec;29(3):181–3.
  3. Health IT Standards Committee. Provider data migration and patient portability [Internet]. Information Exchange Workgroup. 2014 Aug 28 [cited 2014 Oct 23]. Available from:
  4. Michel J, Hsiao A, Fenick A. Using a scripted data entry process to transfer legacy immunization data while transitioning between electronic medical record systems. Appl Clin Inform. 2014;5(1):284–98.

Submitted by Jeffrey Tripp