More about Cloud Migration
- Petabyte-Scale Storage Success Stories With Cloud Volumes ONTAP
- Azure StorSimple EOL: Using Cloud Volumes ONTAP for Hybrid Cloud Management
- Refactoring Applications to Kubernetes in Cloud Migrations
- The War Over the Cloud Has Ended (and the Winner Might Surprise You)
- Modern Data Estate: What IT Leaders Need To Know
- Strategies for AWS Migration: The New 7th R Explained
- Cloud Computing Deployment Models and Architectures
- Cloud Adoption Strategy: What’s the Best Approach for Your Organization?
- What is Cloud Migration? Strategy, Process and Tools
- Hybrid Cloud Strategy: A Winning Model for Enterprises
- Google Cloud Architecture and Building Your Own Solution Architecture
- SnapMirror in the Cloud: New Use Cases for NetApp’s Data Replication Technology
- 8 Digital Transformation Technologies and Their Business Impact
- What Is Digital Transformation in Banking?
- Digital Transformation in Healthcare: 4 Key Trends
- Digital Transformation: Examples from 5 Industries
- The Future of Cloud Computing: 5 Trends You Must Know About
- 5 Types of Digital Transformation and the Technologies that Power Them
- Digital Transformation Strategy: 6 Tips for Success
- Kubernetes Data Mobility with Cloud Volumes ONTAP
- Application Migration to Azure: 4 Approaches and One Migration Tool
- Cloud First Strategy: Challenges, Considerations, and Best Practices
- Application Migration to AWS: Free Tools to Ease Your Migration
- Why Cloud Adoption Fails and 6 Tips for Success
- Cloud Application Migration: A Practical Guide
- AWS Snowball vs Snowmobile: Data -Migration Options Compared
- AWS Snowball Edge: Data Shipping and Compute at the Edge
- AWS Snowmobile: Migrate Data to the Cloud With the World’s Biggest Hard Disk
- AWS Snowball Family: Options, Process, and Best Practices
- AWS Cloud Adoption Framework (CAF): 6 Migration Perspectives
- Top 3 Cloud Adoption Frameworks: Your Path To The Cloud
- AWS Snowball Pricing Simplified
- Azure Cloud Adoption Framework: The 9 Methodologies Explained
- Cloud Roadmap: Mapping Out Your Path To The Cloud
- Hybrid Deployment on Google Cloud: Meet Google Anthos
- 3 Ways to Create an Azure Migrate Project
- Azure Migration Step by Step: Discover, Migrate, Optimize, and Monitor
- Migrate from VMware to Azure: The Basics and a Quick Tutorial
- Migrate SQL Server to Azure: Options, Tools, and a Quick Tutorial
- Migrate Databases to Azure: 3 Quick Tutorials
- 4 Ways to Migrate SQL to Azure
- Azure Migration Program: 4 Key Elements
- Azure Migrate: Key Components and a 4-Step Migration Plan
- Cloud Journey: 6 Stages of Cloud Adoption
- 5 Azure Data Migration Tools You Should Be Using
- Azure Migration Tools: One-Click Migration for VMs and Data
- Typical Mistakes and Misconceptions Business Leaders Have About Hybrid and Multicloud
- Google Cloud PostgreSQL: Managed or Self-Managed?
- AWS Storage Gateway: Connecting Your On-Premise Storage to the Amazon Cloud
- Azure PostgreSQL: Managed or Self-Managed?
- Cloud Scalability: How Cloud Volumes ONTAP Stores Petabytes of Data
- AWS Migration: Understanding the Process and Solving 5 Key Challenges
- Google Cloud Pricing vs AWS: A Fair Comparison?
- VMware Cloud Services: A New Option for Hybrid Cloud Management
- AWS PostgreSQL: Managed or Self-Managed?
- Google Cloud MySQL: MySQL as a Service vs. Self Managed in the Cloud
- Azure MySQL: MySQL as a Service vs. Self-Managed in the Cloud
- AWS MySQL: MySQL as a Service vs. Self Managed in the Cloud
- Cloud Adoption for Financial Companies
- VMware on AWS: Architecture and Service Options
- VMware on Google Cloud: A Deployment Roadmap
- VMware on Azure: One-Step Migration to the Cloud
- Better in the Cloud: Workloads Gartner Says You Should Move to the Cloud Now
- 3 Cloud Migration Approaches and Their Pros and Cons
- Oracle on Google Cloud: Two Deployment Options
- How to Upload Files to Azure Blob Storage with AzCopy, PowerShell, and More
- Oracle on Azure: Fully Managed vs Self Managed
- Azure SQL Server: Managed Service vs Self Managed
- Google Partners with NetApp on New Bare Metal Solution in the Cloud
- AWS Database Migration Service: Copy-Paste Your Database to Amazon
- SQL Server on Google Cloud: Two Deployment Options
- GCP Migration with Cloud Volumes ONTAP
- SQL Server in AWS: Two Deployment Options
- Azure Database Migration Service: Automate Your Cloud DB Migration
- Azure Managed Service Provider: How to Save Time and Reduce Cloud Overhead
- Hybrid Cloud Storage: The Best of Both Worlds
- Hybrid Cloud Storage Solutions for AWS Compared: Storage Gateway vs Cloud Volumes ONTAP
- AWS Oracle: Two Options for Migrating Your Oracle DB to Amazon
- Migrating Physical Server to AWS: Now Free with AWS CloudEndure
- Azure Hybrid Cloud:
Azure in Your Local Data Center
- What Is a Lift and Shift Cloud Migration?
- Google Cloud Migration Tools: Copying 1GB or 500TB? Learn How
- VMC on AWS Vs. Cloud Volumes ONTAP
- AWS Cloud Migration Services: Don’t Migrate Alone
- 5 Steps to the Cloud: AWS Migration Checklist
- Setting up Storage Gateway with Amazon EC2
- VMware Cloud on AWS: How Fujitsu Saves Millions Using Cloud Volumes ONTAP
- SQL Server End of Life Challenges and How Cloud Volumes ONTAP Can Help
- The Cloud Tiering Service Architecture: How We Get Cold Data from Your Data Center to the Cloud
- Cloud Data Integration 101: Benefits, Challenges, and Tools
- Google Cloud Migration: Why and How to Make the Move
- What’s the Easiest Way to Start Using the Cloud? Three Cloud Onboarding Approaches
- AWS vs Azure vs Google Cloud: Choosing the Best Cloud Provider for You
- Cloud Migration Tools: Transferring Your Data with Ease
- Azure Migration Strategy: Four Steps to the Cloud
- 11-Step Azure Migration Checklist
- AWS Migration Strategy: The 6 Rs in Depth
- Officeworks Adopts a Cloud-First Strategy with Cloud Volumes ONTAP
- Azure Migration: The Keys to a Successful Enterprise Migration to Azure
- Cloud Volumes ONTAP: Cloud Migration Case Studies
January 7, 2020
Topics: Cloud Volumes ONTAP Cloud StorageData MigrationElementary5 minute read
According to a Gartner survey, 81% of public cloud users leverage more than one cloud provider. With the explosion of multicloud and hybrid cloud deployment as the primary type of cloud migration strategy, there is a growing need to integrate disparate cloud systems. Cloud data integration can help multiple applications—in the public cloud, private cloud, and on-premises—consistently synchronize and share data.
In this post, we’ll discuss important considerations and challenges facing cloud data integration projects, briefly review cloud integration platforms that can help, and show how NetApp Cloud Volumes ONTAP can help share storage between clouds and on-premises systems.
In this article, you will learn:
- What is cloud data integration
- Cloud data integration benefits and challenges
- Evaluating an Integration Platform as a Service (iPaaS) for your project
- Cloud data integration and Cloud Volumes ONTAP
What is Cloud Data Integration?
Cloud data integration is the practice of integrating data used by disparate systems, between or within public or private clouds, or between cloud-based and on-premise systems. The goal is to create unified data stores that can be accessed efficiently and transparently by all relevant users and applications.
There are mature tools for data integration within public cloud providers or private cloud platforms, for example within AWS or an OpenStack data center. The main challenge begins when organizations need to integrate multiple public clouds, set up hybrid cloud environments, integrate legacy on-premise systems with cloud workloads, or lift and shift legacy workloads into the cloud.
Without automation and central control, cloud data integration can be painful. Administrators need to manually set up multiple integrations, test them and verify data is being transferred correctly. They also need to convert or transform data to suit the file format, data structure or data model expected by different cloud systems. This is why many organizations use Integration Platforms as a Service (iPaaS) that have pre-built adaptors or connectors for many IT systems, making cloud integration faster, easier and less error prone. We briefly review a few of these systems below.
Cloud Data Integration Benefits and Challenges
Integrating data between cloud systems, and between cloud and on-premises systems, can have several important advantages:
- Synchronizing data—ensuring IT systems and applications operating on the same data or entities have a consistent view of the data with frequent or real-time updates.
- Automating workflows—integration can help automate organizational processes that involve manual copying of data or manual data entry, and standardize how data should be treated on its way from one application to another.
- Eliminate redundant data—it is very common for IT systems to store the same data several times, for the benefit of different applications or organizational processes. Integration helps to eliminate duplications and use a shared data store, reducing storage costs and synchronization efforts.
- Flexibility and scalability—integrating systems gives operational staff many opportunities to improve processes, and identify new systems that can provide more value to internal and external customers.
However, cloud data integration projects may face several important challenges:
- Data movement—moving data between clouds, and between cloud and on-premise systems, can be time-consuming and error-prone, or even unfeasible in some cases, depending on the data volumes and the required data transfer frequency. Cloud data integration won’t work without solid strategies to transfer data in a timely manner.
- No standardization—there is no standard approach or protocol for integrating data between cloud systems, not to mention cloud and on-premise systems. Each cloud platform, service or resource tends to have different data schemas and formats. Data connectors or adaptors need to be constantly updated, as new cloud services are introduced or as applications are updated or modified.
- Architectural issues—cloud systems are often architected with scalability or performance in mind, not around data integration. For example, in a system that rapidly scales up or down, with data stored on dozens or hundreds of cloud instances, it may be challenging to synchronize with external systems.
- ETL—in traditional data integration projects, complex Extract Transform Load (ETL) workflows were set up to clean data and transform it into the precise format needed by target systems. Many cloud systems work with unstructured data or provide a flexible data model for structured data. However, they still need data to be cleaned, treated and converted into the desired format. Integration strategies must consider how ETL can be performed without slowing down the integration or adding a lot of complexity.
Evaluating Cloud Integration Platforms for Data Integration
While simple data integration projects can be performed ad hoc, with scripting or home-grown automation, enterprise-scale projects will almost always require an integration platform.
Use the following considerations to evaluate cloud integration platforms for your project:
- Specific application support—does the platform support the applications you are running today? Is it extensible and able to support any applications or data formats you may adopt in the future, and what is the effort involved in building a custom connector or integrating additional data sources via API?
- How mapping works—most integration platforms offer a visual interface for mapping data fields between source and target systems. Check if the interface is friendly for the individuals who will carry out data mapping, who may not have a technical background. Also make sure it is powerful enough to handle your organization’s data formats, rules and exceptions.
- Security and compliance—consider the security requirements and industry standards applicable to each of your datasets. See if the integration platform can fulfill those requirements, both for one-time data transfers and ongoing data synchronization.
- Data cleaning, preparation and integrity—check if the integration platform takes responsibility for preparing data, converting it to the target format and verifying its integrity. If not, you will need to have other tools or strategies in place to create the data stream your target application expects.
Cloud Data Integration and Cloud Volumes ONTAP
NetApp Cloud Volumes ONTAP can help integrate your data as part of your cloud migration strategy. Cloud Volumes ONTAP is built on the raw cloud resources on AWS, Azure, and Google. And thanks to Cloud Manager, it has a single-pane UI from where you can control, automate, and orchestrate all your storage resources, no matter where they're located, whether it's in a hybrid or multicloud architecture.
In addition, NetApp SnapMirror® helps to replicate, migrate, and synchronize data across different data sources, platforms and clouds, making it easier for you to integrate your data across every storage environment you use.