BlueXP Blog

Google Cloud Migration Tools: Copying 1GB or 500TB? Learn How

Written by Yifat Perry, Technical Content Manager | Mar 5, 2020 5:00:00 AM

Like all cloud providers, Google Cloud has gone to great efforts to make it easy to migrate workloads to its public cloud platform. Google Cloud provides several services that help you manage data transfer from on-premises systems to its cloud services.

In this post, we’ll provide an overview of the Google Cloud data transfer services and solutions, and provide best practices for using them effectively in your Google Cloud migration. In addition, we’ll show how NetApp Cloud Volumes ONTAP can help migrate on-premises storage to Google Cloud.

In this article, you will learn:

Google Storage Transfer Service: Migrating Data to Google Cloud Object Storage

Storage Transfer Service is a Google product that enables you to move large volumes of data from an on-premises database to Google’s managed object storage service, Google Cloud Storage:

  • Move or backup data to a Google Cloud Storage bucket, from your on-premise storage or from other cloud providers.
  • Move data between Google Cloud Storage buckets, to make it available to different users or applications.
  • Move data in bulk as part of a data processing or analytics process.

By default, the Transfer Service copies a file from a data source if it doesn’t exist in the target, or if it has a different version compared to the object in the target bucket, and retains objects in the source bucket.

Beyond this default behavior, the Transfer Service provides additional features to make data transfers more convenient: 

  • Scheduling a data transfer as a one-time or recurring event.
  • Deleting objects in the destination storage bucket if the same objects don’t exist in the source.
  • Deleting items from a data source after transferring them.

Google Storage Transfer Service Best Practices

Google provides a few tips for working more effectively with the Storage Transfer Service: 

  • Identical agent setup—it is a must to have all agents, installed on all machines you are transferring data from, and see the same NFS drives with the same relative paths.
  • More agents means higher performance—data transfers are parallelized across all agents, so use several agents, and add more if possible, to make better use of available bandwidth.
  • Review errors—take the time to review errors during transfers, to avoid much bigger problems later. Storage Transfer Service lets you see a sample of the data being transferred and view examples of errors directly in the Cloud Console. You can also see a full record of all errors in data transfer. Errors might result from applications writing to the source or destination during the transfer, permission issues, or other problems. Perform troubleshooting and if necessary, roll back and retry the data transfer.
  • Bandwidth caps—keep in mind you might have other workloads consuming bandwidth. If necessary, set a cap to ensure data transfers do not take over all your bandwidth and stop other workloads from functioning.
  • Use Stackdriver Monitoring—for long-running transfers, use StackDriver to monitor agent health and throughput, and get alerts when an agent is malfunctioning. Acting on failures is very important when you perform a transfer that takes days or weeks, to prevent interruptions that can delay your project.

Learn more in our in-depth article about the Google Storage Transfer Service.

BigQuery Data Transfer Service

The BigQuery Data Transfer Service schedules and automates data movement from SaaS applications to Google BigQuery, Google’s elastically scalable data warehouse. This lets you set the foundation for a cloud-based data warehouse without a major development project.  

BigQuery Transfer Service can operate immediately with no installation—just set up an account, verify access, and information starts flowing into BigQuery.

BigQuery Data Transfer Service originally supported Google application sources like Google Ads, Campaign Manager, Google Ad Manager, YouTube and Google Play, and is adding connectors that let you transfer data from sources like Teradata and Amazon S3.

Google Transfer Appliance: Transferring over 100TB of Data to Google Cloud

Google provides the Transfer Appliance, a secure, high capacity storage server that is shipped from Google to your datacenter. You can fill it with data and ship it back to an ingest location, for it to be uploaded to Google Cloud Storage. The appliance comes in two sizes—100TB and 480TB, and encrypts data automatically to ensure data is secure. 

Here is the process for transferring huge volumes of data in bulk using the Transfer Appliance:

  • Request one or more appliances by submitting a request online.
  • Receive the appliance—it is shipped by deliver carriers in a tamper-resistant package.
  • Transfer data to the appliance—install it in your data center and capture data, the appliance will automatically encrypt ingested data.
  • Ship the appliance back to Google—use the provided return label to send the device back to a Google ingestion center.
  • Google uploads your data to Cloud Storage—you will be notified when it is ready.
  • Access your data—log into Google Console to decrypt the data and choose a storage bucket to load it into.
  • Google securely erases the appliance—Google follows the NIST-800-88 standard to securely erase the appliance ensuring there is no trace or record of your data.

gsutil Command Line Utility: Copying Smaller Data Volumes

The gsutil utility is the standard tool for small- to midsize data transfers of less than a few terabytes. You can use gsutil to transfer data from a private data centre to Google Cloud. gsutil provides all the basic features you need to control your Cloud Storage instances, including:

  • Copying data to and from a local file system or between Cloud Storage buckets
  • Renaming and moving objects
  • Performing real-time incremental syncs, like Linux rsync, to a Cloud Storage bucket

Google recommends including gsutil in your default path when you use Cloud Shell, or using Cloud SDK, which includes gsutil by default. Read our article about How to Use the gsutil Command-Line Tool for Google Cloud Storage.

Migrating Storage to Google Cloud with Cloud Volumes ONTAP

NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload.

In particular, Cloud Volumes ONTAP assists with cloud migration. NetApp’s data replication tools SnapMirror® and Cloud Sync service will get your data to the cloud.
View full post