Using an object storage service is often the primary choice in cloud computing. Its benefits, including ease of use and small operational overhead, are extremely appealing to cloud customers. In Google Cloud, the object storage service is provided by the suitably named Google Cloud Storage service.
In this article we are going to explain how to use Google Cloud Storage via the command-line tool gsutil, and other ways to interact with Google Cloud Storage. This can be also helpful for users who are getting started with the new NetApp® Cloud Volumes ONTAP for Google Cloud.
To get started now, click the links below:
In object storage, files are simply referenced as objects and organized into buckets—logical namespaces that are able to hold one or more objects. While a bucket can have multiple objects, an object can only belong to just one bucket. The object storage service offered by Google Cloud is called Google Cloud Storage. Google Cloud Storage comes with built-in capabilities to manage and automate bucket lifecycle rules. In Google Cloud Storage, the availability of the data can be configured by choosing a storage class for the bucket and its objects: Standard, Nearline, and Coldline. In addition, the data location can be specified, which also impacts the availability, by choosing between region, dual-region and multi-region. Google Cloud Storage comes with built-in capabilities to manage and automate bucket lifecycle rules.
Using cloud computing resources has never been easier. The number of tools and overall methods one has at their disposal to interact with these resources is useful to a range of different users.
In Google Cloud Platform, there are three different methods that can be used:
What would you use if you had multiple files to store in object storage in the cloud? The gsutil command-line tool of course! Now let’s see how to do it.
In this section we are going to explore how the gsutil command-line tool works in practice. This tool can help with the transfer of several objects, including folders, and is also well suited for automating tasks that are not so easy to achieve via the Google Cloud Console.
1. Start by logging in to the Google Cloud Console (the web user interface) and create a new Google Cloud Platform project with billing configured and enabled.
2. Download and install the latest version of Google Cloud SDK from the official documentation website. Depending on your machine operating system, the installation process may vary slightly, but in general should be quite straightforward.
3. You can verify that the install went successfully by opening up your machine terminal and executing the command gsutil -v. You will be greeted with the version of the gsutil installed on your machine (Fig. 1). If not, please make sure to go back to steps 1 and 2 before proceeding further.
Fig.1 gsutil version.
4. To start using gsutil, you first need to authenticate it with your Google Cloud Platform account. Issue the command gcloud auth login and you will be directed to your browser to authenticate. Once the process is successful, both your email and active GCP project will be displayed in the terminal (Fig. 2).
Fig.2 Authenticate to GCP using gcloud.
Creating a Google Cloud Storage bucket is the first step before any files (i.e. objects) can be uploaded to Google Cloud Storage. The bucket is a virtual namespace for our objects and, as you might recall, an object can only belong to a single bucket.
1. Using your machine terminal, issue the command gsutil help mb (Fig. 2.). This help command will provide you all the syntax details of the mb (make bucket) option. You can use the help command with any of the gsutil options, which is an invaluable resource for providing information and detailed explanations on all the mandatory and optional flags.
Fig.3 gsutil help pages
2. Creating the Google Cloud Storage bucket is, as you might have noticed from the previous output (Fig. 3 above), quite simple. Use the command gsutil mb gs://<YOUR_BUCKET_NAME> to create it (Fig. 4). You will need to make sure that <YOUR_BUCKET_NAME> is a unique name, not just in your GCP account but in the entire global Google Cloud Platform.
Fig.4 Create a GCP Bucket using gsutil
The process of adding and removing data from a Google storage bucket is quite simple and fast, with the only potential bottleneck being the speed of your internet connection. If you are used to the terminal, you may be familiar with the concept of the commands cp (copy) and rm (remove). The Google Cloud Storage command-line tool gsutil uses the exact same concept.
1. Start by changing the directory to the folder in your local machine where the files you want to upload are located. In our example case, this would be a folder in our Desktop called “photos” (Fig. 5).
2. In order to copy all the files in the current directory to the storage bucket we previously created, we will use the command gsutil cp * gs://<YOUR_BUCKET_NAME> (Fig. 5).
One important aspect to note is that when performing an upload of multiple large files, using the flag -m, to perform a parallel (multi-threaded/multi-processing) copy, will significantly improve the performance.
Also, if you wish to include local subfolders in your upload, you will be required to use the flag -r, i.e., to perform a recursive copy.
3. You can verify the files were properly created by using the command gsutil ls gs://<YOUR_BUCKET_NAME> and list the contents of the bucket (Fig. 5).
Fig.5 Copying multiple files using gsutil
4. Likewise, if you wish to remove the objects from the storage bucket, you can issue the command gsutil rm gs://<YOUR_BUCKET_NAME>/* to remove all the files inside the bucket (Fig. 6).
5. Clean up the environment by deleting the Cloud Storage bucket you created, by using the subcommand rb (remove bucket), i.e., with the command gsutil rb gs://<YOUR_BUCKET_NAME> (Fig. 6).
Fig.6 Removing files and bucket using gsutil
You've just seen how to interact with the Google Cloud Platform's console, command-line tools, and SDK’s.
Now that you know how to install and authenticate with Google Cloud using your machine terminal, there's a lot more on Google Cloud and Google Cloud Storage with Cloud Volumes ONTAP. Follow the links below to learn how to: