Make the most of your GCP or AWS credits using Spell cluster migration

One feature of the Spell is how easily you can switch between cloud providers using the platform. We support both GCP and AWS with minimal vendor lock-in — as a result, swapping a Spell organization between these two clouds is an easy, relatively painless process. This is a very useful feature for teams not yet committed to a single specific cloud provider, or for teams with cloud compute credits on both clouds that they’d like to use.

In this blog post we’ll show you how to use it!

First, some background on clusters

When you create an organizational account in Spell for Teams, one of the first very things you do is create and configure your cluster. A Spell cluster is an isolated VPC within your GCP project or AWS account that encapsulates Spell's userland resources. Clusters are easy to create — just run the spell cluster init gcp or the spell cluster init aws command and follow the instructions in the interactive prompt to complete cluster setup:

A Spell organization may have at most one cluster at a time (enterprise customers may use multiple organizations). The cluster currently configured for the organization is listed in the sidebar in the Spell web console.

Deleting a cluster is a similarly easy (though more destructive) operation: just run spell cluster delete. However, it’s not as destructive as you might think at first.

Spell has two tiers of infrastructural resources: "cluster level" resources, which live inside of the Spell cluster, and "organization level" resources, which live inside Spell’s private production environment. Because organization-level resources are not part of the cluster, they are not deleted when you delete the cluster. As a result, they will be carried over to any new cluster that you create — including one created on a completely different cloud provider!

Moving between clouds in action

Most of Spell’s internal components are actually organization-level resources. They will persist across clusters, no problem. This makes moving from an AWS environment to a GCP environment, or vice versa, really easy.

Our starting point is an organization named aws-org, which has an already-initialized cluster named aws-spell2-cluster attached:

Let’s now swap to a GCP cluster (note that this requires having GCP authentication set up on your local machine):

$ spell cluster delete
Are you SURE you want to delete the spell cluster aws-org? [y/N]: y
# ...some interactive prompts...
# VERY IMPORTANT: make sure to answer NO when asked if you'd like
# to delete your Spell bucket!
$ spell cluster create gcp
# ...some more interactive prompts...
Your cluster gcp-org is initialized!

The last thing we need to do to complete the migration is move the resources in the old AWS cluster’s S3 bucket into the new GCP cluster’s GCS bucket. Luckily, Google’s gsutil CLI tool makes this trivially easy to do. In my case, I ran the following command:

$ gsutil cp -R \
    s3://aleksey-local-aws-bucket \
    gs://aleksey-local-gcp-bucket

(notice that moving from GCP to AWS is merely a matter of reversing these two lines of code)

If we now check back in on the web console, our runs page looks exactly the same! The only difference is that where once there was aws-org, now there is gcp-org instead:

Visiting the clusters detail page confirm that we are now rocking a GCP cluster instead of an AWS one:

And if we schedule a new run (spell run --machine-type cpu sleep 1000) it will pick up the next available run ID, just like we’d expect it to:

All that being said, you should keep in mind that there are two things that don’t transfer over:

  • The old cluster's machine types (including private machine types) will be deleted and will need to be recreated.
  • The old cluster's Spell model servers will be terminated and will not carry over. You will need to reinitialize them yourself.

And that’s it! That’s all you need!

Ready to Get Started?

Create an account in minutes or connect with our team to learn how Spell can accelerate your business.