Bucket Management

For users on the Spell for Teams plan, we deploy Spell in your cloud and provide the same cluster management tools backing our own internal infrastructure. You can directly mount data from your S3/GS buckets into your runs. This document explains how to accomplish that.

How resources are managed

When you set up Spell within your cluster, the setup script creates a new bucket that is used exclusively for outputs from your runs and Jupyter workspaces. Spell will never write/modify any other buckets. The name of this bucket is displayed under Bucket Name on the cluster dashboard. The contents of this bucket will be displayed under runs in Spell resources.

Adding a private bucket to a cluster

If a bucket is public you can mount resources from that bucket into your runs directly. See the section "Mounting Resources" of the Resources guide for more information.

If a bucket is private, before using it you must first add it to SpellFS by running spell cluster add-bucket:

$ spell cluster add-bucket
All of this will be done within your project 'spell-external' - continue? [Y/n]: y
This command will
    - List your buckets to generate an options menu of buckets that can be added to Spell
    - Add list and read permissions for that bucket to the service account associated with the cluster
    - Ensure that the interoperable S3 access credentials associated with the cluster are able to read this bucket.
Please choose a bucket: spell-my-cluster
Bucket spell-my-cluster]has been added to cluster 1!

Now, when you run spell ls s3/ (AWS cluster) or spell ls gs/ (GCP cluster), you will see the new bucket:

$ spell ls gs/
-        -              spell-my-cluster

After doing this, you can mount private bucket the same way you would mount any other resource, using the mount flag:

spell run --mount gs/spell-my-cluster/file:file 'cat file'