Resources

Resources are the generic name for datasets, models, or other files which can be made available to Spell runs, workspaces, hyperparameter searches, and model servers. Spell keeps these organized for you in a remote filesystem, SpellFS.

Browsing resources

To view resources in the web console, visit the resources page. To view resources using the Spell CLI, use spell ls:

$ spell ls
-        -              public
-        -              uploads
-        -              runs

Resources are stored in an object storage bucket that Spell creates and manages for you (specifically, an S3 bucket on AWS; a Cloud Storage bucket on GCP; or an Azure Blob Storage container on Azure). Our virtual filesystem, SpellFS, handles moving files in bucket storage to and from your model servers or runs automatically.

Resources are split up into a handful of different top-level folders. There are three folders that are created by default: public, runs, and uploads.

The public folder contains a few simple example datasets used by certain Spell tutorials.

The runs contains run outputs. Spell will automatically back up any new files written to the current working directory within a Spell run to a subfolder within this directory. For example:

$ spell ls runs
    -        Jan 31 17:02   1
    -        Jan 31 17:27   4
$ spell ls runs/4
34       Jan 31 17:06   images
210      Jan 31 17:20   data

If a run doesn’t generate any outputs or if it was removed using spell rm, it won’t be listed. To learn more, refer to the section "Saving resources" in the run guide.

The third folder is uploads. This folder contains any files you uploaded to Spell. See the section "Uploading resources" for more details.

Organizations that use the workspace export feature will have an additional workspace_exports folder in their resources. To learn more about this feature see the section "Publishing workspace files to SpellFS" in the workspaces guide.

Organizations that use Spell's private bucket mounting feature will have an additional s3, gs, or azblob folder in their resources. To learn more about this feature refer to the page Bucket Management.

Uploading resources

Use the spell upload command to upload your datasets to SpellFS.

For example, suppose you have a folder foo containing the file bar.py. After running:

$ spell upload foo
Total upload size: 31B
Uploading to uploads/foo  [####################################]  100%
Upload of foo (/Users/aleksey/Desktop/foo) to 'uploads/foo' complete.

The folder foo and all of its contents (in this case, just the one file bar.py) will now be on SpellFS from inside of the uploads folder. You can look around using spell ls (or view your resources in the web console):

$ spell ls uploads/
-        Mar 10 18:57   foo

$ spell ls uploads/foo/
31       Mar 10 18:58   bar.py

You can also choose to upload the folder to a renamed location using the --name option:

$ spell upload foo/ --name alternative_foo

$ spell ls uploads/
-        Mar 10 19:05   alternative_foo
-        Mar 10 18:57   foo

spell upload can also be used to upload an individual file. In this case you should specify a name (otherwise the name of the folder containing the file will be used):

$ spell upload foo/bar.py --name alternative_bar.py

$ spell ls uploads/

-        Mar 10 19:06   alternative_bar.py
-        Mar 10 19:05   alternative_foo
-        Mar 10 18:57   foo

Downloading resources

To download a resource using the Spell CLI, use spell cp command. For example:

$ spell cp runs/4
✔ Copied 10 files

To download a resource using the web console, navigate to the resource you want to download in the browser and click on the download button.

Deleting resources

You can delete any resource in the uploads or runs folders using spell rm.

$ spell rm uploads/foo uploads/alternative_foo

spell rm accepts a list of directory paths as input, and supports ranges (e.g. spell rm /runs/1-10). It does not support wildcards (?, *, etcetera).

Mounting resources

Using SpellFS resources inside of a run, workspace, or model server requires mounting the data to the instance. This is done using the --mount flag on the spell run, spell hyper, spell jupyter, spell server serve, and spell server update commands.

To learn more, refer to the section "Mounting resources" in the run guide, "Creating a workspace using the web console" in the workspace guide, or "Managing model server mounts" in the model server guide.

(Advanced) Creating resource links

Resource links are a convenient way to refer to specific files or directories within your Spell resources. They are very similar to file system symbolic links. If you frequently find yourself referring to a deeply nested file, such as runs/2493/checkpoints/models/face-detection/weights, you can create a link:

$ spell link face-weights runs/2493/checkpoints/models/face-detection/weights
Successfully created symlink:
face-weights ---> runs/2493/checkpoints/models/face-detection/weights    Mar 11 10:58

The link alias can then be used in other Spell commands. For example, to mount the file in a new run you can use the face-weights alias:

$ spell run -m face-weights:model/weights python train.py

To create a link use the spell link command, pass the desired link alias as the first argument and the resource path as the second argument. The resource path must be an existing file or directory within a Spell resource (i.e., a run output, upload, or public dataset). For example:

$ spell link coco-train public/image/coco2014/train2014/
Successfully created symlink:
coco-train ---> public/image/coco2014/train2014    Mar 11 11:07

(Advanced) Viewing resource links

There are three ways to view all of your links.

Running the spell link command with no arguments:

$ spell link
face-weights ---> runs/2493/checkpoints/models/face-detection/weights    Mar 11 10:58
coco-train ---> public/image/coco2014/train2014                          Mar 11 11:07

Running the spell ls command with no arguments:

$ spell ls
-        -              public
-        -              runs
-        -              uploads
-        -              face-weights -> runs/2493/checkpoints/models/face-detection/weights
-        -              coco-train -> public/image/coco2014/train2014

Visiting the resource page on your web console:

(Advanced) Deleting resource links

You can delete a link using the spell unlink command:

$ spell unlink face-weights
Successfully removed link with alias face-weights.