Resources are the generic name for datasets, models, or other files which can be made available to Spell runs, workspaces, hyperparameter searches, and model servers. Spell keeps these organized for you in a remote filesystem, SpellFS.
To view resources in the web console, visit the resources page. To view resources using the Spell CLI, use
$ spell ls - - public - - uploads - - runs
Resources are stored in an object storage bucket that Spell creates and manages for you (specifically, an S3 bucket on AWS; a Cloud Storage bucket on GCP; or an Azure Blob Storage container on Azure). Our virtual filesystem, SpellFS, handles moving files in bucket storage to and from your model servers or runs automatically.
Resources are split up into a handful of different top-level folders. There are three folders that are created by default: public, runs, and uploads.
The public folder contains a few simple example datasets used by certain Spell tutorials.
The runs contains run outputs. Spell will automatically back up any new files written to the current working directory within a Spell run to a subfolder within this directory. For example:
$ spell ls runs - Jan 31 17:02 1 - Jan 31 17:27 4 $ spell ls runs/4 34 Jan 31 17:06 images 210 Jan 31 17:20 data
If a run doesn’t generate any outputs or if it was removed using
spell rm, it won’t be listed. To learn more, refer to the section "Saving resources" in the run guide.
The third folder is uploads. This folder contains any files you uploaded to Spell. See the section "Uploading resources" for more details.
Organizations that use the workspace export feature will have an additional
workspace_exports folder in their resources. To learn more about this feature see the section "Publishing workspace files to SpellFS" in the workspaces guide.
Organizations that use Spell's private bucket mounting feature will have an additional
azblob folder in their resources. To learn more about this feature refer to the page Bucket Management.
spell upload command to upload your datasets to SpellFS.
For example, suppose you have a folder
foo containing the file
bar.py. After running:
$ spell upload foo Total upload size: 31B Uploading to uploads/foo [####################################] 100% Upload of foo (/Users/aleksey/Desktop/foo) to 'uploads/foo' complete.
foo and all of its contents (in this case, just the one file
bar.py) will now be on SpellFS from inside of the
uploads folder. You can look around using
spell ls (or view your resources in the web console):
$ spell ls uploads/ - Mar 10 18:57 foo
$ spell ls uploads/foo/ 31 Mar 10 18:58 bar.py
You can also choose to upload the folder to a renamed location using the
$ spell upload foo/ --name alternative_foo
$ spell ls uploads/ - Mar 10 19:05 alternative_foo - Mar 10 18:57 foo
spell upload can also be used to upload an individual file. In this case you should specify a name (otherwise the name of the folder containing the file will be used):
$ spell upload foo/bar.py --name alternative_bar.py
$ spell ls uploads/
- Mar 10 19:06 alternative_bar.py - Mar 10 19:05 alternative_foo - Mar 10 18:57 foo
To download a resource using the Spell CLI, use
spell cp command. For example:
$ spell cp runs/4 ✔ Copied 10 files
To download a resource using the web console, navigate to the resource you want to download in the browser and click on the download button.
You can delete any resource in the
runs folders using
$ spell rm uploads/foo uploads/alternative_foo
spell rm accepts a list of directory paths as input, and supports ranges (e.g.
spell rm /runs/1-10). It does not support wildcards (
Using SpellFS resources inside of a run, workspace, or model server requires mounting the data to the instance. This is done using the
--mount flag on the
spell server serve, and
spell server update commands.
To learn more, refer to the section "Mounting resources" in the run guide, "Creating a workspace using the web console" in the workspace guide, or "Managing model server mounts" in the model server guide.
(Advanced) Creating resource links
Resource links are a convenient way to refer to specific files or directories within your Spell resources. They are very similar to file system symbolic links. If you frequently find yourself referring to a deeply nested file, such as
runs/2493/checkpoints/models/face-detection/weights, you can create a link:
$ spell link face-weights runs/2493/checkpoints/models/face-detection/weights Successfully created symlink: face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58
The link alias can then be used in other Spell commands. For example, to mount the file in a new run you can use the
$ spell run -m face-weights:model/weights python train.py
To create a link use the
spell link command, pass the desired link alias as the first argument and the resource path as the second argument. The resource path must be an existing file or directory within a Spell resource (i.e., a run output, upload, or public dataset). For example:
$ spell link coco-train public/image/coco2014/train2014/ Successfully created symlink: coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
Resource links created in Spell for Teams are shared across the entire team.
(Advanced) Viewing resource links
There are three ways to view all of your links.
spell link command with no arguments:
$ spell link face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58 coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
spell ls command with no arguments:
$ spell ls - - public - - runs - - uploads - - face-weights -> runs/2493/checkpoints/models/face-detection/weights - - coco-train -> public/image/coco2014/train2014
Visiting the resource page on your web console:
(Advanced) Deleting resource links
You can delete a link using the
spell unlink command:
$ spell unlink face-weights Successfully removed link with alias face-weights.
In Spell for Teams, only the original creator or an admin of the organization is allowed to delete the link.