Resources
Resources are the generic name for datasets, models, or other files which can be made available to Spell runs, workspaces, hyperparameter searches, and model servers. Spell keeps these organized for you in a remote filesystem, SpellFS.
Browsing resources
To view resources in the web console, visit the resources page. To view resources using the Spell CLI, use spell ls
:
$ spell ls
- - public
- - uploads
- - runs
Resources are stored in an object storage bucket that Spell creates and manages for you (specifically, an S3 bucket on AWS; a Cloud Storage bucket on GCP; or an Azure Blob Storage container on Azure). Our virtual filesystem, SpellFS, handles moving files in bucket storage to and from your model servers or runs automatically.
Resources are split up into a handful of different top-level folders. There are three folders that are created by default: public, runs, and uploads.
The public folder contains a few simple example datasets used by certain Spell tutorials.
The runs contains run outputs. Spell will automatically back up any new files written to the current working directory within a Spell run to a subfolder within this directory. For example:
$ spell ls runs
- Jan 31 17:02 1
- Jan 31 17:27 4
$ spell ls runs/4
34 Jan 31 17:06 images
210 Jan 31 17:20 data
If a run doesn’t generate any outputs or if it was removed using spell rm
, it won’t be listed. To learn more, refer to the section "Saving resources" in the run guide.
The third folder is uploads. This folder contains any files you uploaded to Spell. See the section "Uploading resources" for more details.
Organizations that use the workspace export feature will have an additional workspace_exports
folder in their resources. To learn more about this feature see the section "Publishing workspace files to SpellFS" in the workspaces guide.
Organizations that use Spell's private bucket mounting feature will have an additional s3
, gs
, or azblob
folder in their resources. To learn more about this feature refer to the page Bucket Management.
Uploading resources
Use the spell upload
command to upload your datasets to SpellFS.
For example, suppose you have a folder foo
containing the file bar.py
. After running:
$ spell upload foo
Total upload size: 31B
Uploading to uploads/foo [####################################] 100%
Upload of foo (/Users/aleksey/Desktop/foo) to 'uploads/foo' complete.
The folder foo
and all of its contents (in this case, just the one file bar.py
) will now be on SpellFS from inside of the uploads
folder. You can look around using spell ls
(or view your resources in the web console):
$ spell ls uploads/
- Mar 10 18:57 foo
$ spell ls uploads/foo/
31 Mar 10 18:58 bar.py
You can also choose to upload the folder to a renamed location using the --name
option:
$ spell upload foo/ --name alternative_foo
$ spell ls uploads/
- Mar 10 19:05 alternative_foo
- Mar 10 18:57 foo
spell upload
can also be used to upload an individual file. In this case you should specify a name (otherwise the name of the folder containing the file will be used):
$ spell upload foo/bar.py --name alternative_bar.py
$ spell ls uploads/
- Mar 10 19:06 alternative_bar.py
- Mar 10 19:05 alternative_foo
- Mar 10 18:57 foo
Downloading resources
To download a resource using the Spell CLI, use spell cp
command. For example:
$ spell cp runs/4
✔ Copied 10 files
To download a resource using the web console, navigate to the resource you want to download in the browser and click on the download button.
Deleting resources
You can delete any resource in the uploads
or runs
folders using spell rm
.
$ spell rm uploads/foo uploads/alternative_foo
spell rm
accepts a list of directory paths as input, and supports ranges (e.g. spell rm /runs/1-10
). It does not support wildcards (?
, *
, etcetera).
Mounting resources
Using SpellFS resources inside of a run, workspace, or model server requires mounting the data to the instance. This is done using the --mount
flag on the spell run
, spell hyper
, spell jupyter
, spell server serve
, and spell server update
commands.
To learn more, refer to the section "Mounting resources" in the run guide, "Creating a workspace using the web console" in the workspace guide, or "Managing model server mounts" in the model server guide.
(Advanced) Creating resource links
Resource links are a convenient way to refer to specific files or directories within your Spell resources. They are very similar to file system symbolic links. If you frequently find yourself referring to a deeply nested file, such as runs/2493/checkpoints/models/face-detection/weights
, you can create a link:
$ spell link face-weights runs/2493/checkpoints/models/face-detection/weights
Successfully created symlink:
face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58
The link alias can then be used in other Spell commands. For example, to mount the file in a new run you can use the face-weights
alias:
$ spell run -m face-weights:model/weights python train.py
To create a link use the spell link
command, pass the desired link alias as the first argument and the resource path as the second argument. The resource path must be an existing file or directory within a Spell resource (i.e., a run output, upload, or public dataset). For example:
$ spell link coco-train public/image/coco2014/train2014/
Successfully created symlink:
coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
Note
Resource links created in Spell for Teams are shared across the entire team.
(Advanced) Viewing resource links
There are three ways to view all of your links.
Running the spell link
command with no arguments:
$ spell link
face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58
coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
Running the spell ls
command with no arguments:
$ spell ls
- - public
- - runs
- - uploads
- - face-weights -> runs/2493/checkpoints/models/face-detection/weights
- - coco-train -> public/image/coco2014/train2014
Visiting the resource page on your web console:
(Advanced) Deleting resource links
You can delete a link using the spell unlink
command:
$ spell unlink face-weights
Successfully removed link with alias face-weights.
Note
In Spell for Teams, only the original creator or an admin of the organization is allowed to delete the link.