What is a Resource
Resources are the generic name for datasets, models, or any other files that can be made available to a run. Spell keeps these organized for you in a remote filesystem, SpellFS.
Viewing your resources
To look through your available resources, use
spell ls or go to your resources page:
$ spell ls - - public - - uploads - - runs
Initially, you’ll see three directories:
runs. Read more about each of these resource types below.
public directory contains pre-packaged versions of commonly used datasets and a few pre-trained models.
Within public, the resources are organized in a series of directories based on category of the dataset.
$ spell ls public - - audio - - face - - finance - - healthcare - - image - - models - - social - - text - - tutorial - - video
$ spell ls public/video - - bdd100k (Berkeley DeepDrive...
To learn more about how to use public datasets and models in your Spell runs and workspaces, read about Mounting Resources in the runs guide.
runs directory keeps the saved outputs from all of your runs, listed by run id. If a run doesn’t generate any outputs or if it was removed using
spell rm, it won’t be listed.
$ spell ls runs - Jan 31 17:02 1 - Jan 31 17:27 4
To view resources within a run, use
spell ls with the path. This will return a table with the size of the resource, the date created, and the name of the directory or file.
$ spell ls runs/4 34 Jan 31 17:06 images 210 Jan 31 17:20 data
Uploads contains all of the files which you have uploaded to Spell directly. This is the recommended flow for uploading large datasets to Spell. See the next section for more details on how this works.
(Advanced) Workspace exports
There is a special fourth folder,
workspace_exports, containing files exported to SpellFS from workspaces. To learn more about this feature see the section "Publishing workspace files to SpellFS" in the workspaces guide.
spell upload command to upload your datasets to SpellFS.
For example, suppose you have a folder
foo containing the file
bar.py. After running:
$ spell upload foo Total upload size: 31B Uploading to uploads/foo [####################################] 100% Upload of foo (/Users/aleksey/Desktop/foo) to 'uploads/foo' complete.
foo and all of its contents (in this case, just the one file
bar.py) will now be on SpellFS from inside of the
uploads folder. You can look around using
spell ls (or view your resources in the web console):
$ spell ls uploads/ - Mar 10 18:57 foo
$ spell ls uploads/foo/ 31 Mar 10 18:58 bar.py
You can also choose to upload the folder to a renamed location using the
$ spell upload foo/ --name alternative_foo
$ spell ls uploads/ - Mar 10 19:05 alternative_foo - Mar 10 18:57 foo
spell upload can also be used to upload an individual file. In this case you should specify a name (otherwise the name of the folder containing the file will be used):
$ spell upload foo/bar.py --name alternative_bar.py
$ spell ls uploads/
- Mar 10 19:06 alternative_bar.py - Mar 10 19:05 alternative_foo - Mar 10 18:57 foo
Any resource can also be downloaded with the
spell cp command. For example:
$ spell cp runs/4 ✔ Copied 10 files
Deleting uploaded resources
You can delete any resource in the
uploads folder using
$ spell rm uploads/foo uploads/alternative_foo
spell rm accepts a list of directory paths as input, and supports ranges (e.g.
spell rm /runs/1-10). It does not support wildcards (
Using SpellFS resources inside of a run or workspace requires mounting the data to the instance. This is done using e.g. the
--mount flag on the
spell run command.
To learn how to mount resources to a run, see the section "Mounting resources" in the runs guide. To learn how to mount resources to a workspace, see the section "Creating a workspace using the web console" in the workspace guide.
(Advanced) Creating resource links
Resource links are a convenient way to refer to specific files or directories within your Spell resources. They are very similar to file system symbolic links. If you frequently find yourself referring to a deeply nested file, such as
runs/2493/checkpoints/models/face-detection/weights, you can create a link:
$ spell link face-weights runs/2493/checkpoints/models/face-detection/weights Successfully created symlink: face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58
The link alias can then be used in other Spell commands. For example, to mount the file in a new run you can use the
$ spell run -m face-weights:model/weights python train.py
To create a link use the
spell link command, pass the desired link alias as the first argument and the resource path as the second argument. The resource path must be an existing file or directory within a Spell resource (i.e., a run output, upload, or public dataset). For example:
$ spell link coco-train public/image/coco2014/train2014/ Successfully created symlink: coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
Resource links created in Spell for Teams are shared across the entire team.
(Advanced) Viewing resource links
There are three ways to view all of your links.
spell link command with no arguments:
$ spell link face-weights ---> runs/2493/checkpoints/models/face-detection/weights Mar 11 10:58 coco-train ---> public/image/coco2014/train2014 Mar 11 11:07
spell ls command with no arguments:
$ spell ls - - public - - runs - - uploads - - face-weights -> runs/2493/checkpoints/models/face-detection/weights - - coco-train -> public/image/coco2014/train2014
Visiting the resource page on your web console:
(Advanced) Deleting resource links
You can delete a link using the
spell unlink command:
$ spell unlink face-weights Successfully removed link with alias face-weights.
In Spell for Teams, only the original creator or an admin of the organization is allowed to delete the link.