What is a Resource

Resources are the generic name for datasets, models, or any other files that can be made available to a run. Spell keeps these organized for you in a remote filesystem, SpellFS.

Viewing your resources

To look through your available resources, use spell ls or go to your resources page:

$ spell ls
-        -              public
-        -              uploads
-        -              runs

Initially, you’ll see three directories: public , uploads, and runs. Read more about each of these resource types below.

Public

The public directory contains pre-packaged versions of commonly used datasets and a few pre-trained models.

Within public, the resources are organized in a series of directories based on category of the dataset.

$ spell ls public
-        -              audio
-        -              face
-        -              finance
-        -              healthcare
-        -              image
-        -              models
-        -              social
-        -              text
-        -              tutorial
-        -              video

$ spell ls public/video
-        -              bdd100k (Berkeley DeepDrive...

To learn more about how to use public datasets and models in your Spell runs and workspaces, read about Mounting Resources in the runs guide.

Runs

The runs directory keeps the saved outputs from all of your runs, listed by run id. If a run doesn’t generate any outputs or if it was removed using spell rm, it won’t be listed.

$ spell ls runs
    -        Jan 31 17:02   1
    -        Jan 31 17:27   4

To view resources within a run, use spell ls with the path. This will return a table with the size of the resource, the date created, and the name of the directory or file.

$ spell ls runs/4
34       Jan 31 17:06   images
210      Jan 31 17:20   data

Uploads

Uploads contains all of the files which you have uploaded to Spell directly. This is the recommended flow for uploading large datasets to Spell. See the next section for more details on how this works.

(Advanced) Workspace exports

There is a special fourth folder, workspace_exports, containing files exported to SpellFS from workspaces. To learn more about this feature see the section "Publishing workspace files to SpellFS" in the workspaces guide.

Uploading resources

Use the spell upload command to upload your datasets to SpellFS.

For example, suppose you have a folder foo containing the file bar.py. After running:

$ spell upload foo
Total upload size: 31B
Uploading to uploads/foo  [####################################]  100%
Upload of foo (/Users/aleksey/Desktop/foo) to 'uploads/foo' complete.

The folder foo and all of its contents (in this case, just the one file bar.py) will now be on SpellFS from inside of the uploads folder. You can look around using spell ls (or view your resources in the web console):

$ spell ls uploads/
-        Mar 10 18:57   foo

$ spell ls uploads/foo/
31       Mar 10 18:58   bar.py

You can also choose to upload the folder to a renamed location using the --name option:

$ spell upload foo/ --name alternative_foo

$ spell ls uploads/
-        Mar 10 19:05   alternative_foo
-        Mar 10 18:57   foo

spell upload can also be used to upload an individual file. In this case you should specify a name (otherwise the name of the folder containing the file will be used):

$ spell upload foo/bar.py --name alternative_bar.py

$ spell ls uploads/

-        Mar 10 19:06   alternative_bar.py
-        Mar 10 19:05   alternative_foo
-        Mar 10 18:57   foo

Downloading resources

Any resource can also be downloaded with the spell cp command. For example:

$ spell cp runs/4
✔ Copied 10 files

Deleting uploaded resources

You can delete any resource in the uploads folder using spell rm.

$ spell rm uploads/foo uploads/alternative_foo

spell rm accepts a list of directory paths as input, and supports ranges (e.g. spell rm /runs/1-10). It does not support wildcards (?, *, etcetera).

Mounting resources

Using SpellFS resources inside of a run or workspace requires mounting the data to the instance. This is done using e.g. the --mount flag on the spell run command.

To learn how to mount resources to a run, see the section "Mounting resources" in the runs guide. To learn how to mount resources to a workspace, see the section "Creating a workspace using the web console" in the workspace guide.

(Advanced) Creating resource links

Resource links are a convenient way to refer to specific files or directories within your Spell resources. They are very similar to file system symbolic links. If you frequently find yourself referring to a deeply nested file, such as runs/2493/checkpoints/models/face-detection/weights, you can create a link:

$ spell link face-weights runs/2493/checkpoints/models/face-detection/weights
Successfully created symlink:
face-weights ---> runs/2493/checkpoints/models/face-detection/weights    Mar 11 10:58

The link alias can then be used in other Spell commands. For example, to mount the file in a new run you can use the face-weights alias:

$ spell run -m face-weights:model/weights python train.py

To create a link use the spell link command, pass the desired link alias as the first argument and the resource path as the second argument. The resource path must be an existing file or directory within a Spell resource (i.e., a run output, upload, or public dataset). For example:

$ spell link coco-train public/image/coco2014/train2014/
Successfully created symlink:
coco-train ---> public/image/coco2014/train2014    Mar 11 11:07

(Advanced) Viewing resource links

There are three ways to view all of your links.

Running the spell link command with no arguments:

$ spell link
face-weights ---> runs/2493/checkpoints/models/face-detection/weights    Mar 11 10:58
coco-train ---> public/image/coco2014/train2014                          Mar 11 11:07

Running the spell ls command with no arguments:

$ spell ls
-        -              public
-        -              runs
-        -              uploads
-        -              face-weights -> runs/2493/checkpoints/models/face-detection/weights
-        -              coco-train -> public/image/coco2014/train2014

Visiting the resource page on your web console:

(Advanced) Deleting resource links

You can delete a link using the spell unlink command:

$ spell unlink face-weights
Successfully removed link with alias face-weights.