Getting Started
API ReferenceSpell CLI Documentation
Welcome to the spell command line interface documentation. Below you will find a description of each command, including the various options and arguments.
spell
Usage: spell [OPTIONS] COMMAND [ARGS]...
Options:
--docs Open documentation in the browser and exit
-h, --help Show this message and exit.
--version Show the version and exit.
Commands:
archive Specify one or more Run IDs to archive
cluster Manage external clusters
cp Retrieve a file or directory
curl Interact directly with the Spell API via `curl`
feedback Provide feedback to the Spell team
help Display help information
hyper Create hyperparameter searches
info Describes a run. Displays info such as start and end time as well as run
parameters such as apts, pips, and mounts.
jupyter Create a jupyter workspace
keys Manage public SSH keys registered with Spell
kill Kill a current run.
kube-cluster Manage a serving cluster
label Adds and remove labels from a run
link List symlinks, find symlinks by alias, or create a symlink to a resource
path.
login Login to your Spell account.
logout Log out of current session
logs Retrieve logs for a run
ls List resource files
model Manage Spell models
passwd Set a new password for your Spell account
ps Display run statuses
repos List repositories
rm Specify one or more uploaded resources to delete. For example:
uploads/[directory]
run Execute a new run
server Manage model servers
stats Display performance statistics for a run
status Display account and billing information
stop Stop a run with status 'Running'
unlink Remove a link.
upload Upload a file or directory
whoami Display current user information
workflow Execute workflows
spell archive
Usage: spell archive [OPTIONS] RUN_IDS...
Archive one or more runs. Use to remove a finished or failed run by specifying its
RUN_ID.
The removed runs will no longer show up in `ps`. The outputs of removed runs and
removed uploads will no longer appear in `ls` or be mountable on another run with
`--mount`.
Options:
-q, --quiet Suppress logging
--help Show this message and exit.
spell cluster
Usage: spell cluster [OPTIONS] COMMAND [ARGS]...
Manage external clusters on Spell
Options:
--help Show this message and exit.
Commands:
add-bucket Adds a cloud storage bucket to SpellFS
add-docker-registry Configures your cluster to enable runs with docker images
in the private registry hosted by your cloud provider (ECR
or GCR respectively)
create-kube-cluster Sets up a GKE/EKS kubernetes cluster in your Spell VPC.
Required for model serving.
delete Deletes a given cluster
delete-docker-registry Removes your cluster's access to docker images in the
private registry hosted by your cloud provider (ECR or GCR
respectively).
init Create a cluster
kube-cluster-add-user Grant another AWS User in your account permissions to...
kubectl Issue kubectl commands against the serving cluster
list List all clusters
machine-type Manage machine types
node-group Manage kube cluster node groups
rotate-storage-key Rotates the storage key for storage accounts in Azure
clusters
set-instance-permissions Sets the cloud machine instance permissions for the
cluster
unset-instance-permissions Unsets the cloud machine instance permissions for the
cluster
update Makes sure your Spell cluster is fully up to date and able
to support the latest features
update-kube-cluster Update an existing GKE/EKS kubernetes cluster in your
Spell VPC.
spell cluster add-bucket
Usage: spell cluster add-bucket [OPTIONS]
This command adds a cloud storage bucket (S3 or GS) to SpellFS, which enables
interaction with the bucket objects via ls, cp, and mounts. It will also updates the
permissions of that bucket to allow Spell read access to it
Options:
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and Secret
as well as your Region. You will be prompted to confirm the Key
and Region are correct before continuing. This key will be used to
adjust IAM permissions of the role associated with the cluster
that the bucket is being added to.
--bucket TEXT Name of bucket
--help Show this message and exit.
spell cluster add-docker-registry
Usage: spell cluster add-docker-registry [OPTIONS]
This command enables pulling docker images from a private registry. Read permissions
to the registry will be added to the IAM role associated with the cluster.
Options:
--cluster-name TEXT Name of cluster to add registry permissions to
--repo TEXT Name of repository. ECR only
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and
Secret as well as your Region. You will be prompted to confirm
the Key and Region are correct before continuing. This key will
be used to adjust IAM permissions of the role associated with the
cluster that needs access to the registry.
--help Show this message and exit.
spell cluster create-kube-cluster
Usage: spell cluster create-kube-cluster [OPTIONS]
Sets up a GKE or EKS kubernetes cluster in your Spell VPC. This cluster is required
for model serving. Spell will automatically create a CPU node group for you which
will have at least one machine running at all times.
Options:
-p, --profile TEXT AWS profile to pull credentials from
--nodes-min INTEGER Minimum number of nodes in the model serving cluster
(default 1)
--nodes-max INTEGER Minimum number of nodes in the model serving cluster
(default 2)
--node-disk-size INTEGER Size of disks on each node in GB (default 50GB)
--use-existing This is an advanced option to use an existing EKS/GKE
cluster instead of creating a new one. It will reapply
kubernetes configurations. Because Spell sets up your
cluster in a particular manner this option is only likely to
work with clusters created exactly the way we set ours up.
This flag is likely only valuable if you experienced an
error the first time you tried to run this command, but the
kube cluster creation succeeded.
--aws-zones TEXT Allows AWS clusters to explicitly list the availability
zones used for the EKS cluster. List the desired AZs as
comma separated values, ex: 'us-east-1a,us-east-1c,us-
east-1d'. NOTE: Most users will NOT have to do this. This is
useful if there are problems with one or more of the AZs in
the region of your cluster.
--from-file FILENAME A path to a YAML or JSON file to load the arguments and
options from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell cluster delete
Usage: spell cluster delete [OPTIONS]
Facilitates the deletion of your Spell cluster by removing the associated
infrastructure on Spell as well as deleting all associated cloud resources. It will
OPTIONALLY delete the data in your output bucket - including run outputs.
Options:
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and Secret
as well as your Region. You will be prompted to confirm the Key
and Region are correct before continuing. This key will be used to
destroy the VPC, IAM Roles, and optionally the S3 bucket created
for the cluster.
--help Show this message and exit.
spell cluster delete-docker-registry
Usage: spell cluster delete-docker-registry [OPTIONS]
This command removes your cluster's access to docker images in the private registry
hosted by your cloud provider.
Options:
--repo TEXT Name of repository. ECR only
--cluster-name TEXT Name of cluster to remove registry permissions from
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and
Secret as well as your Region. You will be prompted to confirm
the Key and Region are correct before continuing. This key will
be used to adjust IAM permissions of the role associated with the
cluster that has access to the registry.
--help Show this message and exit.
spell cluster init
Usage: spell cluster init [OPTIONS] COMMAND [ARGS]...
Create a new aws/gcp/azure cluster for your org account
Set up a cluster to use machines in your own AWS/GCP/Azure account
Options:
--help Show this message and exit.
Commands:
aws Sets up an AWS VPC as a Spell cluster
az Sets up an Azure VNet as a Spell cluster
gcp Sets up GCP VPC as a Spell cluster
spell cluster init aws
Usage: spell cluster init aws [OPTIONS]
This command sets an AWS VPC of your choosing as an external Spell cluster. This
will let your organization run runs in that VPC, so your data never leaves your VPC.
You set an S3 bucket of your choosing for all run outputs to be written to. After
this cluster is set up you will be able to select the types and number of machines
you would like Spell to create in this cluster.
NOTE: This command uses your AWS credentials, found in ~/.aws/credentials to create
the necessary AWS resources for Spell to access and manage those machines. Your AWS
credentials will need permission to setup these resources.
Options:
-n, --name TEXT This will be used by Spell for you to identify the cluster
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and Secret
as well as your Region. You will be prompted to confirm the Key
and Region are correct before continuing. This key will be used to
create all the resources necessary for Spell to manage machines in
your external VPC. It must have permissions to create these
resources.
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster init az
Usage: spell cluster init az [OPTIONS]
This command creates an Azure VNet of your choosing as an external Spell cluster.
This will let your organization create runs in that VNet, so your data never leaves
your VNet. You create an Azure Blob Container of your choosing for all run outputs
to be written to. After this cluster is set up you will be able to select the types
and number of machines you would like Spell to create in this cluster.
Options:
-n, --name TEXT This will be used for you to identify the cluster
-r, --resource-group TEXT This will be the name of the Resource Group Spell will
create and store all its resources in within your Azure
account
-s, --service-principal TEXT Optionally give custom name to created Service Principal
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster init gcp
Usage: spell cluster init gcp [OPTIONS]
This command creates a Spell cluster within a GCP VPC of your choosing as an
external Spell cluster. This will let your organization run runs in that VPC, so
your data never leaves your VPC. You set an GCS bucket of your choosing for all run
outputs to be written to. After this cluster is set up you will be able to select
the types and number of machines you would like Spell to create in this cluster.
NOTE: This command uses your GCP credentials, activated by running `gcloud auth
application-default login`, to create the necessary GCP resources for Spell to
access and manage those machines. Your GCP credentials will need permission to set
up these resources.
Options:
-n, --name TEXT Name used by Spell for you to identify this GCP cluster
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster kube-cluster-add-user
Usage: spell cluster kube-cluster-add-user [OPTIONS] USER
Grant another AWS User in your account permissions to manage your kube cluster.
These permissions are required for anyone to update the cluster or create/remove
node groups. The user must be in the same account you use to manage this cluster,
which we will deduce from the AWS profile you provide (you will be prompted to
confirm).
Options:
-p, --profile TEXT AWS profile to pull credentials from
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell cluster kubectl
Usage: spell cluster kubectl [OPTIONS] [ARGS]...
Issue kubectl commands against the serving cluster
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell cluster list
Usage: spell cluster list [OPTIONS]
Options:
--help Show this message and exit.
spell cluster machine-type
Usage: spell cluster machine-type [OPTIONS] COMMAND [ARGS]...
Manage groups of similar machines which can be used for training runs and workspaces
on Spell
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
add Creates a new machine type for executing Spell Runs and Workspaces
delete Delete a machine type
get-token Gets the auth token for a Private machine type
list List all your machine types
scale Change the limits for number of machines of this machine type
spell cluster machine-type add
Usage: spell cluster machine-type add [OPTIONS]
Options:
--name TEXT Name to give this machine type
--instance-type TEXT The type of machine to use e.g. 'CPU', 'K80'. If you
skip this you will be prompted with options
--spot Spot/Preemptible instances can be significantly
cheaper than on demand instances, however AWS/GCP can
terminate them at any time. If your run is terminated
prematurely we will keep all data and save it for you
with a final run status of Interrupted.
--default-auto-resume Configure the default auto resume behavior for runs on
this machine type. Runs can explicitly opt in or out
of auto resume, this will be the default used for runs
that don't specify. NOTE: This is only supported for
spot instances currently.
--storage-size INTEGER Disk size in GB
--additional-images [TensorFlow 2]
By default all machines support TensorFlow 1, PyTorch,
and Conda. If you require any additional frameworks
you can select them here. Additional frameworks will
increase the time it takes to spin up new machines.
--min-machines INTEGER Minimum number of machines to keep available at all
times regardless of demand
--max-machines INTEGER Maximum number of machines of this machine type
--idle-timeout INTEGER Grace period to wait before terminating idle machines
(minutes)
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster machine-type delete
Usage: spell cluster machine-type delete [OPTIONS] NAME
Options:
-f, --force Do not prompt for confirmation
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster machine-type get-token
Usage: spell cluster machine-type get-token [OPTIONS] NAME
Options:
--renew Renew token. This will invalidate the previous token
--help Show this message and exit.
spell cluster machine-type list
Usage: spell cluster machine-type list [OPTIONS]
Options:
--help Show this message and exit.
spell cluster machine-type scale
Usage: spell cluster machine-type scale [OPTIONS] NAME
Options:
--min-machines INTEGER Minimum number of machines to keep available at all times
regardless of demand. Omit to leave unchanged
--max-machines INTEGER Maximum number of machines of this machine type. Omit to leave
unchanged
--idle-timeout INTEGER Grace period to wait before terminating idle machines
(minutes). Omit to leave unchanged
--from-file FILENAME A path to a YAML or JSON file to load the arguments and
options from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell cluster node-group
Usage: spell cluster node-group [OPTIONS] COMMAND [ARGS]...
Manage node groups used for model serving cluster nodes
With no subcommand, display all your node groups
Options:
--help Show this message and exit.
Commands:
add Add a new node group to a Spell model serving cluster
delete Delete a node group
list Display all your node groups
scale Adjust minimum and maximum node counts for a given node group
spell cluster node-group add
Usage: spell cluster node-group add [OPTIONS]
Deploy a GKE node pool or eksctl node group for model serving
Options:
-p, --profile TEXT Specify an AWS profile to pull credentials from to perform
the NodeGroup create operation
--name TEXT Name of the node group [required]
--instance-type TEXT Instance type to use for the nodes
--accelerator NAME[:COUNT] (GCP only) Accelerator to attach to nodes, can be
specified multiple times for multiple accelerator types
--min-nodes INTEGER Minimum number of autoscaled nodes in the node group
--max-nodes INTEGER Maximum number of autoscaled nodes in the node group
--spot Use spot instances for node group nodes
--disk-size INTEGER Size of disks on each node in GB
--config-file FILE Path to file containing eksctl NodeGroup or GKE node pool
spec. Note this is an advanced option for users who want
to specify a custom node group or node pool configuration.
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster node-group delete
Usage: spell cluster node-group delete [OPTIONS] NODE_GROUP_NAME
Delete a node group
Options:
-p, --profile TEXT AWS profile to pull credentials from
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster node-group list
Usage: spell cluster node-group list [OPTIONS]
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster node-group scale
Usage: spell cluster node-group scale [OPTIONS] NODE_GROUP_NAME
Adjust autoscaling min/max nodes for a node group
Options:
-p, --profile TEXT Specify an AWS profile to pull credentials from to perform the
NodeGroup scale operation
--min-nodes INTEGER Minimum number of autoscaled nodes in the node group
--max-nodes INTEGER Maximum number of autoscaled nodes in the node group
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell cluster rotate-storage-key
Usage: spell cluster rotate-storage-key [OPTIONS]
This command rotates the cluster storage key for Azure
Options:
--cluster-name TEXT Name of cluster to add registry permissions to
--help Show this message and exit.
spell cluster set-instance-permissions
Usage: spell cluster set-instance-permissions [OPTIONS]
This command sets the Instance Profile / Service Account Spell will give to cloud
instances on your cluster. This can be useful for allowing your Spell runs to access
cloud resources that are normally private like RDS or DynamoDB. If there is already
a custom instance permission set on this cluster it will be replaced with the new
one.
For AWS this requires an IAM Role and an IAM Instance Profile that match (details
here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-
ec2_instance-profiles.html) and an IAM Role which has "ec2.amazonaws.com" as a
trusted entity (details here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html)
For GCP this can be any IAM Service Account within your GCP project. Note that this
command will grant the iam.serviceAccountUser role on the input service account
specifically to your Spell cluster's service account. More details available here:
https://cloud.google.com/iam/docs/service-accounts#user-role
Options:
-p, --profile TEXT This AWS profile will be used to get your Access Key
ID and Secret as well as your Region. You will be
prompted to confirm the Key and Region are correct
before continuing. This key will be used to adjust IAM
permissions of the role associated with the cluster
that the bucket is being added to.
--iam-role-arn TEXT AWS IAM Role ARN (Required for AWS Clusters, must
match Instance Profile)
--iam-instance-profile-arn TEXT
AWS IAM Instance Profile ARN (Required for AWS
Clusters, must match Role)
--iam-service-account TEXT GCP IAM Service Account (Required for GCP Clusters)
--help Show this message and exit.
spell cluster unset-instance-permissions
Usage: spell cluster unset-instance-permissions [OPTIONS]
This command unsets the Instance Profile / Service Account stored on your Spell
Cluster. Please see the `spell cluster set-instance-permissions` command for more
details.
Options:
-p, --profile TEXT This AWS profile will be used to get your Access Key ID and Secret
as well as your Region. You will be prompted to confirm the Key
and Region are correct before continuing. This key will be used to
adjust IAM permissions of the role associated with the cluster
that the bucket is being added to.
--help Show this message and exit.
spell cluster update
Usage: spell cluster update [OPTIONS]
This command makes sure your Spell cluster is fully up to date and able to support
the latest features
Options:
-p, --profile TEXT AWS profile to pull credentials from
--help Show this message and exit.
spell cluster update-kube-cluster
Usage: spell cluster update-kube-cluster [OPTIONS]
Update an existing GKE/EKS kubernetes cluster in your Spell VPC.
Options:
-p, --profile TEXT AWS profile to pull credentials from
--aws-zones TEXT Allows AWS clusters to explicitly list the availability
zones used for the EKS cluster. List the desired AZs as
comma separated values, ex: 'us-east-1a,us-east-1c,us-
east-1d'. NOTE: Most users will NOT have to do this. This
is useful if there are problems with one or more of the AZs
in the region of your cluster.
-f, --force Force a first-time update
--update-grafana-password Just update Grafana password.
--from-file FILENAME A path to a YAML or JSON file to load the arguments and
options from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell cp
Usage: spell cp [OPTIONS] SOURCE_PATH [LOCAL_DIR]
Copy a file or directory from a finished run, uploaded resource, or public dataset
specified by SOURCE_PATH to a LOCAL_DIR.
The contents of SOURCE_PATH will be downloaded from Spell and written to LOCAL_DIR.
If LOCAL_DIR is not provided the current working directory will be used as a
default. If SOURCE_PATH is a directory, the contents of the directory will be
written to LOCAL_DIR.
Options:
-f, --force Overwrite all duplicate files.
-i, --ignore Ignore all duplicate files.
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell curl
Usage: spell curl [OPTIONS] [ARGS]...
Interact directly with the Spell API via `curl`.
Spell automatically adds Authentication and Content-Type headers. URL path arguments
are prependend with the API protocol and host. Other arguments are passed to `curl`
as-is.
Example: spell curl /v1/users/me
Options:
--help Show this message and exit.
spell feedback
Usage: spell feedback [OPTIONS] [MESSAGE]...
Submit feedback to the Spell team to identify bugs and improve Spell's products.
We at Spell would love to hear from you! Both positive and negative feedback is
appreciated. If reporting a bug, please provide as much detail as possible such as
git repo, framework, runid, and any steps to reproduce. If reporting a feature
request, we'll let you know as soon as it's been added.
Options:
--help Show this message and exit.
spell help
Usage: spell help [OPTIONS]
Display help information
Options:
--help Show this message and exit.
spell hyper
Usage: spell hyper [OPTIONS] COMMAND [ARGS]...
Create hyperparameter searches on Spell
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
bayesian Execute a hyperparameter bayesian search
grid Execute a hyperparameter grid search
kill Kill a hyperparameter search
random Execute a hyperparameter random search
stop Stop a hyperparameter search
spell hyper bayesian
Usage: spell hyper bayesian [OPTIONS] COMMAND [ARGS]...
Execute a hyperparameter bayesian search for COMMAND remotely on Spell's
infrastructure
The bayesian command is used to create parallelized bayesian optimization
hyperparameter optimization with num_runs number of total runs, parallelized in sets
of parallel_runs
Options:
-n, --num-runs INTEGER Maximum number of runs for the hyperparameter search
[required]
-r, --parallel-runs INTEGER Number of parallel runs to use for each iteration
[required]
--metric TEXT Metric name that will be used [required]
-a, --metric-agg [avg|min|max|last]
[required]
-p, --param NAME=MIN:MAX[:int|float]
Specify a hyperparameter for the run in the form: a
range (MIN:MAX) If unspecified, type float is assumed.
NAME should appear in the COMMAND surrounded by colons
(i.e., ":NAME:" to indicate where the VALUEs should be
substituted when creating each run.
--label TEXT Label to add to the run. Can specify more than one.
--tensorboard-dir PATH The path where tensorboard files will be read from
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-d, --description TEXT Description of the hyperparameter search. If
unspecified defaults to the current commit message
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the hyperparameter
search
-t, --machine-type [CPU|K80|V100]
Machine type to run on
--framework TEXT Machine learning framework to use. For a full list of
available frameworks see
https://spell.ml/docs/run_overview#dependency-
management-and-customization
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell hyper grid
Usage: spell hyper grid [OPTIONS] COMMAND [ARGS]...
Execute a hyperparameter grid search for COMMAND remotely on Spell's infrastructure
The grid command is used to create numerous runs simultaneously to perform a
hyperparameter grid search. A run will be created for all possible combinations of
parameters provided with the --param option. All other options are the same as the
spell run command and will apply to every run created in the hyperparameter search.
Options:
--param NAME=VALUE[,VALUE,VALUE,...]
Specify a hyperparameter for the run. A run will be
created for all hyperparameter value combinations
provided. NAME should appear in the COMMAND surrounded
by colons (i.e., ":NAME:" to indicate where the VALUE
values should be substituted when creating each run.
--json-param NAME='[VALUE,VALUE,...]'
Specify a hyperparameter for the run. This should be
formatted as a JSON array. This can be used instead of
--param if you want a list of values which contain
commas. A run will be created for all hyperparameter
value combinations provided. NAME should appear in the
COMMAND surrounded by colons (i.e., ":NAME:" to
indicate where the VALUE values should be substituted
when creating each run.
--label TEXT Label to add to the run. Can specify more than one.
--tensorboard-dir PATH The path where tensorboard files will be read from
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-d, --description TEXT Description of the hyperparameter search. If
unspecified defaults to the current commit message
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the hyperparameter
search
-t, --machine-type [CPU|K80|V100]
Machine type to run on
--framework TEXT Machine learning framework to use. For a full list of
available frameworks see
https://spell.ml/docs/run_overview#dependency-
management-and-customization
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--timeout INTEGER If the run is still running after this many minutes we
will stop the run and save any outputs
--stop-condition METRIC_NAME OPERATOR VALUE[:MIN_INDEX]
METRIC_NAME is the name of a metric the run produces.
OPERATOR is <, >, <=, or >=. VALUE is a float. During
the run if there is a metric value that meets the
condition the run will be stopped. You can optionally
provide a MIN_INDEX integer. In that case, only values
of the metric starting with that index will be
considered.
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell hyper kill
Usage: spell hyper kill [OPTIONS] HYPER-SEARCH-ID
Kill a hyperparameter search specified by HYPER-SEARCH-ID.
All runs in the hyperparameter search are killed. Any runs that are already in a
final state are unaffected.
Options:
-q, --quiet Suppress logging
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell hyper random
Usage: spell hyper random [OPTIONS] COMMAND [ARGS]...
Execute a hyperparameter random search for COMMAND remotely on Spell's
infrastructure
The random command is used to create numerous runs simultaneously to perform a
hyperparameter search. As many runs as specified with --num-runs will be created and
each hyperparameter specified with the --param option will be sampled to determine a
specific value for each run. All other options are the same as the spell run
command and will apply to every run created in the hyperparameter search.
Options:
-n, --num-runs INTEGER Total number of runs to create for the hyperparameter
search [required]
-p, --param NAME=VALUE[,VALUE,VALUE,...] | NAME=MIN:MAX[:linear|log|reverse_log[:int|float]]
Specify a hyperparameter for the run. Each run will
sample this random parameter specification to
determine a specific value for the run. The parameter
values can be provided as either a list of values
(from which one value will be randomly selected each
run) or a range (MIN:MAX) and optional scaling
('linear', 'log', or 'reverse_log') and type ('int' or
'float'). If unspecified, a linear scaling and type
float are assumed. NAME should appear in the COMMAND
surrounded by colons (i.e., ":NAME:" to indicate where
the VALUEs should be substituted when creating each
run.
--json-param NAME='[VALUE,VALUE,...]'
Specify a hyperparameter for the run. This should be
formatted as a JSON array. This can be used instead of
--param if you want a list of values which contain
commas. A run will be created for all hyperparameter
value combinations provided. NAME should appear in the
COMMAND surrounded by colons (i.e., ":NAME:" to
indicate where the VALUE values should be substituted
when creating each run.
--label TEXT Label to add to the run. Can specify more than one.
--tensorboard-dir PATH The path where tensorboard files will be read from
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-d, --description TEXT Description of the hyperparameter search. If
unspecified defaults to the current commit message
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the hyperparameter
search
-t, --machine-type [CPU|K80|V100]
Machine type to run on
--framework TEXT Machine learning framework to use. For a full list of
available frameworks see
https://spell.ml/docs/run_overview#dependency-
management-and-customization
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--timeout INTEGER If the run is still running after this many minutes we
will stop the run and save any outputs
--stop-condition METRIC_NAME OPERATOR VALUE[:MIN_INDEX]
METRIC_NAME is the name of a metric the run produces.
OPERATOR is <, >, <=, or >=. VALUE is a float. During
the run if there is a metric value that meets the
condition the run will be stopped. You can optionally
provide a MIN_INDEX integer. In that case, only values
of the metric starting with that index will be
considered.
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell hyper stop
Usage: spell hyper stop [OPTIONS] HYPER-SEARCH-ID
Stop a hyperparameter search specified by HYPER-SEARCH-ID.
All runs in the hyperparameter search that are running are sent a stop signal that
ends current execution and transitions them to the "Saving" state. Stopped runs will
continue to transition through the "Pushing" and "Saving" steps after stopping. If
runs have not started yet, they are killed. Any runs that are already in a final
state are unaffected.
Options:
-q, --quiet Suppress logging
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell info
Usage: spell info [OPTIONS] RUN_ID
Displays information about a run including the start and end time as well run
parameters such as apts, pips, and mounts.
Options:
--help Show this message and exit.
spell jupyter
Usage: spell jupyter [OPTIONS] NAME
Create a Jupyter workspace on Spell.
The jupyter command is used to create a Jupyter workspace. If executed from within a
git respository, the HEAD commit (or optionally a different commit specified with
--commit-ref) will be copied into the Jupyter workspace. Alternatively, the
--github-url option can be used to specify a GitHub repository to copy into the
workspace. The other options can be used to further configure the Jupyter workspace
environment. Once created, the local web browser will open to the Jupyter workspace
in the Spell web console.
Options:
--lab Use Jupyter Lab (default: Jupyter Notebook)
--idle-kernel-timeout INTEGER Timeout in seconds after which to stop idle kernels
--no-activity-timeout INTEGER Timeout in seconds after which to stop the Jupyter
workspace if there are no running kernels and no user
activity
-t, --machine-type [CPU|K80|V100]
Machine type to run on
--framework TEXT Machine learning framework to use. For a full list of
available frameworks see
https://spell.ml/docs/run_overview#dependency-
management-and-customization
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the workspace
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
-d, --description TEXT Description of the workspace. If unspecified defaults
to the current commit message
-f, --force Skip interactive prompts
-v, --verbose Print additional information
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell keys
Usage: spell keys [OPTIONS] COMMAND [ARGS]...
Manage public SSH keys registered with Spell
Options:
--help Show this message and exit.
Commands:
add Add a public SSH key to your account
generate Generate SSH key pair for your account
list Display all your public SSH keys
rm Remove a public SSH key from your account
spell keys add
Usage: spell keys add [OPTIONS]
Add a public SSH key to your account
Options:
-f, --file FILE location of public key file
-t, --title TEXT title for the public key
--help Show this message and exit.
spell keys generate
Usage: spell keys generate [OPTIONS]
Generate SSH key pair for your account
Options:
-t, --title TEXT title for the public key
-f, --force overwrite local key file if it already exists
--help Show this message and exit.
spell keys list
Usage: spell keys list [OPTIONS]
Display all your public SSH keys
Options:
--help Show this message and exit.
spell keys rm
Usage: spell keys rm [OPTIONS] [TITLE]
Remove a public SSH key from your account
Options:
--help Show this message and exit.
spell kill
Usage: spell kill [OPTIONS] RUN_IDS...
Kill a current run specified by RUN_ID. To batch kill, input multiple RUN_IDs
separated by spaces.
A killed run is sent a kill signal that ends current execution and immediately
transitions to the "Killed" state once the signal has been received. Killed runs do
not execute any steps after being killed, so a killed run will not, for example,
execute the "Pushing" or "Saving" steps if killed when in the "Running" status.
Options:
-q, --quiet Suppress logging
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell kube-cluster
Usage: spell kube-cluster [OPTIONS] COMMAND [ARGS]...
Manage a model serving kubernetes cluster (EKS or GKE)
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
add-user Grant another AWS User in your account permissions to manage your kube...
create Sets up a GKE/EKS kubernetes cluster in your Spell VPC. Required for model
serving.
kubectl Issue kubectl commands against the serving cluster
node-group Manage kube cluster node groups
update Update an existing GKE/EKS kubernetes cluster in your Spell VPC.
spell kube-cluster add-user
Usage: spell kube-cluster add-user [OPTIONS] USER
Grant another AWS User in your account permissions to manage your kube cluster.
These permissions are required for anyone to update the cluster or create/remove
node groups. The user must be in the same account you use to manage this cluster,
which we will deduce from the AWS profile you provide (you will be prompted to
confirm).
Options:
-p, --profile TEXT AWS profile to pull credentials from
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell kube-cluster create
Usage: spell kube-cluster create [OPTIONS]
Sets up a GKE or EKS kubernetes cluster in your Spell VPC. This cluster is required
for model serving. Spell will automatically create a CPU node group for you which
will have at least one machine running at all times.
Options:
-p, --profile TEXT AWS profile to pull credentials from
--nodes-min INTEGER Minimum number of nodes in the model serving cluster
(default 1)
--nodes-max INTEGER Minimum number of nodes in the model serving cluster
(default 2)
--node-disk-size INTEGER Size of disks on each node in GB (default 50GB)
--use-existing This is an advanced option to use an existing EKS/GKE
cluster instead of creating a new one. It will reapply
kubernetes configurations. Because Spell sets up your
cluster in a particular manner this option is only likely to
work with clusters created exactly the way we set ours up.
This flag is likely only valuable if you experienced an
error the first time you tried to run this command, but the
kube cluster creation succeeded.
--aws-zones TEXT Allows AWS clusters to explicitly list the availability
zones used for the EKS cluster. List the desired AZs as
comma separated values, ex: 'us-east-1a,us-east-1c,us-
east-1d'. NOTE: Most users will NOT have to do this. This is
useful if there are problems with one or more of the AZs in
the region of your cluster.
--from-file FILENAME A path to a YAML or JSON file to load the arguments and
options from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell kube-cluster kubectl
Usage: spell kube-cluster kubectl [OPTIONS] [ARGS]...
Issue kubectl commands against the serving cluster
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell kube-cluster node-group
Usage: spell kube-cluster node-group [OPTIONS] COMMAND [ARGS]...
Manage node groups used for model serving cluster nodes
With no subcommand, display all your node groups
Options:
--help Show this message and exit.
Commands:
add Add a new node group to a Spell model serving cluster
delete Delete a node group
list Display all your node groups
scale Adjust minimum and maximum node counts for a given node group
spell kube-cluster node-group add
Usage: spell kube-cluster node-group add [OPTIONS]
Deploy a GKE node pool or eksctl node group for model serving
Options:
-p, --profile TEXT Specify an AWS profile to pull credentials from to perform
the NodeGroup create operation
--name TEXT Name of the node group [required]
--instance-type TEXT Instance type to use for the nodes
--accelerator NAME[:COUNT] (GCP only) Accelerator to attach to nodes, can be
specified multiple times for multiple accelerator types
--min-nodes INTEGER Minimum number of autoscaled nodes in the node group
--max-nodes INTEGER Maximum number of autoscaled nodes in the node group
--spot Use spot instances for node group nodes
--disk-size INTEGER Size of disks on each node in GB
--config-file FILE Path to file containing eksctl NodeGroup or GKE node pool
spec. Note this is an advanced option for users who want
to specify a custom node group or node pool configuration.
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell kube-cluster node-group delete
Usage: spell kube-cluster node-group delete [OPTIONS] NODE_GROUP_NAME
Delete a node group
Options:
-p, --profile TEXT AWS profile to pull credentials from
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell kube-cluster node-group list
Usage: spell kube-cluster node-group list [OPTIONS]
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell kube-cluster node-group scale
Usage: spell kube-cluster node-group scale [OPTIONS] NODE_GROUP_NAME
Adjust autoscaling min/max nodes for a node group
Options:
-p, --profile TEXT Specify an AWS profile to pull credentials from to perform the
NodeGroup scale operation
--min-nodes INTEGER Minimum number of autoscaled nodes in the node group
--max-nodes INTEGER Maximum number of autoscaled nodes in the node group
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell kube-cluster update
Usage: spell kube-cluster update [OPTIONS]
Update an existing GKE/EKS kubernetes cluster in your Spell VPC.
Options:
-p, --profile TEXT AWS profile to pull credentials from
--aws-zones TEXT Allows AWS clusters to explicitly list the availability
zones used for the EKS cluster. List the desired AZs as
comma separated values, ex: 'us-east-1a,us-east-1c,us-
east-1d'. NOTE: Most users will NOT have to do this. This
is useful if there are problems with one or more of the AZs
in the region of your cluster.
-f, --force Force a first-time update
--update-grafana-password Just update Grafana password.
--from-file FILENAME A path to a YAML or JSON file to load the arguments and
options from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell label
Usage: spell label [OPTIONS] RUN_ID LABEL_NAME
Used to add and remove labels from runs
Options:
--remove Remove input label from run
--help Show this message and exit.
spell link
Usage: spell link [OPTIONS] [ALIAS] [RESOURCE_PATH]
With no argument, displays all existing symlinks.
With a single argument, searches through server database for a symlink with a
specific alias ALIAS.
With two arguments, creates a symlink with alias ALIAS to a finished run uploaded
resource, or public dataset with path RESOURCE_PATH. Note: ALIAS must NOT contain
any of the following: / . " \ [ ] : ; | = ,
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell login
Usage: spell login [OPTIONS]
Login to your Spell account.
By default, authentication is handled by the Spell web console in your web browser.
Alternatively, identity and password can be specified using command line options.
If you don't have an account with Spell, please create one at https://spell.ml.
Options:
--identity TEXT Spell username or email address
--password TEXT Spell password
--help Show this message and exit.
spell logout
Usage: spell logout [OPTIONS]
Log out of current session
Options:
--help Show this message and exit.
spell logs
Usage: spell logs [OPTIONS] RUN-ID
Retrieve logs for a run specified by RUN_ID.
Streams logs for the specified run. For runs with a large number of log lines the
`--tail N` option allows the user to print only the last N lines. When following
with `--follow` use `Ctrl + C` to detach.
Options:
-d, --delay Replay delay between log entries
-f, --follow Follow log output
-n, --tail INTEGER Show the last NUM lines
-v, --verbose Print additional information
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell ls
Usage: spell ls [OPTIONS] [PATH]
List resource files for datasets, run outputs, and uploads.
Resources are the generic name for datasets, models, or any other files that can be
made available to a run. Spell keeps these organized for you in a remote filesystem
that is browsable with the `ls` command, with the resources placed into directories
that reflect their origin.
There are many ways resources are generated. The user can upload resources directly
with `spell upload` or execute a run with `spell run` that writes files to disk.
Spell also provides a number of publicly-accessible datasets.
Options:
-h Display file sizes in human-readable format
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell model
Usage: spell model [OPTIONS] COMMAND [ARGS]...
Spell models are a way for you to track your progress as you work on training
models. As you run new training runs you can push those outputs to existing model
names, and we will track those over time for you or create new models. You can take
any model version and serve it.
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
create Create a new model
describe Describe a model
list List all existing models
rm Deletes a model or model version
update-description Update a model's description
spell model create
Usage: spell model create [OPTIONS] NAME[:VERSION] RESOURCE
Specify a name for the model, if you use an existing model name a new version will
be created for you, otherwise a new model will be created. If a version is specified
in NAME:VERSION form, a model will be created with the given version name. If a
version is omitted, it will be given an autoincremented integer version. Resource
should be a path to a top level resource such as 'runs/168'.
Options:
-f, --file PATH WITHIN RESOURCE[:MOUNT PATH]
If the run output contains unnecessary information
(like intermediate checkpoint files), you can
whitelist only the needed files with this flag. You
can optionally specify a path for each to be mounted
within a future model server as well. If you omit this
we will just use '<MODEL NAME>/<PATH WITHIN RESOURCE>'
-d, --description TEXT Description of the model
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell model describe
Usage: spell model describe [OPTIONS] MODEL[:VERSION]
Describe a model or a model version
Options:
--help Show this message and exit.
spell model list
Usage: spell model list [OPTIONS]
List all existing models
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell model rm
Usage: spell model rm [OPTIONS] NAME[:VERSION]
Marks a model version as deleted. If version is omitted the whole model is marked as
deleted.
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell model update-description
Usage: spell model update-description [OPTIONS] MODEL:VERSION DESCRIPTION
Update a model's description
Options:
--help Show this message and exit.
spell passwd
Usage: spell passwd [OPTIONS]
Set a new password for your Spell account.
Prompts for the user's current password before allowing the user to specify a new
password.
Options:
--help Show this message and exit.
spell ps
Usage: spell ps [OPTIONS]
Display all user runs and their details
Displays information about all of the user's active and historical runs, displayed
oldest to newest. It can be used to keep track of current active runs or find a
historical run.
Options:
-n, --number INTEGER The number of runs to display
--trunc / --no-trunc Truncate command output
-r, --repository TEXT Repository name to filter runs by
-l, --label TEXT Label name to filter runs by. Multiple usages will return runs
with ANY of the input labels
-v, --verbose Include extra columns such as git commit hash and description
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell repos
Usage: spell repos [OPTIONS]
List all owner repositories
A repository is defined by the root commit of a git repository. Thus, the family of
all Git commits that originate from the same root commit belong to the same
repository. Repositories do not need to be managed or created -- they are created by
the run command when necessary.
Options:
--help Show this message and exit.
spell rm
Usage: spell rm [OPTIONS] RESOURCES...
Remove one or more uploaded resources.
The removed resources will no longer show up in `ls` or be mountable with `--mount`.
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell run
Usage: spell run [OPTIONS] COMMAND [ARGS]...
Execute COMMAND remotely on Spell's infrastructure
The run command is used to create runs and is likely the command you'll use most
while using Spell. It is intended to be emulate local development. Any code,
software, binaries, etc., that you can run locally on your computer can be run on
Spell - you simply put `spell run` in front of the same commands you would use
locally and they will run remotely. The various options can be used to customize the
environment in which COMMAND will run.
Options:
-t, --machine-type [CPU|K80|V100]
Machine type to run on
--framework TEXT Machine learning framework to use. For a full list of
available frameworks see
https://spell.ml/docs/run_overview#dependency-
management-and-customization
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the run
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
--tensorboard-dir PATH The path where tensorboard files will be read from
-d, --description TEXT Description of the run. If unspecified defaults to the
current commit message
--label TEXT Label to add to the run. Can specify more than one.
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-b, --background Do not print logs
--distributed N Execute a distributed run using N machines of the
specified machine type.
--auto-resume / --disable-auto-resume
Enable or disable auto-resume. NOTE: This is only
supported for spot instance machine types
--timeout INTEGER If the run is still running after this many minutes we
will stop the run and save any outputs
--stop-condition METRIC_NAME OPERATOR VALUE[:MIN_INDEX]
METRIC_NAME is the name of a metric the run produces.
OPERATOR is <, >, <=, or >=. VALUE is a float. During
the run if there is a metric value that meets the
condition the run will be stopped. You can optionally
provide a MIN_INDEX integer. In that case, only values
of the metric starting with that index will be
considered.
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell server
Usage: spell server [OPTIONS] COMMAND [ARGS]...
Manage model servers
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
describe Describe a model server
healthcheck cURL the health URL of a model server
list List all your model servers
logs Get logs from a model server
predict cURL the predict URL of a model server
rm Remove a model server
serve Create a new model server using a model
start Start a model server
status Get the status of all pods for this server.
stop Stop a model server
update Update a custom model server
spell server describe
Usage: spell server describe [OPTIONS] NAME
Describe a model server with the specified NAME
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server healthcheck
Usage: spell server healthcheck [OPTIONS] NAME CURL-ARGS
Issue a cURL command against the health endpoint of a model server
This is a GET request. Further cURL arguments can be provided using "--". Example:
spell server healthcheck myserver -- -H "Content-Type: appliation/json" -d '{data:
[1,2,3]}'
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server list
Usage: spell server list [OPTIONS]
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server logs
Usage: spell server logs [OPTIONS] NAME
Get logs for the model server with the specified NAME
Options:
-f, --follow Follow log output
-p, --pod TEXT The ID of the pod you would like logs for. Omit to get a list of
all pods.
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell server predict
Usage: spell server predict [OPTIONS] NAME CURL-ARGS
Issue a cURL command against the predict endpoint of a model server
This is a POST request. Further cURL arguments can be provided using "--". Example:
spell server predict myserver -- -H "Content-Type: appliation/json" -d '{data:
[1,2,3]}'
Options:
--json TEXT Pass a json object to the predict URL. If a path to a file is
provided, it will be read and passed
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server rm
Usage: spell server rm [OPTIONS] NAME
Remove the model server with the specified NAME
Options:
-f, --force Remove the server even if it is running
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell server serve
Usage: spell server serve [OPTIONS] MODEL:VERSION ENTRYPOINT
Create a new model server based on an existing model and entrypoint to a Python
predictor able to serve said model.
Options:
--name TEXT Name of the model server. Defaults to the model name.
--config FILENAME Path to a YAML for JSON file which wil be passed
through to the Predictor
--node-group TEXT Node group to schedule the server to. Defaults to
initial node group.
--classname TEXT Name of the Predictor class to use. Only required if
more than one predictor exist in the Python module
used
--conda-file FILE Path to conda specification file or YAML environment
file
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the model server
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
-d, --description TEXT Description of the model server. If unspecified
defaults to the current commit message
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--min-pods INTEGER The autoscaler will never scale to fewer pods than
this.
--max-pods INTEGER The autoscaler will never scale to more pods than
this.
--target-requests-per-second FLOAT
The autoscaler will scale up pods if the average
number of HTTP(S) requests per second to a pod exceeds
this value.
--target-cpu-utilization FLOAT If average pod CPU usage goes higher than this times
the cpu-request the autoscaler will spin up a new pod.
--cpu-request FLOAT The amount of vCPU cores you expect each pod to use
--cpu-limit FLOAT The maximum amount of vCPU cores a pod can use
--ram-request INTEGER The amount of RAM you expect each pod to use in MB
--ram-limit INTEGER The maximum amount of RAM a pod can use in MB. It will
be terminated if it exceeds this.
--gpu-limit INTEGER Maximum number of GPUs allowable to each pod. This
defaults to 1 if the node group has GPUs
--num-processes INTEGER The number of processes to run the model server on. By
default this is (2 * numberOfCores) + 1 or equal to
the available GPUs if applicable
--enable-batching Enable server-side batching. Without specifying
further options this enables batching with the default
options
--max-batch-size INTEGER The maximum batch size. Default is 100. By using this
flag, you will enable batching.
--request-timeout INTEGER The maximum amount of time in milliseconds to wait
before processing a request. Default is 500ms. By
using this flag, you will enable batching.
--validate Validate the structure of your predictor class. All
Python packages required to import your predictor must
be in your Python environment
--debug Launch the server in debug mode. For security
purposes, this should not be used in production
--open / --no-open Open the server's webpage in a browser after creation
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell server start
Usage: spell server start [OPTIONS] NAME
Start the model server with the specified NAME
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server status
Usage: spell server status [OPTIONS] NAME
Get the status of all pods for this server.
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell server stop
Usage: spell server stop [OPTIONS] NAME
Stop the model server with the specified NAME
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell server update
Usage: spell server update [OPTIONS] SERVER-NAME
Update a custom model server
Options:
--model NAME:VERSION New model to use
--entrypoint TEXT Choose a new entrypoint for the server
--config FILENAME Path to a YAML for JSON file which wil be passed
through to the Predictor
--node-group TEXT Node group to schedule the server to. Defaults to
initial node group.
--classname TEXT Name of the Predictor class to use. Only required if
more than one predictor exist in the Python module
used
--conda-file FILE Path to conda specification file or YAML environment
file
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the model server
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
--update-spell-version Update the version of Spell running the model server
-d, --description TEXT Description of the model server. If unspecified
defaults to the current commit message
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-m, --mount RESOURCE[:MOUNT_PATH]
Attach a resource file or directory (e.g., from a run
output, public dataset, or upload). The resource
(specified by a Spell resource path) is required. An
optional mount path within the container can also be
specified, separated by a colon from the resource
name. If the mount path is omitted, it defaults to the
base name of the resource (e.g., 'mnist' for
'public/image/mnist'). Example: --mount
runs/42:/mnt/data
--min-pods INTEGER The autoscaler will never scale to fewer pods than
this.
--max-pods INTEGER The autoscaler will never scale to more pods than
this.
--target-requests-per-second FLOAT
The autoscaler will scale up pods if the average
number of HTTP(S) requests per second to a pod exceeds
this value.
--target-cpu-utilization FLOAT If average pod CPU usage goes higher than this times
the cpu-request the autoscaler will spin up a new pod.
--cpu-request FLOAT The amount of vCPU cores you expect each pod to use
--cpu-limit FLOAT The maximum amount of vCPU cores a pod can use
--ram-request INTEGER The amount of RAM you expect each pod to use in MB
--ram-limit INTEGER The maximum amount of RAM a pod can use in MB. It will
be terminated if it exceeds this.
--gpu-limit INTEGER Maximum number of GPUs allowable to each pod. This
defaults to 1 if the node group has GPUs
--num-processes TEXT The number of processes to run the model server on. By
default this is (2 * numberOfCores) + 1 or equal to
the number of available GPUs if applicable. To use the
default, enter "default"
--enable-batching / --disable-batching
Enable or disable server-side batching
--max-batch-size INTEGER The maximum batch size. Default is 100. By using this
flag, you will enable batching.
--request-timeout INTEGER The maximum amount of time in milliseconds to wait
before processing a request. Default is 500ms. By
using this flag, you will enable batching.
--validate Validate the structure of your predictor class. All
Python packages required to import your predictor must
be in your Python environment
--debug-on / --debug-off Launch the server in debug mode. For security
purposes, this should not be used in production
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell stats
Usage: spell stats [OPTIONS] RUN_ID
Display performance statistics for a run specified by RUN_ID
A run must have status 'Running' to display performance statistics. CPU statistics
(e.g., memory usage, CPU utilization) and, if applicable for the machine type, GPU
statistics (e.g., power and GPU memory utilization) are displayed.
Options:
-f, --follow Continually stream stats for the run
--help Show this message and exit.
spell status
Usage: spell status [OPTIONS]
Display account and billing information
Display account and billing information, such as current plan, total run time, and
free credit usage.
Options:
--help Show this message and exit.
spell stop
Usage: spell stop [OPTIONS] RUN_IDS...
Stop runs currently in the "Running" state specified by RUN_IDS.
A stopped run is sent a stop signal that ends current execution and transitions to
the "Saving" state once the signal has been received. Stopped runs will execute any
and all steps after being stopped, so a stopped run will, for example, execute both
the "Pushing" and "Saving" steps after stopping. If "stop" is called on a run that
has not yet entered the "Running" state, the run is killed (see documentation of
"kill" for more details).
Options:
-q, --quiet Suppress logging
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell unlink
Usage: spell unlink [OPTIONS] ALIAS
Remove an existing symlink by referencing a particular link alias.
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell upload
Usage: spell upload [OPTIONS] PATH
Upload the contents of PATH to uploads/NAME
The contents of PATH will be uploaded to Spell and will be accessible at
uploads/NAME. If NAME is not provided and path is a directory, NAME defaults to the
the directory name of PATH. If NAME is not provided and PATH is a file, the user is
prompted for a NAME.
Options:
-n, --name NAME upload name
--cluster-name TEXT cluster name
-c, --compress compress before uploading
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell whoami
Usage: spell whoami [OPTIONS]
Display current user information.
Display information about the currently logged in user, such as username, email, and
various other metadata.
Options:
--help Show this message and exit.
spell workflow
Usage: spell workflow [OPTIONS] COMMAND [ARGS]...
Execute workflows on Spell
Options:
--docs Open documentation in the browser and exit
--help Show this message and exit.
Commands:
create Create a new workflow
kill Kill Workflow
rm Archive a workflow
stop Stop Workflow
spell workflow create
Usage: spell workflow create [OPTIONS] COMMAND [ARGS]...
Creates a new workflow
Options:
--local Execute command locally instead of remotely on Spell's
infrastructure
-r, --repo LABEL=PATH[:COMMIT_HASH]
Add a git repository at a specific commit to this
workflow by specifying LABEL=PATH, where PATH is a
path to a local git repository on disk and LABEL is a
label to refer to this snapshot in future run
requests. COMMIT_HASH can optionally be specified by
LABEL=PATH[:COMMIT_HASH]. If no COMMIT_HASH is
specified, the currently checked out commit of the
repo will be used.
--github-repo LABEL=URL[:REF] Add a GitHub repository at a specific ref to this
workflow by specifying LABEL=URL, where URL is a url
to a GitHub repository and LABEL is a label to refer
to this snapshot in future run requests. REF can
optionally be specified by LABEL=PATH[:REF], it can be
a branch, commit hash, or label, and will be resolved
to a commit hash immediately. If no REF is specified,
master will be used.
--conda-file FILE Path to conda specification file or YAML environment
file
--docker-image, --docker_image, --from TEXT
Custom docker image to run from. Specify image as
<domain>/<repository>/<image_name>:<tag>. Default to
docker.io if <domain>/ is omitted. To use images from
Google Container Registry, first run 'spell cluster
add-docker-registry' as a gcp cluster owner.
--pip TEXT Single dependency to install using pip
--pip-req TEXT Requirements file to install using pip
--apt TEXT Dependency to install using apt
-e, --env TEXT Add an environment variable to the workflow
-c, --commit-ref TEXT Git commit hash to use
--github-url TEXT GitHub URL of a repository to use
--github-ref TEXT commit hash, branch, or tag of the repository to pull
(default: 'master')
-d, --description TEXT Description of the workflow. If unspecified defaults
to the current commit message
-f, --force Skip interactive prompts
-v, --verbose Print additional information
-b, --background Do not print logs
--from-file FILENAME A path to a YAML or JSON file to load the arguments
and options from
--save-command FILE Save the arguments and options to this command to a
file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell workflow kill
Usage: spell workflow kill [OPTIONS] WORKFLOW-ID
Kill a workflow run specified by WORKFLOW-ID.
This will kill all the runs created by workflow including the managing and child.
Cannot kill --local workflow runs
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.
spell workflow rm
Usage: spell workflow rm [OPTIONS] WORKFLOW-ID
Archive a workflow run specified by WORKFLOW-ID.
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--help Show this message and exit.
spell workflow stop
Usage: spell workflow stop [OPTIONS] WORKFLOW-ID
Stop a workflow run specified by WORKFLOW-ID.
This will stop all in progress runs and cancel all queued runs for workflow.
Cannot stop --local workflow runs
Options:
--from-file FILENAME A path to a YAML or JSON file to load the arguments and options
from
--save-command FILE Save the arguments and options to this command to a file
--docs Open documentation in the browser and exit
--help Show this message and exit.