Fine-grained access control in Spell using private pip packages

Every user in Spell is associated with an organization, and every user inside of an organization has the same permissions. This simple (flat) permissions model is convenient when everyone in the organization is working on the same project and needs the same level of access, but can be a limitation for larger, more heterogeneous teams.

In this article, we will cover Spell's access control model, then demonstrate how you can use Spell’s private pip packages feature to introduce per-user permissions into your Spell account via a custom authentication library.

Spell’s access control model

The topmost unit of authorization within Spell is the organization. Every Spell user is a member of at least one organization, but there is no limit on how many organizations a single user can be a part of (besides practicality). In the web console, you can toggle your currently active organization using the drop-down menu on the top right:

On the Spell CLI, you can toggle your currently active organization using the spell owner command (this setting is persisted between sessions):

$ spell owner
 aleksey
 spellrun
 external-gcp
 ➔ external-aws
 spell-org

Startups on the Spell for Teams plan will usually have a single organization for the whole team. Larger companies on the Enterprise plan will usually have multiple organizations — one for each team on Spell.

Every organization is associated with a cluster — a set of cloud resources Spell has deployed into a VPC in your account to be used for all the things you do on Spell. One important aspect of cluster management is setting a custom instance profile (CIP). This is an IAM role (actually technically an instance profile, on AWS; on GCP, this is a service account) that the machine backing every run, workspace, hyperparameter search, and workflow executed within the cluster will be provisioned with. For example, here is an example of an organization, external-aws, with a custom instance profile configured for its cluster:

Custom instance profiles can be used to configure team-wide AWS or GCP permissions. For example, if your team makes heavy use of ephemeral AWS EMR Spark clusters, you can use a CIP to give everyone on the team access to AWS EMR — just attach the necessary permissions to the CIP role. To learn more, refer to the AWS setup and/or GCP setup docs.

CIPs expose a flat permissions model — they can only be used to configure permissions for the entire team. Spell does not currently support fine-grained, per-user permissions directly. We have found that flat organization-wide authorization settings work well for most of the teams using Spell today.

For teams that want fine-grained permissions on Spell, we recommend implementing this yourself, using a common SecOps pattern to do so: an authentication package.

Authentication packages

All companies face unique technical challenges, and will set up their backend infrastructure differently to address them. Some companies use Redshift for analytics, others BigQuery, still others lean on Spark. Some use Postgres as their database of records, others MariaDB, and so on.

To minimize the amount of work end users (data scientists, production engineers, etcetera) have to do to authenticate to the very specific configuration of their particular organization, it is very common for DevOps engineers to create and manage one or more authentication libraries for internal use.

An authentication library (or helper library, or glue library, or whatever you want to call it — there is, that I know of, no set name for this concept) is a middleware package that encapsulates a set of helper functions for working with your company’s specific infrastructure.

To give a concrete example, suppose that you have set up an analytics SQL database, one that everyone on your data science team queries. The database has two users: a read_write power user, only used for administrative tasks, and a read_only user that lacks write access, used for day-to-day queries. You want to enforce that only certain members of the team have access to the administrative user.

One way to do that would be to provide the team with a middleware Python package containing a connect method they can use to access the database. At call time, your library will verify that the IAM user calling your connect method actually has access to the database user they are trying to connect as. If the library finds that the user doesn’t actually have access, you can print out a nice error message explaining why they can’t connect to the database (a much better developer experience than having AWS throw a cryptic IAM error).

In other cases authentication middleware is less about access control and more about convenience. For example, there may be a common “flow” many different people on your team use which you consolidate using a centralized setup_workspace method. Or alternatively, you may want to enforce that everyone’s queries "play nice" with one another, e.g. that no one user’s SQL query monopolizes the database resources for too long — so you add a query_with_timeout method to your package, and tell your users to use that method (instead of querying the database directly themselves).

These types of packages are particularly common in data science and machine learning teams because the folks that build the infrastructure (data engineers) and the folks that use it (data analysts, data scientists, machine learning engineers) are typically different people on different teams. Glue code acts as an interface that lets engineers and scientists cooperate on what the "right" way of doing things is, and formalize those best practices team-wide.

To learn more, I recommend checking the following tech talk: "Building and managing a centralized Kubeflow platform at Spotify", which talks about this and related data science team organization topics.

Implementing fine-grained permissions on Spell using an authentication package

You can implement fine-grained permissions on Spell by installing an authentication package into your runs or workspaces using Spell’s private pip packages feature.

To demonstrate how this works, I created a private GitHub repository, spellml/aws-credstash-demo.

Inside this repo is a small Python authentication package I wrote (quite literally named middleware). middleware uses credstash to retrieve the login credentials for a AWS RDS Postgres database.

credstash (GitHub repository) is a small CLI utility package for storing and managing access to secrets on AWS. It uses a master key from the AWS Key Management Service to encrypt and decrypt arbitrary strings encrypted at rest in DynamoDB. We use credstash for secrets management for our infrastructure at Spell, and find that it works quite well (to learn more about credstash, see the blog post "Using AWS KMS to manage secrets in your Infrastructure").

The primary file of the middleware package is cli.py. cli.py is a small click CLI wrapper that uses credstash to retrieve database login credentials, then uses psql to shell out to the database:

To add this private package to my organization, I go to the Settings > Integrations page in the web console, and complete the web-based login flow there to give Spell access to my package:

Now whenever I pass this repository URL to the --pip flag on e.g. spell run, my authentication package will be up and ready to go in the run environment!

spell run \
  --pip "git+https://github.com/spellml/aws-credstash-demo" \
  "middleware --help"
Everything up-to-date
💫 Casting spell #369…
✨ Stop viewing logs with ^C
✨ Machine_Requested… done
✨ Building… done
✨ Run is running
Usage: middleware [OPTIONS] COMMAND [ARGS]...
Options:
 --help  Show this message and exit.

The spellml/aws-credstash-demo repository is now public, so you can check out the code for yourself — or even use it as the starting point for your own auth library.

Ready to Get Started?

Create an account in minutes or connect with our team to learn how Spell can accelerate your business.