Skip to main content

Managing Secrets

When tasks or container builds need access to private package repositories, API credentials, or database keys, Flyte stores and delivers these credentials securely. In flyte-sdk, secret management is split into two layers: the control plane interface (flyte.remote.Secret) for creating, retrieving, listing, and deleting secrets, and the task/image execution interface (flyte.Secret) for injecting secrets into runtime environments.

Managing Secrets with Python SDK​

To manage secrets stored on the Flyte control plane, use flyte.remote.Secret. All methods are decorated with @syncify, enabling both synchronous and asynchronous execution.

import flyte
from flyte.remote import Secret

# Initialize the Flyte client
flyte.init(
    endpoint="dns:///localhost:8080",
    insecure=True,
    project="my-project",
    domain="development",
)

# 1. Create a regular string secret
Secret.create(name="openai-api-key", value="sk-live-secret-token", type="regular")

# 2. Create an image pull secret with binary credentials
with open("docker-config.json", "rb") as f:
    Secret.create(name="regcred", value=f.read(), type="image_pull")

# 3. Retrieve secret metadata and cluster distribution status
sec = Secret.get(name="openai-api-key")
print(f"Secret: {sec.name}, Type: {sec.type}")
print(f"JSON metadata: {sec.to_json()}")

# 4. List all secrets across the configured project and domain
for secret_item in Secret.listall(limit=50):
    print(secret_item.name, secret_item.type)

# 5. Delete a secret
Secret.delete(name="openai-api-key")

Asynchronous Remote Operations​

In async functions or event loops, invoke the .aio attribute generated by @syncify:

import asyncio
from flyte.remote import Secret

async def main():
    # Asynchronously create
    await Secret.create.aio(name="github-pat", value="ghp_exampleToken123")

    # Asynchronously retrieve
    sec = await Secret.get.aio(name="github-pat")

    # Asynchronously iterate through paginated secrets
    async for s in Secret.listall.aio(limit=100):
        print(s.name, s.type)

    # Asynchronously delete
    await Secret.delete.aio(name="github-pat")

asyncio.run(main())

Control Plane Protocol​

Under the hood, flyte.remote.Secret delegates to flyte.remote._client._protocols.SecretService, which handles gRPC calls against the Flyte control plane:

  • CreateSecret(CreateSecretRequest): Uploads a SecretSpec (generic string/binary or image pull) scoped by SecretIdentifier (organization, project, domain, and secret name).
  • GetSecret(GetSecretRequest): Retrieves definition_pb2.Secret containing secret_metadata (timestamps, overall status, and cluster presence status).
  • ListSecrets(ListSecretsRequest): Returns a token-paginated list of secrets.
  • DeleteSecret(DeleteSecretRequest): Removes the secret identifier from the control plane.

Managing Secrets with Flyte CLI​

You can also manage secrets directly from the command line using the flyte CLI commands defined in cli/_create.py, cli/_get.py, and cli/_delete.py.

Create Secrets​

Create a secret using an inline value or an interactive hidden prompt:

# Prompt for secret value securely (input hidden)
flyte create secret my-api-token

# Provide value directly
flyte create secret my-api-token --value "s3cr3t-v4lu3"

# Upload binary credentials from a file as an image pull secret
flyte create secret regcred --from-file ~/.docker/config.json --type image_pull

Inspect and List Secrets​

# Retrieve a specific secret (outputs metadata as JSON)
flyte get secret my-api-token

# List all secrets in the current project and domain
flyte get secret

Delete Secrets​

flyte delete secret my-api-token

Secret Types: Regular vs Image Pull​

The type argument in Secret.create accepts values of SecretTypes = Literal["regular", "image_pull"]:

  1. "regular" (maps to definition_pb2.SecretType.SECRET_TYPE_GENERIC): Intended for API keys, passwords, connection strings, and certificates injected into task execution containers or image build steps.
  2. "image_pull" (maps to definition_pb2.SecretType.SECRET_TYPE_IMAGE_PULL_SECRET): Intended for container registry authentication credentials used by cluster container runtimes when pulling private images.
from flyte.remote import Secret

# Generic credential for runtime or build time
Secret.create(name="aws-access-key", value="AKIAIOSFODNN7EXAMPLE", type="regular")

# Kubernetes image pull secret format (JSON) for private registry access
Secret.create(name="registry-auth", value=b'{"auths": {"...": {...}}}', type="image_pull")

Consuming Secrets in Tasks and Builds​

To declare that a task or container image requires a secret, use flyte.Secret from flyte._secret (or plain string secret names).

Task Environment Variables​

Pass secrets to @task(secrets=...). If not explicitly overridden via as_env_var, flyte.Secret automatically transforms the key into an uppercase environment variable where hyphens become underscores.

import os
import flyte
from flyte import Secret, task

# Implicit string key: converts "my-secret" -> "MY_SECRET"
@task(secrets="my-secret")
def task_with_simple_secret() -> str:
    return os.environ["MY_SECRET"]

# Explicit environment variable mapping
@task(secrets=Secret(key="my-openai-api-key", as_env_var="OPENAI_API_KEY"))
def task_with_custom_env() -> str:
    return os.environ["OPENAI_API_KEY"]

# Multiple secrets
@task(secrets=[
    Secret(key="db-password", as_env_var="DB_PASS"),
    Secret(key="api-token"),  # Injected as API_TOKEN
])
def multi_secret_task():
    pass

Build-Time Secret Mounts​

Secrets can also be mounted into flyte.Image package installation steps to pull private Git repositories or private package feeds without baking credentials into the image layer:

import flyte
from flyte import Secret, task

image = (
    flyte.Image.from_debian_base()
    .with_pip_packages(
        "git+https://$GITHUB_PAT@github.com/flyteorg/private-repo.git",
        secret_mounts=[Secret(key="GITHUB_PAT")]
    )
    .with_apt_packages(
        "custom-pkg",
        secret_mounts=[Secret(key="apt-secret", mount="/etc/apt/apt-secret")]
    )
)

@task(image=image)
def run_private_pkg_task() -> int:
    import private_repo
    return private_repo.compute()

Troubleshooting & Constraints​

Write-Only Secret Retrieval​

Secret.get(name) returns the Secret object containing metadata, creation timestamps, and per-cluster presence status:

sec = Secret.get("openai-api-key")
print(sec.name)  # "openai-api-key"
print(sec.type)  # "regular"

The Flyte control plane does not store or return plaintext secret values on GetSecret calls. Secret.get(name) cannot be used to read back previously saved secret values.

Environment Variable Format Validation​

When supplying as_env_var to flyte.Secret(key=..., as_env_var=...), the name is validated against the regular expression ^[A-Z_][A-Z0-9_]*$. Providing lowercase characters or special characters (such as hyphens) will raise a ValueError:

# Raises ValueError: Invalid environment variable name: invalid-env-var
Secret(key="api-key", as_env_var="invalid-env-var")

Client Initialization​

Calling Secret.create, Secret.get, Secret.listall, or Secret.delete prior to initializing client configuration with flyte.init(...) or CLI configuration raises InitializationError via ensure_client().

Reusable Tasks Constraint​

Tasks defined with reusable=True cannot define custom or overridden secrets directly on the @task decorator. Reusable tasks inherit secret configurations from their parent execution environment. To supply task-level secrets, ensure reusable is not set to True.