Configuring Azure Blob Storage (ABFS)

To use Azure Blob Storage (ABFS) in flyte-sdk, you configure credentials via environment variables. When you use a path with the abfs:// or abfss:// protocol in storage operations like get or put, flyte-sdk automatically detects the protocol and initializes the ABFS configuration.

Authenticating with Account Name and Key

The most direct way to authenticate is by providing your Azure Storage account name and its associated access key.

import os
import asyncio
from flyte import storage

# Set environment variables for authentication
os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = "myaccount"
os.environ["AZURE_STORAGE_ACCOUNT_KEY"] = "my-secret-key"

async def download_data():
    # flyte-sdk automatically uses ABFS.auto() to load credentials
    # when it sees the abfs:// protocol
    local_path = await storage.get("abfs://my-container/data.csv")
    print(f"Downloaded to: {local_path}")

asyncio.run(download_data())

Authenticating with a Service Principal

For production environments, you can authenticate using an Azure Service Principal by providing the tenant ID, client ID, and client secret.

import os
import asyncio
from flyte import storage

# Set environment variables for Service Principal authentication
os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = "myaccount"
os.environ["AZURE_TENANT_ID"] = "your-tenant-id"
os.environ["AZURE_CLIENT_ID"] = "your-client-id"
os.environ["AZURE_CLIENT_SECRET"] = "your-client-secret"

async def upload_data():
    # The ABFS configuration is resolved implicitly
    remote_path = await storage.put("local_file.txt", to_path="abfs://my-container/remote_file.txt")
    print(f"Uploaded to: {remote_path}")

asyncio.run(upload_data())

Configuration Reference

The ABFS class in storage/_config.py maps the following environment variables to its configuration fields:

Environment Variable	ABFS Field	Description
AZURE_STORAGE_ACCOUNT_NAME	account_name	The name of your Azure Storage account.
AZURE_STORAGE_ACCOUNT_KEY	account_key	The access key for the storage account.
AZURE_TENANT_ID	tenant_id	The Azure AD tenant ID.
AZURE_CLIENT_ID	client_id	The Service Principal client ID.
AZURE_CLIENT_SECRET	client_secret	The Service Principal client secret.

Programmatic Configuration

While flyte-sdk is designed to load configuration automatically via ABFS.auto(), you can manually instantiate the ABFS class if you need to pass it to specific internal components.

from flyte.storage import ABFS

# Manually create a configuration object
config = ABFS(
    account_name="myaccount",
    account_key="my-key"
)

# Generate kwargs for fsspec
fsspec_args = config.get_fsspec_kwargs()
# Resulting dict includes 'config' with credentials and 'client_options'

Troubleshooting and Gotchas

• Implicit Loading: Configuration is loaded at the moment a storage function (like storage.get or storage.put) is called with an abfs:// path. If environment variables are missing, the operation may fail or attempt an anonymous connection.
• Credential Priority: The ABFS class does not enforce a specific authentication method if both account keys and service principal details are provided. Ensure you only set the variables required for your chosen method to avoid ambiguous authentication states.
• Obstore Integration: flyte-sdk uses obstore for high-performance streaming. For abfs paths, put_stream and get_stream use an internal bypass (_put_stream_obstore_bypass) to handle async operations, which relies on specific fsspec filesystem structures.