General Storage Settings

Configure provider-independent storage settings

When you need to pass storage configuration into Flyte, start with storage.Storage for the settings shared by all providers. It is a frozen dataclass, so configure values when you construct it rather than mutating an existing instance:

import datetime

from flyte.storage import Storage

storage_config = Storage(
    retries=5,
    backoff=datetime.timedelta(seconds=10),
    enable_debug=True,
    attach_execution_metadata=True,
)

Storage defines these defaults in storage/_config.py:

Setting	Default	Role in the Storage data model
retries	3	Common retry count. S3 uses it as max_retries when creating filesystem arguments.
backoff	datetime.timedelta(seconds=5)	Common initial retry delay. S3 uses it as init_backoff.
enable_debug	False	Stored on the configuration object; no other inspected source currently consumes it.
attach_execution_metadata	True	Stored on the configuration object; no other inspected source currently consumes it.

The @dataclass(..., frozen=True) declaration also gives Storage generated representation and equality behavior while preventing in-place changes after construction. attach_execution_metadata is part of the public configuration shape, but its default currently has no demonstrated runtime effect in the inspected flyte-sdk sources.

Construct explicitly or load common values from the environment

Use explicit constructor arguments when you need typed Python values such as a datetime.timedelta:

import datetime

from flyte.storage import Storage

storage_config = Storage(
    retries=3,
    backoff=datetime.timedelta(seconds=5),
    enable_debug=False,
)

Storage.auto() instead reads the three entries in Storage._KEY_ENV_VAR_MAPPING:

export UNION_STORAGE_RETRIES=5
export UNION_STORAGE_BACKOFF_SECONDS=10
export UNION_STORAGE_DEBUG=true

from flyte.storage import Storage

storage_config = Storage.auto()

Internally, Storage._auto_as_kwargs() calls os.getenv for UNION_STORAGE_RETRIES, UNION_STORAGE_BACKOFF_SECONDS, and UNION_STORAGE_DEBUG. For values that are present, it calls flyte.config.set_if_exists to add them to the constructor keyword arguments; an unset variable is omitted, allowing the dataclass default to apply.

The environment path does not perform explicit type conversion in Storage. Values returned by os.getenv are strings, and set_if_exists only conditionally adds them. In particular, the source does not convert UNION_STORAGE_BACKOFF_SECONDS into a datetime.timedelta or parse UNION_STORAGE_DEBUG into a Boolean. A value such as "false" therefore remains a non-empty string rather than becoming the Boolean False.

Provider auto-configuration has an important distinction. S3.auto() calls super()._auto_as_kwargs() and therefore includes the common UNION_STORAGE_* settings before adding its endpoint and credential values. GCS.auto() and ABFS.auto() read their provider-specific variables but do not call Storage._auto_as_kwargs(), so their .auto() methods do not incorporate the common environment settings in the inspected implementation.

Register storage with Flyte initialization

Pass a Storage instance—or normally a provider subclass such as S3, GCS, or ABFS—to the storage parameter of flyte.init:

import datetime

import flyte
from flyte.storage import S3

flyte.init(
    storage=S3(
        retries=5,
        backoff=datetime.timedelta(seconds=10),
        endpoint="http://localhost:4566",
        access_key_id="minio",
        secret_access_key="miniostorage",
    ),
)

The storage parameter in flyte._initialize.init is typed as Storage | None. During initialization, flyte-sdk places the supplied object unchanged into _InitConfig as storage=storage. get_storage() then returns that value:

from flyte._initialize import get_storage

storage_config = get_storage()

get_storage() requires initialization. If no initialization configuration exists, it raises InitializationError with the StorageNotInitializedError identifier and directs the caller to call flyte.init() with a valid endpoint or API key.

Understand the provider boundary

Storage is the common data model, not a filesystem constructor. Its get_fsspec_kwargs(anonymous=False, **kwargs) method is the provider hook documented as returning filesystem-constructor keyword arguments, but the base implementation always returns an empty dictionary:

from flyte.storage import Storage

storage_config = Storage()
filesystem_kwargs = storage_config.get_fsspec_kwargs()
assert filesystem_kwargs == {}

Use a matching provider subclass for remote filesystem behavior. The runtime function get_configured_fsspec_kwargs(protocol, anonymous=False) in storage/_storage.py selects the configured subclass for s3, gs, or abfs/abfss. If the active configuration is not an instance of the matching provider, it uses that provider's .auto() configuration instead. Unknown protocols return {}.

For example, the S3 override consumes the inherited common fields and builds the retry arguments:

import datetime

from flyte.storage import S3

storage_config = S3(
    retries=5,
    backoff=datetime.timedelta(seconds=10),
)
filesystem_kwargs = storage_config.get_fsspec_kwargs()

In S3.get_fsspec_kwargs, the values are removed from the optional keyword arguments with these defaults:

retries = kwargs.pop("retries", self.retries)
backoff = kwargs.pop("backoff", self.backoff)

They are then placed into the returned retry_config as max_retries and backoff.init_backoff. The S3 implementation also fixes backoff.base at 2, max_backoff at datetime.timedelta(seconds=16), and retry_timeout at datetime.timedelta(minutes=3). It returns client options containing timeout: "99999s" and allow_http: True. S3 credentials and endpoint are placed in a config mapping when supplied; anonymous=True adds skip_signature: True to that mapping.

GCS.get_fsspec_kwargs removes the optional anonymous keyword and otherwise returns the remaining keyword arguments unchanged. ABFS.get_fsspec_kwargs similarly handles its Azure configuration and returns client options, but the common retry/backoff translation shown above is specific to the S3 override.

Trace settings into filesystem construction

The storage flow is:

flyte.init(storage=...) -> _InitConfig.storage
                         -> get_storage()
                         -> get_configured_fsspec_kwargs(protocol)
                         -> provider.get_fsspec_kwargs(...)
                         -> fsspec.filesystem(protocol, **configured_kwargs)

get_underlying_filesystem obtains the protocol from the path when necessary, calls get_configured_fsspec_kwargs, merges any direct **kwargs, and constructs the filesystem with fsspec.filesystem:

configured_kwargs = get_configured_fsspec_kwargs(protocol, anonymous=anonymous)
configured_kwargs.update(kwargs)

return fsspec.filesystem(protocol, **configured_kwargs)

Consequently, a configured S3 object is the path by which the base retries and backoff fields reach the fsspec/obstore retry configuration. A plain Storage object does not select S3, GCS, or Azure behavior by itself.

Provider and execution edge cases

• Use a provider subclass for provider operations. The base Storage.get_fsspec_kwargs() returns {}. In protocol-specific lookup, get_configured_fsspec_kwargs falls back to S3.auto(), GCS.auto(), or ABFS.auto() when the active configuration is absent or is not the matching type.
• Protocol-specific lookup can work before initialization. For a supplied protocol, get_configured_fsspec_kwargs catches InitializationError from get_storage() and falls back to provider auto-configuration. Its no-protocol branch calls get_storage() directly, so it still requires initialization.
• Hybrid execution requires an exact provider type. _run_hybrid retrieves get_storage() and rejects anything for which type(storage) is not exactly S3, GCS, or ABFS. A plain Storage instance—and a further subclass—is rejected with ValueError("Unsupported storage type: ...").
• The raw data path is separate from the storage object. After the provider check, _run_hybrid raises a ValueError if _run_base_dir is unset and reports the required run-context form, flyte.with_runcontext(run_base_dir='s3://bucket/metadata/outputs').
• Treat environment values carefully. The common environment variables are read as strings, without the conversions implied by their names or annotations. Prefer explicit construction when you need int, bool, or datetime.timedelta values.