Configuring Google Cloud Storage (GCS)

Configure an explicit GCS storage object

To associate GCS configuration with Flyte storage operations, construct the publicly exported GCS class and pass it to flyte.init through the storage parameter:

import flyte
from flyte.storage import GCS

flyte.init(
    endpoint="https://flyte.example.com",
    storage=GCS(gsutil_parallelism=True),
)

GCS is a frozen dataclass subclass of Storage. Its only provider-specific field is gsutil_parallelism, whose default is False:

from flyte.storage import GCS

storage = GCS()
assert storage.gsutil_parallelism is False

The storage argument on flyte.init accepts a Storage | None value. After initialization, storage dispatch uses the configured object when the requested protocol is gs and the object is a GCS instance.

Use gs:// paths

The storage protocol name for Google Cloud Storage is gs, not gcs. A GCS URI causes get_underlying_filesystem to derive the protocol and construct the corresponding fsspec filesystem:

from flyte.storage import get_underlying_filesystem

filesystem = get_underlying_filesystem(path="gs://my-bucket/data/file.parquet")

The runtime path is:

1. get_underlying_filesystem(path=...) derives gs from the path when protocol is not supplied.
2. get_configured_fsspec_kwargs("gs") retrieves the initialized storage configuration.
3. If that configuration is a GCS, the dispatcher calls its get_fsspec_kwargs method. Otherwise, it creates a configuration with GCS.auto().
4. The resulting arguments are passed to fsspec.filesystem("gs", **configured_kwargs).

Explicit keyword arguments supplied to get_underlying_filesystem are applied after the configured arguments, so they take precedence when the same key is present:

from flyte.storage import get_underlying_filesystem

filesystem = get_underlying_filesystem(
    path="gs://my-bucket/data",
    project="my-gcp-project",
)

The storage layer passes such additional arguments through to the selected fsspec backend. The GCS class itself does not validate or transform them.

Configure GCP_GSUTIL_PARALLELISM

GCS.auto() reads the GCP_GSUTIL_PARALLELISM environment variable. If it is unset, the dataclass default remains False; if it is present and considered set, its value is placed in the gsutil_parallelism field:

import os
from flyte.storage import GCS

os.environ["GCP_GSUTIL_PARALLELISM"] = "true"
configuration = GCS.auto()

assert configuration.gsutil_parallelism == "true"

The value is read with os.getenv and passed through set_if_exists; it is not converted from text to a Python boolean. For example, the string "false" is non-empty and therefore remains the string "false", rather than becoming False:

import os
from flyte.storage import GCS

os.environ["GCP_GSUTIL_PARALLELISM"] = "false"
configuration = GCS.auto()

assert configuration.gsutil_parallelism == "false"

Use an explicit boolean when constructing GCS directly if the configuration object itself must contain a boolean:

from flyte.storage import GCS

configuration = GCS(gsutil_parallelism=False)

Important forwarding limitation

Although GCS stores gsutil_parallelism, its current fsspec adapter does not emit that field. get_fsspec_kwargs removes an anonymous key, then returns the remaining keyword arguments unchanged:

from flyte.storage import GCS

configuration = GCS(gsutil_parallelism=True)
assert configuration.get_fsspec_kwargs(anonymous=True) == {}

Consequently, setting gsutil_parallelism=True or setting GCP_GSUTIL_PARALLELISM changes the GCS configuration object but does not add a gsutil_parallelism argument to the fsspec.filesystem call made by get_underlying_filesystem. The current implementation should therefore not be treated as an active fsspec configuration switch for parallelism.

The same adapter removes anonymous without adding an anonymous-access option. The anonymous argument is therefore not configured by GCS.get_fsspec_kwargs itself:

from flyte.storage import GCS

configuration = GCS()
assert configuration.get_fsspec_kwargs(anonymous=True) == {}

Automatic versus explicit configuration

When no initialized GCS object matches a gs request, the dispatcher uses GCS.auto(). This makes the environment-based form useful without constructing the object yourself:

import os
from flyte.storage import get_configured_fsspec_kwargs

os.environ["GCP_GSUTIL_PARALLELISM"] = "true"
kwargs = get_configured_fsspec_kwargs("gs")
assert kwargs == {}

The empty result in this example follows from the current GCS.get_fsspec_kwargs implementation: the environment value is captured by GCS.auto(), but the adapter does not return gsutil_parallelism. GCS.auto() also does not merge the generic Storage settings for retries, backoff, or debug mode.

For an explicitly initialized configuration, use flyte.init(storage=GCS(...)) before resolving storage paths. The gs dispatcher checks the configured provider type; an initialized S3 or ABFS object is not used for a gs:// path, and the dispatcher falls back to GCS.auto().

Authentication and backend dependencies

GCS does not implement credential discovery or inject Google credentials. Filesystem creation is delegated to fsspec, and the storage layer also registers gs among the object-store protocols supported by obstore. Credentials and any Google Cloud client configuration must therefore be available to the selected fsspec or obstore backend and its ambient environment; GCS itself provides no Google-specific credential-file or credential-environment setting.

Hybrid execution requirements

GCS is one of the exact storage types accepted by the hybrid execution path, alongside S3 and ABFS. Hybrid execution also requires a run base directory. Configure a remote base path through the run context before running the task; the source error shows the expected shape as s3://bucket/metadata/outputs, and a GCS deployment should use a gs:// URI instead.

The provider check uses exact type membership, so a subclass of GCS is not accepted by that hybrid check. Use a GCS instance directly when hybrid execution is required.