URI Resolution Internals

Abstract versus physical URIs

A flyte://... value is an abstract URI at the point where it appears in a Flyte literal. RemoteFSPathResolver does not upload data or select an object-store implementation; it is a process-local lookup table that can replace that abstract value with a concrete path such as an s3:// or gs:// URI (or a local path) when a mapping has been registered.

The resolver identifies its abstract scheme through the class attribute protocol in storage/_remote_fs.py:

class RemoteFSPathResolver:
    protocol = "flyte://"

The module also defines REMOTE_PLACEHOLDER = "flyte://data". The source comment says that this abstraction is “not really a filesystem” and that users cannot specify the remote path yet. The constant is not used by the resolver or elsewhere in the repository, and the resolver does not create mappings automatically.

Resolver state and API

RemoteFSPathResolver stores mappings in the class-level dictionary _flyte_path_to_remote_map. Thus, all callers in one Python process share the same registry rather than receiving a resolver instance scoped to a task or run. Both operations use the class-level threading.Lock:

• add_mapping(flyte_uri, remote_path) stores the value under the exact URI string. Adding another value for that same string replaces the previous value and returns nothing.
• resolve_remote_path(flyte_uri) performs an exact lookup and returns the mapped string, or None when the URI is absent.

The API-level lifecycle can be seen directly with the class methods:

from flyte.storage._remote_fs import RemoteFSPathResolver

RemoteFSPathResolver.add_mapping("flyte://data/example", "s3://bucket/example")
physical_path = RemoteFSPathResolver.resolve_remote_path("flyte://data/example")
assert physical_path == "s3://bucket/example"

This is a registration-and-lookup example, not a production call path found in the repository: repository-wide usage contains no call to RemoteFSPathResolver.add_mapping. Code outside this checkout, or another integration not present here, would need to populate the registry before producing a flyte://... value that is expected to resolve.

The lock makes each individual read and write mutually exclusive, but the class has no clear/reset operation, persistence mechanism, or multi-operation transaction. Mappings therefore remain in memory for the process lifetime and are shared across its users.

Resolution during literal conversion

The active consumer is modify_literal_uris in types/_type_engine.py. It mutates a FlyteIDL Literal in place. Its traversal handles the literal shapes that can contain nested values:

• collection literals: recursively process each item;
• map literals: recursively process each map value;
• union scalars: recursively process the union value;
• blob scalars and structured-dataset scalars: resolve their URI when it starts with RemoteFSPathResolver.protocol.

The relevant branches are:

def modify_literal_uris(lit: Literal):
    """
    Modifies the literal object recursively to replace the URIs with the native paths in case they are of
    type "flyte://"
    """
    from flyte.storage._remote_fs import RemoteFSPathResolver

    if lit.HasField("collection"):
        for literal in lit.collection.literals:
            modify_literal_uris(literal)
    elif lit.HasField("map"):
        for k, v in lit.map.literals.items():
            modify_literal_uris(v)
    elif lit.HasField("scalar"):
        if (
            lit.scalar.HasField("blob")
            and lit.scalar.blob.uri
            and lit.scalar.blob.uri.startswith(RemoteFSPathResolver.protocol)
        ):
            lit.scalar.blob.uri = RemoteFSPathResolver.resolve_remote_path(lit.scalar.blob.uri)
        elif lit.scalar.HasField("union"):
            modify_literal_uris(lit.scalar.union.value)
        elif (
            lit.scalar.HasField("structured_dataset")
            and lit.scalar.structured_dataset.uri
            and lit.scalar.structured_dataset.uri.startswith(RemoteFSPathResolver.protocol)
        ):
            lit.scalar.structured_dataset.uri = RemoteFSPathResolver.resolve_remote_path(
                lit.scalar.structured_dataset.uri
            )

The check is a string-prefix check followed by an exact dictionary lookup. The helper does not normalize URIs, rewrite arbitrary string fields, or inspect offloaded metadata URIs. If no mapping exists, resolve_remote_path returns None, and the helper assigns that result directly to the protobuf URI field; it does not check for the missing mapping or retain the original flyte:// value.

TypeEngine.to_literal makes this traversal part of the standard Python-to-literal path. It first selects a transformer, optionally performs the transformer's type assertion, and serializes the value. Only then does it call modify_literal_uris:

@classmethod
async def to_literal(
    cls, python_val: typing.Any, python_type: Type[T], expected: types_pb2.LiteralType
) -> literals_pb2.Literal:
    transformer = cls.get_transformer(python_type)

    if transformer.type_assertions_enabled:
        transformer.assert_type(python_type, python_val)

    lv = await transformer.to_literal(python_val, python_type, expected)

    modify_literal_uris(lv)
    return lv

Consequently, the resolver is not a general-purpose path conversion applied to every Python string. It is reached after transformer serialization and only changes matching blob and structured-dataset URI fields, including those reached through the supported recursive literal structures.

Dataframe integration

Structured datasets have an additional active entry path. DataFrameTransformer.encode obtains a dataframe encoder, calls its encode method, fills in structured-dataset metadata when necessary, and constructs a scalar structured-dataset literal. It then invokes the same helper:

sd_model = await handler.encode(sd, structured_literal_type)

if sd_model.metadata is None:
    sd_model.metadata = literals_pb2.StructuredDatasetMetadata(structured_dataset_type=structured_literal_type)
if sd_model.metadata and sd_model.metadata.structured_dataset_type is None:
    sd_model.metadata.structured_dataset_type = structured_literal_type
sd_model.metadata.structured_dataset_type.format = handler.supported_format
lit = literals_pb2.Literal(scalar=literals_pb2.Scalar(structured_dataset=sd_model))

# Because the handler.encode may have uploaded something, and because the sd may end up living inside a
# dataclass, we need to modify any uploaded flyte:// urls here.
modify_literal_uris(lit)  # todo: verify that this can be removed.
sd._literal_sd = sd_model
sd._already_uploaded = True
return lit

The comment and the TODO are part of the source's current behavior: dataframe encoding currently performs the rewrite, but the code does not establish that this call is permanent architecture.

For example, the basic dataframe encoder normally obtains a concrete destination from internal_ctx().raw_data.get_random_remote_path() when dataframe.uri is empty, writes a .csv file there, and returns that URI in the structured dataset:

if not dataframe.uri:
    from flyte._context import internal_ctx

    ctx = internal_ctx()
    uri = ctx.raw_data.get_random_remote_path()
else:
    uri = typing.cast(str, dataframe.uri)

if not storage.is_remote(uri):
    Path(uri).mkdir(parents=True, exist_ok=True)
path = os.path.join(uri, ".csv")
df = typing.cast(pd.DataFrame, dataframe.val)
df.to_csv(path, index=False, storage_options=get_pandas_storage_options(uri=path))
structured_dataset_type.format = CSV
return literals_pb2.StructuredDataset(
    uri=uri, metadata=literals_pb2.StructuredDatasetMetadata(structured_dataset_type)
)

The resolver matters in this path only when an encoder has produced a flyte:// URI. The dataframe transformer wraps the returned structured dataset in a literal and passes it through modify_literal_uris before marking the dataframe as uploaded.

Relationship to the storage layer

RemoteFSPathResolver is separate from the ordinary raw-data and upload mechanisms. RawDataPath.get_random_remote_path uses the configured path and fsspec protocol handling: for a local prefix it builds an absolute local path, while for another protocol it joins the prefix and a generated UUID using the filesystem separator. It returns that physical destination directly; it does not register a flyte:// mapping.

Likewise, File.from_local chooses either the explicitly supplied remote_destination or internal_ctx().raw_data.get_random_remote_path(). For non-file protocols it calls storage.put or storage.put_stream, and the resulting physical path is placed directly on the File object:

remote_path = remote_destination or internal_ctx().raw_data.get_random_remote_path()
protocol = get_protocol(remote_path)

if "file" in protocol:
    if remote_destination is None:
        path = str(Path(local_path).absolute())
    else:
        async with aiofiles.open(local_path, "rb") as src:
            async with aiofiles.open(remote_path, "wb") as dst:
                await dst.write(await src.read())
        path = str(Path(remote_path).absolute())
else:
    path = await storage.put(str(local_path), remote_path)

These paths explain the distinction between the two mechanisms: the storage layer generates or receives a concrete destination and performs I/O, whereas RemoteFSPathResolver only translates a previously registered abstract key during literal processing.

Limitations and troubleshooting

When diagnosing a URI that did not resolve, the implementation narrows the likely causes:

• The strings must match exactly. flyte://data/x and flyte://data/x/ are different dictionary keys. There is no scheme validation in add_mapping, URI normalization, alias handling, or special conversion between s3:// and gs://.
• A mapping must already be present in the same process. The dictionary is class-level and in-memory. No persistence, task/run scoping, environment-variable configuration, or resolver-specific configuration key is defined by the inspected code.
• Registration overwrites silently. A second add_mapping call for an identical URI replaces the first remote path without a status return.
• Missing entries are not handled by the traversal. resolve_remote_path returns None; modify_literal_uris assigns that return value directly for matching blob or structured-dataset fields.
• Only particular literal fields are rewritten. Collections, maps, and unions are traversed, but arbitrary strings and unrelated URI-bearing fields are not processed by modify_literal_uris.
• The registry has no cleanup API. The source exposes only add_mapping and resolve_remote_path, so stale mappings and registry growth are not addressed by this class.

Finally, REMOTE_PLACEHOLDER should not be treated as a configured default destination. It is an unused module constant, and the source comment explicitly describes the remote-filesystem abstraction as incomplete. In the repository version documented here, physical paths normally come from the raw-data context, dataframe handlers, or storage upload functions; URI resolution occurs only when a flyte:// value reaches the literal-rewriting paths and a matching external registration exists.