Working with Single Files

Create a single-file reference

Use File[T] when a Flyte value should identify one file without loading its contents into Python. For example, construct a typed reference to an existing object and then open it explicitly:

from pandas import DataFrame
from flyte.io import File

csv_file = File[DataFrame](path="s3://my-bucket/data.csv")

async with csv_file.open() as f:
    content = await f.read()

with csv_file.open_sync() as f:
    content = f.read()

File is publicly exported from flyte.io and is implemented in io/_file.py. It is a generic Pydantic model. Its main fields are path, optional name, format, optional hash, and an optional hash_method. path may be a local path or a remote URI. If name is omitted, the model validator sets it to Path(path).name.

Constructing File[DataFrame](path=...) only creates the reference. It does not check that the path exists, download it, or upload it. Use open, open_sync, exists_sync, or download when you want an I/O operation.

Reference an existing file

For a path that already identifies the file, use from_existing_remote (despite its name, it accepts local paths as well):

from flyte.io import File

local_file = File.from_existing_remote("/tmp/data.csv")
remote_file = File.from_existing_remote("s3://my-bucket/data.csv")

named_file = File(path="s3://my-bucket/data.csv", name="daily.csv", format="csv")

from_existing_remote(remote_path, file_cache_key=None) simply constructs a File. It does not perform existence validation or transfer. file_cache_key is stored as hash; when it is omitted, Flyte's discovery hashing uses the File object's attributes rather than the contents of the referenced file.

The same reference-based behavior is used by directory lookup. Dir.get_file builds a File[T](path=...), checks the filesystem separately, and returns None when the path does not exist:

file = await directory.get_file("data.csv")
if file:
    async with file.open() as f:
        data = await f.read()

CLI file arguments follow the same pattern. FileParamType.convert validates that a local argument exists and is a regular file, accepts remote values without a local existence check, and returns File.from_existing_remote(value).

Read and write asynchronously

open is an asynchronous context manager. It selects the filesystem from self.path, uses aiofiles.open for local file paths, and otherwise prefers an asynchronous fsspec filesystem implementation before falling back to fs.open:

from flyte.io import File

file = File(path="s3://my-bucket/data.csv")

async with file.open("rb") as f:
    data = await f.read()

The default mode is "rb". For remote paths, the mode must contain b; otherwise open raises:

ValueError: Mode must include 'b' for binary access, when using remote files.

Use the open options supported by File.open when reading remote data. block_size, cache_type, cache_options, compression, and additional keyword arguments are passed to the fsspec opening path. The documented cache types are "readahead", "mmap", "bytes", and "none":

async with file.open(
    "rb",
    block_size=1024 * 1024,
    cache_type="readahead",
    cache_options={},
    compression="gzip",
) as f:
    data = await f.read()

For local asynchronous files, the implementation uses aiofiles.open(self.path, mode=mode, **kwargs). The local branch does not pass the fsspec cache, block-size, or compression options to aiofiles; those options apply to the non-local fsspec path.

To write directly to a destination selected by Flyte's raw-data context, create a new remote reference and open it in binary write mode:

from pandas import DataFrame
from flyte.io import File

async def write_csv() -> File[DataFrame]:
    file = File[DataFrame].new_remote()
    async with file.open("wb") as f:
        f.write(b"value\n1\n")
    return file

new_remote requires Flyte initialization. It asks internal_ctx().raw_data for a random remote path, so the resulting File already points at the location being written. Returning that same reference does not trigger another upload.

Read and write synchronously

Use open_sync when the surrounding code is synchronous:

from flyte.io import File

file = File(path="/tmp/data.csv")

with file.open_sync("rb") as f:
    data = f.read()

with file.open_sync("wb") as f:
    f.write(b"value\n1\n")

open_sync obtains the filesystem with storage.get_underlying_filesystem and delegates to fs.open. It accepts the same named options—block_size, cache_type, cache_options, compression, and extra keyword arguments—but unlike asynchronous open, it has no special local aiofiles branch and no hashing-writer wrapper.

Upload a local file

When content has already been written locally, use the asynchronous from_local factory:

from pandas import DataFrame
from flyte.io import File

async def publish_csv() -> File[DataFrame]:
    return await File[DataFrame].from_local(
        "/tmp/data.csv",
        remote_destination="s3://my-bucket/data.csv",
    )

from_local(local_path, remote_destination=None, hash_method=None) first verifies that local_path exists and raises ValueError("File not found: ...") if it does not. With a remote destination it uses the storage layer's upload operation. With no destination, it obtains a random path from the configured raw-data context.

If the selected destination uses the local file protocol and no explicit destination was provided, from_local keeps the absolute source path instead of copying it. An explicit local destination copies the bytes to that destination. It does not create missing parent directories for an explicit local destination.

The actual parameter name is remote_destination. Do not use optional, as shown in an outdated File docstring example; optional is not in the method signature.

Choose a creation method

Goal	API	I/O at factory call	Initialization required
Point at an existing local or remote path	File.from_existing_remote(path) or File(path=path)	None	No
Create a destination and stream bytes into it	File[T].new_remote() followed by open("wb")	new_remote only obtains a path; writing occurs in open	Yes
Upload a local source	await File[T].from_local(local_path, remote_destination=...)	Validates and copies/uploads the source	Yes

Check existence and download

Check existence synchronously with exists_sync:

from flyte.io import File

file = File.from_existing_remote("s3://my-bucket/data.csv")
if file.exists_sync():
    print(file.name)

Download is asynchronous and returns the local destination as a string:

local_path = await file.download("/tmp/data.csv")

When local_path is omitted, download asks the storage layer for a random local path. For a local source it copies with aiofiles; for a remote source it calls storage.get. The local-copy branch reads the complete source with await src.read() before writing it. There is no download_sync method; the source contains a TODO noting that synchronous download still needs to be implemented.

Preserve cache and hash information

File distinguishes a known cache key from an active hashing method:

• A string passed to new_remote(hash_method=...) is placed directly in hash; it is treated as a known cache value, not as the name of a hashing algorithm.
• A HashMethod object is retained in hash_method.
• When from_local uploads with a hashing method, the remote upload can use AsyncHashingReader and record the resulting hash. A PrecomputedValue uses its predefined result and avoids streaming the source through a hasher.

For a remote asynchronous open, if hash_method is set and hash is still unset, the synchronous fallback wraps the handle in HashingWriter and assigns self.hash after the context exits. open_sync does not contain this wrapper logic, so hashing behavior is not symmetric between the two APIs. Hashing follows the bytes passed to write; it is not automatically a digest of some separate source file.

Use File in tasks and Flyte values

A task can accept a typed file reference and read it itself:

from pandas import DataFrame
from flyte.io import File

async def read_file(file: File[DataFrame]):
    async with file.open() as f:
        return await f.read()

FileTransformer connects File to Flyte's TypeEngine. It deliberately performs no I/O. Its get_literal_type returns a single-part blob type. to_literal places File.path in the blob URI, File.format in the blob metadata, and the optional File.hash in the literal hash. to_python_value requires a blob scalar with SINGLE dimensionality and reconstructs a File, deriving its name from Path(uri).name. A PythonPickle blob is excluded by guess_python_type.

The generic type does not currently populate the literal format: get_literal_type sets format="" and includes a source TODO about deriving the format from the generic type. The instance's format is still emitted by to_literal.

This separation means task-boundary serialization carries a path-like blob reference, while opening, uploading, and downloading remain explicit operations. It is also why container output handling calls await File.from_local(output_path) when the declared output type is a File: the generated local output must be published before it becomes the returned reference.

Troubleshoot common failures

• Initialization error from new_remote or from_local: call flyte.init(...) with the storage/raw-data configuration needed by the execution environment. Plain construction and local reference operations do not require these factory methods.
• Remote text mode rejected: include b in the mode passed to asynchronous open, such as "rb" or "wb". The explicit binary-mode check is only in the non-local asynchronous path.
• Unexpected missing-file behavior: from_existing_remote does not validate existence. from_local does validate its local source and raises ValueError when it is absent.
• Hash not populated after synchronous writing: open_sync delegates directly to the filesystem and does not install the asynchronous method's remote HashingWriter behavior.
• Wrong upload keyword: use remote_destination=... with from_local; optional=... is not accepted.
• Unexpected filename: name defaults to Path(path).name, and FileTransformer.to_python_value applies the same basename derivation to a blob URI. Provider-specific URI forms may therefore produce a different logical name than expected.
• Looking for synchronous download: File currently provides only async def download; download_sync is not defined. The repository snapshot also contains no discovered test, Markdown documentation, or external example files for this API, so the implementation and its docstrings are the concrete reference for these behaviors.