Overview of File and Dir

File and Dir as data references

When a task should accept or return data without hard-coding whether it lives on a worker's disk or in object storage, use File for one file and Dir for a directory. Both are generic Pydantic models: File[DataFrame] and Dir[DataFrame] express the intended file format while the instance stores a path or URI. They do not contain the file contents.

from pandas import DataFrame
from flyte.io import Dir, File

csv_file = File[DataFrame](path="s3://my-bucket/data.csv")
data_dir = Dir[DataFrame](path="s3://my-bucket/data/")

Both models expose these reference fields:

• path: str: a local path or a remote URI.
• name: Optional[str]: if omitted, the pre_init model validator derives it from Path(path).name.
• format: str: defaults to an empty string.
• hash: Optional[str]: an optional value used by Flyte's cache and discovery handling.

File also has hash_method, which is excluded from the serialized model. File and Dir implement Mashumaro's SerializableType methods (_serialize and _deserialize) and provide schema_match for Flyte type handling.

The generic parameter does not make either class perform parsing. In particular, Dir's docstring states that users are responsible for I/O; its type transformer does not automatically upload or download a directory.

Choose how to create a reference

Use direct construction when you already have a path and want a reference immediately. This performs no upload:

from flyte.io import File

file = File.from_existing_remote("s3://my-bucket/data.csv")

File.from_existing_remote stores the URI and optional file_cache_key as hash. Dir.from_existing_remote has the corresponding behavior for directories:

from flyte.io import Dir

data_dir = Dir.from_existing_remote("s3://bucket/data/", dir_cache_key="abc123")

You can also construct a local reference directly with File(path="/tmp/data.csv") or Dir(path="/tmp/data/"). The explicit from_existing_remote names the intent, but it does not inspect or transfer the remote object.

Create output references and upload local data

For a file that a task will write directly to configured remote storage, call File.new_remote() after Flyte has been initialized:

@env.task
async def write_file() -> File[DataFrame]:
    df = pd.DataFrame(...)
    file = File.new_remote()
    async with file.open("wb") as f:
        df.to_csv(f)
    return file

new_remote obtains a randomized path from internal_ctx().raw_data.get_random_remote_path(). It is guarded by requires_initialization, so it needs Flyte's initialized raw-data context. The returned reference is already pointed at the generated destination; the documented pattern writes to it and does not perform another upload.

When the source is already on local disk and should be uploaded or copied, use the asynchronous factory:

remote_file = await File[DataFrame].from_local(
    "/tmp/data.csv",
    "s3://bucket/data.csv",
)

remote_dir = await Dir[DataFrame].from_local(
    "/tmp/data_dir/",
    "s3://bucket/data/",
    dir_cache_key="abc123",
)

File.from_local first checks that the source exists, then either copies to a local file destination or calls the storage layer for a remote destination. If no destination is supplied, it asks the initialized raw-data context for one. A string hash_method is treated as a precomputed hash value; a HashMethod can be used while bytes are streamed so the resulting hash is stored on the returned File.

Dir.from_local calls storage.put(..., recursive=True) and preserves the basename of the local directory as name. Its synchronous upload method, from_local_sync, currently raises NotImplementedError; use the asynchronous factory for uploads.

Open a file without choosing a storage API

Use File.open as an asynchronous context manager or File.open_sync as a synchronous context manager. The same reference works with a local path or a remote URI:

from pandas import DataFrame
from flyte.io import File

csv_file = File[DataFrame](path="s3://my-bucket/data.csv")

async with csv_file.open() as f:
    content = await f.read()

with csv_file.open_sync() as f:
    content = f.read()

Both methods accept mode, block_size, cache_type, cache_options, compression, and additional filesystem options. The default mode is "rb", and the default cache type is "readahead"; pass cache_type="none" to omit caching options.

File.open selects a filesystem with storage.get_underlying_filesystem(path=self.path). For the file protocol it uses aiofiles. For other protocols it first tries an asynchronous fsspec filesystem's open_async; if that is unavailable, it falls back to synchronous fs.open. The remote path branch requires binary mode and raises ValueError when "b" is absent:

async with csv_file.open("rb") as f:
    data = await f.read()

The synchronous method always delegates to the selected filesystem's open. The async local branch passes the mode and extra keyword arguments to aiofiles; the async remote open_async branch passes the path and mode directly, so not every cache, block-size, or compression option is forwarded through that branch.

Traverse a directory as File references

Dir composes with File: walking a directory yields File[T] children rather than raw path strings. Use the asynchronous or synchronous pair that matches the surrounding code:

from pandas import DataFrame
from flyte.io import Dir

data_dir = Dir[DataFrame](path="s3://my-bucket/data/")

async for file in data_dir.walk():
    async with file.open() as f:
        content = await f.read()

for file in data_dir.walk_sync():
    with file.open_sync() as f:
        content = f.read()

walk obtains the filesystem from storage.get_underlying_filesystem. With an AsyncFileSystem, it calls the filesystem's _walk; otherwise it uses fs.walk. Local child paths are joined with os.path.join, while remote child paths are rebuilt with the filesystem separator and unstrip_protocol. Each result is created as File[T](path=full_file).

For a non-recursive list, use list_files or list_files_sync:

files = await data_dir.list_files()
for file in files:
    if file.exists_sync():
        pass

files = data_dir.list_files_sync()

list_files collects results from walk(recursive=False), and list_files_sync collects results from walk_sync(recursive=False). To resolve one known child, use get_file or get_file_sync; each returns a File[T] when the path exists and None otherwise:

file = await data_dir.get_file("data.csv")
if file:
    async with file.open() as f:
        content = await f.read()

The two lookup implementations join paths differently: asynchronous lookup uses fs.sep, while synchronous lookup uses os.path.join and then calls File.exists_sync().

Local and remote transfer behavior

The storage protocol selected from path determines the underlying filesystem. File.exists_sync delegates to that filesystem, while Dir.exists uses an asynchronous filesystem's _exists when available and otherwise calls exists. The asynchronous file download API returns the destination path:

local_file = await csv_file.download("/tmp/myfile.csv")

For a local source, File.download copies bytes with aiofiles; for a remote source, it calls storage.get. File has no implemented download_sync method—the source marks synchronous download as a TODO.

Directories have both async and sync download methods, but their remote support differs:

local_dir = await data_dir.download("/tmp/my_data/")

Dir.download returns the original local path when no copy is requested, copies a local directory to a different destination with shutil.copytree, and otherwise calls storage.get(..., recursive=True). Dir.download_sync supports local copying but raises NotImplementedError for remote paths.

Storage configuration therefore matters for generated paths and transfers. File.new_remote and File.from_local require Flyte initialization because they use the initialized raw-data context when a destination is generated. Dir.from_local delegates destination and recursive upload behavior to storage.put.

Container task integration

ContainerTask treats File and Dir values as path-backed inputs. Its command rendering requires path-like syntax such as /var/inputs/infile, rather than a template such as {{.inputs.infile}}:

if input_val and type(input_val) in [File, Dir]:
    if not path_k:
        raise AssertionError(
            "File and Directory commands should not use the template syntax "
            "like this: {{.inputs.infile}}\n"
            "Please use a path-like syntax, such as: /var/inputs/infile."
        )
    local_flyte_file_or_dir_path = input_val.path
    remote_flyte_file_or_dir_path = os.path.join(self._input_data_dir, k)
    volume_binding[local_flyte_file_or_dir_path] = {
        "bind": remote_flyte_file_or_dir_path,
        "mode": "rw",
    }

For declared container outputs, ContainerTask._get_output uses the asynchronous factories: File.from_local(output_path) for a File output and Dir.from_local(output_path) for a Dir output. Consequently, a container output written under the container output directory becomes a Flyte reference through the configured storage layer.

Flyte serialization shape

The registered FileTransformer represents a File as a Flyte single-part blob containing its URI, declared format, and optional hash. DirTransformer represents a Dir as a multipart blob with the corresponding URI, format, and hash fields. The transformers reject values of the wrong class and reject the opposite blob shape when reconstructing values.

Although both models carry format and are generic over T, their transformer get_literal_type implementations currently leave the declared blob format empty. The source includes a TODO that generic format propagation is not yet implemented. The type parameter is therefore useful at the Python API boundary—such as File[DataFrame] and Dir[DataFrame]—but does not currently populate the Flyte literal format.

Current limitations to account for

Area	Current behavior
Initialization	File.new_remote and File.from_local are guarded by requires_initialization and need an initialized raw-data context. Direct construction and from_existing_remote do not use that guard.
File download	File.download is asynchronous only; there is no implemented download_sync.
Remote directory download	Dir.download_sync raises NotImplementedError for remote paths.
Directory upload	Dir.from_local_sync always raises NotImplementedError.
walk_sync filtering	walk_sync accepts file_pattern, defaulting to "*", but the implementation never applies it.
Async versus sync walking	Async walk(recursive=False) changes max_depth to 2; sync walking passes the caller's max_depth unchanged. Their non-recursive behavior is therefore not identical.
Remote async opening	Remote async access requires binary mode. Also, the open_async branch receives only the path and mode, unlike the fallback branch's broader fsspec options.
Directory abstraction	Dir is a reference and multipart literal; it does not automatically upload or download directory contents. Use its explicit factories and transfer methods.
Format serialization	File and Dir retain a format field, but the transformers currently declare an empty literal format rather than deriving it from the generic type.