How to Manage Data and File Paths

Allocate a task’s raw-data locations

When a Flyte task needs a location for transient inputs or outputs, use the RawDataPath attached to the active TaskContext instead of inventing a path. In task code, flyte.ctx() exposes that context; higher-level file and storage APIs use the same raw-data object automatically.

import flyte


env = flyte.TaskEnvironment("raw-data-example")


@env.task
def show_raw_data_prefix() -> str:
    task_context = flyte.ctx()
    if task_context is None:
        raise ValueError("This task requires an active task context")
    return task_context.raw_data_path.path

TaskContext.raw_data_path is a RawDataPath; its path field is the configured storage prefix. output_path and run_base_dir are separate TaskContext fields used for run metadata and task output locations. flyte.ctx() returns None when no task context is installed.

Configure local raw-data storage

For a local run, pass raw_data_path to flyte.with_runcontext. The public option is annotated as str | None and is documented as currently supported only for local runs:

import flyte


env = flyte.TaskEnvironment("local-data-example")


@env.task
def local_task() -> str:
    task_context = flyte.ctx()
    if task_context is None:
        raise ValueError("This task requires an active task context")
    return task_context.raw_data_path.path


if __name__ == "__main__":
    flyte.with_runcontext(
        mode="local",
        raw_data_path="/tmp/flyte-raw-data",
    ).run(local_task)

The runner installs the path before local execution by calling RawDataPath.from_local_folder. The constructor behavior differs by argument type:

import pathlib

from flyte.models import RawDataPath


existing_string_path = RawDataPath.from_local_folder("/tmp/flyte-data")
created_path = RawDataPath.from_local_folder(pathlib.Path("/tmp/flyte-data/nested"))
temporary_path = RawDataPath.from_local_folder()

• A pathlib.Path is created recursively with mkdir(parents=True, exist_ok=True) and then stored as a string.
• Omitting the argument (None) creates a directory with tempfile.mkdtemp().
• A string is accepted as-is; from_local_folder does not create that directory.
• Other argument types raise ValueError("Invalid local path ...").

Use a pathlib.Path when the SDK should create a missing local directory. If the path comes from the public with_runcontext(raw_data_path=...) option, pass the string form expected by that option or create the directory yourself first.

Generate a unique destination

Call get_random_remote_path on the active raw-data object when you need a destination prefix:

from flyte.models import RawDataPath


raw_data = RawDataPath.from_local_folder()
unnamed_destination = raw_data.get_random_remote_path()
named_destination = raw_data.get_random_remote_path(file_name="result.csv")

Each call adds a 128-bit random UUID rendered as hexadecimal below raw_data.path. The result is not deterministic, so do not use it as a stable child-task identifier.

For a local prefix, the result is an absolute path. With file_name, RawDataPath creates the parent directory and touches the target file if it does not already exist. Without file_name, it returns the random directory path without creating that directory. For a non-local prefix, the method uses the protocol detected by fsspec, joins with that filesystem’s separator, and returns the path without creating a remote object.

That distinction matters when writing directly to a generated remote location: obtaining the string does not upload or create anything. The eventual filesystem writer must do so.

Let file and storage APIs choose the path

Most task code does not need to call get_random_remote_path directly. When no destination is supplied, these SDK APIs obtain the current context’s raw-data path and allocate a destination for you.

Create a file to write

File.new_remote is the direct way to create a file reference for a file that the task will write:

import flyte
from flyte.io import File


env = flyte.TaskEnvironment("file-output-example")


@env.task
async def make_file() -> File:
    file = File.new_remote()
    async with file.open("wb") as output:
        await output.write(b"hello")
    return file

File.new_remote() calls internal_ctx().raw_data.get_random_remote_path() and stores the returned string in the new File. It is decorated with requires_initialization, so storage initialization and an available upload location are required before calling it.

Upload a local file

File.from_local first checks that the source exists. If remote_destination is omitted, it generates a destination from the raw-data context; if a destination is supplied, that destination is used instead.

from flyte.io import File


async def upload_file() -> File:
    return await File.from_local(
        "/tmp/data.csv",
        remote_destination="s3://my-bucket/data.csv",
    )

The method is asynchronous and accepts local_path: str | pathlib.Path, an optional remote_destination, and an optional hash_method. A missing local source raises ValueError("File not found: ...") before a destination is generated. If the generated destination is local, the implementation can use the source path directly; an explicitly supplied local destination causes a copy.

Upload files and streams through storage

flyte.storage.put generates a raw-data destination when to_path is false. For a file upload it appends the source filename; for a recursive directory upload it does not append a filename:

import flyte.storage as storage


async def upload_data() -> str:
    return await storage.put("/tmp/data.csv")


async def upload_directory() -> str:
    return await storage.put("/tmp/data", recursive=True)

storage.put_stream follows the same rule and accepts an optional name:

import flyte.storage as storage


async def upload_stream(data: bytes) -> str:
    return await storage.put_stream(data, name="payload.bin")

Pass to_path to either API when the destination must be explicit. Otherwise, both use internal_ctx().raw_data.get_random_remote_path(...).

Understand context propagation

TaskContext is a frozen, keyword-only dataclass. Its required fields include action, version, raw_data_path, output_path, run_base_dir, and report; task runtime code creates it rather than requiring task authors to construct one. _internal.runtime.taskrunner.convert_and_run creates a TaskContext, installs it with ctx.replace_task_context(tctx), converts inputs, runs the task, and converts outputs inside that context.

The context bridge exposes the raw-data object in two related ways:

• flyte.ctx() returns data.task_context for user task code.
• internal_ctx().raw_data first returns data.task_context.raw_data_path, then falls back to a context-level raw-data path. If neither is available, it raises ValueError("Raw data path has not been set in the context.").

Use TaskContext.replace when you need a modified context value. It returns a new context rather than mutating the frozen instance. Supplying data merges keys into a copy of the existing mapping:

from flyte.models import TaskContext


async def add_context_value(task_context: TaskContext) -> TaskContext:
    return task_context.replace(data={"source": "generated"})

A data replacement therefore preserves existing custom keys. Passing data=None skips that merge; it does not clear the existing mapping.

Nested local task execution reuses the parent task’s raw_data_path. Child output paths are derived separately from the parent context’s run_base_dir and child action information. Consequently, use raw_data_path for shared transient data allocation and do not treat a random raw-data path as the deterministic identity of a child action.

Remote runs also construct a RawDataPath, but not from the local raw_data_path run-context option. _Runner derives the remote prefix as:

raw_data_path = f"{output_path}/rd/{random_id}"
raw_data_path_obj = RawDataPath(path=raw_data_path)

Here output_path is the configured run_base_dir, and random_id is the first six characters of a newly generated UUID. Remote execution requires a non-None run_base_dir; the runner raises a ValueError directing the caller to set flyte.with_runcontext(run_base_dir=...) when it is absent.

Troubleshoot path allocation

flyte.ctx() is None

The task is not executing with an installed TaskContext. Only dereference raw_data_path from task code when running inside a task context. Lower-level APIs that use internal_ctx().raw_data raise ValueError("Raw data path has not been set in the context.") when no task-level or context-level path exists.

File.new_remote reports missing initialization or upload configuration

File.new_remote requires initialization. Configure Flyte storage and ensure the active context has an upload/raw-data location before calling it. Supplying an explicit destination to APIs such as File.from_local or storage.put is the alternative when the operation’s contract permits it.

A local directory is still missing

A string passed to RawDataPath.from_local_folder is stored without creating its directory. Use pathlib.Path for recursive creation, pass None for an automatically created temporary directory, or create the string path before use.

A remote path does not exist yet

get_random_remote_path only allocates a name for non-local filesystems. It does not create a remote directory or object. Upload or write the data through File, storage.put, storage.put_stream, or the relevant filesystem operation.

Remote execution rejects the run context

Set run_base_dir in flyte.with_runcontext. The documented raw_data_path option is currently for local runs; remote execution derives its raw-data prefix beneath run_base_dir.