Data Hashing and Caching

Cache identity follows the File lifecycle

A File cache identity can either be supplied up front or computed while bytes move through an I/O operation. In File.new_remote(), a plain str is stored as the known hash, while an object satisfying HashMethod is retained as hash_method for later use:

@classmethod
@requires_initialization
def new_remote(cls, hash_method: Optional[HashMethod | str] = None) -> File[T]:
    ctx = internal_ctx()
    known_cache_key = hash_method if isinstance(hash_method, str) else None
    method = hash_method if isinstance(hash_method, HashMethod) else None

    return cls(
        path=ctx.raw_data.get_random_remote_path(),
        hash=known_cache_key,
        hash_method=method,
    )

The distinction is important: a string is not interpreted as a hashlib algorithm or calculated from the file. It is an already-known cache key. A HashMethod is an incremental accumulator that receives data from one of the hashing I/O wrappers. File.hash_method is the working mechanism; File.hash is the value that eventually travels with the File object.

The hashing helpers live in the private flyte.io._hashing_io module. flyte.io._file imports HashMethod, PrecomputedValue, HashingWriter, and AsyncHashingReader directly, but these helpers are not the normal public entry point for application code. Applications generally select hashing through File.new_remote() or File.from_local().

Accumulators: known values and content-derived values

HashMethod is a runtime-checkable protocol with two operations required by the wrappers: update(data: memoryview, /) and result(). It also declares reset(), but the implementation comments specify that reset() is optional and the writers do not require it. Consequently, a caller can provide an accumulator other than the built-in HashlibAccumulator as long as it supplies the operations used by the wrappers.

HashlibAccumulator adapts a hashlib-compatible object. Its update() forwards the memoryview, and result() returns the wrapped object's hexdigest(). The class method from_hash_name() uses Python's hashlib.new(), so the algorithm name must be accepted by Python's hashlib:

from flyte.io._hashing_io import HashlibAccumulator

accumulator = HashlibAccumulator.from_hash_name("sha256")

For a cache key that is already known, PrecomputedValue implements the same protocol without examining data. Its update() is a no-op and result() always returns the constructor value. That makes it appropriate only when the caller already has a value that identifies the file contents or cache identity:

from flyte.io._hashing_io import PrecomputedValue

known_key = PrecomputedValue("already-computed-cache-key")

PrecomputedValue does not validate the source file and does not compute a digest. It simply carries the supplied value through the File workflow.

Hashing the bytes that pass through I/O

The wrappers decorate an underlying file-like handle and delegate the actual I/O to it. They convert data to a memoryview before calling HashMethod.update() and return the underlying handle's read or write result.

HashingWriter updates the accumulator on every write() before calling the wrapped handle. writelines() invokes write() once for each item, so those writes are included as well. Bytes-like values are passed to memoryview() directly. For str, the wrapper encodes only the value used for hashing: it chooses the explicit encoding, then the underlying handle's .encoding, and finally UTF-8. The original string is still passed unchanged to the underlying text handle. errors defaults to "strict".

AsyncHashingWriter has the same data-conversion and ordering behavior, but awaits the underlying write() and each item in writelines(). Its flush() and close() methods accept an underlying method that is either synchronous or awaitable, using inspect.isawaitable() to decide whether to await the result.

The reader variants apply the same idea in the opposite direction. HashingReader hashes values returned by read(), readline(), readlines(), and iteration through __next__, while returning those original values to the caller. Empty and None results are not accumulated. Iteration calls next() on the underlying handle rather than implementing a separate read loop, preserving the underlying iterator's behavior. AsyncHashingReader provides corresponding async methods; its readlines() uses the handle's readlines() when available and otherwise collects lines through async iteration. Its __anext__() prefers the underlying async iterator and otherwise uses readline(), treating "" and b"" as end-of-stream.

For example, the synchronous writer's core operation is deliberately small:

def write(self, data):
    mv = self._to_bytes_mv(data)
    self._acc.update(mv)
    return self._fh.write(data)

This is an incremental, path-dependent hash: a reader includes only data actually returned by the calls made through the wrapper, and a writer includes only data passed through its write() or writelines() methods.

Automatic hashing with File

Remote output created with new_remote()

File.new_remote() creates a remote reference with a generated path. If it receives a HashMethod, the method is retained until the file is opened for writing. In the remote synchronous-handle branch of File.open(), flyte-sdk wraps the handle only when self.hash_method is set and self.hash is still unset:

if self.hash_method and self.hash is None:
    logger.debug(f"Wrapping file handle with hashing writer using {self.hash_method}")
    fh = HashingWriter(file_handle, accumulator=self.hash_method)
    yield fh
    self.hash = fh.result()
    fh.close()
else:
    yield file_handle
    file_handle.close()

The hash is assigned after the context body finishes. Supplying a known string instead means hash is already populated, so this condition is false and the handle is not wrapped. The source also has a separate async-filesystem path that yields the async handle directly; the hashing-writer block shown above is in the fallback path using fs.open().

Uploading a local file with from_local()

File.from_local() accepts hash_method as either HashMethod | str. It first separates a string into hash_value and retains protocol instances as hash_method. A missing local source raises ValueError.

For a remote destination using the local-file protocol, the method copies the source to the destination with aiofiles; when a HashMethod is present, it wraps the destination with HashingWriter and obtains the result from that wrapper. For other protocols, it streams the local source through AsyncHashingReader into storage.put_stream() and then records the reader's result:

if hash_method:
    if not isinstance(hash_method, PrecomputedValue):
        async with aiofiles.open(local_path, "rb") as src:
            src_wrapper = AsyncHashingReader(src, accumulator=hash_method)
            path = await storage.put_stream(src_wrapper, to_path=remote_path)
            hash_value = src_wrapper.result()
    else:
        path = await storage.put(str(local_path), remote_path)
        hash_value = hash_method.result()

The PrecomputedValue branch still uploads the source, but it uses storage.put() rather than reading through AsyncHashingReader, then takes the supplied value from result(). Thus a precomputed key avoids content hashing during the upload; it does not avoid the upload itself.

A content-derived upload can be requested as follows. from_local() is asynchronous and requires the initialized Flyte storage/raw-data context used by its @requires_initialization decorator:

from flyte.io import File
from flyte.io._hashing_io import HashlibAccumulator


async def upload_file():
    return await File.from_local(
        "/tmp/data.bin",
        hash_method=HashlibAccumulator.from_hash_name("sha256"),
    )

A caller with a trusted cache identity can instead pass PrecomputedValue:

from flyte.io import File
from flyte.io._hashing_io import PrecomputedValue


async def upload_with_known_key():
    return await File.from_local(
        "/tmp/data.bin",
        hash_method=PrecomputedValue("known-cache-key"),
    )

When no hash argument is supplied, File.from_local() uploads or copies without a hashing wrapper. The method's documentation states that cache-key computation then falls back to the File object's attributes, such as path, name, and format. When the configured destination is local and remote_destination is omitted, the implementation optimizes by retaining the local absolute path rather than copying it; that branch does not run the hashing wrapper.

Serialization into Flyte literals

The computed or supplied value matters because FileTransformer.to_literal() copies python_val.hash into the top-level Flyte literal hash field while placing the file location in the blob URI. to_python_value() reads that literal hash back into the reconstructed File:

return literals_pb2.Literal(
    scalar=literals_pb2.Scalar(
        blob=literals_pb2.Blob(
            metadata=literals_pb2.BlobMetadata(
                type=types_pb2.BlobType(
                    format=python_val.format,
                    dimensionality=types_pb2.BlobType.BlobDimensionality.SINGLE,
                )
            ),
            uri=python_val.path,
        )
    ),
    hash=python_val.hash if python_val.hash else None,
)

The working hash_method is not the serialized identity; the resulting hash is. This is the integration point through which a hash calculated during upload or remote writing becomes available as literal metadata for Flyte's task input/output processing.

Dir has a parallel hash field and transformer behavior, but it does not use the hashing reader/writer wrappers to derive a directory-content digest. Dir.from_local() uploads recursively and stores only the caller-supplied dir_cache_key:

output_path = await storage.put(from_path=local_path_str, to_path=remote_path, recursive=True)
return cls(path=output_path, name=dirname, hash=dir_cache_key)

Consequently, directory cache identity is an explicit string choice, rather than an automatically computed hash of the directory contents.

Operational constraints

Hashing is tied to the wrapper, not to the file path. A reader that is only partially consumed hashes only the bytes returned so far; opening a File does not by itself calculate its content hash. Similarly, data written through a saved reference to the underlying handle, rather than through HashingWriter, is absent from the accumulator. HashingWriter advances the accumulator before invoking the underlying write(), so a failed write can leave the accumulator containing bytes that were not successfully persisted.

Text handling also affects the value: the digest is calculated over the selected encoded representation, while the original str is delegated to the text handle. Use an explicit encoding when the default resolution—handle encoding, then UTF-8—would not match the identity you need.

Finally, File.open_sync() currently yields the raw handle and does not provide equivalent HashingReader or HashingWriter integration; the source marks synchronous download support as a TODO. For automatic content hashing, use the asynchronous File.from_local() path or the supported remote writing path in File.open() rather than assuming that every File access updates its hash.