Code Packaging and Versioning

When deploying tasks to remote clusters, rebuilding entire container images for every code change slows down iteration and clutters image registries. flyte-sdk decouples container environments from user code by packaging source trees and task definitions into versioned code bundles, generating serialization metadata, and inflating artifacts inside target containers at runtime.

Three core models coordinate this lifecycle:

• CodeBundle: Represents packaged, hashed archives (either tarballs or pickled templates) and their extraction targets.
• SerializationContext: Holds task compilation settings, command entrypoints, and remote storage template paths.
• TaskContext: Provides runtime access to execution metadata, storage paths, and bundle locations inside running tasks.

---

Code Packaging with CodeBundle

CodeBundle is an immutable, frozen dataclass defined in flyte.models. It records the computed hash digest of task artifacts, the remote or local archive locations, and the target extraction path.

from flyte.models import CodeBundle

# Creating a code bundle pointing to a remote tarball
bundle = CodeBundle(
    computed_version="d41d8cd98f00b204e9800998ecf8427e",
    destination=".",
    tgz="s3://my-flyte-bucket/code/bundle-d41d8cd98f00b204e9800998ecf8427e.tar.gz",
)

Model Structure and Validation

CodeBundle enforces that at least one distribution payload is specified. During initialization, its __post_init__ method validates that either tgz (compressed tarball path) or pkl (pickled TaskTemplate path) is provided:

@dataclass(frozen=True, kw_only=True)
class CodeBundle:
    computed_version: str
    destination: str = "."
    tgz: str | None = None
    pkl: str | None = None
    downloaded_path: pathlib.Path | None = None

    def __post_init__(self):
        if self.tgz is None and self.pkl is None:
            raise ValueError("Either tgz or pkl must be provided")

    def with_downloaded_path(self, path: pathlib.Path) -> CodeBundle:
        return replace(self, downloaded_path=path)

If you construct CodeBundle(computed_version="v1") without specifying tgz or pkl, flyte-sdk immediately raises a ValueError("Either tgz or pkl must be provided").

Generating Tarball Bundles (build_code_bundle)

In standard deployment and remote run workflows (flyte._run and flyte._deploy), the SDK uses build_code_bundle (located in flyte._code_bundle.bundle) to discover source files, compute a combined digest, create a compressed .tar.gz archive, and upload it to the control plane.

import pathlib
from flyte._code_bundle import build_code_bundle

# Build, hash, and upload a code bundle from a source root directory
bundle = await build_code_bundle(
    from_dir=pathlib.Path("/path/to/project"),
    extract_dir=".",
    copy_style="loaded_modules",
)

build_code_bundle applies ignore rules (StandardIgnore and GitIgnore) via list_files_to_bundle to exclude virtual environments (.venv, venv), bytecode caches (__pycache__), build directories (dist, build), and repository metadata (.git).

The copy_style parameter controls file selection:

• "loaded_modules" (default): Inspects active Python imports in sys.modules within the project root to package only dependencies referenced by the tasks.
• "all": Packages all non-ignored files within the project root.
• "none": Skips project files.

To ensure deterministic hashing across environments, create_bundle writes archives using mtime=0 with pigz (if available on the host system) or standard gzip.

Generating Pickled Bundles (build_pkl_bundle)

For dynamic task execution or fast sub-action dispatch where full source trees are unnecessary, flyte-sdk packages the serialized task definition as a pickled object using build_pkl_bundle:

from flyte._code_bundle import build_pkl_bundle
from flyteidl.core.tasks_pb2 import TaskTemplate

task_template = TaskTemplate(...)
code_bundle = await build_pkl_bundle(
    o=task_template,
    upload_to_controlplane=True,
)

build_pkl_bundle serializes the task via cloudpickle into a .pkl.gz archive with mtime=0.

Constraint: build_pkl_bundle raises a ValueError if you set upload_to_controlplane=True and provide an upload_from_dataplane_base_path simultaneously. When packaging on the dataplane, set upload_to_controlplane=False and supply the dataplane base path.

---

Task Serialization with SerializationContext

During task registration and template compilation, flyte-sdk uses SerializationContext (from flyte.models) to inject task versions, code bundle paths, and placeholder template arguments.

import pathlib
from flyte.models import SerializationContext, CodeBundle

sctx = SerializationContext(
    version="v1.2.0",
    project="flytesnacks",
    domain="development",
    org="flyte",
    code_bundle=CodeBundle(
        computed_version="hash123",
        tgz="s3://bucket/code.tar.gz",
    ),
    root_dir=pathlib.Path("/workspace"),
)

Entrypoint Resolution and Placeholders

SerializationContext provides defaults for task container inputs and outputs:

• input_path: Defaults to "{{.input}}" (Flyte engine input substitution template).
• output_path: Defaults to "{{.outputPrefix}}".
• interpreter_path: Defaults to "/opt/venv/bin/python".

The get_entrypoint_path method calculates the runtime entrypoint script relative to the container interpreter:

sctx = SerializationContext(
    version="v1",
    interpreter_path="/opt/conda/bin/python",
)
entrypoint = sctx.get_entrypoint_path()
# Returns: "/opt/conda/bin/runtime.py"

During serialization in flyte._internal.runtime.task_serde, these values configure the container command and arguments embedded in the registered TaskTemplate protobuf.

---

Runtime Execution with TaskContext

When a task executes inside a cluster container or local runner, execution parameters are captured in TaskContext. Tasks access this context through the global flyte.ctx() accessor.

import flyte
from flyte import task

@task
def process_data() -> str:
    ctx = flyte.ctx()
    if ctx and ctx.is_in_cluster():
        print(f"Running action {ctx.action.name} with version {ctx.version}")
        print(f"Raw data path: {ctx.raw_data_path.base_path}")
    return "completed"

Context Properties and Inspection

TaskContext contains key runtime metadata:

Attribute	Type	Description
action	ActionID	Identifies the specific execution run and sub-action hierarchy.
version	str	The version string of the task being executed.
raw_data_path	RawDataPath	Configured storage locations for file inputs, outputs, and intermediate data.
mode	Literal["local", "remote", "hybrid"]	The execution environment mode (defaults to "remote").
code_bundle	CodeBundle | None	The bundle containing task code and destination directories.
data	Dict[str, Any]	Execution dictionary accessible via key indexing ctx["custom_key"].

Environment Checks and Context Updates

To determine if code is running remotely inside a cluster pod rather than a local test runner, call ctx.is_in_cluster():

ctx = flyte.ctx()
if ctx.is_in_cluster():
    # Remotely running inside container pod
    pass

The replace method creates a copy with updated attributes or merged dictionary values:

updated_ctx = ctx.replace(data={"retry_count": 2})
assert updated_ctx["retry_count"] == 2

---

End-to-End Bundle Inflation Lifecycle

The following diagram illustrates how flyte-sdk packages, serializes, transfers, and extracts code across execution phases:

[Local / Deployment Phase]
   1. Source Files -> list_files_to_bundle() (Applies StandardIgnore & GitIgnore)
   2. Archive Creation -> create_bundle() (gzip with mtime=0)
   3. Upload & Hash -> upload_file.aio() -> CodeBundle(tgz="s3://...", computed_version="...")
   4. Serialization -> SerializationContext(code_bundle=...) -> TaskTemplate Protobuf
                                |
                                v
[Remote Pod / Runtime Phase]
   5. Container Startup -> runtime.py entrypoint starts
   6. Inflation -> download_bundle(bundle)
         - If bundle.tgz: storage.get() -> runs `tar -xvf <bundle> -C <destination>`
         - If bundle.pkl: storage.get() -> loads pickled TaskTemplate
   7. Code Bundle Updated -> bundle.with_downloaded_path(downloaded_path)
   8. Context Initialization -> TaskContext(code_bundle=bundle) set in contextvars
   9. Execution -> taskrunner invokes task function with flyte.ctx() available

Unpacking at Runtime (download_bundle)

When worker pods launch via flyte._bin.runtime or entrypoints.py, download_bundle in flyte._code_bundle.bundle fetches the bundle archive using the storage backend (flyte.storage) and extracts it:

# Unpacking logic inside download_bundle
if bundle.tgz:
    downloaded_bundle = dest / os.path.basename(bundle.tgz)
    if not downloaded_bundle.exists():
        await storage.get(bundle.tgz, str(downloaded_bundle.absolute()))
        process = await asyncio.create_subprocess_exec(
            "tar", "-xvf", str(downloaded_bundle), "-C", str(dest),
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
        )
        stdout, stderr = await process.communicate()
        if process.returncode != 0:
            raise RuntimeError(stderr.decode())

Once extracted, bundle.with_downloaded_path(...) binds the target directory to TaskContext.code_bundle, the destination directory is added to sys.path, and the user's task entrypoint runs.