Skip to main content

Generating Unique Execution IDs

When you execute tasks in flyte-sdk, each execution needs a unique identifier. This is crucial for tracking, logging, and managing the lifecycle of tasks and their sub-components. The flyte-sdk addresses this requirement primarily through the ActionID class.

Understanding ActionID​

The ActionID class serves as the fundamental identifier for any action within a run. It's a dataclass designed to encapsulate all necessary information to uniquely pinpoint a specific task execution.

@rich.repr.auto
@dataclass(frozen=True, kw_only=True)
class ActionID:
    """
    A class representing the ID of an Action, nested within a Run. This is used to identify a specific action on a task.
    """

    name: str
    run_name: str | None = None
    project: str | None = None
    domain: str | None = None
    org: str | None = None

As seen in its definition, ActionID includes several attributes:

  • name: The primary name of the action.
  • run_name: The name of the overall run to which this action belongs.
  • project, domain, org: Optional attributes that provide hierarchical context for the action, useful in multi-tenant or organized environments.

A key behavior of ActionID is how it handles the run_name. If you instantiate an ActionID without explicitly providing a run_name, it automatically defaults to the value of name. This ensures that every action always has an associated run name, even if it's the same as its own action name.

You can create an ActionID in several ways. For instance, to generate a completely random and unique ID, use the create_random class method:

from flyte.models import ActionID

random_action_id = ActionID.create_random()
print(f"Random Action ID: {random_action_id.name}")
# Example output: Random Action ID: random_string_of_chars

More commonly, ActionID instances are constructed at runtime, often populated from command-line arguments or environment variables. For example, during the initial setup of a task execution, flyte-sdk constructs an ActionID like this:

# From _bin/runtime.py
action = ActionID(name=name, run_name=run_name, project=project, domain=domain, org=org)

Here, name, run_name, project, domain, and org would be derived from the execution environment. Similarly, if a task is run without a predefined name, flyte-sdk can automatically assign a random ActionID:

# From _run.py
if self._name is None:
    action = ActionID.create_random()
else:
    action = ActionID(name=self._name)

Generating Deterministic Sub-Action IDs​

When a task needs to spawn sub-tasks or perform internal actions that require their own unique, yet predictable, identifiers, ActionID provides mechanisms to create "sub-actions." This is particularly useful for ensuring idempotency or traceability across multiple runs of the same parent task with identical inputs.

The new_sub_action method allows you to create a new ActionID that is a derivative of the current one. If you don't provide a name, it will generate a random one:

from flyte.models import ActionID

parent_action = ActionID(name="my_parent_task_run")
sub_action_random = parent_action.new_sub_action()
print(f"Parent: {parent_action.name}, Sub-action (random): {sub_action_random.name}")

For scenarios requiring deterministic naming, such as when a parent task executes a child task multiple times with the same parameters, the new_sub_action_from method is invaluable. This method generates a new ActionID name by hashing a combination of inputs, ensuring that the same inputs always produce the same sub-action ID.

# From models.py
def new_sub_action_from(self, task_call_seq: int, task_hash: str, input_hash: str, group: str | None) -> ActionID:
    """Make a deterministic name"""
    import hashlib

    from flyte._utils.helpers import base36_encode

    components = f"{self.name}-{input_hash}-{task_hash}-{task_call_seq}" + (f"-{group}" if group else "")
    logger.debug(f"----- Generating sub-action ID from components: {components}")
    # has the components into something deterministic
    bytes_digest = hashlib.md5(components.encode()).digest()
    new_name = base36_encode(bytes_digest)
    return self.new_sub_action(new_name)

This method takes several parameters to construct a unique hash:

  • task_call_seq: An integer representing the sequence number of the task call.
  • task_hash: A hash representing the task definition itself.
  • input_hash: A hash representing the inputs provided to the sub-action.
  • group: An optional string to further categorize the sub-action.

By combining these components, flyte-sdk ensures that if a parent task calls a sub-action with the exact same task_call_seq, task_hash, input_hash, and group, it will always result in the same ActionID for the sub-action. This is critical for caching, retries, and ensuring consistent behavior across executions.

Managing Task Context with TaskContext​

While ActionID provides the unique identifier, the TaskContext class aggregates all the necessary contextual information for a task's execution. It acts as a central repository for parameters, paths, and other runtime details that a task might need.

@rich.repr.auto
@dataclass(frozen=True, kw_only=True)
class TaskContext:
    """
    A context class to hold the current task executions context.
    This can be used to access various contextual parameters in the task execution by the user.

    :param action: The action ID of the current execution. This is always set, within a run.
    :param version: The version of the executed task. This is set when the task is executed by an action and will be
      set on all sub-actions.
    """

    action: ActionID
    version: str
    raw_data_path: RawDataPath
    input_path: str | None = None
    output_path: str
    run_base_dir: str
    report: Report
    group_data: GroupData | None = None
    checkpoints: Checkpoints | None = None
    code_bundle: CodeBundle | None = None
    compiled_image_cache: ImageCache | None = None
    data: Dict[str, Any] = field(default_factory=dict)
    mode: Literal["local", "remote", "hybrid"] = "remote"
    interactive_mode: bool = False

The most crucial attribute of TaskContext is action, which is an instance of ActionID. This links the entire context to a specific, uniquely identified execution. Other important attributes include:

  • version: The version of the task being executed.
  • raw_data_path, input_path, output_path, run_base_dir: Paths related to data storage and execution directories.
  • report: An object for reporting execution status and metrics.
  • mode: Indicates the execution environment (e.g., "local", "remote", "hybrid").

TaskContext instances are created at the beginning of a task's execution, gathering all relevant information. For example, when a task is prepared to run, its context is assembled:

# From _run.py
tctx = TaskContext(
    action=action, # This is the ActionID instance
    checkpoints=checkpoints,
    code_bundle=code_bundle,
    output_path=output_path,
    version=version if version else "na",
    raw_data_path=raw_data_path_obj,
    compiled_image_cache=image_cache,
    run_base_dir=self._run_base_dir,
    report=flyte.report.Report(name=action.name),
)

Immutability and Updating TaskContext​

TaskContext is defined as a frozen dataclass, meaning that once an instance is created, its attributes cannot be directly modified. This immutability ensures that the context remains consistent throughout a task's execution and prevents accidental changes.

If you need to create a TaskContext with updated or additional information, you must use the replace method. This method returns a new TaskContext instance with the specified attributes changed, leaving the original instance untouched.

from flyte.models import ActionID, TaskContext
from flyte.models.common import RawDataPath, Report # Assuming these imports for a runnable example

# Create an initial TaskContext
initial_action = ActionID(name="my_initial_task")
initial_context = TaskContext(
    action=initial_action,
    version="v1.0",
    raw_data_path=RawDataPath(path="/tmp/raw"),
    output_path="/tmp/output",
    run_base_dir="/tmp/run",
    report=Report(name="initial_report")
)

print(f"Original output path: {initial_context.output_path}")

# Create a new context with an updated output path
updated_context = initial_context.replace(output_path="/new/output/path")

print(f"Updated output path: {updated_context.output_path}")
print(f"Original context's output path (unchanged): {initial_context.output_path}")

# The replace method also handles merging dictionary data
context_with_data = initial_context.replace(data={"key1": "value1"})
print(f"Context data: {context_with_data.data}")

context_with_more_data = context_with_data.replace(data={"key2": "value2"})
print(f"Context with merged data: {context_with_more_data.data}")

This pattern of immutability and explicit replacement is a common design choice in flyte-sdk to maintain data integrity and predictability.

Configuration and Considerations​

Several environment variables and parameters influence the creation and behavior of ActionID and TaskContext:

  • Environment Variables for ActionID: You can set RUN_NAME and ACTION_NAME environment variables to pre-define the run_name and name attributes of an ActionID if they are not provided via command-line arguments.
  • run_base_dir Requirement: The run_base_dir parameter within TaskContext is critical. If it is not set when running a task, flyte-sdk will raise a ValueError, indicating a fundamental setup requirement for task execution.
  • Optional TaskContext Attributes: Many attributes within TaskContext, such as input_path, group_data, checkpoints, code_bundle, and compiled_image_cache, are optional. Their presence and values depend on the specific execution mode (e.g., local vs. remote) and the features being utilized by the task. For instance, code_bundle and compiled_image_cache are typically None during local runs.

Understanding how ActionID uniquely identifies task executions and how TaskContext provides a comprehensive, immutable snapshot of the execution environment is key to developing robust and traceable workflows with flyte-sdk.