Task Execution Errors

Where each execution error comes from

When a task ends unsuccessfully, first determine whether the failure came back as a serialized execution error or as a terminal remote-action phase. flyte-sdk uses both paths:

Exception	Origin in flyte-sdk	Selection condition
`OOMError`	Server `ExecutionError.USER`, converted by `convert_error_to_native`	The error code contains `OOM`, matched case-insensitively
`TaskInterruptedError`	Server `ExecutionError.USER`, converted by `convert_error_to_native`	The error code contains the case-sensitive substring `Interrupted`
`RetriesExhaustedError`	Server `ExecutionError.USER`, converted by `convert_error_to_native`	The error code contains the case-sensitive substring `RetriesExhausted`
`TaskTimeoutError`	Remote controller terminal phase	The action reaches `PHASE_TIMED_OUT`
`RunAbortedError`	Remote controller terminal phase	The action reaches `PHASE_ABORTED`

All five classes are in errors.py. They inherit from RuntimeUserError, which supplies kind="user"; that value is the SDK's execution-error classification and does not by itself identify which runtime resource or component caused the failure.

Inspect the exception metadata

BaseRuntimeError.__init__ stores the error code, classification, and optional worker, and passes the root-cause message to RuntimeError. Consequently, the most useful first inspection is the exception message together with code, kind, and worker:

try:
    run_task()
except flyte.errors.OOMError as exc:
    print(str(exc))
    print(exc.code)
    print(exc.kind)
    print(exc.worker)

The same attributes are available on TaskInterruptedError and RetriesExhaustedError, as well as on the phase-based errors. Access these classes through flyte.errors; the five classes are not listed as top-level exports in flyte.__all__.

The constructors are not interchangeable. OOMError, TaskInterruptedError, and RetriesExhaustedError do not define their own constructors, so they inherit RuntimeUserError(code, message, worker=None). The remote controller creates the other two with only a message:

flyte.errors.OOMError(code="OOM", message="container exceeded its memory limit", worker="worker-1")
flyte.errors.TaskInterruptedError(code="Interrupted", message="action interrupted", worker="worker-1")
flyte.errors.RetriesExhaustedError(code="RetriesExhaustedError", message="all attempts failed", worker="worker-1")
flyte.errors.TaskTimeoutError("action timed out")
flyte.errors.RunAbortedError("action was aborted")

The first three lines reflect the inherited public constructor shape. In normal execution, flyte-sdk constructs them from a server ExecutionError; the last two lines reflect the message-only constructors implemented by TaskTimeoutError and RunAbortedError. Those constructors set their own codes (TaskTimeoutError and RunAbortedError) and the user kind.

How errors are converted and raised

For a server-reported task failure, the path is:

ExecutionError (or Error wrapper)
    -> convert_error_to_native
    -> specialized flyte.errors exception
    -> local or remote controller raises the exception

_internal/runtime/convert.py implements the central conversion function. It returns an existing Python exception unchanged, unwraps the local Error wrapper, and cleans the incoming code before constructing the native exception:

def convert_error_to_native(err: execution_pb2.ExecutionError | Exception | Error) -> Exception | None:
    if not err:
        return None

    if isinstance(err, Exception):
        return err

    if isinstance(err, Error):
        err = err.err

    user_code, server_code = _clean_error_code(err.code)
    match err.kind:
        case execution_pb2.ExecutionError.USER:
            if "OOM" in err.code.upper():
                return flyte.errors.OOMError(code=user_code, message=err.message, worker=err.worker)
            elif "Interrupted" in err.code:
                return flyte.errors.TaskInterruptedError(code=user_code, message=err.message, worker=err.worker)
            elif "RetriesExhausted" in err.code:
                return flyte.errors.RetriesExhaustedError(code=user_code, message=err.message, worker=err.worker)
            return flyte.errors.RuntimeUserError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.SYSTEM:
            return flyte.errors.RuntimeSystemError(code=user_code, message=err.message, worker=err.worker)
    return None

The complete function also handles UNKNOWN errors and several other user error codes. The important diagnostic detail for these five exceptions is that matching uses the original err.code, while the exception receives user_code returned by _clean_error_code. That helper splits at the first |; therefore, a server code such as RetriesExhaustedError|my-user-code can select RetriesExhaustedError while the resulting exception's code is the cleaned code rather than the complete incoming string.

The matching is textual rather than enum-based. OOM is checked using err.code.upper(), but Interrupted and RetriesExhausted are checked with case-sensitive substring tests. A differently cased server code can therefore become a generic RuntimeUserError instead of one of the specialized classes.

Remote failures and missing error artifacts

handle_action_failure in _internal/controllers/remote/_controller.py starts with action.err or action.client_err. If the action is in PHASE_FAILED and neither is present, it loads the error artifact at:

<run_output_base>/<action-name>/1

The implementation uses io.error_path(...) and io.load_error(...). If loading fails, it creates a RuntimeSystemError describing the load failure. It then calls convert_error_to_native; if conversion returns None, it returns RuntimeSystemError("UnableToConvertError", ...). The remote caller raises the returned exception after submit_action reports a failed action.

Local execution follows the same conversion rule. _internal/controllers/_local_controller.py receives err from direct_dispatch, calls convert_error_to_native(err), and raises the converted exception. If conversion produces no exception, the local fallback is RuntimeSystemError("BadError", "Unknown error").

Remote phase errors are handled before generic failure conversion

In the remote nested-task execution path, _internal/controllers/remote/_controller.py checks terminal phases immediately after submit_action:

if n.phase == run_definition_pb2.PHASE_ABORTED:
    logger.warning(f"Action {n.action_id.name} was aborted, aborting current Action{current_action_id.name}")
    raise flyte.errors.RunAbortedError(
        f"Action {n.action_id.name} was aborted, aborting current Action {current_action_id.name}"
    )

if n.phase == run_definition_pb2.PHASE_TIMED_OUT:
    logger.warning(
        f"Action {n.action_id.name} timed out, raising timeout exception Action {current_action_id.name}"
    )
    raise flyte.errors.TaskTimeoutError(
        f"Action {n.action_id.name} timed out, raising exception in current Action {current_action_id.name}"
    )

if n.has_error() or n.phase == run_definition_pb2.PHASE_FAILED:
    exc = await handle_action_failure(action, _task.name)
    raise exc

This ordering means an aborted or timed-out phase is translated to its dedicated exception even when there is no serialized execution error. It also distinguishes cancellation of the Python operation from a server-reported interruption: when submit_action raises asyncio.CancelledError, the controller calls cancel_action(action) and re-raises CancelledError; it does not translate that event into TaskInterruptedError.

Previously failed traced actions use convert_error_to_native while rebuilding TraceInfo(error=...). Trace reuse can consequently expose a specialized exception without rerunning the failed trace.

Troubleshooting by origin

What you observe	Inspect next	flyte-sdk behavior
`OOMError`	`str(exc)`, `exc.code`, and `exc.worker`; retain the original server code if available	A user-kind code containing `OOM` is converted to `OOMError`
`TaskTimeoutError`	The action phase and the configured `Timeout.max_runtime`	A remote `PHASE_TIMED_OUT` is raised directly with code `TaskTimeoutError`
`TaskInterruptedError`	The server error code's spelling and casing, plus `exc.worker`	Only a user-kind code containing exact-case `Interrupted` is specialized
`RetriesExhaustedError`	`exc.code`, the task retry count, and the server error code	A user-kind code containing exact-case `RetriesExhausted` is specialized
`RunAbortedError`	The remote action phase and exception message	A remote `PHASE_ABORTED` is raised directly with code `RunAbortedError`
Generic `RuntimeUserError` instead of a specialized class	The incoming error code before cleaning	The converter's string checks did not match, or the error kind was not `ExecutionError.USER`
Generic `RuntimeSystemError` after a remote failure	Whether `action.err`/`action.client_err` was present and whether the error artifact loaded	Missing inline errors trigger a load from `<run_output_base>/<action-name>/1`; load or conversion failures use a system error

Configure the conditions that lead to these errors

Retries and exhausted attempts

Configure task retries with an integer or RetryStrategy. The RetryStrategy dataclass has count, backoff, and backoff_factor fields. The source docstring shows the integer form:

@task(retries=5)
def my_task():
    pass

At serialization time, get_proto_retry_strategy in _internal/runtime/task_serde.py writes RetryStrategy(retries=retries.count) into task metadata. It explicitly rejects an integer passed directly to that serializer with AssertionError; task definitions normalize integer retry settings before this internal serialization call.

Task retries are distinct from remote controller system retries. create_remote_controller constructs a controller with max_system_retries=5, and the controller uses that setting for system/controller-processing retries. Exceeding that limit produces a generic RuntimeSystemError; it is not RetriesExhaustedError. The specialized exception is selected only when the server reports a user-kind error code containing RetriesExhausted.

Runtime timeout

Use Timeout when you need to specify the maximum runtime, and optionally the queue limit:

from datetime import timedelta


timeout = Timeout(max_runtime=timedelta(minutes=5), max_queued_time=timedelta(minutes=10))

@env.task(timeout=timeout)
async def my_task():
    pass

Timeout also accepts an integer number of seconds or a timedelta directly through timeout_from_request. get_proto_timeout serializes timeout_from_request(timeout).max_runtime as a protobuf duration. It does not serialize max_queued_time in this function, so the timeout value represented in the task metadata is the maximum runtime. When the remote action reaches PHASE_TIMED_OUT, the remote controller raises TaskTimeoutError.

The same task metadata contains retries, timeout, and interruptible=task.interruptable in _internal/runtime/task_serde.py. A server-reported interruption is still identified by its execution-error code and converted to TaskInterruptedError; the interruptable metadata setting is not itself an exception.

Implementation limits to account for

The specialized server-decoded classes depend on error-code text. OOM matching is case-insensitive, while Interrupted and RetriesExhausted are case-sensitive.
The cleaned exc.code can differ from the original server code because _clean_error_code removes the server-injected prefix before the first |.
TaskTimeoutError and RunAbortedError accept only message; do not pass them the (code, message, worker) arguments used by the inherited-code classes.
The retry docstring in _retry.py shows RetryStrategy(count=5, max_backoff=10, backoff=2), but the dataclass fields implemented there are backoff and backoff_factor; max_backoff is not an implemented field.
The repository search associated with these findings found no matching Python test files, Markdown documentation, or example files for these errors. The documented behavior above is therefore taken from the exception definitions, conversion path, controller paths, and configuration serializers rather than from repository test cases.

Where each execution error comes from​

Inspect the exception metadata​

How errors are converted and raised​

Remote failures and missing error artifacts​

Remote phase errors are handled before generic failure conversion​

Troubleshooting by origin​

Configure the conditions that lead to these errors​

Retries and exhausted attempts​

Runtime timeout​

Implementation limits to account for​