Deployment and Environment Errors

When deploying tasks and environments or running them on remote clusters, failures can occur during local module discovery, remote image building, task specification deployment, or container execution. The flyte-sdk defines structured exception classes derived from RuntimeUserError in errors.py to identify and handle each failure scenario.

import flyte.errors

# Handle environment and deployment exceptions
try:
    # Trigger deployment, remote execution, or module discovery
    ...
except flyte.errors.ModuleLoadError as e:
    print(f"Module discovery failed: {e.message}")
except flyte.errors.ImageBuildError as e:
    print(f"Container image build failed: {e.message}")
except flyte.errors.DeploymentError as e:
    print(f"Task deployment failed: {e.message}")
except flyte.errors.ImagePullBackOffError as e:
    print(f"Worker pod failed to pull image on node {e.worker}: {e.message}")
except flyte.errors.InvalidImageNameError as e:
    print(f"Malformed image specifier: {e.message}")
except flyte.errors.PrimaryContainerNotFoundError as e:
    print(f"Missing primary container in execution pod: {e.message}")

---

Module Discovery and Pre-Deployment Errors

During task discovery and deployment scans, the SDK dynamically discovers and imports Python modules using importlib.util. If a target file contains syntax errors or missing dependencies, the loader raises flyte.errors.ModuleLoadError.

Dynamic Module Loading with ModuleLoadError

In _utils/module_loader.py, _load_module_from_file attempts to compile and execute each candidate .py file. If an exception is encountered during execution, it wraps the underlying error:

from pathlib import Path
import flyte.errors
from flyte._utils.module_loader import load_python_modules

# Scan a directory for tasks and workflows
loaded_modules, failed_paths = load_python_modules(Path("./my_project"), recursive=True)

for path, error_msg in failed_paths:
    print(f"Failed to load {path}: {error_msg}")

Inside _utils/module_loader.py, _load_module_from_file wraps loading failures:

try:
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    module_path = os.path.dirname(os.path.abspath(file_path))
    sys.path.append(module_path)
    spec.loader.exec_module(module)
except Exception as e:
    raise flyte.errors.ModuleLoadError(f"Failed to load module from {file_path}: {e}") from e

ModuleLoadError initializes with super().__init__("ModuleLoadError", message, "user"), marking the error code as "ModuleLoadError" and the error category as user-facing.

---

Deployment and Image Build Pipeline Errors

Missing Preconditions and Task Registration with DeploymentError

flyte.errors.DeploymentError is raised in _deploy.py when deployment preconditions are not satisfied or when the gRPC task_service.DeployTask call fails.

Precondition Validation

When using --copy-style none, code is not bundled into the image at deployment time, requiring an explicit version identifier. If version is omitted, _deploy.py raises DeploymentError:

# In flyte._deploy
version = deployment_plan.version
if copy_style == "none" and not version:
    raise flyte.errors.DeploymentError("Version must be set when copy_style is none")

gRPC Service Registration

When deploying tasks over gRPC, _deploy.py catches non-idempotent gRPC failures:

try:
    await get_client().task_service.DeployTask(task_service_pb2.DeployTaskRequest(task_id=task_id, spec=spec))
    logger.info(f"Deployed task {task.name} with version {task_id.version}")
except grpc.aio.AioRpcError as e:
    if e.code() == grpc.StatusCode.ALREADY_EXISTS:
        logger.info(f"Task {task.name} with image {image_uri} already exists, skipping deployment.")
        return spec
    raise
except Exception as e:
    logger.error(f"Failed to deploy task {task.name} with image {image_uri}: {e}")
    raise flyte.errors.DeploymentError(
        f"Failed to deploy task {task.name} file{task.source_file} with image {image_uri}, Error: {e!s}"
    ) from e

Remote Image Builder Failures with ImageBuildError

When using Union or remote builder infrastructure to build container images, _internal/imagebuild/remote_builder.py raises flyte.errors.ImageBuildError if the remote builder backend is disabled or if the build execution fails:

# Checking remote image builder availability
try:
    import flyte.remote as remote

    remote.Task.get(
        name=IMAGE_TASK_NAME,
        project=IMAGE_TASK_PROJECT,
        domain=IMAGE_TASK_DOMAIN,
        auto_version="latest",
    )
except Exception as e:
    msg = "remote image builder is not enabled. Please contact Union support to enable it."
    raise flyte.errors.ImageBuildError(msg) from e

If the remote build execution finishes in a state other than run_definition_pb2.PHASE_SUCCEEDED, an ImageBuildError is raised with the execution URL:

if run_details.action_details.raw_phase != run_definition_pb2.PHASE_SUCCEEDED:
    raise flyte.errors.ImageBuildError(f"❌ Build failed in {elapsed} at {run.url}")

---

Container Runtime and Execution Environment Errors

When tasks execute remotely on Kubernetes worker nodes, the control plane reports execution errors via execution_pb2.ExecutionError protobuf messages. The converter function convert_error_to_native in _internal/runtime/convert.py maps these protobuf error codes to native Python exception types.

                  ┌───────────────────────────────┐
                  │ execution_pb2.ExecutionError  │
                  └──────────────┬────────────────┘
                                 │
                     convert_error_to_native()
                                 │
     ┌───────────────────────────┼───────────────────────────┐
     ▼                           ▼                           ▼
"ImagePullBackOff"        "InvalidImageName"     "PrimaryContainerNotFound"
     │                           │                           │
     ▼                           ▼                           ▼
ImagePullBackOffError    InvalidImageNameError   PrimaryContainerNotFoundError

Protocol-to-Native Error Conversion

convert_error_to_native inspects err.code and instantiates the matching error class, preserving the code, message, and worker metadata:

# In _internal/runtime/convert.py
user_code, server_code = _clean_error_code(err.code)
match err.kind:
    case execution_pb2.ExecutionError.USER:
        if "PrimaryContainerNotFound" in err.code:
            return flyte.errors.PrimaryContainerNotFoundError(
                code=user_code, message=err.message, worker=err.worker
            )
        elif "InvalidImageName" in err.code:
            return flyte.errors.InvalidImageNameError(code=user_code, message=err.message, worker=err.worker)
        elif "ImagePullBackOff" in err.code:
            return flyte.errors.ImagePullBackOffError(code=user_code, message=err.message, worker=err.worker)
        return flyte.errors.RuntimeUserError(code=user_code, message=err.message, worker=err.worker)

Runtime Environment Exception Classes

Unlike DeploymentError, ImageBuildError, and ModuleLoadError (which take a single message argument), runtime environment exceptions inherit RuntimeUserError.__init__(self, code: str, message: str, worker: str | None = None):

• ImagePullBackOffError: Raised when a Kubernetes worker pod fails to pull the container image specified for the task (for example, due to private registry authentication errors, missing image tags, or network timeouts).
• InvalidImageNameError: Raised when the task's configured container image string contains invalid characters, an unsupported format, or an unparseable registry URI.
• PrimaryContainerNotFoundError: Raised when the pod specification generated for the task execution fails to define or expose the designated primary execution container.

import flyte.errors

def inspect_runtime_error(exc: flyte.errors.RuntimeUserError):
    print(f"Error Code:    {exc.code}")
    print(f"Error Message: {exc.message}")
    print(f"Error Kind:    {exc.kind}")
    print(f"Worker Node:   {exc.worker}")

---

Troubleshooting

CLI Module Load Failures

Symptom: flyte deploy aborts before registering tasks, outputting a table of failed module paths.

Loaded 3 modules with, but failed to load 2 paths:
Modules:
  Path: my_project/experimental.py | Err: Failed to load module from ...: No module named 'optional_pkg'
Failed to load 2 files. Use --ignore-load-errors to ignore these errors.

Solution:

1. Fix syntax or missing dependency imports in the reported files.
2. If non-essential scripts or optional plugin modules reside in the scanned folder, pass --ignore-load-errors to skip unloadable modules:

flyte deploy --ignore-load-errors

Missing Version with copy_style="none"

Symptom: flyte.errors.DeploymentError: Version must be set when copy_style is none

Solution: Provide an explicit version string in deployment arguments when --copy-style none is configured:

flyte deploy --copy-style none --version v1.2.0

Task Already Exists During Registration

Behavior: When DeployTask receives grpc.StatusCode.ALREADY_EXISTS, the deployment engine in _deploy.py logs an informational message and skips re-deploying that task without raising a DeploymentError:

Task my_task with image registry.example.com/app:v1 already exists, skipping deployment.

No mitigation is required; the deployment workflow continues to process subsequent tasks.