The Pickling Fallback Mechanism
When TypeEngine cannot find a registered transformer for a Python type, flyte-sdk does not stop at type inference. After checking the type itself, generic origins, the method-resolution order, and dataclass handling, TypeEngine.get_transformer emits a pickle warning and returns a FlytePickleTransformer:
class TypeEngine:
@classmethod
def get_transformer(cls, python_type: Type) -> TypeTransformer:
v = cls._get_transformer(python_type)
if v is not None:
return v
if hasattr(python_type, "__mro__"):
class_tree = inspect.getmro(python_type)
for t in class_tree:
v = cls._get_transformer(t)
if v is not None:
return v
if dataclasses.is_dataclass(python_type):
return cls._DATACLASS_TRANSFORMER
display_pickle_warning(str(python_type))
from flyte.types._pickle import FlytePickleTransformer
return FlytePickleTransformer()
The dataclass check comes after the registered-type and MRO searches. The comment in types/_type_engine.py says this ordering gives users a chance to register a transformer for dataclass-like objects. Thus, pickle is the final fallback for an unregistered type, not the first transformer selected for every Python object.
TypeEngine.to_literal and TypeEngine.to_python_value use the selected transformer for ordinary conversion. to_literal optionally calls assert_type, invokes transformer.to_literal, and then applies modify_literal_uris; deserialization obtains a transformer for the expected Python type and calls its to_python_value.
class TypeEngine:
@classmethod
async def to_literal(
cls, python_val: typing.Any, python_type: Type[T], expected: types_pb2.LiteralType
) -> literals_pb2.Literal:
transformer = cls.get_transformer(python_type)
if transformer.type_assertions_enabled:
transformer.assert_type(python_type, python_val)
lv = await transformer.to_literal(python_val, python_type, expected)
modify_literal_uris(lv)
return lv
Consequently, application code normally reaches the pickle path by passing an unsupported type through the type engine rather than by constructing a FlytePickle value itself.
The fallback type and its dynamic generic form
FlytePickle is defined in types/_pickle.py and re-exported as types.FlytePickle. Its docstring explicitly describes it as an internal type: types that Flyte cannot recognize become FlytePickle, and users are told not to use the type directly.
The class has no instance state. The unspecialized class reports type(None) from python_type(). Subscription creates a subclass dynamically: _SpecificFormatClass inherits from FlytePickle, sets __origin__ = FlytePickle, and returns the Python type supplied to FlytePickle[T] from its own python_type() method. This gives the type engine a generic-shaped type while keeping one transformer registered for the fallback family.
class FlytePickle(typing.Generic[T]):
@classmethod
def python_type(cls) -> typing.Type:
return type(None)
@classmethod
def __class_getitem__(cls, python_type: typing.Type) -> typing.Type:
if python_type is None:
return cls
class _SpecificFormatClass(FlytePickle):
__origin__ = FlytePickle
@classmethod
def python_type(cls) -> typing.Type:
return python_type
return _SpecificFormatClass
types/_pickle.py registers FlytePickleTransformer() with TypeEngine at module import. The fallback branch also imports the transformer lazily and constructs one when lookup has failed, so the final decision remains explicit in TypeEngine.get_transformer.
Serialization lifecycle
FlytePickle.to_pickle uses cloudpickle.dumps to turn the complete value into bytes. It computes an MD5 digest from those bytes and uses the digest as the leaf name of a locally generated path. The method creates the parent directory, writes the bytes asynchronously with aiofiles, and uploads the local file through storage.put:
class FlytePickle(typing.Generic[T]):
@classmethod
async def to_pickle(cls, python_val: typing.Any) -> str:
h = hashlib.md5()
str_bytes = cloudpickle.dumps(python_val)
h.update(str_bytes)
uri = storage.get_random_local_path(file_path_or_file_name=h.hexdigest())
os.makedirs(os.path.dirname(uri), exist_ok=True)
async with aiofiles.open(uri, "w+b") as outfile:
await outfile.write(str_bytes)
return await storage.put(str(uri))
The call to storage.put does not provide a destination. The active Flyte raw-data context therefore supplies the remote path, so a task or run must have a usable raw-data/output storage configuration. The MD5 value names the local staging file; the method does not itself implement upload deduplication or an integrity check during deserialization.
from_pickle handles both local and remote URIs. For a remote URI it allocates another local path and calls storage.get; it then reads the complete file asynchronously and passes the bytes to cloudpickle.loads:
class FlytePickle(typing.Generic[T]):
@classmethod
async def from_pickle(cls, uri: str) -> typing.Any:
if storage.is_remote(uri):
local_path = storage.get_random_local_path()
await storage.get(uri, str(local_path), False)
uri = str(local_path)
async with aiofiles.open(uri, "rb") as infile:
data = cloudpickle.loads(await infile.read())
return data
Both methods are asynchronous because staging and object-storage operations are asynchronous. They also materialize the serialized payload in memory: to_pickle holds the result of cloudpickle.dumps, and from_pickle reads the complete file before calling cloudpickle.loads.
The PythonPickle literal contract
FlytePickleTransformer subclasses TypeTransformer[FlytePickle] and names its wire format "PythonPickle". Its to_literal method rejects None with AssertionError("Cannot pickle None Value."), delegates serialization to FlytePickle.to_pickle, and returns a scalar containing a single-part blob. The blob metadata records both the format and dimensionality:
class FlytePickleTransformer(TypeTransformer[FlytePickle]):
async def to_literal(
self,
python_val: T,
python_type: Type[T],
expected: types_pb2.LiteralType,
) -> literals_pb2.Literal:
if python_val is None:
raise AssertionError("Cannot pickle None Value.")
meta = literals_pb2.BlobMetadata(
type=types_pb2.BlobType(
format=self.PYTHON_PICKLE_FORMAT, dimensionality=types_pb2.BlobType.BlobDimensionality.SINGLE
)
)
remote_path = await FlytePickle.to_pickle(python_val)
return literals_pb2.Literal(scalar=literals_pb2.Scalar(blob=literals_pb2.Blob(metadata=meta, uri=remote_path)))
get_literal_type(t) describes the same single-part PythonPickle blob and adds {"python_class_name": str(t)} as literal metadata. That metadata records the string form of the source type, while the payload itself remains a cloudpickle byte stream.
On the reverse path, to_python_value reads lv.scalar.blob.uri and delegates to FlytePickle.from_pickle. It expects the scalar/blob shape produced by to_literal; a differently shaped literal is not converted into a custom pickle-specific error. guess_python_type is intentionally narrow: it returns FlytePickle only for a single-part blob whose format is exactly PythonPickle; other blob shapes or formats raise ValueError.
The transformer’s assert_type performs no validation:
class FlytePickleTransformer(TypeTransformer[FlytePickle]):
def assert_type(self, t: Type[T], v: T):
# Every type can serialize to pickle, so we don't need to check the type here.
...
This means passing type assertions does not prove that a particular runtime value can actually be serialized. A failure from cloudpickle.dumps can still occur later. None is the explicit exception handled by to_literal, so optionality should be represented through the type-system path rather than relying on a pickle payload for None.
Dictionary fallback is a separate encoding
Dictionaries do not automatically use the FlytePickleTransformer. DictTransformer normally emits MessagePack binary literals for the relevant dictionary representation. Its dict_to_binary_literal method catches TypeError from MessagePackEncoder.encode; only when allow_pickle is enabled does it call FlytePickle.to_pickle:
class DictTransformer:
@staticmethod
async def dict_to_binary_literal(v: dict, python_type: Type[dict], allow_pickle: bool) -> Literal:
try:
encoder = MessagePackEncoder(python_type)
msgpack_bytes = encoder.encode(v)
return Literal(scalar=Scalar(binary=Binary(value=msgpack_bytes, tag=MESSAGEPACK)))
except TypeError as e:
if allow_pickle:
remote_path = await FlytePickle.to_pickle(v)
return Literal(
scalar=Scalar(
generic=_json_format.Parse(json.dumps({"pickle_file": remote_path}), struct_pb2.Struct())
),
metadata={"format": "pickle"},
)
raise TypeTransformerFailedError(f"Cannot convert `{v}` to Flyte Literal.\nError Message: {e}")
For this path, the literal is a generic protobuf containing pickle_file, with literal metadata {"format": "pickle"}. It is not the single-part PythonPickle blob contract used by FlytePickleTransformer. During dictionary deserialization, DictTransformer.to_python_value checks for that metadata, extracts pickle_file, and calls FlytePickle.from_pickle. If pickle is not enabled, the original MessagePack conversion error is wrapped in TypeTransformerFailedError instead.
The allow_pickle decision is read from an Annotated dictionary type’s metadata: DictTransformer.is_pickle looks for an OrderedDict metadata value and returns its allow_pickle entry, defaulting to False. This makes dictionary pickling an explicit opt-in fallback after the normal dictionary encoding fails.
Integration with files and the CLI
The PythonPickle format is kept distinct from ordinary file handling. io/_file.py excludes blobs with format PythonPickle in FileTransformer.guess_python_type, so a pickle blob is not reverse-identified as a regular File.
The CLI makes the same distinction in cli/_params.py. For a single-part blob, it compares the format with FlytePickleTransformer.PYTHON_PICKLE_FORMAT and returns PickleParamType; other single-part blobs become FileParamType, while non-single blobs become DirParamType:
def literal_type_to_click_type(lt, python_type):
if lt.HasField("blob"):
if lt.blob.dimensionality == BlobType.BlobDimensionality.SINGLE:
if lt.blob.format == FlytePickleTransformer.PYTHON_PICKLE_FORMAT:
return PickleParamType()
return FileParamType()
return DirParamType()
Trade-offs and operational constraints
The package documentation in types/__init__.py states the central trade-off: pickle is not human-readable, can pass objects between Python-written Flyte tasks, cannot represent pickled objects in the UI, and may be inefficient for large datasets. The implementation explains the latter cost concretely: serialization creates the entire byte string in memory, writes the whole payload, and uploads it as one object.
The warning emitted by display_pickle_warning adds a compatibility constraint: pickle can only be used to send objects between the exact same version of Python, and the warning strongly recommends using a Python type supported by Flyte. This makes the fallback Python-specific rather than a portable representation for cross-language consumers.
A custom transformer or a supported Flyte type is therefore the better choice when the value must be portable, inspectable, rendered by the UI, or efficient at larger scale. TypeEngine.get_transformer checks registered transformers before dataclass handling and pickle fallback, so registering a purpose-built transformer lets the value use an explicit literal representation instead of silently taking the opaque PythonPickle path. The fallback is useful for Python-to-Python transport of otherwise unsupported values, but it trades representation and portability for convenience.