Serializing Celery tasks with orjson

I came across this Celery GitHub issue today. The OP was wondering if it’s currently possible to configure Celery with support for orjson serializer.

As it turns out, it is not possible out of the box. And the reasoning provided is to prevent yet another dependency from being added to the already growing list of Celery dependencies.

The OP is posting the following reasons for wanting orjson support:

• The default Python JSON module is slow
• It has no support for dataclass, datetime, numpy, UUID types

Both of these issues have affected my Celery implementations as well. Less of an issue due to performance, because last I checked I am already using Python. 😂

But moreso due to the missing support for datetime and UUID. These are quite common considerations in just about every Python codebase these days.

All jokes aside, I thought I’d share a working example of orjson serialization in Celery. Cheers!

# system imports
from typing import Any

# dependency imports
from celery import Celery
from kombu import serialization

import orjson

def initialize_orjson_serializer(celery: Celery) -> None:
    def deserialize(data: bytes | str) -> tuple[Any, Any, Any]:
        return orjson.loads(data)

    def serialize(data: Any) -> str:
        return orjson.dumps(data).decode("utf-8")

    # accept tasks serialized with the following content types
    celery.conf.accept_content = ["json"]

    # accept results serialized with the following content types
    celery.conf.result_accept_content = ["json"]

    # celery will serialize outgoing tasks with this content type
    celery.conf.task_serializer = "json"

    # register json as a content type
    serialization.register(
        "json",
        serialize,
        deserialize,
        "json",
    )