Tutorial#

Adaptix analyzes your type hints and generates corresponding transformers based on the retrieved information. You can flexibly tune the conversion process following DRY principle.

Installation#

Just use pip to install the library

pip install adaptix==3.0.0b5

Integrations with 3-rd party libraries are turned on automatically, but you can install adaptix with extras to check that versions are compatible.

There are two variants of extras. The first one checks that the version is the same or newer than the last supported, the second (strict) additionally checks that the version same or older than the last tested version.

Extras	Versions bound
`attrs`	`attrs >= 21.3.0`
`attrs-strict`	`attrs >= 21.3.0, <= 23.2.0`
`sqlalchemy`	`sqlalchemy >= 2.0.0`
`sqlalchemy-strict`	`sqlalchemy >= 2.0.0, <= 2.0.29`
`pydantic`	`pydantic >= 2.0.0`
`pydantic-strict`	`pydantic >= 2.0.0, <= 2.7.0`

Extras are specified inside square brackets, separating by comma.

So, this is valid installation variants:

pip install adaptix[attrs-strict]==3.0.0b5
pip install adaptix[attrs, sqlalchemy-strict]==3.0.0b5

Introduction#

The central object of the library is Retort. It can create models from mapping (loading) and create mappings from the model (dumping).

from dataclasses import dataclass

from adaptix import Retort


@dataclass
class Book:
    title: str
    price: int


data = {
    "title": "Fahrenheit 451",
    "price": 100,
}

# Retort is meant to be global constant or just one-time created
retort = Retort()

book = retort.load(data, Book)
assert book == Book(title="Fahrenheit 451", price=100)
assert retort.dump(book) == data

All typing information is retrieved from your annotations, so is not required from you to provide any additional schema or even change your dataclass decorators or class bases.

In the provided example book.author == "Unknown author" because normal dataclass constructor is called.

It is better to create a retort only once because all loaders are cached inside it after the first usage. Otherwise, the structure of your classes will be analyzed again and again for every new instance of Retort.

If you don’t need any customization, you can use the predefined load and dump functions.

Nested objects#

Nested objects are supported out of the box. It is surprising, but you do not have to do anything except define your dataclasses. For example, you expect that the author of the Book is an instance of a Person, but in the dumped form it is a dictionary.

Declare your dataclasses as usual and then just load your data.

from dataclasses import dataclass

from adaptix import Retort


@dataclass
class Person:
    name: str


@dataclass
class Book:
    title: str
    price: int
    author: Person


data = {
    "title": "Fahrenheit 451",
    "price": 100,
    "author": {
        "name": "Ray Bradbury",
    },
}

retort = Retort()

book: Book = retort.load(data, Book)
assert book == Book(title="Fahrenheit 451", price=100, author=Person("Ray Bradbury"))
assert retort.dump(book) == data

Lists and other collections#

Want to load a collection of dataclasses? No changes are required, just specify the correct target type (e.g List[SomeClass] or Dict[str, SomeClass]).

from dataclasses import dataclass
from typing import List

from adaptix import Retort


@dataclass
class Book:
    title: str
    price: int


data = [
    {
        "title": "Fahrenheit 451",
        "price": 100,
    },
    {
        "title": "1984",
        "price": 100,
    },
]

retort = Retort()
books = retort.load(data, List[Book])
assert books == [Book(title="Fahrenheit 451", price=100), Book(title="1984", price=100)]
assert retort.dump(books, List[Book]) == data

Fields also can contain any supported collections.

Retort configuration#

There are two parameters that Retort constructor takes.

debug_trail is responsible for saving the place where the exception was caused. By default, retort saves all raised errors (including unexpected ones) and the path to them. If data is loading or dumping from a trusted source where an error is unlikely, you can change this behavior to saving only the first error with trail or without trail. It will slightly improve performance if no error is caused and will have more impact if an exception is raised. More details about working with the saved trail in Error handling

strict_coercion affects only the loading process. If it is enabled (this is the default state) type will be converted only two conditions passed:

There is only one way to produce casting
No information will be lost

So this mode forbids converting dict to list (dict values will be lost), forbids converting str to int (we do not know which base must be used), but allows to converting str to Decimal (base always is 10 by definition).

Strict coercion requires additional type checks before calling the main constructor, therefore disabling it can improve performance.

Retort recipe#

Retort also supports a more powerful and more flexible configuration system via recipe. It implements chain-of-responsibility design pattern. The recipe consists of providers, each of which can precisely override one of the retort’s behavior aspects.

from dataclasses import dataclass
from datetime import datetime, timezone

from adaptix import Retort, loader


@dataclass
class Book:
    title: str
    price: int
    created_at: datetime


data = {
    "title": "Fahrenheit 451",
    "price": 100,
    "created_at": 1674938508.599962,
}

retort = Retort(
    recipe=[
        loader(datetime, lambda x: datetime.fromtimestamp(x, tz=timezone.utc)),
    ],
)

book = retort.load(data, Book)
assert book == Book(
    title="Fahrenheit 451",
    price=100,
    created_at=datetime(2023, 1, 28, 20, 41, 48, 599962, tzinfo=timezone.utc),
)

Default datetime loader accepts only str in ISO 8601 format, loader(datetime, lambda x: datetime.fromtimestamp(x, tz=timezone.utc)) replaces it with a specified lambda function that takes int representing Unix time.

Providers at the start of the recipe have higher priority because they overlap subsequent ones.

from dataclasses import dataclass

from adaptix import Retort, loader


@dataclass
class Foo:
    value: int


def add_one(data):
    return data + 1


def add_two(data):
    return data + 2


retort = Retort(
    recipe=[
        loader(int, add_one),
        loader(int, add_two),
    ],
)

assert retort.load({"value": 10}, Foo) == Foo(11)

Basic providers overview#

The list of providers is not limited to loader and dumper, there are a lot of other high-level helpers. Here are some of them.

constructor creates a loader that extracts data from dict and passes it to the given function.
name_mapping renames and skips model fields for the outside world. You can change the naming convention to camelCase via the name_style parameter or rename individual fields via map.
with_property allows dumping properties of the model like other fields.
enum_by_exact_value is the default behavior for all enums. It uses enum values without any conversions to represent enum cases.
enum_by_name allows representing enums by their names.
enum_by_value takes the type of enum values and uses it to load or dump enum cases.

Predicate system#

So far all examples use classes to apply providers but you can specify other conditions. There is a single predicate system that is used by most of the builtins providers.

Basic rules:

If you pass a class, the provider will be applied to all same types.
If you pass an abstract class, the provider will be applied to all subclasses.
If you pass a runtime checkable protocol, the provider will be applied to all protocol implementations.
If you pass a string, it will be interpreted as a regex and the provider will be applied to all fields with id matched by the regex. In most cases, field_id is the name of the field at class definition. Any field_id must be a valid python identifier, so if you pass the field_id directly, it will match an equal string.

Using string directly for predicate often is inconvenient because it matches fields with the same name in all models. So there special helper for this case.

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import List

from adaptix import P, Retort, loader


@dataclass
class Person:
    id: int
    name: str
    created_at: datetime


@dataclass
class Book:
    name: str
    price: int
    created_at: datetime


@dataclass
class Bookshop:
    workers: List[Person]
    books: List[Book]


data = {
    "workers": [
        {
            "id": 193,
            "name": "Kate",
            "created_at": "2023-01-29T21:26:28.026860+00:00",
        },
    ],
    "books": [
        {
            "name": "Fahrenheit 451",
            "price": 100,
            "created_at": 1674938508.599962,
        },
    ],
}

retort = Retort(
    recipe=[
        loader(P[Book].created_at, lambda x: datetime.fromtimestamp(x, tz=timezone.utc)),
    ],
)

bookshop = retort.load(data, Bookshop)

assert bookshop == Bookshop(
    workers=[
        Person(
            id=193,
            name="Kate",
            created_at=datetime(2023, 1, 29, 21, 26, 28, 26860, tzinfo=timezone.utc),
        ),
    ],
    books=[
        Book(
            name="Fahrenheit 451",
            price=100,
            created_at=datetime(2023, 1, 28, 20, 41, 48, 599962, tzinfo=timezone.utc),
        ),
    ],
)

P represents pattern of path at structure definition. P[Book].created_at will match field created_at only if it placed inside model Book

Some facts about P:

P['name'] is the same as P.name
P[Foo] is the same as Foo predicate
P[Foo] + P.name is the same as P[Foo].name
P[Foo, Bar] matches class Foo or class Bar
P could be combined via |, &, ^, also it can be reversed using ~
P can be expanded without limit. P[Foo].name[Bar].age is valid and matches field age located at model Bar, situated at field name, placed at model Foo

Retort extension and combination#

No changes can be made after the retort creation. You can only make new retort object based on the existing one

replace method using to change scalar options debug_trail and strict_coercion

from adaptix import DebugTrail, Retort

external_retort = Retort(
    recipe=[
        # very complex configuration
    ],
)

# create retort to faster load data from an internal trusted source
# where it already validated
internal_retort = external_retort.replace(
    strict_coercion=False,
    debug_trail=DebugTrail.DISABLE,
)

extend method adds items to the recipe beginning. This allows following the DRY principle.

from datetime import datetime

from adaptix import Retort, dumper, loader

base_retort = Retort(
    recipe=[
        loader(datetime, datetime.fromtimestamp),
        dumper(datetime, datetime.timestamp),
    ],
)

specific_retort1 = base_retort.extend(
    recipe=[
        loader(bytes, bytes.hex),
        loader(bytes, bytes.fromhex),
    ],
)

# same as

specific_retort2 = Retort(
    recipe=[
        loader(bytes, bytes.hex),
        loader(bytes, bytes.fromhex),
        loader(datetime, datetime.fromtimestamp),
        dumper(datetime, datetime.timestamp),
    ],
)

You can include one retort to another, it allows to separate creation of loaders and dumpers for specific types into isolated layers.

from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum
from typing import List

from adaptix import Retort, bound, dumper, enum_by_name, loader


class LiteraryGenre(Enum):
    DRAMA = 1
    FOLKLORE = 2
    POETRY = 3
    PROSE = 4


@dataclass
class LiteraryWork:
    id: int
    name: str
    genre: LiteraryGenre
    uploaded_at: datetime


literature_retort = Retort(
    recipe=[
        loader(datetime, lambda x: datetime.fromtimestamp(x, tz=timezone.utc)),
        dumper(datetime, lambda x: x.timestamp()),
        enum_by_name(LiteraryGenre),
    ],
)


# another module and another abstraction level

@dataclass
class Person:
    name: str
    works: List[LiteraryWork]


retort = Retort(
    recipe=[
        bound(LiteraryWork, literature_retort),
    ],
)

data = {
    "name": "Ray Bradbury",
    "works": [
        {
            "id": 7397,
            "name": "Fahrenheit 451",
            "genre": "PROSE",
            "uploaded_at": 1675111113,
        },
    ],
}

person = retort.load(data, Person)
assert person == Person(
    name="Ray Bradbury",
    works=[
        LiteraryWork(
            id=7397,
            name="Fahrenheit 451",
            genre=LiteraryGenre.PROSE,
            uploaded_at=datetime(2023, 1, 30, 20, 38, 33, tzinfo=timezone.utc),
        ),
    ],
)

In this example, loader and dumper for LiteraryWork will be created by literature_retort (note that debug_trail and strict_coercion options of upper-level retort do not affects inner retorts).

Retort is provider that proxies search into their own recipe, so if you pass retort without a bound wrapper, it will be used for all loaders and dumpers, overriding all subsequent providers.

Provider chaining#

Sometimes you want to add some additional data processing before or after the existing converter instead of fully replacing it. This is called chaining.

The third parameter of loader and dumper control the chaining process. Chain.FIRST means that the result of the given function will be passed to the next matched loader/dumper at the recipe, Chain.LAST marks to apply your function after the one generated by the next provider.

import json
from dataclasses import dataclass
from datetime import datetime

from adaptix import Chain, P, Retort, dumper, loader


@dataclass
class Book:
    title: str
    price: int
    author: str


@dataclass
class Message:
    id: str
    timestamp: datetime
    body: Book


data = {
    "id": "ajsVre",
    "timestamp": "2023-01-29T21:26:28.026860",
    "body": '{"title": "Fahrenheit 451", "price": 100, "author": "Ray Bradbury"}',
}

retort = Retort(
    recipe=[
        loader(P[Message].body, json.loads, Chain.FIRST),
        dumper(P[Message].body, json.dumps, Chain.LAST),
    ],
)

message = retort.load(data, Message)
assert message == Message(
    id="ajsVre",
    timestamp=datetime(2023, 1, 29, 21, 26, 28, 26860),
    body=Book(
        title="Fahrenheit 451",
        price=100,
        author="Ray Bradbury",
    ),
)

Validators#

validator is a convenient wrapper over loader and chaining to create a verifier of input data.

from dataclasses import dataclass

from adaptix import P, Retort, validator
from adaptix.load_error import AggregateLoadError, LoadError, ValidationLoadError


@dataclass
class Book:
    title: str
    price: int


data = {
    "title": "Fahrenheit 451",
    "price": -10,
}

retort = Retort(
    recipe=[
        validator(P[Book].price, lambda x: x >= 0, "value must be greater or equal 0"),
    ],
)

try:
    retort.load(data, Book)
except AggregateLoadError as e:
    assert len(e.exceptions) == 1
    assert isinstance(e.exceptions[0], ValidationLoadError)
    assert e.exceptions[0].msg == "value must be greater or equal 0"


class BelowZeroError(LoadError):
    def __init__(self, actual_value: int):
        self.actual_value = actual_value

    def __str__(self):
        return f"actual_value={self.actual_value}"


retort = Retort(
    recipe=[
        validator(P[Book].price, lambda x: x >= 0, lambda x: BelowZeroError(x)),
    ],
)

try:
    retort.load(data, Book)
except AggregateLoadError as e:
    assert len(e.exceptions) == 1
    assert isinstance(e.exceptions[0], BelowZeroError)
    assert e.exceptions[0].actual_value == -10

If the test function returns False, the exception will be raised. You can pass an exception factory that returns the actual exception or pass the string to raise ValidationError instance.

Error handling#

All loaders have to throw LoadError to signal invalid input data. Other exceptions mean errors at loaders themselves. All builtin LoadError children have listed at adaptix.load_error subpackage and designed to produce machine-readable structured errors.

from dataclasses import dataclass

from adaptix import Retort
from adaptix.load_error import AggregateLoadError, LoadError


@dataclass
class Book:
    title: str
    price: int
    author: str = "Unknown author"


data = {
    # Field values are mixed up
    "title": 100,
    "price": "Fahrenheit 451",
}

retort = Retort()

try:
    retort.load(data, Book)
except LoadError as e:
    assert isinstance(e, AggregateLoadError)

By default, all thrown errors are collected into AggregateLoadError, each exception has an additional note describing path of place where the error is caused. This path is called a Struct trail and acts like JSONPath pointing to location inside the input data.

For Python versions less than 3.11, an extra package exceptiongroup is used. This package patch some functions from traceback during import to backport ExceptionGroup rendering to early versions. More details at documentation.

By default, all collection-like and model-like loaders wrap all errors into AggregateLoadError. Each sub-exception contains a trail relative to the parent exception.

Non-guaranteed behavior

Order of errors inside AggregateLoadError is not guaranteed.

You can set debug_trail=DebugTrail.FIRST at Retort to raise only the first met error.

Changing debug_trail to DebugTrail.DISABLE make the raised exception act like any normal exception.

If there is at least one unexpected error AggregateLoadError is replaced by standard ExceptionGroup. For the dumping process any exception is unexpected, so it always will be wrapped with ExceptionGroup

from dataclasses import dataclass
from datetime import datetime

from adaptix import Retort, loader
from adaptix.struct_trail import Attr, get_trail


@dataclass
class Book:
    title: str
    price: int
    created_at: datetime


data = {
    "title": "Fahrenheit 451",
    "price": 100,
    "created_at": "2023-10-07T16:25:19.303579",
}


def broken_title_loader(data):
    raise ArithmeticError("Some unexpected error")


retort = Retort(
    recipe=[
        loader("title", broken_title_loader),
    ],
)

try:
    retort.load(data, Book)
except Exception as e:
    assert isinstance(e, ExceptionGroup)
    assert len(e.exceptions) == 1
    assert isinstance(e.exceptions[0], ArithmeticError)
    assert list(get_trail(e.exceptions[0])) == ["title"]

book = Book(
    title="Fahrenheit 451",
    price=100,
    created_at=None,  # type: ignore[arg-type]
)

try:
    retort.dump(book)
except Exception as e:
    assert isinstance(e, ExceptionGroup)
    assert len(e.exceptions) == 1
    assert isinstance(e.exceptions[0], TypeError)
    assert list(get_trail(e.exceptions[0])) == [Attr("created_at")]

Trail of exception is stored at a special private attribute and could be accessed via get_trail.

As you can see, trail elements after dumping are wrapped in Attr. It is necessary because str or int instances mean that data can be accessed via [].