Status: experimental — proof of concept.
This is a research exploration. The API is unstable, multiple inheritance is unsupported, and critical bugs may still exist. Not recommended for production use.
Automatic memory optimization for Python classes. Apply @thrift and get __slots__, packed bools, interned strings, and compact int/float storage — no manual refactoring.
pip install git+https://github.com/varnie/thriftclass.gitOr from a local clone:
git clone https://github.com/varnie/thriftclass.git
cd thriftclass
pip install .from thriftclass import thrift
@thrift
class Point:
x: float
y: float
label: str
active: bool
visible: bool
p = Point()
p.x = 1.0; p.y = 2.0
p.label = "origin"
p.active = True; p.visible = False
Point.memory_report().show()| Strategy | Default | Effect |
|---|---|---|
__slots__ |
on | Eliminates per-instance __dict__ (~200+ bytes saved) |
| Bool packing | on | Packs multiple bool fields into a single integer bitfield |
| String interning | on | Deduplicates repeated string values via sys.intern() |
| Compact ints | on | Stores int fields in a bytearray buffer instead of Python objects (28→8 bytes each) |
| Compact floats | on | Stores float fields in a bytearray buffer instead of Python objects (24→8 bytes each) |
| Adaptive | off | Monitors real data, then reports — or auto-applies — optimal types (e.g. int16 instead of int64) |
thriftclass is a memory-first library. By replacing standard Python attributes with properties backed by bytearray and struct.pack, it drastically reduces RAM usage but introduces a CPU overhead on attribute access (getter/setter).
- RAM: Up to 40-60% savings for large collections of objects.
- CPU: Attribute reads/writes are 2-4x slower than standard
__slots__.
Always benchmark your specific use case to ensure the CPU overhead is acceptable for your performance budget.
- ✅ YES: You have millions of tiny, interconnected objects with complex logic (e.g., in-memory graphs, game entity trees, custom caches) and your app is hitting RAM limits.
- ❌ NO (Simple Data Transfer): For regular API payloads or configuration, prefer standard
dataclasses.dataclass(slots=True)orpydantic. - ❌ NO (Heavy Math/Matrices): If you are doing heavy numeric matrix calculations, use
numpyorpandas. They store data in pure C-arrays much more efficiently and perform operations at native C speed.
@thrift(slots=True, pack_bools=True, intern_strings=True,
compact_ints=True, compact_floats=True)
class MyClass:
...
@thrift(slots=False) # disable __slots__
@thrift(pack_bools=False) # keep bools as regular attributes
@thrift(compact_ints=False) # keep ints as Python objects
@thrift(compact_floats=False) # keep floats as Python objects
@thrift(check_overflow=True) # raise OverflowError on int out of range@thrift(adaptive=True, adaptive_sample=100)
class HttpRequest:
method: str
status: int
duration: float
path: str
# create real instances — the monitor collects statistics
for _ in range(120):
r = HttpRequest()
r.method = ...
...
# view analysis
report = HttpRequest.__adaptive_monitor__.get_report()
# rebuild with optimal types (int16, float32, etc.)
HttpRequest = HttpRequest.optimize()from dataclasses import dataclass
@thrift
@dataclass
class Config:
host: str = "localhost"
port: int = 8080
debug: bool = False
verbose: bool = False@thrift
class Node:
x: float
y: float
z: float
visited: bool
active: bool
label: str
depth: int
Node.memory_report().show()Output:
┌──────────────────────────────────────────────────────────┐
│ thriftclass — Node │
├──────────────────────────────────────────────────────────┤
│ Memory per instance: │
│ Before : 344 bytes │
│ After : 214 bytes │
│ Saved : 130 bytes (37.8%) │
├──────────────────────────────────────────────────────────┤
│ Optimizations applied: │
│ ✓ __slots__ eliminates per-object __dict__ │
│ ✓ compact ints/floats stored in bytearray buffer │
│ ✓ bool → bitfield packs bool fields into one int │
│ ✓ str interning interned strings share memory │
├──────────────────────────────────────────────────────────┤
│ Fields: │
│ x → float64 (8 bytes in buffer) │
│ y → float64 (8 bytes in buffer) │
│ visited → packed into bitfield │
│ label → interned │
│ depth → int64 (8 bytes in buffer) │
└──────────────────────────────────────────────────────────┘
Six strategies run in sequence, each transforming the class and passing it to the next:
Recreates the class via type(name, bases, namespace) with __slots__ set to all annotated field names. The per-instance __dict__ (~40% of instance memory) is removed. Old class members are copied into the new namespace. For dataclasses on Python 3.10+, uses dataclass(slots=True) to preserve defaults.
Each int/float field becomes a property whose getter/setter call struct.pack_into/unpack_from on a shared bytearray. A Python int (~28 B) becomes an 8‑byte int64; a float (~24 B) becomes an 8‑byte float64. The buffer is lazily initialized on first access with resize-if-too-small logic for inheritance. Field order and offset tracking ensure parent and child fields don't overlap.
Bool slots are replaced with property descriptors that read/write individual bits of a single _bool_flags integer slot via bitmasks (self._bool_flags |= 1 << bit). Inheritance assigns unique bit offsets per level by querying the parent's __thrift_bit_map__, so child bits don't collide with parent bits. Only classes with ≥2 bool fields are packed.
Patches __setattr__ and __init__ in-place (no class recreation, avoiding __slots__ conflicts). Every string value assigned to an interned field is run through sys.intern(), so identical strings (e.g. "INFO" across thousands of log entries) share one object. Safe for low-cardinality fields; high-cardinality fields (timestamps, UUIDs) leak memory since interned strings live for the process lifetime.
Wraps the class to collect field values during normal usage. After enough samples, analyses ranges and cardinality, then rebuilds the class with tighter types (e.g. int16 instead of int64, float32 instead of float64, interning for low-cardinality strings). Call .optimize() on the class to get the rebuilt version — existing instances keep the old layout.
Prints a bordered table showing bytes before/after, which strategies applied, and per-field storage details. Available as a classmethod on any thriftified class.
- Inheritance: parent and child share a single compact buffer for
int/floatfields. Bool packing uses independent bit offsets per level, so bools don't collide. Avoid overriding__init__manually in thriftified subclasses. - String interning: high-cardinality strings (timestamps, UUIDs, unique messages) accumulate in the intern table forever — they're never GC'd. Disable with
intern_strings=Falsefor such fields. - Overflow:
intvalues beyondint64range (±9.2×10¹⁸) raisestruct.error. Enablecheck_overflow=Truefor early detection. - Dataclass: works when
@thriftis placed above@dataclass(@thrift/@dataclass). The opposite order may fail because thrift modifies annotations before dataclass processes them. - The adaptive monitor rebuilds the class via a new type — existing instances keep the old layout. Call
.optimize()before creating production instances.
pip install -e ".[dev]"
python -m pytest tests/ -v