Project Overview

loom-kernel is a framework kernel for building backend systems with a common set of typed building blocks:

  • Core for models, commands, use cases, routing, logging, and shared infrastructure contracts.

  • REST for HTTP APIs, auto-CRUD, jobs, and FastAPI-based delivery.

  • ETL for declarative batch pipelines with backend-specific runners.

  • Streaming for typed topic-oriented flows with Kafka transport and a Bytewax runtime adapter.

The three public subsystems solve different problems, but they share the same design rules:

  • author with typed Python objects

  • resolve configuration from YAML

  • keep runtime concerns behind adapters

  • prefer explicit contracts over reflection-heavy magic

The companion demo repository for the streaming subsystem is dummy-loom-streaming.

At a glance

Area

What it is

Main runtime

Core

Shared kernel for models, commands, use cases, routing, logging, config, and engine primitives.

Internal kernel services

REST

Request/response application layer with typed use cases, auto-CRUD, jobs, and callbacks.

FastAPI + Celery

ETL

Declarative batch pipelines over tables and files, with backend-specific execution.

Polars + Spark

Streaming

Typed topic-oriented flows for microservices and event-driven processing.

Kafka + Bytewax

REST

REST is the request/response part of the kernel.

Use it when you want:

  • typed use cases with declarative dependencies

  • model-driven auto-CRUD

  • background jobs and callbacks

  • FastAPI delivery with minimal ceremony

The public entrypoint is loom.rest, and the quickstart is focused on models, interfaces, and use-case wiring.

REST also covers asynchronous/background work through Celery jobs and callbacks.

ETL

ETL is the batch-processing part of the kernel.

Use it when you want:

  • typed ETL parameters

  • declarative pipeline/process/step graphs

  • backend-specific execution on Polars or Spark

  • file and table targets with explicit write modes

The public entrypoint is loom.etl, and the quickstart is focused on ETLStep, ETLProcess, and ETLPipeline.

The two supported execution paths are:

  • Polars for local and lake-style batch processing

  • Spark for distributed batch processing

The model is table-centric: sources, targets, schemas, and write modes are the core abstractions.

Streaming

Streaming is the topic-oriented part of the kernel.

Use it when you want:

  • typed StreamFlow declarations

  • Kafka topics as sources and sinks

  • synchronous and asynchronous steps

  • fan-out, routing, and batch collection

  • execution through the Bytewax adapter

The public entrypoint is loom.streaming, with transport support in loom.streaming.kafka and runtime support in loom.streaming.bytewax.

Streaming is the right fit for microservices, fan-out, branching, and message-driven workflows.

The design goal is consistent authoring across domains:

  • declare behavior in Python types

  • resolve configuration from YAML

  • keep runtime adapters separate from authoring APIs

  • preserve strong typing and explicit contracts

How the parts fit together

Part

Authoring surface

Runtime surface

Core

loom.core.*

shared kernel services

REST

loom.rest.*

FastAPI adapters

ETL

loom.etl.*

Polars / Spark runners

Streaming

loom.streaming.*

Kafka / Bytewax adapters

Streaming in practice

Streaming is split into three public layers:

  1. loom.streaming for flow declarations.

  2. loom.streaming.kafka for transport contracts and codecs.

  3. loom.streaming.bytewax for the execution adapter.

The public flow authoring API uses StreamFlow, Process, FromTopic, IntoTopic, CollectBatch, With, WithAsync, Fork, Broadcast, and Router.

Documentation structure