loom.etl.runner

Runner API for compiling and executing ETL pipelines.

class loom.etl.runner.ETLRunner(reader, writer, observers=(), dispatcher=None, checkpoint_store=None)[source]

Bases: object

Wire storage config, backend I/O, and executor into one callable.

Parameters:

  • reader (SourceReader) – Source reader implementation.

  • writer (TargetWriter) – Target writer implementation.

  • observers (Sequence[ETLRunObserver]) – Lifecycle observers wrapped in a composite observer.

  • dispatcher (ParallelDispatcher | None) – Parallel task dispatcher.

  • checkpoint_store (CheckpointStore | None)

classmethod from_config(config, obs_config=None, *, spark=None, dispatcher=None, cleaner=None)[source]

Build an ETLRunner from resolved config objects.

Parameters:
Return type:

ETLRunner

classmethod from_yaml(path, *, spark=None, dispatcher=None)[source]

Load config from a YAML file and build an ETLRunner.

Parameters:

  • path (str)

  • spark (SparkSession | None)

  • dispatcher (ParallelDispatcher | None)

Return type:

ETLRunner

classmethod from_spark(spark, obs_config=None, *, dispatcher=None, cleaner=None)[source]

Build an ETLRunner using Spark as the execution engine.

Parameters:
Return type:

ETLRunner

classmethod from_dict(storage, observability=None, *, spark=None, dispatcher=None, cleaner=None)[source]

Build an ETLRunner from pre-resolved plain Python dicts.

Parameters:

  • storage (dict[str, Any])

  • observability (dict[str, Any] | None)

  • spark (SparkSession | None)

  • dispatcher (ParallelDispatcher | None)

  • cleaner (TempCleaner | None)

Return type:

ETLRunner

run(pipeline, params, *, include=None, correlation_id=None, attempt=1, last_attempt=True)[source]

Compile, optionally filter, and execute pipeline.

Parameters:
Return type:

None

cleanup_correlation(correlation_id)[source]

Remove all CORRELATION-scope intermediates for correlation_id.

Parameters:

correlation_id (str)

Return type:

None

exception loom.etl.runner.InvalidStageError(include)[source]

Bases: ValueError

Raised when include matches no step or process in the compiled plan.

Parameters:

include (frozenset[str]) – The set of names that produced no match.

Return type:

None