nirs4all-io¶
The dataset-assembly bridge of the nirs4all ecosystem. nirs4all-io turns a
user input — a directory, a list of files, or a config dict — into a
pipeline-ready dataset. It owns the dataset-level concepts (X / Y / metadata
roles, train / test / folds, multi-source, relational joins, signal / task-type
inference, and a declarative convention system) and produces the
SpectroDataset shape that the main nirs4all
library models.
By architectural design it never decodes vendor file bytes itself: vendor
spectroscopy reads are delegated to
nirs4all-formats. It also
has no runtime dependency on nirs4all — the only touch-point is a lazy
import of the SpectroDataset class at materialization time.
Note
The current published wheel reads the CSV family of tabular inputs; the
vendor-format reader path (delegated to nirs4all-formats, never re-parsed
here) and additional tabular backends (numpy / Parquet / Excel) land with the
broader load path.
The pipeline¶
Every input flows through the same four stages:
any input ──► RESOLVE ──► INFER ──► CONFIGURE ──► MATERIALIZE ──► SpectroDataset
(InputSet) (DatasetPlan, scored) (DatasetSpec)
RESOLVE — normalize whatever you passed (path, glob, list, dict, arrays) into a concrete
InputSet.INFER — inspect the inputs and propose a confidence-scored
DatasetPlan(roles, columns, structure, signal/task type), with an evidence trace per decision. Scores are for ranking/triage, not calibrated probabilities.CONFIGURE — settle on a versioned, machine-validatable
DatasetSpec, the canonical contract.MATERIALIZE — build the target: the assembled structural summary (default), or a
nirs4allSpectroDataset.
Quickstart¶
import nirs4all_io as nio
# Inspect a directory and get a scored recommendation (DatasetPlan)
plan = nio.infer("data/mango/", conventions=["nirs4all-classic"])
print(plan.recommendations)
print(plan.overall_score)
# Materialize the assembled structural summary (no nirs4all needed)
summary = nio.load("data/mango/", conventions=["nirs4all-classic"])
# Or build a real nirs4all SpectroDataset
ds = nio.load("data/mango/", target="spectrodataset", conventions=["nirs4all-classic"])
# An explicit DatasetSpec / config dict also works
summary = nio.load({"sources": [{"id": "x", "role": "features", "input": "X.csv"}]})
See Getting started for a fuller walk-through and Installation to install the package.
The nirs4all ecosystem¶
Main Python modelling library — pipelines, SpectroDataset, predictions.
Rust readers for ~58 NIRS/spectroscopy file formats (nirs4all-io reads vendor files through this).
Portable C-ABI PLS/NIRS engine (libn4m) + bindings.
Curated DOI-pinned NIRS dataset catalog (n4a-datasets).
Canonical portable aggregate distribution (Rust, Python, R, WASM, MATLAB/Octave).
Reproducible, OOF/leakage-safe ML coordinator.
Typed sample-aligned multi-source data contracts.