data.manydataset

data.manydataset

Variable-length sequence datasets and padding collates for recurrent models.

Ported from spotPython (spotpython.data.manydataset and the padding collates from spotpython.data.lightdatamodule) without any Lightning dependency. Each dataset item is one complete sequence (e.g. one operating curve of a compressor map), so batches must be padded with the collate classes provided here.

Requires the torch optional extra (pip install 'spotoptim[torch]').

Classes

Name	Description
ManyToManyDataset	Sequence dataset with one target value per time step.
ManyToOneDataset	Sequence dataset with a single target value per sequence.
PadSequenceManyToMany	Padding collate for `ManyToManyDataset` batches.
PadSequenceManyToOne	Padding collate for `ManyToOneDataset` batches.

ManyToManyDataset

data.manydataset.ManyToManyDataset(
    df_list,
    target,
    drop=None,
    dtype=torch.float32,
)

Sequence dataset with one target value per time step.

Each element of df_list is one variable-length sequence; item i is the pair (features_i, targets_i) with shapes (T_i, n_features) and (T_i,).

Parameters

Name	Type	Description	Default
df_list	List[pd.DataFrame]	List of pandas DataFrames, one per sequence.	required
target	str	The target column name.	required
drop	Optional[Union[str, List[str]]]	Column(s) to drop from the DataFrames before extracting features. If a listed column is missing, no column is dropped. Defaults to None.	`None`
dtype	`torch`.`dtype`	Data type for the tensors. Defaults to `torch.float32`.	`torch.float32`

Attributes

Name	Type	Description
data	List[pd.DataFrame]	DataFrames with the `drop` columns removed.
target	List[`torch`.`Tensor`]	Per-sequence target tensors, shape `(T_i,)`.
features	List[`torch`.`Tensor`]	Per-sequence feature tensors, shape `(T_i, n_features)`.

Examples

import pandas as pd
from spotoptim.data.manydataset import ManyToManyDataset

df1 = pd.DataFrame({"x": [1.0, 2.0, 3.0], "y": [2.0, 4.0, 6.0]})
df2 = pd.DataFrame({"x": [4.0, 5.0], "y": [8.0, 10.0]})
ds = ManyToManyDataset([df1, df2], target="y")
print(len(ds))
features, targets = ds[0]
print(features.shape, targets.shape)

2
torch.Size([3, 1]) torch.Size([3])

ManyToOneDataset

data.manydataset.ManyToOneDataset(
    df_list,
    target,
    drop=None,
    dtype=torch.float32,
)

Sequence dataset with a single target value per sequence.

Like ManyToManyDataset, but item i pairs the full feature sequence with the scalar target taken from the first row of sequence i.

Parameters

Name	Type	Description	Default
df_list	List[pd.DataFrame]	List of pandas DataFrames, one per sequence.	required
target	str	The target column name.	required
drop	Optional[Union[str, List[str]]]	Column(s) to drop from the DataFrames before extracting features. If a listed column is missing, no column is dropped. Defaults to None.	`None`
dtype	`torch`.`dtype`	Data type for the tensors. Defaults to `torch.float32`.	`torch.float32`

Attributes

Name	Type	Description
data	List[pd.DataFrame]	DataFrames with the `drop` columns removed.
target	List[`torch`.`Tensor`]	Per-sequence scalar target tensors.
features	List[`torch`.`Tensor`]	Per-sequence feature tensors, shape `(T_i, n_features)`.

Examples

import pandas as pd
from spotoptim.data.manydataset import ManyToOneDataset

df1 = pd.DataFrame({"x": [1.0, 2.0, 3.0], "y": [5.0, 5.0, 5.0]})
df2 = pd.DataFrame({"x": [4.0, 5.0], "y": [7.0, 7.0]})
ds = ManyToOneDataset([df1, df2], target="y")
features, target = ds[1]
print(features.shape, target)

torch.Size([2, 1]) tensor(7.)

PadSequenceManyToMany

data.manydataset.PadSequenceManyToMany()

Padding collate for ManyToManyDataset batches.

Pads features and targets of a batch of variable-length sequences with zeros to the longest sequence in the batch and records the true lengths, as required by torch.nn.utils.rnn.pack_padded_sequence.

Examples

import pandas as pd
from torch.utils.data import DataLoader
from spotoptim.data.manydataset import ManyToManyDataset, PadSequenceManyToMany

df1 = pd.DataFrame({"x": [1.0, 2.0, 3.0], "y": [2.0, 4.0, 6.0]})
df2 = pd.DataFrame({"x": [4.0, 5.0], "y": [8.0, 10.0]})
ds = ManyToManyDataset([df1, df2], target="y")
dl = DataLoader(ds, batch_size=2, shuffle=False, collate_fn=PadSequenceManyToMany())
x, lengths, y = next(iter(dl))
print(x.shape, lengths.tolist(), y.shape)

torch.Size([2, 3, 1]) [3, 2] torch.Size([2, 3])

PadSequenceManyToOne

data.manydataset.PadSequenceManyToOne()

Padding collate for ManyToOneDataset batches.

Pads the feature sequences with zeros and stacks the scalar targets.

Examples

import pandas as pd
from torch.utils.data import DataLoader
from spotoptim.data.manydataset import ManyToOneDataset, PadSequenceManyToOne

df1 = pd.DataFrame({"x": [1.0, 2.0, 3.0], "y": [5.0, 5.0, 5.0]})
df2 = pd.DataFrame({"x": [4.0, 5.0], "y": [7.0, 7.0]})
ds = ManyToOneDataset([df1, df2], target="y")
dl = DataLoader(ds, batch_size=2, shuffle=False, collate_fn=PadSequenceManyToOne())
x, lengths, y = next(iter(dl))
print(x.shape, lengths.tolist(), y.shape)

torch.Size([2, 3, 1]) [3, 2] torch.Size([2])

Functions

Name	Description
load_pooled_sequence_data	Pool several DataFrames into one concatenated sequence dataset.
load_sequence_data	Group a DataFrame into a variable-length sequence dataset.

load_pooled_sequence_data

data.manydataset.load_pooled_sequence_data(
    data_list,
    target,
    group_by,
    drop=None,
    input_features=None,
    feature_scaling=None,
    target_scaling=None,
    dataset_type='many_to_many',
)

Pool several DataFrames into one concatenated sequence dataset.

Applies load_sequence_data to every DataFrame (e.g. one compressor map each) and concatenates the resulting sequence datasets — the pooled training set of the schu25a “global training” workflow (ported from load_pretrain_data in src/rnn/utils.py).

Parameters

Name	Type	Description	Default
data_list	List[pd.DataFrame]	Input DataFrames; one per map.	required
target	str	The target column name.	required
group_by	str	Column whose values define the sequences.	required
drop	Optional[Union[str, List[str]]]	Column(s) to drop from the groups before extracting features. Defaults to None.	`None`
input_features	Optional[List[str]]	Columns scaled by `feature_scaling`; see `load_sequence_data`. Defaults to None.	`None`
feature_scaling		Optional sklearn-style scaler, applied per DataFrame. Defaults to None.	`None`
target_scaling		Optional sklearn-style scaler, applied per DataFrame. Defaults to None.	`None`
dataset_type	str	“many_to_many” or “many_to_one”. Defaults to “many_to_many”.	`'many_to_many'`

Returns

Name	Type	Description
ConcatDataset	`ConcatDataset`	The concatenation of the per-DataFrame sequence datasets.

Examples

import pandas as pd
from spotoptim.data.manydataset import load_pooled_sequence_data

df1 = pd.DataFrame({"line": [1, 1, 2], "x": [0.1, 0.2, 0.3], "y": [1.0, 2.0, 3.0]})
df2 = pd.DataFrame({"line": [1, 1], "x": [0.4, 0.5], "y": [4.0, 5.0]})
pooled = load_pooled_sequence_data([df1, df2], target="y", group_by="line", drop="line")
print(len(pooled))

load_sequence_data

data.manydataset.load_sequence_data(
    data,
    target,
    group_by,
    drop=None,
    input_features=None,
    feature_scaling=None,
    target_scaling=None,
    dataset_type='many_to_many',
)

Group a DataFrame into a variable-length sequence dataset.

Splits data by the values of group_by (e.g. one compressor speed line per group) and wraps the groups in a ManyToManyDataset or ManyToOneDataset. Ported from the schu25a study’s load_data helper (src/rnn/utils.py); the experiment-specific column defaults were removed.

Parameters

Name	Type	Description	Default
data	pd.DataFrame	Input data; one row per time step.	required
target	str	The target column name.	required
group_by	str	Column whose values define the sequences.	required
drop	Optional[Union[str, List[str]]]	Column(s) to drop from the groups before extracting features. Defaults to None.	`None`
input_features	Optional[List[str]]	Columns scaled by `feature_scaling`. Only used when `feature_scaling` is given; the dataset features are always all columns except `drop` and `target`. Defaults to None.	`None`
feature_scaling		Optional sklearn-style scaler; applied in place to `data[input_features]` via `fit_transform`. Defaults to None.	`None`
target_scaling		Optional sklearn-style scaler; applied in place to `data[target]` via `fit_transform`. Defaults to None.	`None`
dataset_type	str	Dataset flavor. Options: - “many_to_many”: one target value per time step. - “many_to_one”: one scalar target per sequence. Defaults to “many_to_many”.	`'many_to_many'`

Returns

Name	Type	Description
tuple	Tuple[`Dataset`, pd.DataFrame]	`(dataset, data)` — the sequence dataset and the (possibly scaled) DataFrame.

Raises

Name	Type	Description
	ValueError	If `dataset_type` is not “many_to_many” or “many_to_one”.

Examples

import pandas as pd
from spotoptim.data.manydataset import load_sequence_data

df = pd.DataFrame({
    "line": [1, 1, 1, 2, 2],
    "x": [0.1, 0.2, 0.3, 0.4, 0.5],
    "y": [1.0, 2.0, 3.0, 4.0, 5.0],
})
ds, df = load_sequence_data(df, target="y", group_by="line", drop="line")
print(len(ds), ds[0][0].shape)

2 torch.Size([3, 1])