Autoformer get_lagged_subsequences always true if condition

[Marco-Pi]

System Info

transformers 4.53.2, MacOS, python 3.12.10

Who can help?

@elisim @kashif

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

During training of the Autoformer model I get this error:
ValueError: lags cannot go further than history length, found lag 37 while history length is only 70
This exception is raised when calling the create_network_inputs function of the Autoformer class, specifcally when the function call get_lagged_subsequences function (at the time of writing line 1411 of modeling_autoformer.py):
lagged_sequence = self.get_lagged_subsequences(sequence=inputs, subsequences_length=subsequences_length)

The signature of get_lagged_subsequences is:
def get_lagged_subsequences( self, sequence: torch.Tensor, subsequences_length: int, shift: int = 0 ) -> torch.Tensor:

The arguments passed during the above function call has the following values:

• inputs is the concatenation of past and future values
• subsequences_length is the sum between the context length and the prediction length
• shift is kept to 0

Inside get_lagged_subsequences there is this line of code (at the time of writing 1505):
if max(indices) + subsequences_length > sequence_length:

where:

• max(indices) is the maximum value of the lags_sequences
• subsequences_length is the one passed as parameter, thus the sum between the context length and the prediction length
• sequence_length is initialised as sequence_length = sequence.shape[1], thus is equal to the context_length

So in the best case when max(indices) is equal to zero, we get that the if condition became:
0 + (context_length + prediction_length) > context_length
Since the model always predict at least one value the prediction length is always greater than zero, so this condition is always true. The problem is that having this condition always being true block the code since when the if condition is verified an exception is raised:
if reshaped_lagged_sequence.shape[1] != time_feat.shape[1]: raise ValueError( f"input length {reshaped_lagged_sequence.shape[1]} and time feature lengths {time_feat.shape[1]} does not match")

Here is a minimal code that allow to reproduce the error, please note that for every value of lags_sequence the exception is raised here I am using the default ones loaded from huggingface/autoformer-tourism-monthly config:

from datasets import Dataset
import torch
import numpy as np

cfg = AutoConfig.from_pretrained("huggingface/autoformer-tourism-monthly")
model = AutoformerModel(cfg)

X, y, T, M, T_future = (np.random.rand(500, 60), np.random.rand(500, 10), np.random.rand(500, 60, 6), np.random.rand(500, 60), np.random.rand(500, 10, 6))

train_ds = Dataset.from_dict({
    'past_values':          X,
    'future_values':        y,
    'past_time_features':   T, 
    'past_observed_mask':   M,
    'future_time_features': T_future
})

training_args = TrainingArguments(
    output_dir="./autoformer_checkpoints",
    save_steps=100,
    logging_steps=50,
    num_train_epochs=50,
    learning_rate=1e-4,
    report_to="none"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
)

trainer.train()

Expected behavior

The expected behavior is to have a meaningful if condition or remove the if line if not actually needed, in order to being able to train and use the Autoformer model

[jg100]

@Marco-Pi I feel like it may be intended behavior to have a restriction like that on the lag, I'm relatively new to the codebase so I'll let someone a little more senior comment on the true intentionality.

The only thing I'd suggest is maybe adding a more comprehensive error message. I agree, it's somewhat misleading and feels like the Autoformer does not work right out of the box lol.

But with some sample data tweaking and resizing you should be able to get it to work.

Here is my personal test code, which worked as expected, I was using:

import torch
from transformers import AutoConfig, AutoformerForPrediction

# Load pretrained config and model
config = AutoConfig.from_pretrained("huggingface/autoformer-tourism-monthly")
model = AutoformerForPrediction.from_pretrained("huggingface/autoformer-tourism-monthly")
model.eval()

# ==== Prepare mock input batch ====
batch_size = 1
context_length = config.context_length
prediction_length = config.prediction_length
num_time_features = config.num_time_features
num_static_cat = config.num_static_categorical_features
num_static_real = config.num_static_real_features
lags_sequence = config.lags_sequence
print("Lags sequence: ", lags_sequence)
print("Context length", context_length)
required_history_length = context_length + max(lags_sequence)

# Create random but correctly shaped input tensors
# This is the important part, all the test values need to be shaped with the calculated required history length

past_values = torch.randn(batch_size, required_history_length)
past_time_features = torch.randn(batch_size, required_history_length, num_time_features)
future_time_features = torch.randn(batch_size, prediction_length, num_time_features)
past_observed_mask = torch.ones(batch_size, required_history_length)
static_categorical_features = torch.zeros(batch_size, num_static_cat, dtype=torch.long)
static_real_features = torch.randn(batch_size, num_static_real)

# Batch dict for generate()
batch = {
    "past_values": past_values,
    "past_time_features": past_time_features,
    "future_time_features": future_time_features,
    "past_observed_mask": past_observed_mask,
    "static_categorical_features": static_categorical_features,
    "static_real_features": static_real_features,
}

# ==== Run model prediction ====
with torch.no_grad():
    outputs = model.generate(**batch)

# ==== Print result ====
print("Forecast shape:", outputs.sequences.shape)  # [batch_size, prediction_length, num_samples]
print("Sample mean forecast:", outputs.sequences[0].mean().item())

For your code it might look something like this:

context_length = cfg.context_length
max_lag = max(cfg.lags_sequence)
required_past_length = context_length + max_lag
prediction_length = cfg.prediction_length
num_features = cfg.num_time_features


num_samples = 500

X = np.random.rand(num_samples, required_past_length).astype(np.float32)             # past_values
y = np.random.rand(num_samples, prediction_length).astype(np.float32)                # future_values
T = np.random.rand(num_samples, required_past_length, num_features).astype(np.float32)  # past_time_features
M = np.ones((num_samples, required_past_length), dtype=np.float32)                   # past_observed_mask
T_future = np.random.rand(num_samples, prediction_length, num_features).astype(np.float32)

[jg100]

It might even be worth adding a helper function somewhere to format the dataset