Empty target for stream still looks for coordinate embedding network

[kctezcan]

What happened?

In model.py, predict_decoders(), we have in a loop over the stream names:

for stream_name in self.stream_names:
    ...
    # embed token coords
    tc_embed = self.embed_target_coords[stream_name]
    tc_tokens = tc_embed(t_coords)

If we have a stream tht is only used as a forcing, i.e. does not have any targets, it does not have network in embed_target_coords, so self.embed_target_coords[stream_name] returns an error.

Solution: we should add a check in the loop:

for stream_name in self.stream_names:
    ...
    if stream_name not in self.embed_target_coords.keys():
        continue

    # embed token coords
    tc_embed = self.embed_target_coords[stream_name]
    tc_tokens = tc_embed(t_coords)

What are the steps to reproduce the bug?

run for example era5 as forcing and decode synop

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[kctezcan]

Follow up: i nthe same loop we have more checks which set the pred to an empty tensor:

# skip when coordinate embeddings yields nan (i.e. the coord embedding network diverged)
if torch.isnan(tc_tokens).any():
    logger.warning( ... )
    pred = torch.tensor([], device=tc_tokens.device)

or

# skip empty lengths
elif tc_tokens.shape[0] == 0:
    pred = torch.tensor([], device=tc_tokens.device)

after the loop, the pred goes nto this line: pred = torch.split(pred, t_coords_lens, dim=1)
which gives an error, since pred is empty.

Possibly these should also just skip the iteration with continue

[clessig]

This is related to #1724. CC @shmh40

We need to consistently remove empty output streams so that these do not appear for pred networks, in output processing, .... We can either create cf.streams_input and cf.streams_output or consistently check for len(cf.streams[i].target_channels) = 0. The latter one is easier where we are and I think one should try this first.