Proposal to Update Traffic Light Detection Message to handle V2I information

[YoshiRi]

This discussion is related to V2X implementation introduced by @yuasabe.

Background

In the current Autoware system, traffic signal information is obtained by integrating camera recognition and V2X (V2I/V2N) communications via the autoware_traffic_light_arbiter module. However, the existing ROS 2 message type, TrafficLightGroupArray (as defined in autoware_perception_msgs), only maintains a list of TrafficLightElement representing the current signal status. It does not handle predictive information regarding future signal changes (i.e., scheduling information). As a result, predictive controls—such as preemptive deceleration based on upcoming signal changes—are not fully exploited.

New Design Proposal

This proposal aims to enhance the conventional TrafficLightGroupArray by incorporating predictive information about future traffic signal states, thereby enabling more advanced traffic signal coordination and control.

Here shows the simple comparison between previous messages and updated messages.

Before

After

Addition of a FutureTrafficLightElement List

Each TrafficLightGroup will be extended to include a list of Future TLEs (Future TrafficLightElements). In addition to the current signal information, this list will contain predictions of future signal states. With this enhancement, planned signal control becomes possible, such as preemptively decelerating in anticipation of an upcoming signal change.

Definition of FutureTrafficLightElement

• Header (frame/stamp) Setting:
  Each Future TLE will include a header that specifies an absolute timestamp and the source of the information (frame id). Since the TrafficLightGroup itself does not include a header, managing Future TLEs with absolute time rather than relative time is necessary to ensure time synchronization and consistency when integrating data from multiple V2I or V2V sources.

• List of TrafficLightElements:
  Similar to the standard TLE, Future TLEs will maintain a list of predicted signal elements. Since traffic signals operate on a scheduled basis, when the next state is predictable, this information can be preloaded into the system. In cases where no predictive information is available (e.g., through local estimation), this list is expected to remain empty.

• Introduction of a Probability Field:
  Due to variations in the communication environment and signal characteristics, the reliability of received V2I data may fluctuate. Therefore, it is proposed that each Future TLE include a numerical value indicating its reliability—expressed as a probability (for example, in the range of 0.0 to 1.0). This value can be derived from experimental data or based on catalog values from the communication devices. In cases where a valid value cannot be computed, an invalid marker (such as -1.0) should be used.

With these design changes, predictive information about future signal states can be integrated into the system, paving the way for enhanced traffic signal coordination and ultimately contributing to safer and more efficient autonomous driving.

[mitsudome-r]

Thanks for the proposal.

I have some comments regarding the design:

• Header: I believe just having the stamp field would be enough. We probably don't really care about the source of information in Planning modules when we get the message. If we really need the source information, it should still be defined as a separate string field rather than using header message because frame_id is mainly used to define the TF (coordinate) frame of the message.
• Probability: Do we actually need this? Is there a case where a planning module would have to care about the reliability of the future state to make decisions? We don't even have the probability field for the current detected traffic light state, and I'm not sure if we want to have that field in general. If it cannot compute the future state, then we can just have empty future_elements or put 0 in the timestamp.

[YoshiRi]

@mitsudome-r
Thanks for your feedback.

Header:
It might indeed be problematic to confuse this field with the TF frame information. This message is not only passed to the planning module but also serves as an interface for data integration within the perception module. From that viewpoint, it is important to record the source of the information. In some algorithms, the vehicle itself may populate this field, and depending on V2V conditions or the specific location, multiple V2I sources might be involved.

Probability:
I also believe that this field should be reserved because it is useful for information integration within the perception system. The current TLR message does not include this attribute mainly because probability values are stored within the TrafficLightElement. However, during previous V2I demonstrations, we encountered cases where invalid data was received, leading to uncertainty in the predicted timestamps. In such cases, having a dedicated field to convey that uncertainty becomes necessary.

[yuasabe]

Thank you for the proposal. I would like to add some of my comments on the proposed changes to the traffic signal related messages.

About using std_msgs/Header:
The absolute time that a traffic signal state is expected to begin is the most important value. This can be represented using builtin_interfaces/Time time.

But as mentioned in the proposal, adding a "information origin" field such as string information_source can be beneficial especially when aggregating traffic signal information from multiple sources such as V2I infrastructure, V2V data, manual overvides, etc. This will give downstream modules (e.g. behavior planning, visualization, logging) the ability to prioritize sources, handle uncertainty, and detect contradictions or dropouts. However, as @mitsudome-r -san mentioned above, the frame_id inside header is not suitable for this purpose. Adding a string information_source will be better.

About adding the probability field
I think naming it reliability better fits its purpose than probability. SPaT messages often contain deterministic schedules (e.g. signal ends in 7.4s), but the transport or source may be fallible (e.g. time drift, packet loss, timestamp jitter). The origin of the information could be less trustworthy as well, such as a V2V broadcast from another vehicle as opposed to V2I-based info from a trusted RSU.

The reliability filed could encode information such as:

• Data freshness (was this SPaT message received just now or 2 seconds ago?)
• Signal source trustworthiness (e.g., RSU > V2V)
• Internal system diagnostics (was the data inferred or directly observed?)

with a scale of [0.0, 1.0].

I have some other comments as well which I will add on in another fresh comment.

[yuasabe]

Thank you for the proposal. I'd like to share some feedback on the proposed updates to the traffic signal–related message definitions.

Clarifying the Semantic Differences

Referring to the proposed TrafficLightGroup message:

[autoware_perception_msgs/TrafficLightGroup]
int64 traffic_light_group_id
autoware_perception_msgs/TrafficLightElement[] elements
autoware_perception_msgs/FutureTrafficLightElement[] future_elements

There’s an important semantic distinction between the two arrays:

• TrafficLightElement[] elements represents simultaneous signal aspects at the current moment.
• FutureTrafficLightElement[] future_elements represents a chronological sequence of future signal states.

In the current signal case (elements), multiple TrafficLightElements may be valid at once — for example, a red circle and a green right arrow would indicate “stop straight/left, but protected right turn allowed.” These elements are interpreted together, as a single-time-point snapshot of the traffic signal logic.

On the other hand, in the proposed future signal (future_elements) structure, each FutureTrafficLightElement corresponds to a specific future time point, and contains a set of simultaneous elements valid at that moment. Thus:

• The outer array (future_elements[]) is ordered chronologically.
• The inner array (elements[] within each future entry) contains concurrent signal aspects for that future time.

This dual use of similar field names (like elements) to express conceptually different relationships — simultaneity vs. temporal sequencing — could be confusing to developers. It may also lead to misinterpretation, such as treating the future entries as parallel hypotheses rather than an ordered timeline of states.

Proposed Refinement

To make the temporal semantics clearer and the structure more extensible, I suggest the following message definition:

autoware_perception_msgs/TrafficLightGroup
int64 traffic_light_group_id
autoware_perception_msgs/TrafficLightElement[] current_elements
autoware_perception_msgs/FutureTrafficLightState[] future_predictions

autoware_perception_msgs/FutureTrafficLightState
builtin_interfaces/Time time                          # Absolute time this state is expected
autoware_perception_msgs/TrafficLightElement[] simultaneous_elements  # Valid signals at this time
float32 reliability                                    # Confidence level [0.0–1.0]
string information_source                             # e.g., "V2I_SPAT", "V2V", etc.

This structure:

• Clearly separates current state from future states
• Avoids overloaded naming like “elements” across different temporal contexts
• Introduces reliability to capture the trustworthiness of the source, distinct from probabilistic prediction
• Adds information_source to help downstream consumers (e.g., planners) reason about the origin of the signal data

Let me know what you think about these changes, and I’d appreciate feedback.
@YoshiRi @mitsudome-r

[YoshiRi]

@yuasabe cc: @MasatoSaeki
Thanks for you neat suggestion.
In my understandings, your suggestion can be separated into following options:

1. rename elements to current_elements
2. rename FutureTrafficLightElement into FutureTrafficLightState and future_predictions
3. divide header into time and frame information
4. redefine uncertainty parameters as reliability

I totally agree and accept 2, 3 and 4 suggestion.
So, my concern is only about 1.

Though it seems clearer with new variable name, this change is update and could affect many packages using this message types.
It would be clear that compared to futurestate, it is obvious that the elements means the current traffic_light_estimate state.

Can we keep elements variable name as current style and just adding comments to explain what we meant to?

@MasatoSaeki
How do you think about changing TrafficLightGroup variable name? I will rethink my opinion if you think it is quite easy to rename elements to current_elements for current code base and our tools.

[MasatoSaeki]

@YoshiRi @yuasabe
My opinion is autoware_perception_msgs/TrafficLightElement[] elements is better

I think it is ok to change name elements to current_elements, but elements is better (enough) because type name is autoware_perception_msgs/TrafficLightElement[] and I would not like to complicate name. autoware_perception_msgs/TrafficLightElement[] and autoware_perception_msgs/FutureTrafficLightState[] is enough different, so we do not need to make names very obvious.

BTW, autoware_perception_msgs/FutureTrafficLightState[] predictions is better because of simple. (predictions include "future" mean I think)

[yuasabe]

@YoshiRi @MasatoSaeki
Thank you for your comments.
I have created a PR to autoware_msgs to update the items discussed above: autowarefoundation/autoware_msgs#134

The newest relations between the different traffic light related messages defined in autoware_perception_msgs is shown below:

[YoshiRi]

IMO, Remaining TODO is like following:

• Merge new message PR
• Update traffic_light_arbiter to handle V2I messages

Please tell me if there are other tasks to be done in this discussion.