[FEATURE]update the trace schema to align with OTel guidelines

[YANG-DB]

Is your feature request related to a problem?
Trace Schema Alignment: Review and update the trace schema to align with OTel guidelines

What solution would you like?
A clear and concise description of what you want to happen.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

[paulstn]

Field Name from Log Record (OTEL)	Description	Simple Schema	DataPrepper Scehma	Comments
traceId	A 16-byte identifier unique to a trace. All spans in a trace share the same traceId.	traceId (keyword)	traceId (keyword)
spanId	An 8-byte identifier unique to a span within a trace.	spanId (keyword)	spanId (keyword)
traceState	Additional tracing metadata using the W3C Trace Context format.	"traceState": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } },	❌ Not Present
parentSpanId	The spanId of the parent span. If this is a root span, the field is empty.	parentSpanId(keyword)	parentSpanId(keyword)
flags	A bit field for trace flags, context propagation, and remote determination. Bits 0-7 are W3C trace flags, bits 8-9 determine if the parent span is remote.
name	The human-readable name of the span (e.g., method or operation name).	name(keyword)	name(keyword)
		❌ Not Present	traceGroup(keyword)	What is the difference between traceGroup and name?
kind	The role of the span in a distributed system. Possible values: INTERNAL (1), SERVER (2), CLIENT (3), PRODUCER (4), CONSUMER (5).	kind(keyword)	kind(keyword)
startTimeUnixNano	The span start time in nanoseconds since UNIX epoch.	startTime(date_nanos)	startTime(date_nanos)
endTimeUnixNano	The span end time in nanoseconds since UNIX epoch. Expected to be >= startTimeUnixNano.	endTIme(date_nanos)	endTIme(date_nanos)
			traceGroupFields.endTime(date_nanos)	Why do we need multiple fields for the same?
			traceGroupFields.durationInNanos(long)	Why do we need this?
			durationInNanos(long)	Duplicate of above field? Is this for precalculation?
			traceGroupFields.statusCode(keyword)	What is this?
attributes	A collection of key-value pairs providing metadata about the span.	"attributes": { "type": "object" }	{ "span_attributes_map": { "mapping": { "type":"keyword" }, "path_match":"span.attributes.*" } }	Why do we need span.attributes? Can't we just have attributes?
		attributes.data_stream.dataset (keyword)	❌ Not Present
		attributes.data_stream.namespace (keyword)	❌ Not Present
		attributes.data_stream.type (keyword)	❌ Not Present
droppedAttributesCount	The number of attributes that were discarded due to length limits.	droppedAttributesCount(long)
events	A collection of timestamped annotations on the span.	events(nested)	events(nested)	Why do we need data_stream attributes here and not present in data prepper schema. https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md [Lior Perry pointed to thsi] This should ideally be constant_keyword which only stores once for an index. [https://opensearch.org/docs/latest/field-types/supported-field-types/constant-keyword/] We should use this? Having these fields helps doing multi index query and filter based on attribute types.
events.timeUnixNano	The timestamp when the event occurred (nanoseconds since UNIX epoch).	events.@timestamp(date_nanos)	events.time	Different names, can we consolidate?
events.name	A human-readable name for the event.	events.observedTimestmap(date_nanos)	❌ Not Present
events.attributes	Key-value attributes associated with the event.	"events_attributes_map": { "mapping": { "type": "keyword" }, "path_match": "events.attributes.*" }	❌ Not Present
events.droppedAttributesCount	The number of dropped attributes in this event.	❌ Not Present	❌ Not Present
droppedEventsCount	The number of events that were discarded due to size limits.		❌ Not Present	We should include this.
links	References from this span to another span in the same or a different trace.	type: nested.	type: nested.	Shouldn't we define the known fields.
links.traceId	A unique identifier of a trace that this linked span belongs to.	links.traceId	❌ Not Present
links.spanId	A unique identifier for the linked span.	links.spanId	❌ Not Present
links.traceState	The trace state associated with the link.	links.traceState	❌ Not Present
links.attributes	A collection of key-value attributes describing the link.	{ "links_attributes_map": { "mapping": { "type": "keyword" }, "path_match": "links.attributes.*" } }	❌ Not Present
links.droppedAttributesCount	The number of dropped attributes in this link.	❌ Not Present	❌ Not Present
links.flags	A bit field for trace flags and remote context status.	❌ Not Present	❌ Not Present
droppedLinksCount	The number of dropped links due to size restrictions.	droppedLinksCount(long)	❌ Not Present
status.code	The final status of the span. Possible values: UNSET (0), OK (1), ERROR (2).	status.code(long)	status.code(integer)	Should we change simple schema to integer.
status.message	A human-readable error message (if applicable).	status.message(keyword)	status.message(keyword)
resource.attributes	Key-value attributes describing the resource associated with the span.		"resource_attributes_map": { "mapping": { "type":"keyword" }, "path_match":"resource.attributes.*" }
		serviceName(keyword)	serviceName(keyword)	Ideally serviceName should be under resources.attributes.service.name
resource.schemaUrl	Schema URL associated with the resource attributes.	schemaUrl	❌ Not Present	can we shift this to under resources
instrumentationScope.name	The name of the instrumentation that generated this span.	instrumentationScope.name(keyword)	❌ Not Present
instrumentationScope.version	The version of the instrumentation that generated this span.	instrumentationScope.version(keyword)	❌ Not Present
instrumentationScope.schemaUrl	Schema URL for the instrumentation scope.	instrumentationScope.schemaUrl(keyword)	❌ Not Present
❌ Not Present		instrumentationScope.droppedAttributesCount	❌ Not Present
instrumentationScope.attributes	attributes of the instrucmentation scope.	{ "instrumentation_scope_attributes_map": { "mapping": { "type": "keyword" }, "path_match": "instrumentationScope.attributes.*" } }	❌ Not Present

[YANG-DB]

Thanks @paulstn for the detailed review
how would we like to distribute the tasks between us ?

[paulstn]

Rough draft of moving ss4o and data prepper trace scheme to OTel:

Proposed Field	Analysis
name
parentSpanId
startTime
endTime
traceId	Otel Span Context
spanId	Otel Span Context
traceFlags	Otel Span Context
traceState	Otel Span Context
attributes	metadata
droppedAttributesCount
events	structured log message/annotation of span
droppedEventsCount
links
droppedLinksCount
status	code/message
kind
serviceName	should be able to go under attributes
instrumentationScope	can move under events
schemaUrl	can move under attributes

[krisfreedain]

Catch All Triage - 1 2