Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Generate more telemetry via TelemetryAPI receiver #1230

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open

feat: Generate more telemetry via TelemetryAPI receiver #1230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
208e682
feat: Extend telemetryapi receiver to collect metrics, logs, and more…
borchero Mar 26, 2024
c5116cd
Fix
borchero Mar 26, 2024
6326239
Try
borchero Mar 26, 2024
2606caf
No metrics
borchero Mar 26, 2024
937fbe5
try
borchero Mar 26, 2024
f381938
Debug
borchero Mar 26, 2024
5a7184a
Fix
borchero Mar 26, 2024
0efab13
try
borchero Mar 27, 2024
d827957
fix?
borchero Mar 27, 2024
7e5d76c
Fix
borchero Mar 27, 2024
8e504b5
Add zeros
borchero Mar 27, 2024
a19b48d
Fix timestamps
borchero Mar 27, 2024
20ed49d
Fix?
borchero Mar 27, 2024
8b6d1ef
try without 0
borchero Mar 27, 2024
044f053
Try zero-init
borchero Mar 27, 2024
17b6976
Do not send early
borchero Mar 27, 2024
858a060
No add zero
borchero Mar 28, 2024
eae2acf
Try
borchero Mar 28, 2024
6ae04f0
zero-init
borchero Mar 28, 2024
0583116
Better tracing
borchero Mar 29, 2024
23a6c76
Fix
borchero Mar 29, 2024
94036e0
fix
borchero Mar 29, 2024
7458856
Parse logs
borchero Mar 29, 2024
c7af04b
Fix gomod
borchero Mar 29, 2024
4180f2b
Tidy
borchero Mar 29, 2024
a3665a3
Fixes?
borchero Mar 29, 2024
9e029d0
Debug
borchero Mar 29, 2024
93e9b24
Fix?
borchero Mar 29, 2024
82af051
Fix
borchero Mar 29, 2024
f5dc219
fixes
borchero Mar 29, 2024
e208e20
Some stuff
borchero Mar 29, 2024
563900a
Debug
borchero Mar 29, 2024
440c40d
Fix
borchero Mar 29, 2024
da9e17f
Debug
borchero Mar 29, 2024
3902ff8
Try
borchero Mar 29, 2024
b9af0e4
Fix
borchero Mar 29, 2024
0d86b00
No logs
borchero Mar 30, 2024
c9d7a67
More attributes
borchero Mar 30, 2024
6c4932a
Remove config
borchero Mar 30, 2024
52b67ff
Try
borchero Mar 31, 2024
53d1d7b
Service instance id
borchero Mar 31, 2024
bb6ccd1
Fix?
borchero Mar 31, 2024
e55593b
Update docs
borchero Mar 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions collector/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
module github.com/open-telemetry/opentelemetry-lambda/collector

go 1.20
go 1.21

toolchain go1.21.5

replace github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents => ./lambdacomponents

replace github.com/open-telemetry/opentelemetry-lambda/collector/lambdalifecycle => ./lambdalifecycle

replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/coldstartprocessor => ./processor/coldstartprocessor
replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/faasprocessor => ./processor/faasprocessor

replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/decoupleprocessor => ./processor/decoupleprocessor

Expand Down Expand Up @@ -61,7 +63,7 @@ require (
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/logr v1.3.0 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/gobwas/glob v0.2.3 // indirect
Expand Down Expand Up @@ -102,8 +104,8 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.92.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.92.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor v0.92.0 // indirect
github.com/open-telemetry/opentelemetry-lambda/collector/processor/coldstartprocessor v0.92.0 // indirect
github.com/open-telemetry/opentelemetry-lambda/collector/processor/decoupleprocessor v0.0.0-00010101000000-000000000000 // indirect
github.com/open-telemetry/opentelemetry-lambda/collector/processor/faasprocessor v0.92.0 // indirect
github.com/open-telemetry/opentelemetry-lambda/collector/receiver/telemetryapireceiver v0.92.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
Expand Down Expand Up @@ -153,13 +155,13 @@ require (
go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.92.0 // indirect
go.opentelemetry.io/collector/receiver v0.92.0 // indirect
go.opentelemetry.io/collector/receiver/otlpreceiver v0.92.0 // indirect
go.opentelemetry.io/collector/semconv v0.92.0 // indirect
go.opentelemetry.io/collector/semconv v0.97.0 // indirect
go.opentelemetry.io/collector/service v0.92.0 // indirect
go.opentelemetry.io/contrib/config v0.1.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.21.1 // indirect
go.opentelemetry.io/otel v1.21.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/bridge/opencensus v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v0.44.0 // indirect
Expand All @@ -169,14 +171,14 @@ require (
go.opentelemetry.io/otel/exporters/prometheus v0.44.1-0.20231201153405-6027c1ae76f2 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.21.0 // indirect
go.opentelemetry.io/otel/metric v1.21.0 // indirect
go.opentelemetry.io/otel/sdk v1.21.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.21.0 // indirect
go.opentelemetry.io/otel/trace v1.21.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
golang.org/x/exp v0.0.0-20240103183307-be819d1f06fc // indirect
golang.org/x/net v0.20.0 // indirect
golang.org/x/sys v0.16.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
gonum.org/v1/gonum v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17 // indirect
Expand Down
52 changes: 35 additions & 17 deletions collector/go.sum
View file
Open in desktop

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions collector/internal/lifecycle/manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,14 @@ package lifecycle
import (
"context"
"fmt"
"github.com/open-telemetry/opentelemetry-lambda/collector/lambdalifecycle"
"os"
"os/signal"
"path/filepath"
"sync"
"syscall"

"github.com/open-telemetry/opentelemetry-lambda/collector/lambdalifecycle"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this new component propagate context from the collector to the application?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this answers your question but the collector doesn't really pass any context to the application. When the application generates a span A with a parent B that was propagated from an external system, the faas processor identifies span A (by checking for spans with a faas.invocation_id) and inserts the Lambda runtime span (let's call it C) between A and B. That means, C becomes parent of A and B becomes parent of C.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that helps.


"go.uber.org/multierr"
"go.uber.org/zap"

Expand All @@ -33,9 +34,7 @@ import (
"github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents"
)

var (
extensionName = filepath.Base(os.Args[0]) // extension name has to match the filename
)
var extensionName = filepath.Base(os.Args[0]) // extension name has to match the filename

type collectorWrapper interface {
Start(ctx context.Context) error
Expand Down
7 changes: 5 additions & 2 deletions collector/internal/telemetryapi/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,11 @@ func NewClient(logger *zap.Logger) *Client {
}
}

func (c *Client) Subscribe(ctx context.Context, extensionID string, listenerURI string) (string, error) {
func (c *Client) Subscribe(
ctx context.Context,
extensionID string,
listenerURI string,
) (string, error) {
eventTypes := []EventType{
Platform,
// Function,
Expand All @@ -73,7 +77,6 @@ func (c *Client) Subscribe(ctx context.Context, extensionID string, listenerURI
BufferingCfg: bufferingConfig,
Destination: destination,
})

if err != nil {
return "", fmt.Errorf("Failed to marshal SubscribeRequest: %w", err)
}
Expand Down
8 changes: 8 additions & 0 deletions collector/internal/telemetryapi/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ const (
PlatformInitStart EventType = Platform + ".initStart"
// PlatformInitRuntimeDone is used when function initialization ended.
PlatformInitRuntimeDone EventType = Platform + ".initRuntimeDone"
// PlatformInitReport is used to report on function initialization.
PlatformInitReport EventType = Platform + ".initReport"
// PlatformStart is used when function invocation started.
PlatformStart EventType = Platform + ".start"
// PlatformRuntimeDone is used when function invocation ended.
PlatformRuntimeDone EventType = Platform + ".runtimeDone"
// PlatformReport is used to report on function invocation.
PlatformReport EventType = Platform + ".report"
// Function is used to receive log events emitted by the function
Function EventType = "function"
// Extension is used is to receive log events emitted by the extension
Expand Down
4 changes: 2 additions & 2 deletions collector/lambdacomponents/default.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import (
"go.opentelemetry.io/collector/receiver/otlpreceiver"
"go.uber.org/multierr"

"github.com/open-telemetry/opentelemetry-lambda/collector/processor/coldstartprocessor"
"github.com/open-telemetry/opentelemetry-lambda/collector/processor/faasprocessor"
"github.com/open-telemetry/opentelemetry-lambda/collector/receiver/telemetryapireceiver"
)

Expand Down Expand Up @@ -68,7 +68,7 @@ func Components(extensionID string) (otelcol.Factories, error) {
probabilisticsamplerprocessor.NewFactory(),
resourceprocessor.NewFactory(),
spanprocessor.NewFactory(),
coldstartprocessor.NewFactory(),
faasprocessor.NewFactory(),
decoupleprocessor.NewFactory(),
batchprocessor.NewFactory(),
)
Expand Down
28 changes: 15 additions & 13 deletions collector/lambdacomponents/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents

go 1.20
go 1.21

toolchain go1.21.5

require (
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.92.0
Expand All @@ -10,8 +12,8 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.92.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.92.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor v0.92.0
github.com/open-telemetry/opentelemetry-lambda/collector/processor/coldstartprocessor v0.92.0
github.com/open-telemetry/opentelemetry-lambda/collector/processor/decoupleprocessor v0.0.0-00010101000000-000000000000
github.com/open-telemetry/opentelemetry-lambda/collector/processor/faasprocessor v0.92.0
github.com/open-telemetry/opentelemetry-lambda/collector/receiver/telemetryapireceiver v0.92.0
go.opentelemetry.io/collector/exporter v0.92.0
go.opentelemetry.io/collector/exporter/loggingexporter v0.92.0
Expand Down Expand Up @@ -51,7 +53,7 @@ require (
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/logr v1.3.0 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/gobwas/glob v0.2.3 // indirect
Expand Down Expand Up @@ -127,13 +129,13 @@ require (
go.opentelemetry.io/collector/extension/auth v0.92.0 // indirect
go.opentelemetry.io/collector/featuregate v1.0.1 // indirect
go.opentelemetry.io/collector/pdata v1.0.1 // indirect
go.opentelemetry.io/collector/semconv v0.92.0 // indirect
go.opentelemetry.io/collector/semconv v0.97.0 // indirect
go.opentelemetry.io/collector/service v0.92.0 // indirect
go.opentelemetry.io/contrib/config v0.1.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.21.1 // indirect
go.opentelemetry.io/otel v1.21.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/bridge/opencensus v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v0.44.0 // indirect
Expand All @@ -143,21 +145,21 @@ require (
go.opentelemetry.io/otel/exporters/prometheus v0.44.1-0.20231201153405-6027c1ae76f2 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v0.44.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.21.0 // indirect
go.opentelemetry.io/otel/metric v1.21.0 // indirect
go.opentelemetry.io/otel/sdk v1.21.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.21.0 // indirect
go.opentelemetry.io/otel/trace v1.21.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
go.uber.org/zap v1.26.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/exp v0.0.0-20240103183307-be819d1f06fc // indirect
golang.org/x/net v0.20.0 // indirect
golang.org/x/sys v0.16.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
gonum.org/v1/gonum v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231127180814-3a041ad873d4 // indirect
google.golang.org/grpc v1.60.1 // indirect
google.golang.org/protobuf v1.32.0 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Expand All @@ -172,7 +174,7 @@ replace github.com/open-telemetry/opentelemetry-lambda/collector => ../

replace github.com/open-telemetry/opentelemetry-lambda/collector/lambdalifecycle => ../lambdalifecycle

replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/coldstartprocessor => ../processor/coldstartprocessor
replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/faasprocessor => ../processor/faasprocessor

replace github.com/open-telemetry/opentelemetry-lambda/collector/processor/decoupleprocessor => ../processor/decoupleprocessor

Expand Down
Loading