Skip to content

Commit

Permalink
Merge pull request #127 from scribd/maksimt/DEVPLAT-1850/pubsub-sqs
Browse files Browse the repository at this point in the history 
[DEVPLAT-1850] Add SQS support to PubSub
  • Loading branch information
@Neurostep
Neurostep authored Nov 8, 2024
2 parents 4e5ec90 + 720e682 commit a016bc3
Show file tree
Hide file tree
Showing 19 changed files with 1,976 additions and 214 deletions.
37 changes: 37 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ SDK, the Go version.
- [Usage of ORM](#usage-of-orm)
- [PubSub](#pubsub)
- [Kafka specific configuration](#kafka-specific-configuration)
- [SQS specific configuration](#sqs-specific-configuration)
- [Cache](#cache)
- [Redis](#redis-specific-configuration)
- [Experimentation](#experimentation)
Expand Down Expand Up @@ -849,6 +850,23 @@ common: &common
subscriber:
topic: "test-topic"
group_id: ""
sqs:
subscriber:
# Set by APP_PUBSUB_SQS_SUBSCRIBER_ENABLED env variable
enabled: true
# Set by APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL env variable
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
# Set by APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES env variable
max_messages: 10
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WORKERS env variable
workers: 1
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME env variable
wait_time: "10s"
publisher:
# Set by APP_PUBSUB_SQS_PUBLISHER_ENABLED env variable
enabled: true
# Set by APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL env variable
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
```
#### Kafka specific configuration
Expand Down Expand Up @@ -920,6 +938,25 @@ To authenticate the requests to Kafka, Go SDK provides a configuration set for T
| Assumable role | This role will be used to establish connection to AWS MSK ignoring the static credentials | `role` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_ROLE` | string | AWS ARN string |
| Session name | Will be passed to AWS STS when assuming the role | `session_name` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_SESSION_NAME` | string | application |

#### SQS specific configuration

Under the hood, the SQS integration relies on the provided SQS client. To learn more about the AWS services initialization, check the [AWS service configuration](#aws-service-configuration).

**Subscriber**:

| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
|-------------------|-------------------------------------------------------------------------------------------------|-----------------------|---------------------------------------------------|---------|-----------------|
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL` | string | queue URL |
| Max messages | Maximum number of messages to retrieve from the queue per polling iteration | `max_messages` | `APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES` | number | 1,2... |
| Number of workers | Number of workers processing incoming messages | `workers` | `APP_PUBSUB_SQS_SUBSCRIBER_WORKERS` | number | 1,2... |
| Wait time | The maximum amount of time in time.Duration to wait for messages to be available for retrieval. | `wait_time` | `APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME` | string | 10s |

**Publisher**:

| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
|-----------|-----------------------|-----------------------|---------------------------------------------------|---------|-----------------|
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL` | string | queue URL |

### Cache

`go-sdk` provides a convenient way to create an application Cache configuration.
Expand Down
53 changes: 27 additions & 26 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ require (
github.com/DATA-DOG/go-txdb v0.1.9
github.com/DataDog/datadog-go v4.8.2+incompatible
github.com/aws/aws-sdk-go v1.44.327
github.com/aws/aws-sdk-go-v2 v1.26.1
github.com/aws/aws-sdk-go-v2/config v1.27.11
github.com/aws/aws-sdk-go-v2/credentials v1.17.11
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.27.4
github.com/aws/aws-sdk-go-v2/service/sfn v1.19.4
github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6
github.com/aws/aws-sdk-go-v2 v1.32.3
github.com/aws/aws-sdk-go-v2/config v1.28.1
github.com/aws/aws-sdk-go-v2/credentials v1.17.42
github.com/aws/aws-sdk-go-v2/service/s3 v1.66.2
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.32.3
github.com/aws/aws-sdk-go-v2/service/sfn v1.33.3
github.com/aws/aws-sdk-go-v2/service/sqs v1.36.3
github.com/aws/aws-sdk-go-v2/service/sts v1.32.3
github.com/aws/smithy-go v1.22.0
github.com/getsentry/sentry-go v0.12.0
github.com/go-kit/kit v0.9.0
github.com/go-sql-driver/mysql v1.7.1
Expand Down Expand Up @@ -48,24 +49,24 @@ require (
github.com/DataDog/gostackparse v0.7.0 // indirect
github.com/DataDog/sketches-go v1.4.2 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.21.4 // indirect
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.34 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
github.com/aws/aws-sdk-go-v2/service/kinesis v1.18.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.21.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
github.com/aws/smithy-go v1.20.2 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.6 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.22 // indirect
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ec2 v1.187.0 // indirect
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.35.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.3 // indirect
github.com/aws/aws-sdk-go-v2/service/kinesis v1.32.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.33.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.3 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
Expand Down
Loading

0 comments on commit a016bc3

Please sign in to comment.