Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DX-1407] Tyk Streams: Sync response #5047

Merged
merged 67 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
a4e4c2d
add sync response processor content file empty and hook up to menu
Jul 8, 2024
8d096f9
add sync_response tag
Jul 8, 2024
0f7f7b2
add Sync Response draft content
Jul 8, 2024
e943c99
add sync responses guide to menu
Jul 8, 2024
686d985
add link to sync-response guide from processor
Jul 8, 2024
3f47efa
add sync_response output content
Jul 8, 2024
28ad598
add sync_response output to Tyk Streams menu
Jul 8, 2024
c793747
initial draft commit of http_client output unformatted
Jul 8, 2024
f5af224
fix bloblang link in advanced bloblang
Jul 8, 2024
c2b8489
add http client to inputs menu, draft commit
Jul 8, 2024
0cb4f99
add func interpolation link to http clients input
Jul 8, 2024
a5b1b62
add synchronous responses link in http-client output
Jul 8, 2024
1952110
format http_client output
Jul 8, 2024
5acb671
fix styling error
Jul 8, 2024
4295273
fix http_client output link and batch_as_multipart formatting
Jul 9, 2024
b580fd3
fix synchronous responses link
Jul 9, 2024
33bd1c3
add batching.check link to bloblang query
Jul 9, 2024
771547d
add interpolation function links
Jul 9, 2024
a72bbb4
initial commit of http_client input
Jul 9, 2024
eef1859
add http client input
Jul 10, 2024
eb15044
add generate draft content
Jul 10, 2024
38c3ad2
add generate to menu
Jul 10, 2024
268f24e
add generate link to http-client inputs
Jul 10, 2024
8577d35
comment secrets related content in http_client input
Jul 10, 2024
c9dfe38
comment http_client output secret related content
Jul 10, 2024
b7c747e
add styling and comment out secrets access until know whether they ca…
Jul 10, 2024
2a82df5
add http processor, commenting secrets content until org decide if ca…
Jul 10, 2024
3478c24
add bloblang links
Jul 10, 2024
61419fe
add branch link
Jul 10, 2024
c7d5476
add rate limit TODO comments
Jul 10, 2024
b61dee1
add bloblang links
Jul 10, 2024
9c2f059
create scanners folder
Jul 12, 2024
83c296d
add scanners overview draft
Jul 12, 2024
def3934
add formatting
Jul 12, 2024
3e73429
add front matter
Jul 12, 2024
17ce00e
add TODO relating to http_client to give improved concrete example
Jul 12, 2024
ba9723c
fix overview header and add lines document
Jul 12, 2024
880ac00
add scanners menu
Jul 12, 2024
25310de
add header
Jul 12, 2024
4525163
update json_documents with header
Jul 12, 2024
1cfb2ec
add switch scanner
Jul 12, 2024
c306884
add switch scanner to menu
Jul 12, 2024
9e2cbe0
add regular expression matching
Jul 12, 2024
1d0121d
add link to http processor
Aug 19, 2024
c9e86b4
remove version
Aug 19, 2024
31d3415
refactor to remove supported versions and restore authentication fields
Aug 19, 2024
adfbf35
refactor http client output to restore authentication fields
Aug 19, 2024
4dcdf6a
Merge branch 'master' into dx-1407-sync-response
dcs3spp Aug 19, 2024
f22842a
fix broken link for batching
Aug 19, 2024
2caaa47
add csv scanner to menu yaml
Aug 19, 2024
2a9b307
fix broker link
Aug 19, 2024
7a30985
fix broken link to sync responses
Aug 19, 2024
60de3df
fix broken link to date limits
Aug 19, 2024
654c868
remove versions
Aug 20, 2024
ca868d0
update http_client input to remove env var
Aug 20, 2024
8afe8e5
update to fix typo
Aug 20, 2024
0866170
update oauth headers in http processor
Aug 20, 2024
944be43
add oauth2 section headings
Aug 20, 2024
c7c581d
add section headers refactor
Aug 20, 2024
a950638
restore header consistency
Aug 20, 2024
1a92608
remove comment from scanners line
Aug 21, 2024
6b52d77
remove JSON documents scanner
Sep 3, 2024
f9c3b4e
add rate limit links to http processor
Sep 3, 2024
fd6c0cd
Merge branch 'master' into dx-1407-sync-response
dcs3spp Sep 3, 2024
11cc672
update pagination example to remove secrets until Tyk secrets release…
Sep 3, 2024
b8fd112
Merge branch 'dx-1407-sync-response' of github.com:TykTechnologies/ty…
Sep 3, 2024
f48ea84
Merge branch 'master' into dx-1407-sync-response
dcs3spp Sep 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: Generate Input
description: Explains an overview of configuring generate input
tags: [ "Tyk Streams", "Stream Outputs", "Inputs", "Generate" ]
---

Generates messages at a given interval using a [Bloblang]({{< ref "/product-stack/tyk-streaming/configuration/common-configuration/interpolation#bloblang-queries" >}}) mapping executed without a context. This allows you to generate messages for testing your pipeline configs.

```yml
# Config fields, showing default values
input:
label: ""
generate:
mapping: root = "hello world" # No default (required)
interval: 1s
count: 0
batch_size: 1
auto_replay_nacks: true
```

## Examples

## Cron Scheduled Processing

A common use case for the generate input is to trigger processors on a schedule so that the processors themselves can behave similarly to an input. The following configuration reads rows from a PostgreSQL table every 5 minutes.

```yaml
input:
generate:
interval: '@every 5m'
mapping: 'root = {}'
processors:
- sql_select:
driver: postgres
dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable
table: foo
columns: [ "*" ]
```

## Generate 100 Rows

The generate input can be used as a convenient way to generate test data. The following example generates 100 rows of structured data by setting an explicit count. The interval field is set to empty, which means data is generated as fast as the downstream components can consume it.

```yaml
input:
generate:
count: 100
interval: ""
mapping: |
root = if random_int() % 2 == 0 {
{
"type": "foo",
"foo": "is yummy"
}
} else {
{
"type": "bar",
"bar": "is gross"
}
}
```

## Fields

### mapping

A [bloblang]({{< ref "/product-stack/tyk-streaming/configuration/common-configuration/interpolation#bloblang-queries" >}}) mapping to use for generating messages.


Type: `string`

```yml
# Examples

mapping: root = "hello world"

mapping: root = {"test":"message","id":uuid_v4()}
```

### interval

The time interval at which messages should be generated, expressed either as a duration string or as a cron expression. If set to an empty string messages will be generated as fast as downstream services can process them. Cron expressions can specify a timezone by prefixing the expression with `TZ=<location name>`, where the location name corresponds to a file within the IANA Time Zone database.


Type: `string`
Default: `"1s"`

```yml
# Examples

interval: 5s

interval: 1m

interval: 1h

interval: '@every 1s'

interval: 0,30 */2 * * * *

interval: TZ=Europe/London 30 3-6,20-23 * * *
```

### count

An optional number of messages to generate, if set above 0 the specified number of messages is generated and then the input will shut down.


Type: `int`
Default: `0`

### batch_size

The number of generated messages that should be accumulated into each batch flushed at the specified interval.


Type: `int`
Default: `1`

### auto_replay_nacks

Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.


Type: `bool`
Default: `true`
Loading