[DX-1407] Tyk Streams: Sync response

[dcs3spp]

User description

For internal users - Please add a Jira DX PR ticket to the subject!

DX-1407

---

Preview Link

• preview - http_client input
• preview - http_client output
• preview - http processor

Description

Adds Sync Response and dependencies

Note: PR has link errors since depends on batch and dependencies PR

Screenshots (if appropriate)

Checklist

• I have added a preview link to the PR description.
• I have reviewed the guidelines for contributing to this repository.
• I have read the technical guidelines for contributing to this repository.
• Make sure you have started your change off our latest master.
• I labelled the PR

---

PR Type

Documentation

---

Description

• Added comprehensive documentation for various Tyk Streams configurations including inputs, outputs, processors, and scanners.
• Introduced new guides and examples for configuring and using Tyk Streams features like synchronous responses and HTTP clients.
• Updated the menu structure to incorporate new documentation pages and ensure easy navigation.

---

Changes walkthrough 📝

	Relevant files
Documentation	13 files generate.md Add documentation for Generate Input configuration              tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md Added documentation for the Generate Input configuration. Included examples for cron scheduled processing and generating test data. Detailed configuration fields like mapping, interval, count, etc. +126/-0  http-client.md Document Http Client input configuration                                  tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/http-client.md Added documentation for configuring Http Client input. Explained common and advanced configuration fields. Provided examples for basic pagination. +745/-0  http-client.md Document Http Client output configuration                                tyk-docs/content/product-stack/tyk-streaming/configuration/outputs/http-client.md Added documentation for configuring Http Client output. Explained fields like url, verb, headers, and batching options. Discussed performance benefits and response propagation. +822/-0  stdout.md Update stdout output configuration tags                                    tyk-docs/content/product-stack/tyk-streaming/configuration/outputs/stdout.md Added stdout tag to the existing configuration documentation. +1/-1      sync-response.md Document Sync Response output configuration                            tyk-docs/content/product-stack/tyk-streaming/configuration/outputs/sync-response.md Added documentation for Sync Response output configuration. Explained how it returns message payloads to input origins. Provided example usage with http_server input. +38/-0    http.md Document Http processor configuration                                        tyk-docs/content/product-stack/tyk-streaming/configuration/processors/http.md Added documentation for configuring Http processor. Detailed fields like url, verb, headers, and error handling. Included example for branched request processing. +679/-0  sync-response.md Document Sync Response processor configuration                      tyk-docs/content/product-stack/tyk-streaming/configuration/processors/sync-response.md Added documentation for Sync Response processor. Explained how it adds payloads as synchronous responses. Mentioned compatibility with http_server input. +17/-0    csv.md Document CSV scanner configuration                                              tyk-docs/content/product-stack/tyk-streaming/configuration/scanners/csv.md Added documentation for CSV scanner configuration. Explained fields like custom_delimiter and parse_header_row. Described metadata added by the scanner. +57/-0    lines.md Document Lines scanner configuration                                          tyk-docs/content/product-stack/tyk-streaming/configuration/scanners/lines.md Added documentation for Lines scanner configuration. Explained fields like custom_delimiter and max_buffer_size. +32/-0    overview.md Add overview of scanners in Tyk Streams                                    tyk-docs/content/product-stack/tyk-streaming/configuration/scanners/overview.md Added an overview of scanners in Tyk Streams. Explained the purpose and configuration of scanners. Provided examples for using scanners with file inputs. +36/-0    re-match.md Document Regular Expression Match scanner                                tyk-docs/content/product-stack/tyk-streaming/configuration/scanners/re-match.md Added documentation for Regular Expression Match scanner. Explained fields like pattern and max_buffer_size. +37/-0    switch.md Document Switch scanner configuration                                        tyk-docs/content/product-stack/tyk-streaming/configuration/scanners/switch.md Added documentation for Switch scanner configuration. Explained how to dynamically select child scanners. Provided example for switching based on file name. +68/-0    sync-responses.md Add guide on Synchronous Responses in Tyk Streams                tyk-docs/content/product-stack/tyk-streaming/guides/sync-responses.md Added guide on Synchronous Responses in Tyk Streams. Explained routing processed messages back to inputs. Included examples for using sync responses with HTTP server. +133/-0
Configuration changes	1 files menu.yaml Update menu structure for new documentation                            tyk-docs/data/menu.yaml Updated menu structure to include new documentation pages. Added entries for new configurations and guides. +52/-0

---

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[netlify]

✅ PS. Pls add /docs/nightly to the end of url

Name	Link
🔨 Latest commit	f48ea84
🔍 Latest deploy log	https://app.netlify.com/sites/tyk-docs/deploys/66d9d98ae99e1600080b2a9b
😎 Deploy Preview	https://deploy-preview-5047--tyk-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dcs3spp]

Closing PR in draft status and requires scanner component, http processor and generate input

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit b8fd112)

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Key issues to review Documentation Clarity The documentation for the 'generate' input configuration could be enhanced by providing more detailed examples and explanations of each parameter, especially for complex configurations like 'Cron Scheduled Processing' and 'Generate 100 Rows'. This will help users better understand how to use these features effectively. Configuration Options The 'http_client' input configuration documentation is extensive but could benefit from a clearer separation between basic and advanced configurations. It might help to introduce tabs or sections that explicitly divide these for better readability and navigation. Example Completeness The examples provided in the 'http_client' output configuration are useful, but they lack context on how these configurations affect the behavior of the system. Adding use-case driven examples could provide clarity on when to use certain configurations. Error Handling The section on error handling in the HTTP processor could be expanded to include examples of handling different types of HTTP errors and common recovery patterns. This would be particularly useful for users looking to implement robust error handling strategies in their streaming configurations.

[github-actions]

Persistent review updated to latest commit b8fd112

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to b8fd112

Category	Suggestion	Score
Clarification	Clarify the behavior of the count field when set to zero To avoid potential confusion, explicitly mention that the count field, when set to zero, means infinite generation of messages. This clarification can help users better understand the behavior of the generate input configuration. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [16] -count: 0 +count: 0 # Setting to 0 will generate messages indefinitely Suggestion importance[1-10]: 9 Why: This clarification is important for users to understand the behavior of the count field, preventing potential misuse or confusion regarding message generation.	9
Enhancement	Add a practical example of using a cron expression with a timezone For the interval field, it's beneficial to add an example of using a cron expression with a timezone. This would provide clarity on how to use timezones with cron expressions, which is currently mentioned but not exemplified. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [101] -interval: TZ=Europe/London 30 3-6,20-23 * * * +interval: TZ=Europe/London 0 0 * * * # Every midnight, London time Suggestion importance[1-10]: 8 Why: Providing an example of using a cron expression with a timezone enhances the documentation by offering practical guidance, which is beneficial for users unfamiliar with timezone configurations.	8
	Improve the default mapping to be more descriptive and practical Consider using a more descriptive mapping for the default configuration to avoid potential confusion or misuse. The current default mapping of 'root = "hello world"' is too simplistic and might not be useful in real-world scenarios. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [14] -mapping: root = "hello world" # No default (required) +mapping: root = {"message": "hello world", "timestamp": now().format_timestamp()} Suggestion importance[1-10]: 7 Why: The suggestion to use a more descriptive mapping improves clarity and potential utility, making it more practical for real-world scenarios. However, it is not crucial for functionality.	7
	Enhance the complexity of the mapping example to better demonstrate Bloblang's capabilities It's recommended to provide a more complex example for the mapping field to demonstrate the capabilities of Bloblang. The current examples are too simplistic and do not showcase the potential of using Bloblang for complex data transformations. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [77] -mapping: root = {"test":"message","id":uuid_v4()} +mapping: | + root = { + "id": uuid_v4(), + "timestamp": now().unix(), + "data": { + "value": random_int(), + "description": "Sample data" + } + } Suggestion importance[1-10]: 7 Why: The suggestion to provide a more complex example showcases the potential of Bloblang, which can be educational for users. However, it is not essential for basic understanding.	7

---

Previous suggestions

Suggestions up to commit fd6c0cd

Category	Suggestion	Score
Maintainability	Use more descriptive variable names in the mapping configuration To enhance clarity and maintainability, consider using a more descriptive variable name than foo and bar in the mapping field. This change will make the configuration more understandable to new developers or external users. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [50-60] root = if random_int() % 2 == 0 { { - "type": "foo", - "foo": "is yummy" + "type": "exampleType1", + "exampleType1": "is yummy" } } else { { - "type": "bar", - "bar": "is gross" + "type": "exampleType2", + "exampleType2": "is gross" } } Suggestion importance[1-10]: 6 Why: Using more descriptive variable names enhances code readability and maintainability, making it easier for new developers or users to understand the configuration. This is a minor improvement but beneficial for clarity.	6
Possible issue	Validate the interval field to avoid configuration errors For better error handling and user feedback, consider validating the interval field to ensure it's not set to an empty string unless explicitly intended. This validation can prevent configuration errors that might lead to unexpected behavior. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [48] -interval: "" +interval: "1s" # Default to 1 second if not specified Suggestion importance[1-10]: 5 Why: Validating the interval field to ensure it's not unintentionally left empty can prevent unexpected behavior. However, the empty string is a valid configuration for generating data as fast as possible, so this suggestion might not be universally applicable.	5
Enhancement	Provide a default value for the mapping field to prevent runtime errors It's recommended to provide a default value for the mapping field under generate to avoid potential runtime errors if this field is omitted. Consider adding a sensible default that can serve as an example or placeholder. tyk-docs/content/product-stack/tyk-streaming/configuration/inputs/generate.md [14] -mapping: root = "hello world" # No default (required) +mapping: root = "hello world" # Default example Suggestion importance[1-10]: 4 Why: While providing a default value for the mapping field could prevent runtime errors, the field is marked as required, indicating that users should provide their own value. The suggestion doesn't address a critical issue but could improve usability.	4

[dcs3spp]

/release to release-5.5

[tykbot]

Working on it! Note that it can take a few minutes.

[tykbot]

@dcs3spp Succesfully merged PR

[dcs3spp]

/release to release-5.4

[tykbot]

Working on it! Note that it can take a few minutes.

[tykbot]

@dcs3spp Succesfully merged PR

[dcs3spp]

/release to release-5.3

[tykbot]

Working on it! Note that it can take a few minutes.

[tykbot]

@dcs3spp Succesfully merged PR