You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added comprehensive documentation for various Bloblang methods including JWT parsing, object and array manipulation, string manipulation, timestamp manipulation, number manipulation, parsing, and GeoIP methods.
Each method includes detailed parameters and practical examples to guide users through their implementation.
Changes walkthrough 📝
Relevant files
Documentation
7 files
json-web-tokens.md
Add documentation for JWT parsing methods in Bloblang
Possible Redundancy:
The PR includes extensive documentation for various Bloblang methods across multiple files. It's crucial to ensure there's no redundancy in the content, especially where method descriptions might overlap (e.g., parsing methods and specific data type manipulations).
Consistency in Examples:
Ensure that the examples provided in the documentation are consistent in style and format across all files. This includes the presentation of input and output data, as well as the clarity and relevance of the examples to typical use cases.
Technical Accuracy:
Given the technical nature of the documentation, it's important to verify the accuracy of the descriptions, especially the parameters and behavior of Bloblang methods. This includes correct references to external resources like MaxMind database files or RFC standards.
Accessibility of Content:
The documentation should be accessible to both new and experienced users. This includes clear explanations of Bloblang concepts, method functionalities, and potential use cases. Consider adding a glossary or linking to existing Bloblang resources for unfamiliar terms or concepts.
Replace sensitive key data with placeholders to enhance security
It's recommended to avoid including sensitive information such as private keys directly in the documentation. Instead, consider using placeholders and providing instructions on how users can generate or retrieve their own keys securely.
Why: Replacing sensitive key data with placeholders is crucial for security, especially in documentation, to prevent misuse of private keys.
10
Add a validation step after parsing JWTs to ensure claim integrity
To ensure the security and integrity of JWTs, it's important to validate the claims after parsing the JWT. Consider adding a validation step in the examples to promote best security practices.
root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""")
+if not validate_claims(root.claims):+ raise ValueError("Invalid JWT claims")
Suggestion importance[1-10]: 8
Why: Validating JWT claims after parsing is important for security to ensure the integrity and authenticity of the token, promoting best security practices.
8
Enhancement
Expand the fold method example to include complex data structures and operations
In the fold method example, it would be beneficial to demonstrate the use of the fold method with a more complex data structure, such as an array of objects, to show how it can handle more than just simple arithmetic operations.
```coffee
-root.sum = this.foo.fold(0, item -> item.tally + item.value)-# In: {"foo":[3,8,11]}-# Out: {"sum":22}+root.complex_fold = this.data.fold({}, (tally, item) -> tally.merge({item.key: item.value * 2}))+# In: {"data":[{"key":"a", "value":10}, {"key":"b", "value":20}]}+# Out: {"complex_fold":{"a":20, "b":40}}+# Explanation: This example demonstrates merging objects and applying a function to the values, showcasing the versatility of the `fold` method.
<details><summary>Suggestion importance[1-10]: 9</summary>
Why: Demonstrating the `fold` method with a complex data structure showcases its versatility and provides a more comprehensive understanding of its capabilities, which is highly beneficial.
</details></details></td><td align=center>9
</td></tr><tr><td>
<details><summary>Add examples to illustrate the use of numerical methods for clearer understanding</summary>
___
**Consider adding a brief explanation or example of how the <code>.ceil()</code>, <code>.floor()</code>, and <br><code>.round()</code> methods modify the number types. This can help users understand the <br>practical application of these methods in their Bloblang scripts.**
[tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/arithmetic.md [19]](https://github.com/TykTechnologies/tyk-docs/pull/5063/files#diff-5fa5b747d85c0b8e2a34c74ad377891d511b36213d306cd20ed931ccaa36f7b0R19-R19)
```diff
-In order to explicitly coerce numbers into integer types you can use the [.ceil(), .floor(), or .round()]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/numbers" >}}) methods.
+In order to explicitly coerce numbers into integer types you can use the [.ceil(), .floor(), or .round()]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/numbers" >}}) methods. For example, `123.456.ceil()` results in `124`, `123.456.floor()` results in `123`, and `123.456.round()` results in `123`.
Suggestion importance[1-10]: 8
Why: Adding examples for the .ceil(), .floor(), and .round() methods significantly improves user understanding and practical application of these methods in Bloblang scripts.
8
Clarify the behavior of the ceil method for negative numbers
To avoid potential confusion and ensure accuracy in documentation, explicitly mention the behavior of the ceil method when dealing with negative numbers, as the current example might imply incorrect rounding direction for negative decimals.
-Returns the least integer value greater than or equal to a number. If the resulting value fits within a 64-bit integer then that is returned, otherwise a new floating point number is returned.+Returns the least integer value greater than or equal to a number. For positive numbers, it rounds up to the nearest integer. For negative numbers, it rounds up towards zero (e.g., -5.9 becomes -5). If the resulting value fits within a 64-bit integer, that is returned; otherwise, a new floating point number is returned.
Suggestion importance[1-10]: 8
Why: Clarifying the behavior of the ceil method for negative numbers improves the accuracy and comprehensibility of the documentation, which is important for users.
8
Improve the example for the assign method to demonstrate merging behavior with unique keys
The example code block for the assign method should demonstrate the behavior when there is a collision in the merged structures. The current example only shows the case where the destination object's value is overwritten. It would be helpful to include an example where both source and destination have unique keys to illustrate the merging behavior more clearly.
<details><summary>Suggestion importance[1-10]: 8</summary>
Why: The suggestion enhances the clarity of the `assign` method by showing how unique keys are merged, which is valuable for understanding the method's behavior in different scenarios.
</details></details></td><td align=center>8
</td></tr><tr><td>
<details><summary>Add an example of using a custom comparison function in the <code>sort</code> method</summary>
___
**The <code>sort</code> method example should include a demonstration of sorting with a custom <br>comparison function to provide clarity on how to use the optional <code>compare</code> parameter <br>effectively.**
[tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/object-and-arrays.md [674-677]](https://github.com/TykTechnologies/tyk-docs/pull/5063/files#diff-a3691299ea4fb575c0350c4977372bbb8866e8b7e470f2b51f011e7df9238ef5R674-R677)
```diff
```coffee
-root.sorted = this.foo.sort()
-# In: {"foo":["bbb","ccc","aaa"]}
-# Out: {"sorted":["aaa","bbb","ccc"]}
+root.sorted = this.foo.sort((left, right) -> left.length() < right.length())
+# In: {"foo":["longer","short","mid"]}
+# Out: {"sorted":["short","mid","longer"]}
+# Explanation: This example uses a custom comparison function based on string length to sort the array.
<details><summary>Suggestion importance[1-10]: 8</summary>
Why: Including an example with a custom comparison function for the `sort` method illustrates the use of the optional `compare` parameter effectively, enhancing the documentation's usefulness.
</details></details></td><td align=center>8
</td></tr><tr><td rowspan=3><strong>Best practice</strong></td>
<td>
<details><summary>Add error handling to JWT parsing examples for increased robustness</summary>
___
**The example code snippets for JWT parsing should include error handling to manage <br>cases where the JWT is invalid or the parsing fails. This is crucial for robustness, <br>especially in production environments.**
[tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/json-web-tokens.md [21-24]](https://github.com/TykTechnologies/tyk-docs/pull/5063/files#diff-c906fa396ee55f8cb1b40989a680f9ecbaf1b4b1eb3753e0e3b7611708e14afeR21-R24)
```diff
-root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY-----
-... public key data ...
------END PUBLIC KEY-----""")
+try:
+ root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY-----
+ ... public key data ...
+ -----END PUBLIC KEY-----""")
+except Exception as e:
+ root.error = str(e)
Suggestion importance[1-10]: 9
Why: Adding error handling is a best practice that enhances the robustness of the code, especially in production environments where parsing failures need to be managed gracefully.
9
Add a warning about potential integer overflow in type conversion methods
Consider adding a note or warning about the potential for integer overflow when using type conversion methods like int16, int32, etc., especially when dealing with large or unknown input values. This can help users avoid unexpected behavior or errors in their applications.
-Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).+Converts a numerical type into a 16-bit signed integer. Note: Be cautious of integer overflow when dealing with large or unknown input values. This method is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
Suggestion importance[1-10]: 9
Why: Adding a warning about potential integer overflow is crucial for preventing unexpected behavior and errors, especially in advanced use cases.
9
Add a performance note to the from_all method description to aid in efficient usage
It would be beneficial to add a note about the performance implications of using the from_all method, especially when dealing with large batches of data. This can help users make more informed decisions about when and how to use this method effectively.
-Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array.+Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Note: Using `from_all` on large batches may impact performance, so it should be used judiciously.
Suggestion importance[1-10]: 7
Why: Including a note about the performance implications of using the from_all method helps users make informed decisions and use the method more effectively, especially with large data batches.
7
Clarification
Clarify the encoding behavior and options for JSON field mapping to enhance user understanding
It's beneficial to clarify that the default base64 encoding can be overridden, as this might not be immediately clear to all users. A brief mention of how to specify a different encoding method could prevent confusion and errors in data handling.
-Returns the full raw contents of the mapping target message as a byte array. When mapping to a JSON field the value should be encoded using the method [encode]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/encoding-and-encryption#encode" >}}), or cast to a string directly using the method [string]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/type-coercion#string" >}}), otherwise it will be base64 encoded by default.+Returns the full raw contents of the mapping target message as a byte array. When mapping to a JSON field, the value is base64 encoded by default but can be encoded differently using the method [encode]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/encoding-and-encryption#encode" >}}) with your choice of encoding scheme, or cast to a string directly using the method [string]({{< ref "/product-stack/tyk-streaming/guides/bloblang/methods/type-coercion#string" >}}).
Suggestion importance[1-10]: 7
Why: The suggestion clarifies the default encoding behavior and how to override it, which can prevent confusion and errors in data handling.
7
Maintainability
Use a variable for the signing secret to improve code clarity
To improve the clarity and maintainability of the code examples, consider using a variable for the signing secret instead of embedding it directly in the function call. This makes the examples easier to read and modify.
Why: Using a variable for the signing secret improves code readability and maintainability, making it easier to modify and understand.
7
Use more descriptive variable names
To enhance code clarity and maintainability, consider using a more descriptive variable name than ele in the lambda function used with map_each. A name like number or value could make the code more readable and understandable.
Why: Using more descriptive variable names enhances code readability and maintainability, though it is a minor improvement.
6
Possible bug
Add error handling to the map_each function
It is recommended to add error handling for the map_each function to ensure robustness in case of unexpected input types or values that do not support the abs() method. This can prevent runtime errors and improve the reliability of the code.
Why: Adding error handling improves robustness and prevents runtime errors, but the exact error handling mechanism should be verified for compatibility with Bloblang.
7
Clarity
Clarify the output structure in the explode method example for better understanding
The explode method example for objects should clarify the output structure when exploding nested objects. The current example might confuse readers about how the keys and values are represented in the output. A more detailed explanation or a simpler example could enhance understanding.
```coffee
root = this.explode("value")
# In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}}
-# Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}}+# Out: {"foo":{"id":1,"value":2},"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}}}+# Explanation: Each key within the 'value' object becomes a top-level key in the output, with its value retaining the structure from the input but prefixed with the original object's 'id'.
<details><summary>Suggestion importance[1-10]: 7</summary>
Why: The suggestion improves the explanation of the `explode` method, making it easier for readers to understand how nested objects are handled, which enhances clarity.
</details></details></td><td align=center>7
</td></tr><tr><td rowspan=1><strong>Readability</strong></td>
<td>
<details><summary>Improve the readability of code examples by adding comments and breaking down complex expressions</summary>
___
**To enhance readability and maintainability, consider breaking down the long code <br>examples into smaller segments or adding inline comments. This can make the examples <br>easier to understand at a glance, especially for users who are new to Bloblang.**
[tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/encoding-and-encryption.md [23-25]](https://github.com/TykTechnologies/tyk-docs/pull/5063/files#diff-951ad1f1a3244545cb537c828783a9bc2981c01ca57f23faecf6da69235a2971R23-R25)
```diff
```coffee
+# Generate a long string by repeating content 1000 times
let long_content = range(0, 1000).map_each(content()).join(" ")
+# Calculate the length of the uncompressed string
root.a_len = $long_content.length()
+# Calculate the length after gzip compression
root.b_len = $long_content.compress("gzip").length()
<details><summary>Suggestion importance[1-10]: 6</summary>
Why: Adding comments and breaking down complex expressions in code examples enhances readability and helps users, especially those new to Bloblang, understand the examples better.
</details></details></td><td align=center>6
</td></tr></tr></tbody></table>
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
buger
deleted the
merge/release-5.4/cf089bfb21c5da52bb2979900b805a66c869c890
branch
User description
DX-1496 Refactor Blob Lang Methods Page For Indexing (#5036)
refactor methods page for indexing
Co-authored-by: Simon Pears [email protected]
Co-authored-by: Yaara [email protected]
Co-authored-by: Yaara [email protected]
PR Type
documentation
Description
Changes walkthrough 📝
7 files
json-web-tokens.md
Add documentation for JWT parsing methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/json-web-tokens.md
object-and-arrays.md
Add documentation for object and array methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/object-and-arrays.md
strings.md
Add documentation for string manipulation methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/strings.md
timestamps.md
Add documentation for timestamp manipulation methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/timestamps.md
numbers.md
Add documentation for number manipulation methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/numbers.md
parsing.md
Add documentation for parsing methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/parsing.md
geoip.md
Add documentation for GeoIP methods in Bloblangtyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/geoip.md
9 files
encoding-and-encryption.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/encoding-and-encryption.md
...
type-coercion.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/type-coercion.md
...
general.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/general.md
...
regular-expressions.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/regular-expressions.md
...
overview.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/overview.md
...
arithmetic.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/arithmetic.md
...
overview.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/methods/overview.md
...
menu.yaml
...tyk-docs/data/menu.yaml
...
functions.md
...tyk-docs/content/product-stack/tyk-streaming/guides/bloblang/functions.md
...