Support Non-Form GET/POST Requests in Sites Tree Export #8941
Description
Describe the bug:
The Sites Tree File Format export produced by the exim add-on treats all request with bodies as if they are application/x-www-form-urlencoded POST requests using the format key1=value1&key2=value2. It attempts to parse them, strip values, and produce an output like key1=&key2=.
This produces inconsistencies when the request body uses other formats, such as JSON or XML:
- Multi-line bodies are truncated and only the first line is preserved in the
datafield. - Bodies with
=characters are misinterpreted. For example, an XML declaration like<?xml version="1.0"?>becomes<?xml version=after parsing. - The bodies of non-POST requests with bodies (e.g.
PUT,PATCH) are not included, since onlyPOSTrequests go through the body-extraction logic.
Steps to reproduce the behavior:
Run the following automation plan, which uses a requestor job to send different types of HTTP requests, and an export job to export the site tree.
env:
contexts:
- name: Default Context
urls:
- https://httpbingo.org
jobs:
- type: requestor
requests:
- name: GET Request
url: https://httpbingo.org/get?key1=value1&key2=value2
method: GET
- name: POST Request With Form Data
url: https://httpbingo.org/post
method: POST
data: 'key1=value1&key2=value2'
headers:
- "Content-Type: application/x-www-form-urlencoded"
- name: PUT Request
url: https://httpbingo.org/put
method: PUT
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With Multipart Form Data
url: https://httpbingo.org/post
method: POST
data: |
--boundary
Content-Disposition: form-data; name="key1"
value1
--boundary
Content-Disposition: form-data; name="key2"
value2
--boundary--
headers:
- "Content-Type: multipart/form-data; boundary=boundary"
- name: PATCH Request
url: https://httpbingo.org/patch
method: PATCH
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With JSON Data
url: https://httpbingo.org/post
method: POST
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With XML Data
url: https://httpbingo.org/post
method: POST
data: |
<?xml version="1.0"?>
<key1>value1</key1>
<key2>value2</key2>
headers:
- "User-Agent: test"
- 'Content-Type: application/xml'
- type: export
parameters:
type: yaml
source: sitestree
fileName: sitestree.yamlNote: httpbingo.org requires a User-Agent header to respond correctly, but it's omitted here for brevity since the response is not relevant to this report.
The resulting sitestree.yaml file should look like this:
- node: Sites
children:
- node: https://httpbingo.org
url: https://httpbingo.org
method: GET
children:
- node: "GET:get(key1,key2)"
url: https://httpbingo.org/get?key1=value1&key2=value2
method: GET
responseLength: 930
statusCode: 200
- node: PATCH:patch
url: https://httpbingo.org/patch
method: PATCH
responseLength: 1080
statusCode: 200
- node: POST:post()(<?xml version)
url: https://httpbingo.org/post
method: POST
responseLength: 1100
statusCode: 200
data: '%3C%3Fxml+version='
- node: "POST:post()(key1,key2)"
url: https://httpbingo.org/post
method: POST
responseLength: 1103
statusCode: 200
data: key1=&key2=
- node: POST:post()(multipart/form-data)
url: https://httpbingo.org/post
method: POST
responseLength: 1238
statusCode: 200
- node: "POST:post()({\"key1\": \"value1\", \"key2\": \"value2\"})"
url: https://httpbingo.org/post
method: POST
responseLength: 1078
statusCode: 200
data: '%7B%22key1%22%3A+%22value1%22%2C++%22key2%22%3A+%22value2%22%7D='
- node: PUT:put
url: https://httpbingo.org/put
method: PUT
responseLength: 1076
statusCode: 200Note the following behavior:
Expected
GET Request: no body, nodatafield.POST Request with Form Data: keys are kept, values are stripped.POST Request with Multipart Form Data: the body is present but intentionally left out in the export.
Unexpected
PUT RequestandPATCH Requestare missing thedatafield, even though they have bodies.POST Request with JSON Datais mis-parsed (ends with an=character.)POST Request with XML Datais mis-parsed (only the first line is kept, and everything after the first=is removed.)
Expected behavior:
- PUT and PATCH requests include a
datafield as long as they have bodies. - Request bodies are exported with minimal parsing to support non-form requests.
Software versions:
ZAP
Version: 2.16.1
Installed Add-ons: [...[id=exim, version=0.14.0]...]
Screenshots:
No response
Errors from the zap.log file:
No response
Additional context:
Omitting the values of each key in x-www-form-urlencoded bodies seems reasonable, but this approach could lead to inconsistencies with JSON, XML, and possibly other formats. These would need to either:
- Contain the full body
- Or require parsing and sanitizing based on their formats, which (IMHO) adds unnecessary complexity
It might be worth considering keeping the entire body with minimal parsing, unless there's a strong reason to do otherwise.
Would you like to help fix this issue?
- Yes