-
Notifications
You must be signed in to change notification settings - Fork 469
Description
Contrary to my expectations, it seems that when the toxicity of a reset_peer
toxic is set to 0, the toxic still has a residual impact on the proxy.
To pinpoint the issue, I have created a single toxic with toxicity: 0
:
{
"name": "connection_reset",
"type": "reset_peer",
"stream": "downstream",
"toxicity": 0
}
Despite the toxicity set to 0, we can still see some exceptions thrown during the test (~1 every 10 seconds). This only happens when using toxiproxy.
The README states
toxicity: probability of the toxic being applied to a link
A toxicity value of 0 then means that the toxic has no probability of being applied, suggesting that it is the same as not having the toxic defined at all. However, this was not what I observed.
Evidence
To make sure the exceptions were not caused by my server, I conducted a test where I created a single toxic with toxicity: 0
, followed by the deletion of the toxic. I repeated this process many times, with 5-minute waits between each create and delete operation.
- Create the toxic
curl -X POST http://localhost:63264/proxies/my-proxy/toxics \
-H "Content-Type: application/json" \
-d '{
"name": "connection_reset",
"type": "reset_peer",
"stream": "downstream",
"toxicity": 0
}'
- Delete the toxic
curl -X DELETE http://localhost:63264/proxies/my-proxy/toxics/connection_reset
When the toxic is deleted, the proxy operates without issue. However, when it is introduced with toxicity set to 0, it causes some requests to fail with "Connection reset" errors. The chart below highlights this difference.
Conclusion
Based on the documentation, I expected that a toxicity of 0 would mean the toxic would have no effect, but the observed behavior suggests that toxicity: 0
does not completely disable the toxic.
Activity
mattbardal commentedon Dec 13, 2024
Hey @fmmagalhaes! I'm trying to reproduce this behaviour with a newly created Rails app but I haven't noticed the toxic having any residual impact on the proxy when the toxicity is set to 0.
The steps I followed to try and reproduce are as follows. For reference, I used a brand new Rails
8.0.0.1
app running on port 3000 and Toxiproxy version2.11
.I created a proxy along with the same
reset_peer
toxic specified above and used a simple Ruby script to make GET requests and kept track of the number of errors that occur when making the requests to thelocalhost:22220/up
endpoint on the Rails app.Proxy creation:
Toxic creation:
Ruby script:
After running this script a few times with
toxicity: 0
, I did not encounter anyConnection reset by peer
errors. However, when I removed the toxic and recreated it with atoxicity: 0.25
and tried again, this time the Ruby script started encounteringConnection reset by peer
errors.Is there anything I may be missing while trying to reproduce and are you still encountering this behaviour?
fmmagalhaes commentedon Dec 13, 2024
Hello, @mattbardal
On my side, the client is a Spring Boot application. While I couldn’t replicate the issue locally with the same client, it consistently occurs in my CI/CD pipeline during tests that use Toxiproxy. I’ve managed to reproduce it with two independent applications in that environment.
Unfortunately, I can’t disclose much about the client itself, which I understand makes it harder to identify the problem.
I have also been unable to pinpoint the exact conditions under which the issue is reproducible. However, I believe the test I shared earlier still highlights a potential issue with the toxicity mechanism. Would you agree with that assessment?