Testing SMTP Server Error Responses

[IBlasterus]

Sometimes during development you need to write logic for error responses from the SMTP server.
Can you add this feature to Mailpit Test Server?
There could be some startup keys that make Mailpit respond with an error.
For example, 451.

[axllent]

Previously a "chaos monkey" feature was requested (see #110, #144 & #268) to return SMTP errors at random. That proposal was rejected at the time because it was impossible to implement (properly) due to the fact that Mailpit was reliant a third party library for SMTP. Whilst Mailpit is still using a version of that library, it is now within the Mailpit codebase meaning it is now technically possible.

What you are asking for however is something different, so to help me understand your requirements:

1. Are you wanting a 100% chance of failure in your tests, or at random?
2. Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK?
3. Should the error result in a SMTP server disconnection, or...?
4. Anything else you can think of...

The example given previously was MailHog's Jim feature which is what many/all of these previous requests were based on, and which may help provide some suggestions and ideas for this discussion.

Just to be clear, I'm not saying (yet) whether I will add this option, I'm just a lot more open to it now because it is now possible, it just needs to be implemented properly which takes a solid understanding of the requirements, and of course planning. It is also worth mentioning that I am having shoulder surgery in 2 days, so if I am to add this option it would likely only be in January or February next year, which gives me plenty of time to consider the options before then.

[IBlasterus]

Thank you very much for taking the time to answer my question.
I wish you a speedy recovery.

1. Are you wanting a 100% chance of failure in your tests, or at random?

Especially for me need 100% chance of failure.

2. Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK?

Need specific type of failure. For example, for development backend reaction on 451 error.

3. Should the error result in a SMTP server disconnection, or...?

Maybe. My Backend connect to SMTP, send E-mail and disconnect.

4. Anything else you can think of...

Nothing else comes to mind.

[ThomasLandauer]

Just an idea on how to implement this:
Hard-code an email address like [email protected], and if it's used as sender/receiver, return that error.
Any more fancy way (like sending some special keywords in the SMTP session) won't work, since most users probably can't modify that.

[axllent]

Now that's an interesting approach, thanks for the idea @ThomasLandauer 👍

Edit: After some thought and as nice as this idea sounds, it is very limited in that the email would need to have already been processed in order to process the addresses, meaning the only failure one could trigger this way would be after a successful transaction.

[axllent]

I have just released v1.22.0 which adds new "Chaos" functionality (see release notes and links for documentation). The configuration is fairly flexible to allow you to set the error code and probability (which determines whether you want it to trigger never, randomly, or always), and can be set / changed via CLI flags (or environment variables), and if enabled, can be modified via both the web UI and API.

Please let me know how it works for you? Thanks.

[ThomasLandauer]

I've read those issues where people are requesting that chaos feature, but - to be honest - I don't understand their use case, (i.e. how adding unpredictability helps ;-)

Anyway, my use case is: I have a test suite that I run before deploying. The test suite contains a test for a successful connection to my mailserver (represented by Mailpit), and one test for an unsuccessful connection. The current implementation (if I understand it right) would require me to create a separate test suite for the unsuccessful test, and restart Mailpit when switching between theses test suites. This feels even clumsier than the hack I currently have (see #405 (comment))

So what I'd like to have would be a way to "request" an error from within the test. So my idea would be to either hard-code some "special" email address ([email protected]) or subject ("Please answer with error") or any other part of the email itself (X-Mailpit-Give-Error: Yes). Or add some API call (that can be triggered with cURL) which causes an error in the next SMTP session.

[axllent]

Thanks for the feedback @ThomasLandauer. Chaos (engineering) revolves around handling unpredictability (ie: failures at random), which is different to you r case where you want to predict the failure. Predictability however can be achieved with the same functionality by setting the failure trigger to 100(%) probability, guaranteeing it will fail.

If I understand correctly, your test suite spins up an instance of Mailpit when it runs the test. You do not need to spin up a separate instance of Mailpit or do a completely separate test to get a failure, you just need to:

1. Spin up the initial instance of Mailpit with the --enable-chaos flag (or MP_ENABLE_CHAOS=true env)
2. Run your successful test
3. Do a PUT request to the API to set chaos trigger you want to fail and the failure code
4. Run your failure test

This matches your last point exactly, so either I'm confused, or maybe the instructions are not clear?

[ThomasLandauer]

Thanks, this works! :-)

Some suggestions (somewhat unordered - sorry):

• If I hadn't followed this issue, I never would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place). Maybe even add a copy-pastable PUT request somewhere: 'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'
• At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning http://0.0.0.0:8025/ - is this a placeholder or the actual URL? I'm now using http://127.0.0.1:8025/api/v1/chaos
• Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples):

  {
  "AUTH": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "MAIL": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "RCPT": {
  "ErrorCode": 451,
  "Probability": 5
  }
  }

• Does the response to PUT contain a trailing \n? I'm getting:

  - Expected | + Actual
  @@ @@
  -'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}'
  +'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}
  +'
  

[axllent]

Thanks, this works! :-)

Great!

Some suggestions (somewhat unordered - sorry):

* If I hadn't followed this issue, I **never** would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place).

It's currently described & linked to from the project's README, Mailpit's home page, the features page, the configuration page , and the runtime options pages, so I really don't think it can be any more prominent than that, especially considering it's an advanced option which most users will never need or use. ;-)

Maybe even add a copy-pastable PUT request somewhere: `'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'`

Whilst I get your point, the online API documentation differs somewhat from the bundled (in Mailpit itself) API documentation which does - there are actually clear curl examples in the bundled documentation. The reason for the difference is that the website cannot know what IP your Mailpit instance is actually using, it's webroot, or whether it's even running, and I'm concerned that the average user will start posting Github issues because "it's not working" for them.

I will make the information about the bundled API documentation clearer on the website and explain this.

* At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning `http://0.0.0.0:8025/` - is this a placeholder or the actual URL? I'm now using `http://127.0.0.1:8025/api/v1/chaos`

0.0.0.0 is actually the same as 127.0.0.1, which is the same as localhost. As above, I will try make this clearer and more consistent.

* Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples):
  {
  "AUTH": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "MAIL": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "RCPT": {
  "ErrorCode": 451,
  "Probability": 5
  }
  }

Ironically, I had it as AUTH, MAIL and RCPT right up until the last minute before committing - but figured it may cause confusion (plus I needed to then explain each step in the web UI). In the end I went with the more human-readable object keys. Too late to change now, the feature is in the API and changing that would be a breaking change.

* Does the response to PUT contain a trailing `\n`? I'm getting:
  ```
  - Expected | + Actual
  @@ @@
  -'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}'
  +'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}
  +'
  ```

Quite possibly - I don't format the HTTP response manually, it's using Go's built-in JSON encoding to turn a struct (of the current config) into a JSON string. On that note, JSON should never be validated as a 1:1 string, as it it is prone to formatting differences (eg: "pretty"), and the fact that the order of keys can differ per implementation (Go will set them alphabetically, but PHP for instance won't). A JSON string should be converted to JSON and then parsed, and a JSON string with or without trailing lines is valid.