autodiscover/autoconfig errors and tunnings #223

stefaweb · 2018-07-01T06:41:06Z

Hello!

I tried to summarize infos and problems with autodiscover/autoconfig features.

Modoboa server is installed with Modoboa-installer on Debian Stretch.

In DNS, I have:

autodiscover.domain.tld. CNAME mail.domain.tld.
autoconfig.domain.tld. CNAME mail.domain.tld.
_autodiscover._tcp.domain.tld SRV 1 1 443  mail.domain.tld.
_imaps._tcp.domain.tld SRV 1 1 993  mail.domain.tld.
_pop3s._tcp.domain.tld SRV 10 1 995 mail.domain.tld.

At output, autodiscover is not working from outside request.

If I run this command on the server /srv/automx/env/bin/automx-test [email protected], its working for autoconfig and autodiscover.

But in direct from the navigator with https://autodiscover.domain.tld/autodiscover/autodiscover.xml, I got an 500 error.

mail.domain.tld [pid: 17026|app: 0|req: 12/13] 82.67.159.142 () {44 vars in 860 bytes} [Fri Jun 29 12:05:25 2018] GET /autodiscover/autodiscover.xml => generated 0 bytes in 2 msecs (HTTP/2.0 500) 2 headers in 82 bytes (1 switches on core 0)

autoconfig is working fine.

http://autoconfig.domain.tld/mail/[email protected] works fine with automx-test and with a navigator.

Two issues seems related: #151, #174

More on this.

Working session with automx-test:

2018-06-30 10:22:30,954 DEBUG: DOCUMENT_ROOT: /srv/automx/instance
2018-06-30 10:22:30,954 DEBUG: CONTENT_TYPE: application/x-www-form-urlencoded
2018-06-30 10:22:30,955 DEBUG: wsgi.input: <uwsgi._Input object at 0x7f85d1dbd468>
2018-06-30 10:22:30,955 DEBUG: HTTP_HOST: autodiscover.domain.tld
2018-06-30 10:22:30,955 DEBUG: HTTPS: on
2018-06-30 10:22:30,955 DEBUG: wsgi.multithread: False
2018-06-30 10:22:30,955 DEBUG: HTTP_CONTENT_TYPE: application/x-www-form-urlencoded
2018-06-30 10:22:30,955 DEBUG: REQUEST_URI: /mobileconfig
2018-06-30 10:22:30,955 DEBUG: HTTP_ACCEPT: */*
2018-06-30 10:22:30,956 DEBUG: wsgi.version: (1, 0)
2018-06-30 10:22:30,956 DEBUG: wsgi.run_once: False
2018-06-30 10:22:30,956 DEBUG: wsgi.errors: <open file 'wsgi_errors', mode 'w' at 0x7f85d1da6d20>
2018-06-30 10:22:30,956 DEBUG: REMOTE_PORT: 57682
2018-06-30 10:22:30,956 DEBUG: REQUEST_SCHEME: https
2018-06-30 10:22:30,956 DEBUG: UWSGI_APPID: autodiscover.domain.tld|
2018-06-30 10:22:30,956 DEBUG: uwsgi.version: 2.0.14-debian
2018-06-30 10:22:30,957 DEBUG: HTTP_CONTENT_LENGTH: 67
2018-06-30 10:22:30,957 DEBUG: wsgi.file_wrapper: <built-in function uwsgi_sendfile>
2018-06-30 10:22:30,957 DEBUG: HTTP_ACCEPT_ENCODING: identity
2018-06-30 10:22:30,957 DEBUG: --------------- END environ ---------------
2018-06-30 10:22:30,957 DEBUG: Request POST (raw)

Crashed session with direct call from navigator:

2018-06-30 10:23:39,266 DEBUG: DOCUMENT_ROOT: /srv/automx/instance
2018-06-30 10:23:39,266 DEBUG: wsgi.input: <uwsgi._Input object at 0x7f85d1dbd468>
2018-06-30 10:23:39,266 DEBUG: HTTP_DNT: 1
2018-06-30 10:23:39,266 DEBUG: HTTP_HOST: autodiscover.domain.tld
2018-06-30 10:23:39,266 DEBUG: HTTPS: on
2018-06-30 10:23:39,266 DEBUG: wsgi.multithread: False
2018-06-30 10:23:39,266 DEBUG: REQUEST_URI: /autodiscover/autodiscover.xml
2018-06-30 10:23:39,267 DEBUG: HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2018-06-30 10:23:39,267 DEBUG: wsgi.version: (1, 0)
2018-06-30 10:23:39,267 DEBUG: wsgi.run_once: False
2018-06-30 10:23:39,267 DEBUG: wsgi.errors: <open file 'wsgi_errors', mode 'w' at 0x7f85d1da6d20>
2018-06-30 10:23:39,267 DEBUG: REMOTE_PORT: 53541
2018-06-30 10:23:39,267 DEBUG: HTTP_ACCEPT_LANGUAGE: fr-fr
2018-06-30 10:23:39,267 DEBUG: REQUEST_SCHEME: https
2018-06-30 10:23:39,268 DEBUG: UWSGI_APPID: autodiscover.domain.tld|
2018-06-30 10:23:39,268 DEBUG: uwsgi.version: 2.0.14-debian
2018-06-30 10:23:39,268 DEBUG: CONTENT_TYPE:
2018-06-30 10:23:39,268 DEBUG: wsgi.file_wrapper: <built-in function uwsgi_sendfile>
2018-06-30 10:23:39,268 DEBUG: HTTP_ACCEPT_ENCODING: br, gzip, deflate
2018-06-30 10:23:39,269 DEBUG: --------------- END environ ---------------
autodiscover.domain.tld [pid: 25376|app: 0|req: 50/60] xx.xx.xx.xx () {44 vars in 770 bytes} [Sat Jun 30 10:23:39 2018] GET /autodiscover/autodiscover.xml => generated 0 bytes in 8 msecs (HTTP/1.1 500) 2 headers in 82 bytes (2 switches on core 0)

I made these modfications to nginx vhosts:

In /etc/nginx/sites-available/autoconfig.domain.tld.conf add this block:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name autodiscover.domain.tld;
    root /srv/automx/instance;

    ssl_certificate /etc/letsencrypt/live/mail.domain.tld/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mail.domain.tld/privkey.pem; # managed by Certbot
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_verify_depth 3;
    ssl_dhparam /etc/nginx/dhparam.pem;

    client_max_body_size 10M;

    access_log /var/log/nginx/autodiscover.domain.tld-access.log;
    error_log /var/log/nginx/autodiscover.domain.tld-error.log;

    location ~* ^/autodiscover/autodiscover.xml {
        include uwsgi_params;
        uwsgi_pass automx;
    }

    location /mail/config-v1.1.xml {
        include uwsgi_params;
        uwsgi_pass automx;
    }

    location /mobileconfig {
        include uwsgi_params;
        uwsgi_pass automx;
    }
}

In /etc/nginx/sites-available/maildomain.tld.conf change:

location /autodiscover/autodiscover.xml {

with (take care of lower/upper case in URL):

location ~* ^/autodiscover/autodiscover.xml

Result.

If you use these tools, it works:

https://testconnectivity.microsoft.com (click on Outlook Autodiscover)
https://www.mailenable.com/Tools/AutoDiscover/validate.asp

If I try direct with a navigator, it doesn't work.

Currently, autodiscover doesn't work with Outlook Mac, Apple Mail and iPhone Mail as direct call using https produce a 500 error.

The text was updated successfully, but these errors were encountered:

stefaweb · 2018-07-01T09:50:22Z

Found this with nginx debug.

Something generate a Resource temporarily unavailable.

With navigator (tried with Safari, Firefox and Chrome):
https://autodiscover.domain.tld/autodiscover/autodiscover.xml
In /var/log/nginx/autodiscover.domain.tld-error.log.

2018/07/01 11:41:31 [debug] 3347#3347: *1 http finalize request: -4, "/autodiscover/autodiscover.xml?" a:1, c:2
2018/07/01 11:41:31 [debug] 3347#3347: *1 http request count:2 blk:0
2018/07/01 11:41:31 [debug] 3347#3347: *1 http run request: "/autodiscover/autodiscover.xml?"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream check client, write event:1, "/autodiscover/autodiscover.xml"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream recv(): -1 (11: Resource temporarily unavailable)
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream request: "/autodiscover/autodiscover.xml?"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream dummy handler
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream request: "/autodiscover/autodiscover.xml?"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http upstream process header
2018/07/01 11:41:31 [debug] 3347#3347: *1 malloc: 000055E56627CF00:4096
2018/07/01 11:41:31 [debug] 3347#3347: *1 recv: fd:22 82 of 4096
2018/07/01 11:41:31 [debug] 3347#3347: *1 http uwsgi status 500 "500 Internal Server Error"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http uwsgi header: "Content-Type: text/html"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http uwsgi header: "Content-Length: 0"
2018/07/01 11:41:31 [debug] 3347#3347: *1 http uwsgi header done
2018/07/01 11:41:31 [debug] 3347#3347: *1 xslt filter header
2018/07/01 11:41:31 [debug] 3347#3347: *1 HTTP/1.1 500 Internal Server Error

With automx-test:
In /var/log/nginx/autodiscover.domain.tld-error.log.

2018/07/01 11:42:00 [debug] 3348#3348: *5 http finalize request: -4, "/autodiscover/autodiscover.xml?" a:1, c:2
2018/07/01 11:42:00 [debug] 3348#3348: *5 http request count:2 blk:0
2018/07/01 11:42:00 [debug] 3348#3348: *5 post event 000055E5662BDEC0
2018/07/01 11:42:00 [debug] 3348#3348: *5 delete posted event 000055E5662BDEC0
2018/07/01 11:42:00 [debug] 3348#3348: *5 http run request: "/autodiscover/autodiscover.xml?"
2018/07/01 11:42:00 [debug] 3348#3348: *5 http read client request body
2018/07/01 11:42:00 [debug] 3348#3348: *5 SSL_read: 373
2018/07/01 11:42:00 [debug] 3348#3348: *5 http client request body recv 373
2018/07/01 11:42:00 [debug] 3348#3348: *5 http body new buf t:1 f:0 000055E566271760, pos 000055E566271760, size: 373 file: 0, size: 0
2018/07/01 11:42:00 [debug] 3348#3348: *5 http client request body rest 0
2018/07/01 11:42:00 [debug] 3348#3348: *5 event timer del: 18: 1530438180026
2018/07/01 11:42:00 [debug] 3348#3348: *5 http init upstream, client timer: 0
2018/07/01 11:42:00 [debug] 3348#3348: *5 epoll add event: fd:18 op:3 ev:80002005
2018/07/01 11:42:00 [debug] 3348#3348: *5 posix_memalign: 000055E56627BEB0:512 @16
2018/07/01 11:42:00 [debug] 3348#3348: *5 http script copy: "QUERY_STRING"

With https://www.mailenable.com/Tools/AutoDiscover/validate.asp.
In /var/log/nginx/autodiscover.domain.tld-error.log.

2018/07/01 14:27:26 [debug] 3348#3348: *24 http finalize request: -4, "/AutoDiscover/AutoDiscover.xml?" a:1, c:2
2018/07/01 14:27:26 [debug] 3348#3348: *24 http request count:2 blk:0
2018/07/01 14:27:26 [debug] 3348#3348: *24 post event 000055E5662BDEC0
2018/07/01 14:27:26 [debug] 3348#3348: *24 delete posted event 000055E5662BDEC0
2018/07/01 14:27:26 [debug] 3348#3348: *24 http run request: "/AutoDiscover/AutoDiscover.xml?"
2018/07/01 14:27:26 [debug] 3348#3348: *24 http read client request body
2018/07/01 14:27:26 [debug] 3348#3348: *24 SSL_read: 1
2018/07/01 14:27:26 [debug] 3348#3348: *24 SSL_read: 377
2018/07/01 14:27:26 [debug] 3348#3348: *24 http client request body recv 378
2018/07/01 14:27:26 [debug] 3348#3348: *24 http body new buf t:1 f:0 000055E5661F8EF0, pos 000055E5661F8EF0, size: 378 file: 0, size: 0
2018/07/01 14:27:26 [debug] 3348#3348: *24 http client request body rest 0
2018/07/01 14:27:26 [debug] 3348#3348: *24 event timer del: 18: 1530448106482
2018/07/01 14:27:26 [debug] 3348#3348: *24 http init upstream, client timer: 0
2018/07/01 14:27:26 [debug] 3348#3348: *24 epoll add event: fd:18 op:3 ev:80002005
2018/07/01 14:27:26 [debug] 3348#3348: *24 posix_memalign: 000055E5662756F0:512 @16
2018/07/01 14:27:26 [debug] 3348#3348: *24 http script copy: "QUERY_STRING"

stefaweb · 2018-07-02T09:08:30Z

More test.

new block in autoconfig.domain.tld.conf no needed.
we just need to mod location for autodiscover (location ~* ^/autodiscover/autodiscover.xml) to catch lower/upercase.

Still trying to have the right POST in my RestClient but autodiscover works with Windows.

PR #224

nickdbush · 2018-10-02T17:55:37Z

I'm running into a similar issue, currently researching.

tonioo · 2018-11-28T08:34:42Z

@stefaweb @nickdbush Any news about this issue? Unfortunately, I don't have any mac device to reproduce it...

tonioo · 2018-11-28T09:43:07Z

Issue #253 should interest you guys.

gianks · 2019-07-13T12:44:54Z

Hi, any progress?
I have the same problem accessing http://autoconfig with a 500 error returned.
Using https returns a 400 Bad Request: seems that the subdomain autoconfig is not configured to receive its own ssl certificate from LetsEncrypt and meanwhile nginx doesn't redirect to mail.whatsoever (as suggested in this other issue: #222 ) which i tried and anyway responds with a "Page doesn't exists", actually without errors.

horvan · 2019-11-07T22:32:04Z

automx is not installed after auto installer is used. Tested today with ubuntu 18.04 Maybe you should replace automx by z-push available on z-push.org Maybe modoboa will rock again if you do so

dbryar · 2019-11-11T13:42:11Z

automx was installed with everything else for me ok (1.14, RasPi Stretch), but only autoconfigure.example.com returns data to requests while a GET request for autodiscover.example.com returns a 500 internal server error.

Running the test from https://testconnectivity.microsoft.com/ returns a positive result when choosing the 'Outlook Autodiscover' option after making some changes to the setup, namely;

moved all the autodiscover and mobileconfig settings from the mail.example.com site config file to the autoconfig.example.com site config in Nginx
created a location in both the Nginx configs to route /.well-know/acme-challenge

    location /.well-known/acme-challenge {
        root /var/www/html;
    }

ran certbot to create a certificate for autodiscover.example.com and autoconfig.example.com in addition to the existing mail.example.com

sudo certbot certonly \
    --webroot -n --agree-tos --force-renewal \
    -w /var/www/html \
    --email [email protected] \
	-d mail.example.com \
	-d autodiscover.example.com \ 
        -d autoconfig.example.com

move autoconfig to SSL in Nginx (copy the method from the mail.example.com config)
changed the instance for automx from 'modoboa' to 'automx' under autodiscover

piensavirtual · 2024-10-21T13:26:16Z

've been tinkering with autodiscover, and it wasn't working. I reviewed the code in automx_wsgi.py, located in /srv/automx/instance, and made some changes that only affect autodiscover. Now it works perfectly, but I'm not sure if this issue has already been resolved.

description of the issue:

When processing Autodiscover requests using automx, the application encounters an error related to the handling of Unicode strings that contain an XML encoding declaration (). The error occurs when attempting to parse the XML request body in automx_wsgi.py, leading to a failure in handling autodiscover requests.
Error message:

Traceback (most recent call last):
File "/srv/automx/instance/automx_wsgi.py", line 126, in application
tree = etree.parse(fd)
^^^^^^^^^^^^^^^
File "src/lxml/etree.pyx", line 3589, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1975, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1990, in lxml.etree._parseMemoryDocument
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

Steps to reproduce:

Send an Autodiscover POST request with an XML body containing a <?xml version="1.0" encoding="utf-8"?> declaration. This can be done using a client like Outlook or manually using curl:


curl -X POST -d "<?xml version='1.0' encoding='utf-8'?><Autodiscover xmlns='http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006'><Request><EMailAddress>[email protected]</EMailAddress><AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema></Request></Autodiscover>" "https://autodiscover.example.com/autodiscover/autodiscover.xml"

Observe the error in the server logs where lxml raises a ValueError when attempting to parse the request body containing the XML encoding declaration.

Root cause:

The issue arises because lxml expects bytes input when handling XML documents that contain an encoding declaration (e.g., ). The current code in automx_wsgi.py uses StringIO and attempts to process the request body as a Unicode string, which causes the ValueError.
Code location:

In the file automx_wsgi.py, around line 126, the following code is used to handle the parsing of the XML:

python

fd = StringIO(request_body.decode("utf-8").replace('', ''))
tree = etree.parse(fd)

This code attempts to decode the request body as a Unicode string, which is incompatible with the XML encoding declaration.
Proposed solution:

To fix the issue, the request body should be handled as bytes, and BytesIO should be used instead of StringIO when parsing the XML. The updated code would look like this:

python

from io import BytesIO

Ensure request_body is treated as bytes and do not decode it

fd = BytesIO(request_body)
tree = etree.parse(fd)

This modification ensures that lxml receives the correct bytes input for XML documents with encoding declarations.
Additional context:

This issue only affects Autodiscover requests where the client sends an XML body with an encoding declaration. Autoconfig, which typically uses GET requests, is unaffected.
This issue has been observed in environments running Python 3.x with lxml for XML parsing.

stefaweb mentioned this issue Jul 1, 2018

Installer don't generate letsencrypt alternate name for autodiscover #222

Open

kryskool added enhancement feedback-needed labels Jul 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

autodiscover/autoconfig errors and tunnings #223

autodiscover/autoconfig errors and tunnings #223

stefaweb commented Jul 1, 2018 •

edited

Loading

stefaweb commented Jul 1, 2018 •

edited

Loading

stefaweb commented Jul 2, 2018 •

edited

Loading

nickdbush commented Oct 2, 2018

tonioo commented Nov 28, 2018

tonioo commented Nov 28, 2018

gianks commented Jul 13, 2019 •

edited

Loading

horvan commented Nov 7, 2019

dbryar commented Nov 11, 2019 •

edited

Loading

piensavirtual commented Oct 21, 2024 •

edited

Loading

autodiscover/autoconfig errors and tunnings #223

autodiscover/autoconfig errors and tunnings #223

Comments

stefaweb commented Jul 1, 2018 • edited Loading

stefaweb commented Jul 1, 2018 • edited Loading

stefaweb commented Jul 2, 2018 • edited Loading

nickdbush commented Oct 2, 2018

tonioo commented Nov 28, 2018

tonioo commented Nov 28, 2018

gianks commented Jul 13, 2019 • edited Loading

horvan commented Nov 7, 2019

dbryar commented Nov 11, 2019 • edited Loading

piensavirtual commented Oct 21, 2024 • edited Loading

Ensure request_body is treated as bytes and do not decode it

stefaweb commented Jul 1, 2018 •

edited

Loading

stefaweb commented Jul 1, 2018 •

edited

Loading

stefaweb commented Jul 2, 2018 •

edited

Loading

gianks commented Jul 13, 2019 •

edited

Loading

dbryar commented Nov 11, 2019 •

edited

Loading

piensavirtual commented Oct 21, 2024 •

edited

Loading