RMail and malformed UTF-8 headers

[jorrizza]

In broken email messages (mostly spam) the headers aren't always encoded properly (I think?), causing sup to crash. It feeds the UTF-8 string into RMail::Header::Field.parse, which in turn defines an ASCII-8bit regex and matches the string against that. For now I've fixed it like this in rmail-sup's header.rb:

def parse(field)
  field = field.dup.to_str
  field.force_encoding('binary')
  if field =~ EXTRACT_FIELD_NAME_RE

But I've got no idea if it's sup who should send it an ASCII string, or RMail who should coerce it into an ASCII string.

And here's the traceback. The line number in header.rb is off a bit, but you get the idea:

/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/header.rb:81:in `parse'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:233:in `block in parse_header'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:229:in `each'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:229:in `parse_header'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:194:in `parse_low'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:183:in `parse'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:331:in `parse'
/var/lib/gems/1.9.1/gems/rmail-sup-1.0.1/lib/rmail/parser.rb:345:in `read'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/mbox.rb:84:in `block in load_message'
<internal:prelude>:10:in `synchronize'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/mbox.rb:74:in `load_message'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/message.rb:772:in `parsed_message'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/message.rb:261:in `load_from_source!'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/message.rb:360:in `build_from_source'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:204:in `block in poll_from'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/mbox.rb:143:in `poll'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:201:in `poll_from'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:146:in `block (2 levels) in do_poll'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:136:in `each'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:136:in `block in do_poll'
<internal:prelude>:10:in `synchronize'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:135:in `do_poll'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/util.rb:649:in `method_missing'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/modes/poll_mode.rb:15:in `poll'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:58:in `poll_with_sources'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/poll.rb:94:in `poll'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup/util.rb:649:in `method_missing'
/var/lib/gems/1.9.1/gems/sup-0.15.2/bin/sup:220:in `block (2 levels) in <module:Redwood>'
/var/lib/gems/1.9.1/gems/sup-0.15.2/lib/sup.rb:87:in `block in reporting_thread'

[gauteh]

Perhaps it is better to use the fix_encoding! method on the Sup string before it is passed to parse, then use the /u modifier on the EXTRACT_FIELD_NAME_RE regexp (sup-heliotrope/rmail-sup@b5300ac) in place of /n [0].

[0] http://www.ruby-doc.org/core-2.1.0/Regexp.html#class-Regexp-label-Encoding

[gauteh]

The proper fix would be of course to migrate to Mail.. #127.

[gauteh]

I think you are missing the actual error from the trace, could you post the full output from the crash?

[jorrizza]

I don't have the full log anymore. I should have saved it. I've tried finding the message that triggered it, but failed. It's easy to recreate the error in irb, though:

"õ" =~ EXTRACT_FIELD_NAME_RE
Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)

So the full error log would be this error message, followed by the traceback in the original report.

[gauteh]

Could you try the changes in pull-request #214 with a manually installed version of rmail with sup-heliotrope/rmail-sup#3 applied?

[jorrizza]

I set up a separate user account on the same machine the error occurred on and I added the pull requests you mentioned. I did some Bundler stuff to get it all to work. I added my own inbox mbox file to this sup's sources to make sure the malformed headers are present. I ran sup-sync right after the initial setup. This is the result.

$ bin/sup-sync
Scanning mbox:/home/testsup/bunchofmail...
[2014-01-29 16:17:56 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:17:57 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:17:58 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:17:59 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:17:59 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:00 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:01 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:01 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:01 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:01 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
[2014-01-29 16:18:01 +0100] WARNING: content_type ��} is not safe, changed to application/octet-stream
/home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:706:in `===': incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) (Encoding::CompatibilityError)
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:706:in `block in get_tokenize'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:650:in `loop'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:650:in `get_tokenize'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:643:in `get'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:402:in `address_lookahead'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:426:in `address'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:366:in `address_list'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:302:in `parse'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/address.rb:789:in `parse'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/header.rb:859:in `block in address_list_fetch'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/header.rb:855:in `each'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/header.rb:855:in `address_list_fetch'
    from /home/testsup/sup/vendor/rmail-sup-1.0.1.666/lib/rmail/header.rb:688:in `from'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:482:in `message_to_chunks'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:462:in `block in message_to_chunks'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:462:in `map'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:462:in `message_to_chunks'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:265:in `load_from_source!'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/message.rb:362:in `build_from_source'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/poll.rb:204:in `block in poll_from'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/mbox.rb:143:in `poll'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/poll.rb:201:in `poll_from'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/lib/sup/util.rb:632:in `method_missing'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/bin/sup-sync:125:in `block in <top (required)>'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/bin/sup-sync:120:in `each'
    from /home/testsup/sup/vendor/ruby/1.9.1/bundler/gems/sup-6da5139571a5/bin/sup-sync:120:in `<top (required)>'
    from bin/sup-sync:16:in `load'
    from bin/sup-sync:16:in `<main>'

address.rb:706 defines the following regex:

/\A[\200-\377\w!$%&\'*+\/=?^_\`{\}|~#-]+/nm

This goes beyond my poor knowledge of some email RFCs.

[gauteh]

Hm, this is a different, but similar problem..

do you know what address this fails on?

[jorrizza]

Here's the hexdump of the address. Or the contents of @string anyway.

00000000  ef bf bd 20 4f 66 66 69  63 69 61 6c 20 53 69 74  |... Official Sit|
00000010  65 20 3c 68 62 76 76 40  79 61 68 6f 6f 2e 63 6f  |e <hbvv@yahoo.co|
00000020  6d 3e                                             |m>|
00000022

Edit: EF BF BD is UTF-8 for U+FFFD, the replacement character. So that first character might have been something different. I'm looking into it.

Edit 2: Found it!

$ grep -m 1 'Official Site <hbvv@yahoo.com>' ../bunchofmail | hexdump -C
00000000  46 72 6f 6d 3a 20 56 49  41 47 52 41 20 ae 20 4f  |From: VIAGRA . O|
00000010  66 66 69 63 69 61 6c 20  53 69 74 65 20 3c 68 62  |fficial Site <hb|
00000020  76 76 40 79 61 68 6f 6f  2e 63 6f 6d 3e 0a        |vv@yahoo.com>.|
0000002e

Yes, that's most definitely spam. The byte that made sup crash is AE.

[gauteh]

Hm, what version of ruby and rmail are you running?

[jorrizza]

I'm using the rmail from sup-heliotrope/rmail-sup#3 and ruby 1.9.3p484.

[gauteh]

Could you post the entire message somewhere, I'm not able to crash sup with just those bits.

[gauteh]

Preferably as a binary upload

[gauteh]

By the way, you can use the environment variable SUP_BASE to make sup use something else than ~/.sup for home folder