Unicode issues when performing dumpf on Redhat Enterprise 6

When running "blueprint-show -S <blueprint_name>" from a fairly simple blueprint created via blueprint-rules, I got the following error:

Traceback (most recent call last):
  File "/usr/bin/blueprint-show", line 63, in 
    filename = getattr(b, options.generate)(options.relaxed).dumpf()
  File "/usr/lib/python2.6/site-packages/blueprint/frontend/sh.py", line 333, in dumpf
    f.write('{0}\n'.format(out))
  File "/usr/lib64/python2.6/codecs.py", line 691, in write
    return self.writer.write(data)
  File "/usr/lib64/python2.6/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 605131: ordinal not in range(128)

I traced the problem down to the default /etc/services file on RHEL6. There's four lines in this file with 8-bit characters. The file had been modified on the original system, but I checked and these four lines exist in a freshly installed copy of RHEL6 too.

The core of the problem appears to be the code in sh.py around line 333, although I'm not sure exactly why it fails as Blueprint appears to correctly detect the file's contents are non-ascii and attempts to handle it. I wonder if maybe "encode('utf-8', 'ignore')" isn't actually working as expected here? (According to chardet the encoding of this /etc/service file is EUC-JP)

if isinstance(out, unicode):
             out = unicodedata.normalize('NFKD', out).encode('utf-8', 'ignore')
f.write('{0}\n'.format(out))

I found that reworking the code as shown below appears to solve the problem for me, and the content of the resulting script correctly replicates the 8-bit characters from the original. I'm just unsure if this will have other undesirable side effects.

if isinstance(out, unicode):
                f.write(u"{0}\n".format(out))
else:
                f.write('{0}\n'.format(out))

[virtadpt]

I'm running into this problem as well, and it's a show stopper. I'm considering reverting to an earlier version of Blueprint to get stuff done. Perhaps repository bisection will help isolate the problem.

[virtadpt]

One of my co-workers figured out a fix for this problem, though it's not in Blueprint. He patched the /usr/lib/python2.7/site.py file on the machine in question (64-bit Kubuntu v12.04) and changed the default encoding from 'ascii' to 'UTF-8' and it works now. There is a weak implication that it has to do with the default character encoding settings for the system and not Blueprint per se, but more digging will have to be done I think.

[pocesar]

@virtadpt thanks, that worked out for me. Is there a way to "programatically" set it? (like inside blueprint itself). Hacking python libs are too hacky

[virtadpt]

I've no idea. I haven't found a better solution, nor has anyone else mentioned anything about it.