Encoding is mangled in error message for syntax error

[JasonBarnabe]

Given some invalid JS with some non-English (but valid encoding-wise) characters:

js = 'var test = "в себе "шок-контент", на'
=> "var test = \"в себе \"шок-контент\", на"
js.encoding
=> #<Encoding:UTF-8>
js.valid_encoding?
=> true

mini_racer will raise MiniRacer::ParseError (good), but the error message will have its encoding mangled.

begin
  MiniRacer::Context.new(timeout: 500, max_memory: 20_000_000).eval(js)
rescue MiniRacer::ParseError => e
  msg = e.message
end
=> "Uncaught SyntaxError: Unexpected identifier '\xD1\x88\xD0\xBE\xD0\xBA' at <eval>:1:20"
msg.encoding
=> #<Encoding:BINARY (ASCII-8BIT)>
msg.valid_encoding?
=> true
puts msg
(app):16:in 'IO#write': "\xD1" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
        from (app):16:in 'IO#puts'
        from (app):16:in 'Kernel#puts'
        from (app):16:in '<main>'

[tisba]

This is not limited to SyntaxErrors. You can also throw an error in JS with this message and get the same issue.

Shorter reproduction:

MiniRacer::Context.new.eval('ä')

produces

JavaScript at <eval>:1:1: ReferenceError: \xC3\xA4 is not defined (MiniRacer::RuntimeError)
	from JavaScript at <eval>:1:1
	from (irb):3:in 'MiniRacer::Context#eval'
	from (irb):3:in '<main>'
	from <internal:kernel>:168:in 'Kernel#loop'
	from /Users/basti/.gem/ruby/3.4.5/gems/irb-1.15.2/exe/irb:9:in '<top (required)>'
	from /Users/basti/.rubies/ruby-3.4.5/lib/ruby/site_ruby/3.4.0/rubygems.rb:319:in 'Kernel#load'
	from /Users/basti/.rubies/ruby-3.4.5/lib/ruby/site_ruby/3.4.0/rubygems.rb:319:in 'Gem.activate_and_load_bin_path'
	from /Users/basti/.gem/ruby/3.4.5/bin/irb:25:in '<main>'

As a workaround: If you know that your input is UTF-8, you can use String#force_encoding, because \xC3\xA4 in my example is a valid UTF-8 byte sequence. Just the Ruby metadata about the encoding got lost. ("Uncaught SyntaxError: Unexpected identifier '\xD1\x88\xD0\xBE\xD0\xBA' at <eval>:1:20".force_encoding('UTF-8') correctly produces "Uncaught SyntaxError: Unexpected identifier 'шок' at <eval>:1:20").

[bnoordhuis]

Maybe it's specific to the Ruby version? 3.3.4 gives me the expected output with both your examples:

t.rb:3:in `eval': Uncaught SyntaxError: Unexpected identifier 'шок' at <eval>:1:20 (MiniRacer::ParseError)

And:

JavaScript at <eval>:1:1: ReferenceError: ä is not defined (MiniRacer::RuntimeError)

@JasonBarnabe what Ruby version did you test with?

[tisba]

I noticed something interesting. Maybe it's related to the terminal or locale or something?

This is Ruby 3.4.6 on arm64-darwin24 (macOS 15.7.1), get the same under linux in the same terminal via docker run -it --rm ruby:3.4.6 /bin/bash.

One more thing, why I came to my initial conclusion. Via IRB:

irb(main):004> begin; MiniRacer::Context.new.eval('ä'); rescue => e; puts e.message.encoding; end
ASCII-8BIT

whereas:

$ ruby -r mini_racer -e 'begin; MiniRacer::Context.new.eval('ä'); rescue => e; puts e.message.encoding; end'
UTF-8

I'm not exactly sure how Ruby's internal/external string encodings work, but maybe this is not specific to mini_racer after all?

[bnoordhuis]

Oh, interesting. Yes, it's irb, not ruby; it mangles the encoding somehow,.

-EUTF-8 doesn't fix it and -EISO-8859-1 mangles it so bad mini_racer starts raising another exception entirely: Uncaught SyntaxError: Invalid or unexpected token at <eval>:1:1

It doesn't look like a mini_racer issue after all but to double-check, @JasonBarnabe, did you hit this only in irb?

[JasonBarnabe]

I'm on ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [x86_64-linux]

I do see that the code I presented does not occur when run directly with ruby and does occur when run with irb, but I think this is just a detail of how these two methods deal with output. The encoding on the exception string is incorrect in either case, and you can cause an exception simply by concatenating it with a string containing UTF-8 characters, which is likely how I originally encountered this in my Rails app, which is outputting the exception message from mini_racer as a validation error in an ERB template.

encoding.rb:

require 'mini_racer'

begin
  MiniRacer::Context.new(timeout: 500, max_memory: 20_000_000).eval("ш")
rescue MiniRacer::RuntimeError => e
  "ш" + e.message
end

bundle exec ruby ./encoding.rb
./encoding.rb:6:in 'String#+': incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT) (Encoding::CompatibilityError)
	from ./encoding.rb:6:in '<main>'
JavaScript at <eval>:1:1: ReferenceError: ш is not defined (MiniRacer::RuntimeError)
	from ./encoding.rb:4:in 'MiniRacer::Context#eval'
	from ./encoding.rb:4:in '<main>'

[bnoordhuis]

#372