Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quoted string parser reports end of file when encountering an empty line #4691

Open
Alainx277 opened this issue Oct 29, 2024 · 4 comments
Open

Quoted string parser reports end of file when encountering an empty line #4691

Alainx277 opened this issue Oct 29, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@Alainx277
Copy link

Describe the bug

When quoting a string it is possible to have text that goes on multiple lines. If the multiline string contains an empty line however, the parser fails even if the string is properly closed.

To Reproduce

Create a fluentd config like this:

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Run fluentd in dry-run mode:

fluentd --dry-run -c /tmp/fluent.conf
Traceback (most recent call last):
	18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
	17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
	16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
	15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
	11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
	10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
	 9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
	 8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
	 7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
	 6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
	 3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
	 2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
	 1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Expected behavior

The string should be parsed like any other multiline string. In this case I'd expect hello to be set to "world\n\n".

Your Environment

- Fluentd version: 1.16.6

Your Configuration

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Your Error Log

Traceback (most recent call last):
	18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
	17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
	16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
	15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
	11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
	10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
	 9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
	 8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
	 7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
	 6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
	 3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
	 2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
	 1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Additional context

No response
@Alainx277
Copy link
Author

This also happens if a line is terminated with a literal backslash:

<filter test>
  @type record_transformer
  <record>
    hello "world \\
test"
  </record>
</filter>

@kenhys
Copy link
Contributor

kenhys commented Oct 30, 2024

It seems that the limitation of current implementation.
As a workaround, use "\n" explicitly.

@daipom daipom added the bug Something isn't working label Nov 1, 2024
@daipom
Copy link
Contributor

daipom commented Nov 1, 2024

Thanks for your report.
It appears this is a bug of Fluentd.

This works.

<filter test.**>
  @type record_transformer
  <record>
    hello "world
"
  </record>
</filter>

However, the following don't work.

<filter test.**>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

<filter test.**>
  @type record_transformer
  <record>
    hello 'world
'
  </record>
</filter>

<filter test.**>
  @type record_transformer
  <record>
    hello 'world

'
  </record>
</filter>

@daipom
Copy link
Contributor

daipom commented Nov 1, 2024

It looks like single-quoted strings don't support multiline.
https://docs.fluentd.org/configuration/config-file#multiline-support-for-quoted-string-array-and-hash-values

So, we need to check double quoted strings behavior, as @Alainx277 says.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kenhys @daipom @Alainx277