[feature] `Node#inner_text` should not capture `<style>` tag contents

[jaypinho]

Please describe the bug

Per Nokogiri's documentation, the Node#inner_text method (aliased as text and content as well) is meant to capture "the plaintext content for this Node." Given the usage of the method name inner_text, it implies that it works similarly to the JavaScript method of the same name.

However, the JavaScript method explicitly excludes the inner content of any <style> tags that are children of the given node, while Node#inner_text includes it.

Help us reproduce what you're seeing

Example URL (note that you need to curl this link to reproduce the below, not simply examine it in browser dev tools, as runtime JS changes the underlying DOM structure): https://www.binance.com/en/terms

require 'httparty'
require 'nokogiri'
x = HTTParty.get('https://www.binance.com/en/terms').body
y = Nokogiri::HTML.parse(x).at_css("body div main").inner_text

Expected behavior

Expectation: the result should start with Binance Terms of Use...

Actual: it starts with .css-13trade{box-sizing:border-box...

Per the JS docs for innerText, <style> elements are explicitly excluded. See the example here: https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/innerText

Environment

# Nokogiri (1.11.3)
    ---
    warnings: []
    nokogiri:
      version: 1.11.3
      cppflags:
      - "-I/Users/jpinho/.rvm/gems/ruby-2.7.1/gems/nokogiri-1.11.3-x86_64-darwin/ext/nokogiri"
      - "-I/Users/jpinho/.rvm/gems/ruby-2.7.1/gems/nokogiri-1.11.3-x86_64-darwin/ext/nokogiri/include"
      - "-I/Users/jpinho/.rvm/gems/ruby-2.7.1/gems/nokogiri-1.11.3-x86_64-darwin/ext/nokogiri/include/libxml2"
      ldflags: []
    ruby:
      version: 2.7.1
      platform: x86_64-darwin19
      gem_platform: x86_64-darwin-19
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      precompiled: true
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      - 0006-htmlParseComment-treat-as-if-it-closed-the-comment.patch
      - 0007-use-new-htmlParseLookupCommentEnd-to-find-comment-en.patch
      - '0008-use-glibc-strlen.patch'
      - '0009-avoid-isnan-isinf.patch'
      - 0010-parser.c-shrink-the-input-buffer-when-appropriate.patch
      - 0011-update-automake-files-for-arm64.patch
      libxml2_path: "/Users/jpinho/.rvm/gems/ruby-2.7.1/gems/nokogiri-1.11.3-x86_64-darwin/ext/nokogiri"
      iconv_enabled: true
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      precompiled: true
      patches:
      - 0001-update-automake-files-for-arm64.patch
      compiled: 1.1.34
      loaded: 1.1.34
    other_libraries:
      zlib: 1.2.11
      libiconv: '1.15'

Additional context

N/A

[flavorjones]

@jaypinho Thanks for opening this issue, I'll take a deeper look as soon as I have time.

In the meantime, if you have a few minutes, it would be helpful to capture (in the reproduction script) the explicit document structure, so that the content is local to this issue, and perhaps is a smaller reproducible example.

[flavorjones]

@jaypinho I'd like to explore the solution space for a moment, and get your feedback.

I think what you're suggesting is that it's probably OK for Node#text and Node#content to include HTML <style> tag contents (similar to JS textContent), but that Node#inner_text should omit it in order to correctly mirror JS innerText. I think I agree! But I want to make sure we have that shared mental model before proceeding.

I also want to point out, though, that for XML documents, we should not adopt this special handling. Do we agree on that as well?

If so, then I think the shape of a potential solution is:

• if a document is HTML4 or HTML5
• then Node#inner_text should omit <style> text contents
• regardless, Node#text and Node#content should keep current behavior

Does that sound right to you?

[jaypinho]

@flavorjones You've captured it exactly.

A better, more self-contained example than the one I used above:

require 'httparty'
require 'nokogiri'
x = "<html><body><style>.abc {color: red;}</style>Hello world</body></html>"
y = Nokogiri::HTML.parse(x).at_css("body").inner_text
puts y

[flavorjones]

OK! Thanks for confirming my understanding, and for the self-contained example.

I agree the method name is misleading for those familiar with the JS API and acknowledge the docs should perhaps have been clearer on this point; but I'm going to categorize this a "feature request", since the current code is working as designed.

Are you interested in working on a PR for this? If so, I can drop some thoughts here. If not, I'm relatively confident I can find time to address it for a release in the next couple of months.

[stevecheckoway]

One thing that may be worth doing (for someone who is interested and has the time) would be to look at the list of DOM and HTML APIs that are relevant for Nokogiri and see how well Nokogiri's own APIs map on to them. There's a lot there, but an examination of

• Node;
• Document;
• DocumentType;
• DocumentFragment;
• Element;
• Text;
• Comment;
• ProcessingInstruction;
• HTML's extension of Document; and
• HTML's HTMLElement

would cover the most relevant API.

(I'm not really sure what ProcessingInstruction nodes are. The HTML syntax for HTML cannot produce them but the XML syntax can.)

[jaypinho]

Are you interested in working on a PR for this? If so, I can drop some thoughts here. If not, I'm relatively confident I can find time to address it for a release in the next couple of months.

I would like to say yes but am not confident enough in my coding skills (I'm self-taught, and don't know C) to do it. :) I'm happy to wait for whenever you get a chance to take a look - not super urgent on my end. Thank you!