explore further optimizing the HTML5 parser and serializer

[flavorjones]

Ideally we want HTML5 to be the default HTML parser in Nokogiri (see #2331). Some necessary work before we do that is to make sure it's as performant as we can make it.

This issue is open-ended and meant to collect the conversations and optimizations attempts we've made.

gumbo-specific speedup:

• HTML5 documents should not require namespaces in CSS selector queries #2403 (previous work)
• explore ways to speed up HTML5 document serialization #2569 (previous work)
• Implement HTML 5 serialization in C #2596 (previous work)
• perf: use -O2 to compile libgumbo and the nokogiri extension #2639 (previous work)
• perf: avoid gumbo_debug function calls when GUMBO_DEBUG isn't def'd #2723
• perf: update gumbo utf8 decode #2735
• [experiment] gumbo using a malloc arena #2790
• What if we converted libgumbo into a SAX parser and built libxml2 data structures directly?

General speedup:

• perf: use default memory management within libxml2 #2734
• feat: env var to control libxml2 memory management #2843
• Process HTML using Nokolexbor instead of Nokogumbo #3043

[flavorjones]

Let's capture a benchmark at the start of this, on my development machine.

#! /usr/bin/env ruby
# coding: utf-8

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "nokogiri", path: "."
  gem "benchmark-ips"
end

require "nokogiri"
require "benchmark/ips"

filenames = [
  "test/files/GH_1042.html", # 650b
  "test/files/tlm.html", # 70kb
  "big_shopping.html", # 1.9mb
]

inputs = filenames.map { |fn| File.read(fn) }

puts RUBY_DESCRIPTION

inputs.each do |input|
  len = input.length

  Benchmark.ips do |x|
    x.warmup = 0
    x.time = 10

    x.report("html5 parse #{len}") do
      Nokogiri::HTML5::Document.parse(input)
    end
    x.report("html4 parse #{len}") do
      Nokogiri::HTML4::Document.parse(input)
    end
    x.compare!
  end
end

puts "=========="

inputs.each do |input|
  len = input.length
  html4_doc = Nokogiri::HTML4::Document.parse(input)
  html5_doc = Nokogiri::HTML5::Document.parse(input)

  Benchmark.ips do |x|
    x.warmup = 0
    x.time = 10

    x.report("html5 serlz #{len}") do
      html5_doc.to_html
    end
    x.report("html4 serlz #{len}") do
      html4_doc.to_html
    end
    x.compare!
  end
end

(the big_shopping.html file is linked to at #2331 (comment))

ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Calculating -------------------------------------
     html5 parse 656     19.449k (±14.8%) i/s -    147.873k in   9.935697s
     html4 parse 656     24.883k (±13.9%) i/s -    186.143k in   9.922674s

Comparison:
     html4 parse 656:    24883.0 i/s
     html5 parse 656:    19449.1 i/s - same-ish: difference falls within error

Calculating -------------------------------------
   html5 parse 70095    267.387  (±23.6%) i/s -      2.111k in   9.997117s
   html4 parse 70095    478.390  (±17.1%) i/s -      3.371k in   9.996398s

Comparison:
   html4 parse 70095:      478.4 i/s
   html5 parse 70095:      267.4 i/s - 1.79x  (± 0.00) slower

Calculating -------------------------------------
 html5 parse 1929522     12.895  (±15.5%) i/s -    127.000  in  10.053394s
 html4 parse 1929522     37.610  (±23.9%) i/s -    350.000  in  10.011052s

Comparison:
 html4 parse 1929522:       37.6 i/s
 html5 parse 1929522:       12.9 i/s - 2.92x  (± 0.00) slower

==========

Calculating -------------------------------------
     html5 serlz 656     41.608k (±14.9%) i/s -    381.808k in   9.851842s
     html4 serlz 656     59.478k (±21.8%) i/s -    506.898k in   9.775912s

Comparison:
     html4 serlz 656:    59478.5 i/s
     html5 serlz 656:    41608.1 i/s - same-ish: difference falls within error

Calculating -------------------------------------
   html5 serlz 70095    999.372  (±11.1%) i/s -      9.780k in   9.992943s
   html4 serlz 70095      1.259k (±14.8%) i/s -     12.109k in   9.991011s

Comparison:
   html4 serlz 70095:     1259.3 i/s
   html5 serlz 70095:      999.4 i/s - same-ish: difference falls within error

Calculating -------------------------------------
 html5 serlz 1929522    114.104  (± 8.8%) i/s -      1.131k in  10.003148s
 html4 serlz 1929522    108.313  (±12.0%) i/s -      1.066k in   9.999369s

Comparison:
 html5 serlz 1929522:      114.1 i/s
 html4 serlz 1929522:      108.3 i/s - same-ish: difference falls within error

and also some profiling information:

# stackprof-big-shopping.sh
#! /usr/bin/env bash

if [[ $# -lt 1 ]] ; then
  echo "usage: $0 <output-filename>"
  exit 1
fi

cmd=$(rbenv which ruby)

env \
  LD_PRELOAD=$HOME/local/lib/libprofiler.so \
  CPUPROFILE=$1 \
  $cmd ./stackprof-big-shopping.rb | tee $1.log

pprof --gif $cmd $1 > $1.gif
pprof --text $cmd $1 > $1.text

# stackprof-big-shopping.rb
#! /usr/bin/env ruby
# coding: utf-8

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "nokogiri", path: "."
  gem "benchmark-ips"
end

require "nokogiri"
require "benchmark/ips"

input = File.read("big_shopping.html") # 1.9mb
puts "input #{input.length} bytes"

puts RUBY_DESCRIPTION

Benchmark.ips do |x|
  x.warmup = false
  x.time = 10

  x.report("parsing") { Nokogiri::HTML5::Document.parse(input) }
end

text output:

input 1929522 bytes
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Calculating -------------------------------------
             parsing     13.222  (±15.1%) i/s -    130.000  in  10.042170s

top of the stack profile:

Total: 1035 samples
     120  11.6%  11.6%      120  11.6% decode (inline)
      99   9.6%  21.2%      100   9.7% pthread_attr_setschedparam
      88   8.5%  29.7%      833  80.5% gumbo_parse_with_options
      78   7.5%  37.2%       78   7.5% _init@3e000
      65   6.3%  43.5%      464  44.8% gumbo_lex
      53   5.1%  48.6%      176  17.0% read_char
      42   4.1%  52.7%       63   6.1% gumbo_string_buffer_append_codepoint
      40   3.9%  56.5%       40   3.9% get_current_node.isra.0
      35   3.4%  59.9%      195  18.8% handle_token (inline)
      34   3.3%  63.2%      202  19.5% finish_token.isra.0
      25   2.4%  65.6%       61   5.9% insert_text_token.isra.0
      23   2.2%  67.8%       23   2.2% get_adjusted_current_node
      18   1.7%  69.6%       69   6.7% build_tree
      16   1.5%  71.1%      222  21.4% emit_char
      13   1.3%  72.4%       13   1.3% maybe_emit_from_mark
      13   1.3%  73.6%       21   2.0% maybe_resize_string_buffer
      13   1.3%  74.9%       13   1.3% update_position (inline)
      12   1.2%  76.0%       13   1.3% atomic_sub_nounderflow (inline)
      12   1.2%  77.2%       12   1.2% gumbo_tokenizer_set_is_adjusted_current_node_foreign
      11   1.1%  78.3%       12   1.2% handle_text
      10   1.0%  79.2%       38   3.7% __libc_malloc
      10   1.0%  80.2%       47   4.5% tree_traverse.constprop.0
      10   1.0%  81.2%       10   1.0% xmlStrdup (inline)
       9   0.9%  82.0%        9   0.9% __nss_database_lookup
       7   0.7%  82.7%        7   0.7% cfree
       7   0.7%  83.4%        7   0.7% get_char_token_type (inline)
       7   0.7%  84.1%        7   0.7% gumbo_debug
       7   0.7%  84.7%       43   4.2% handle_in_body
       7   0.7%  85.4%        7   0.7% utf8iterator_get_position (inline)
       6   0.6%  86.0%       19   1.8% utf8iterator_next
       6   0.6%  86.6%       54   5.2% xmlFreeNodeList
       5   0.5%  87.1%       27   2.6% handle_attr_value_double_quoted_state
       5   0.5%  87.5%      123  11.9% handle_html_content (inline)
       5   0.5%  88.0%        5   0.5% is_open_element (inline)
       5   0.5%  88.5%       28   2.7% objspace_malloc_increase_body (inline)
       5   0.5%  89.0%        5   0.5% rbimpl_atomic_size_add (inline)
       4   0.4%  89.4%        4   0.4% append_char_to_tag_buffer
       4   0.4%  89.8%       32   3.1% objspace_xmalloc0
       4   0.4%  90.1%       29   2.8% xmlNewText
       3   0.3%  90.4%        3   0.3% _init@33000
       3   0.3%  90.7%        3   0.3% handle_script_data_state
       3   0.3%  91.0%        3   0.3% malloc_usable_size
       3   0.3%  91.3%       12   1.2% ruby_yyparse
       3   0.3%  91.6%       32   3.1% xmlNewPropInternal
       2   0.2%  91.8%        2   0.2% copy_over_original_tag_text (inline)
       2   0.2%  92.0%        4   0.4% finish_tag_name
       2   0.2%  92.2%       30   2.9% gumbo_alloc
       2   0.2%  92.4%        2   0.2% gumbo_tag_lookup
       2   0.2%  92.6%       10   1.0% handle_after_attr_value_quoted_state
       2   0.2%  92.8%       37   3.6% handle_attr_name_state
       2   0.2%  92.9%        3   0.3% node_html_tag_is (inline)
       2   0.2%  93.1%       10   1.0% realloc
       2   0.2%  93.3%        8   0.8% reset_token_start_point (inline)
       2   0.2%  93.5%        2   0.2% secondary_hash (inline)
       2   0.2%  93.7%        2   0.2% utf8_is_control (inline)
       2   0.2%  93.9%        7   0.7% xmlNewNode
       1   0.1%  94.0%        1   0.1% ISEQ_COMPILE_DATA (inline)
       1   0.1%  94.1%        1   0.1% RVALUE_OLD_P_RAW (inline)
       1   0.1%  94.2%        1   0.1% __libc_open64
       1   0.1%  94.3%        1   0.1% __munmap
       1   0.1%  94.4%        4   0.4% adoption_agency_algorithm
       1   0.1%  94.5%        1   0.1% autoload_c_mark
       1   0.1%  94.6%        1   0.1% callable_method_entry
       1   0.1%  94.7%        9   0.9% copy_over_tag_buffer (inline)
       1   0.1%  94.8%        7   0.7% create_element_from_token
       1   0.1%  94.9%        6   0.6% create_node (inline)
       1   0.1%  95.0%       32   3.1% destroy_node_callback
...

[flavorjones]

@stevecheckoway suggested optimizing out calls to gumbo_debug at #2331 (comment)

ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Calculating -------------------------------------
     html5 parse 656     21.011k (±16.5%) i/s -    155.264k in   9.932001s
     html4 parse 656     24.498k (±16.4%) i/s -    178.167k in   9.922181s

Comparison:
     html4 parse 656:    24497.5 i/s
     html5 parse 656:    21011.3 i/s - same-ish: difference falls within error

Calculating -------------------------------------
   html5 parse 70095    293.924  (±22.1%) i/s -      2.355k in  10.008618s
   html4 parse 70095    481.226  (±17.5%) i/s -      3.463k in  10.026732s

Comparison:
   html4 parse 70095:      481.2 i/s
   html5 parse 70095:      293.9 i/s - 1.64x  (± 0.00) slower

Calculating -------------------------------------
 html5 parse 1929522     15.547  (±12.9%) i/s -    152.000  in  10.025299s
 html4 parse 1929522     38.315  (±23.5%) i/s -    354.000  in  10.044622s

Comparison:
 html4 parse 1929522:       38.3 i/s
 html5 parse 1929522:       15.5 i/s - 2.46x  (± 0.00) slower

Already an improvement! A 14% baseline improvement, going from 2.92x slower to 2.46x slower than html4 on the big file.

Here's the stack prof:

input 1929522 bytes
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Calculating -------------------------------------
             parsing     15.184  (±13.2%) i/s -    148.000  in  10.069515s

Total: 1039 samples
      99   9.5%   9.5%      101   9.7% pthread_attr_setschedparam
      87   8.4%  17.9%      813  78.2% gumbo_parse_with_options
      86   8.3%  26.2%       86   8.3% decode (inline)
      69   6.6%  32.8%      155  14.9% read_char
      64   6.2%  39.0%      238  22.9% handle_token (inline)
      58   5.6%  44.6%      432  41.6% gumbo_lex
      55   5.3%  49.9%      201  19.3% finish_token.isra.0
      45   4.3%  54.2%       45   4.3% _init@3e000
      43   4.1%  58.3%       43   4.1% get_current_node.isra.0
      36   3.5%  61.8%       61   5.9% gumbo_string_buffer_append_codepoint
      21   2.0%  63.8%       21   2.0% get_adjusted_current_node
      18   1.7%  65.5%       45   4.3% handle_in_body
      18   1.7%  67.3%       69   6.6% insert_text_token.isra.0
      17   1.6%  68.9%       17   1.6% atomic_sub_nounderflow (inline)
      17   1.6%  70.5%       20   1.9% handle_text
      16   1.5%  72.1%       16   1.5% xmlStrdup (inline)
      14   1.3%  73.4%       42   4.0% objspace_malloc_increase_body (inline)
      13   1.3%  74.7%       13   1.3% __nss_database_lookup
      12   1.2%  75.8%       12   1.2% cfree
      12   1.2%  77.0%       25   2.4% maybe_resize_string_buffer
      11   1.1%  78.1%       11   1.1% get_char_token_type (inline)
      11   1.1%  79.1%       11   1.1% gumbo_tokenizer_set_is_adjusted_current_node_foreign
      11   1.1%  80.2%       31   3.0% handle_attr_value_double_quoted_state
      11   1.1%  81.2%       11   1.1% handle_script_data_state
      11   1.1%  82.3%       50   4.8% tree_traverse.constprop.0
      11   1.1%  83.3%       11   1.1% utf8iterator_get_position (inline)
      10   1.0%  84.3%       33   3.2% __libc_malloc
      10   1.0%  85.3%      141  13.6% handle_html_content (inline)
       9   0.9%  86.1%       76   7.3% build_tree
       9   0.9%  87.0%        9   0.9% update_position (inline)
       7   0.7%  87.7%       67   6.4% xmlFreeNodeList
       6   0.6%  88.3%        6   0.6% maybe_emit_from_mark
       6   0.6%  88.8%        6   0.6% rbimpl_atomic_size_add (inline)
       6   0.6%  89.4%       15   1.4% utf8iterator_next
       5   0.5%  89.9%      215  20.7% emit_char
       5   0.5%  90.4%        5   0.5% malloc_usable_size
       5   0.5%  90.9%       17   1.6% realloc
       4   0.4%  91.2%        4   0.4% has_an_element_in_specific_scope
       4   0.4%  91.6%       34   3.3% xmlNewPropInternal
       3   0.3%  91.9%        4   0.4% append_char_to_tag_buffer
       3   0.3%  92.2%       22   2.1% finish_attribute_name
       3   0.3%  92.5%       13   1.3% gumbo_destroy_attribute
       3   0.3%  92.8%        3   0.3% set_line (inline)
       2   0.2%  93.0%        2   0.2% RB_BUILTIN_TYPE (inline)
       2   0.2%  93.2%        2   0.2% _init@33000
       2   0.2%  93.4%        2   0.2% gumbo_normalized_tagname
       2   0.2%  93.6%        2   0.2% handle_data_state
       2   0.2%  93.7%       10   1.0% handle_tag_name_state
       2   0.2%  93.9%        2   0.2% is_open_element (inline)
       2   0.2%  94.1%        4   0.4% reconstruct_active_formatting_elements
       2   0.2%  94.3%       21   2.0% xmlStrndup
       1   0.1%  94.4%        1   0.1% RB_SPECIAL_CONST_P (inline)
       1   0.1%  94.5%        1   0.1% RCLASS_SUPER (inline)
       1   0.1%  94.6%        1   0.1% RVALUE_WB_UNPROTECTED (inline)
       1   0.1%  94.7%        1   0.1% _dl_rtld_di_serinfo
       1   0.1%  94.8%        1   0.1% add_ctype_to_cc
       1   0.1%  94.9%        1   0.1% callable_method_entry
       1   0.1%  95.0%        1   0.1% copy_over_original_tag_text (inline)

[flavorjones]

@stevecheckoway I tried to do LTO but didn't see a noticeable performance improvement. Here's the patch:

diff --git a/ext/nokogiri/extconf.rb b/ext/nokogiri/extconf.rb
index daf6094..7745e4f 100644
--- a/ext/nokogiri/extconf.rb
+++ b/ext/nokogiri/extconf.rb
@@ -618,6 +618,9 @@ def do_clean
 # gumbo html5 serialization is slower with O3, let's make sure we use O2
 append_cflags("-O2")
 
+# link-time optimization
+append_cflags("-flto")
+
 # always include debugging information
 append_cflags("-g")
 
@@ -725,7 +728,7 @@ def install
         class << recipe
           def configure
             env = {}
-            env["CFLAGS"] = concat_flags(ENV["CFLAGS"], "-fPIC", "-g")
+            env["CFLAGS"] = concat_flags(ENV["CFLAGS"], "-fPIC", "-g", "-flto")
             env["CHOST"] = host
             execute("configure", ["./configure", "--static", configure_prefix], { env: env })
             if darwin?
@@ -751,7 +754,7 @@ def configure
         # The libiconv configure script doesn't accept "arm64" host string but "aarch64"
         recipe.host = recipe.host.gsub("arm64-apple-darwin", "aarch64-apple-darwin")
 
-        cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g")
+        cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g", "-flto")
 
         recipe.configure_options += [
           "--disable-dependency-tracking",
@@ -804,7 +807,7 @@ def configure
       recipe.patch_files = Dir[File.join(PACKAGE_ROOT_DIR, "patches", "libxml2", "*.patch")].sort
     end
 
-    cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g")
+    cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g", "-flto")
 
     if zlib_recipe
       recipe.configure_options << "--with-zlib=#{zlib_recipe.path}"
@@ -853,7 +856,7 @@ def configure
       recipe.patch_files = Dir[File.join(PACKAGE_ROOT_DIR, "patches", "libxslt", "*.patch")].sort
     end
 
-    cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g")
+    cflags = concat_flags(ENV["CFLAGS"], "-O2", "-U_FORTIFY_SOURCE", "-g", "-flto")
 
     if darwin? && !cross_build_p
       recipe.configure_options += ["RANLIB=/usr/bin/ranlib", "AR=/usr/bin/ar"]
@@ -974,9 +977,10 @@ def install
     end
 
     def compile
-      cflags = concat_flags(ENV["CFLAGS"], "-fPIC", "-O2", "-g")
+      cflags = concat_flags(ENV["CFLAGS"], "-fPIC", "-O2", "-g", "-flto")
+      ldflags = concat_flags(ENV["LDFLAGS"], "-flto")
 
-      env = { "CC" => gcc_cmd, "CFLAGS" => cflags }
+      env = { "CC" => gcc_cmd, "CFLAGS" => cflags, "LDFLAGS" => ldflags }
       if config_cross_build?
         if /darwin/.match?(host)
           env["AR"] = "#{host}-libtool"

Is there anything obvious I'm missing? I haven't got much experience with link-time optimization, I may be missing something elementary. (Note that flags passed to append_cflags will also appear in link lines.)

[flavorjones]

Also, for comparison, this is the stack profile for libxml2's HTML4 parser on the same file:

Total: 1046 samples
     182  17.4%  17.4%      186  17.8% pthread_attr_setschedparam
     145  13.9%  31.3%      145  13.9% htmlCurrentChar.part.0
     125  12.0%  43.2%      273  26.1% htmlParseScript
      50   4.8%  48.0%       50   4.8% atomic_sub_nounderflow (inline)
      50   4.8%  52.8%       50   4.8% xmlNextChar
      44   4.2%  57.0%       44   4.2% __nss_database_lookup
      32   3.1%  60.0%      177  16.9% htmlCurrentChar (inline)
      28   2.7%  62.7%       28   2.7% rbimpl_atomic_size_add (inline)
      27   2.6%  65.3%      118  11.3% objspace_malloc_increase_body (inline)
      23   2.2%  67.5%      181  17.3% xmlFreeNodeList
      21   2.0%  69.5%       21   2.0% malloc_usable_size
      18   1.7%  71.2%       18   1.7% _init@3e000
      18   1.7%  72.9%      131  12.5% htmlParseHTMLAttribute
      17   1.6%  74.6%       89   8.5% __libc_malloc
      12   1.1%  75.7%       57   5.4% htmlParseHTMLName
      12   1.1%  76.9%       12   1.1% xmlDictComputeBigKey.part.0
      11   1.1%  77.9%       23   2.2% xmlDictComputeBigKey (inline)
      10   1.0%  78.9%       10   1.0% _init@33000
       9   0.9%  79.7%       30   2.9% objspace_xrealloc.isra.0
       9   0.9%  80.6%        9   0.9% ruby_xmalloc0 (inline)
       9   0.9%  81.5%       42   4.0% xmlDictLookup
       7   0.7%  82.1%        7   0.7% cfree
       7   0.7%  82.8%      389  37.2% htmlParseStartTag
       7   0.7%  83.5%        9   0.9% xmlStrEqual
       6   0.6%  84.0%        6   0.6% htmlSkipBlankChars
       6   0.6%  84.6%       15   1.4% ruby_xmalloc
       5   0.5%  85.1%       28   2.7% htmlParseCharDataInternal.constprop.0
       5   0.5%  85.6%      184  17.6% ruby_sized_xfree
       5   0.5%  86.0%        5   0.5% xmlStrcasecmp
       5   0.5%  86.5%        5   0.5% xmlStrdup (inline)
       4   0.4%  86.9%       28   2.7% bsearch (inline)
       4   0.4%  87.3%        4   0.4% objspace_malloc_gc_stress (inline)
       4   0.4%  87.7%       20   1.9% xmlNewPropInternal
       4   0.4%  88.0%       49   4.7% xmlNewText
       3   0.3%  88.3%        3   0.3% RVALUE_OLD_P_RAW (inline)
       3   0.3%  88.6%        7   0.7% htmlGetEndPriority
       3   0.3%  88.9%      138  13.2% htmlParseAttValue (inline)
       3   0.3%  89.2%       14   1.3% htmlParseHTMLName_nonInvasive.isra.0
       3   0.3%  89.5%      169  16.2% objspace_xmalloc0
       3   0.3%  89.8%       16   1.5% realloc
       3   0.3%  90.1%       19   1.8% ruby_yyparse
       3   0.3%  90.3%        4   0.4% xmlFreeID
       3   0.3%  90.6%       90   8.6% xmlFreeProp
       3   0.3%  90.9%       99   9.5% xmlFreePropList (inline)
       3   0.3%  91.2%      107  10.2% xmlSAX2AttributeInternal
       3   0.3%  91.5%      155  14.8% xmlSAX2StartElement
       2   0.2%  91.7%        2   0.2% RB_BUILTIN_TYPE (inline)
       2   0.2%  91.9%        2   0.2% RVALUE_MARKED (inline)
       2   0.2%  92.1%       13   1.2% gc_mark_stacked_objects (inline)
       2   0.2%  92.3%       17   1.6% htmlCompareStartClose (inline)
       2   0.2%  92.4%      985  94.2% htmlParseContentInternal

It's interesting that pthread_attr_setschedparam dominates both profiles.

[flavorjones]

Pulling on the malloc thread (pun intended), there's a significant performance improvement overall for libxml2 (and so also for html5 serialization) if we don't tell libxml2 to use ruby's memory management functions. I'll ship a PR soon.

edit: PR at #2734

[flavorjones]

@stevecheckoway I'm out of ideas. Anything else you'd like me to explore?