v4.8.0-beta00017

This release contains many bug fixes, performance improvements, and other housekeeping/cleanup tasks. It is the largest update to Lucene.NET 4.8.0 to date.

J2N 2.1 and ICU4N 60.1.0-alpha.436 have had major updates which will lay the groundwork for supporting System.Memory types in Lucene.NET, but those updates will be included in a later release. However, both libraries have had a significant reduction in the number of heap allocations used across their codebases and those improvements have a direct impact on how Lucene.NET performs.

We are including .NET Core support for Lucene.Net.OpenNLP for the first time using the IKVM MavenRefernence feature. The IKVM distribution is quite large and due to disk space limitations of our CI enviornment we had to reference an older version of both IKVM and IKVM.Maven.Sdk. These package references may be upgraded by end users to use a newer version without any risk of breaking changes.

Change Log

Breaking Changes

• Lucene.Net.Util.OfflineSorter: Refactored to base file tracking on FileStream rather than FileInfo, which gives us better control over temp file deletion by specifying the FileOptions.DeleteOnClose option. We also use random access so we don't need to reopen streams over files except in the case of ExternalRefSorter.
• Refactored CharArraySet and CharArrayMap (now CharArrayDictionary) (#762)
• Lucene.Net.Analysis.Kuromoji.Token: Changed these methods into properties: IsKnown() to IsKnown, IsUnknown() to IsUnknown, IsUser() to IsUser.
• Added guard clauses for all TokenAttribute members
• Lucene.Net.Index.IndexReader: De-nested IReaderClosedListener and renamed to IReaderDisposedListener.
• Lucene.Net.Index.IndexWriter: Fixed Dispose() overloads so there is no method signature conflict between the public Dispose(waitForMerges) method and the protected Dispose(disposing) method that can be overridden and called from a finalizer. See #265.
• Lucene.Net.Search.FieldCacheRangeFilter<T>: Changed accessibility from protected internal to private protected. This class was not intended to be subclassed by users. (see #677)
• Lucene.Net.Search.Suggest.Fst.ExternalRefSorter: Changed temp path generation to use FileSupport.CreateTempFile() with named prefix and extension because it more closely matches Lucene and makes the files more easily identifiable.
• Lucene.Net.Util.OfflineSorter: Changed DefaultTempDir() to GetDefaultTempDir().
• Lucene.Net.Analysis.SmartCn.SmartChineseAnalyzer: Changed GetDefaultStopSet() to DefaultStopSet. Marked GetDefaultStopSet() obsolete.
• Lucene.Net.Analysis.Kuromoji.JapaneseAnalyzer: Changed GetDefaultStopSet() and GetDefaultStopTags() to DefaultStopSet and DefaultStopTags, respectively. Marked the old methods obsolete.
• Fixed ArgumentOutOfRange parameters so the message is passed into the 2nd parameter, not the first (which is for argumentName). Fixes #665. Also addressed potential int overflow issues when checking for "index + length must be <= array length".
• Remove .NET 4.5, .NET 4.5.1, and .NET 4.5.2 support and update website framework versions (#650)
• Lucene.Net.IndexWriter.IEvent: Marked internal (as it was in Java). This interface is only used in non-public contexts by Lucene.
• Remove virtual on methods that are being called from constructors (#670 - see PRs linked to this issue)
• Lucene.Net.Util.PriorityQueue<T>: Replaced (int, bool) constructor and removed constructor call to virtual GetSentinelObject() method (#820)
• Lucene.Net.Util: Added ValuePriorityQueue<T> to utilize stack allocations where possible (#826)
• Use factory classes for DirectoryTaxonomyWriter and DirectoryTaxonomyReader to get configs (#847)
• Replicator no longer attempts to deserialize exception types (#968)
• Rename IndexWriter.NextMerge() to GetNextMerge() (#990)
• Lucene.Net.Search.ReferenceContext<T>: Converted to ref struct and reworked TestControlledRealTimeReopenThread.TestStraightForwardDemonstration() to verify functionality (#925)

Bug Fixes

• Lucene.Net.Util.OfflineSorter: Added back original tests using FileInfo and fixed bugs that were preventing the original behavior
• Lucene.Net.Tests.Store.TestRAMDirectory: Fixed teardown to retry file deletion if they are locked by another process.
• Aligned disposable patterns (#746)
• Changed TokenAttribute usage from concrete implementation type to interface type to align with Lucene 4.8.0. We were using the concrete type in several places where it shouldn't have been.
• Lucene.Net.Util.OfflineSorter: Fixed the Sort() and SortPartition() methods so they use the tempDirectory parameter that is passed through the constructor, as was the case in Lucene. Added a constructor overload to specify the directory as a string (a .NET convention).
• Lucene.Net.Analysis.Kuromoji.Util.CSVUtil: Applied SOLR-9413 patch to fix the QuoteEscape() method and add tests. Fixes #660.
• Lucene.Net.Search.Similarities: Statically imported SimilarityBase where appropriate so the Log2 method doesn't have to be qualified (like in Lucene). Fixes #694.
• Fixed a bug where the CharArraySet returned from DefaultStopSet in all analyzers was returning a static writable instance instead of a readonly instance as per the docs.
• Lucene.Net.Tests.Index.TestIndexWriter: Finished port of RandomFailingFieldEnumerable. Fixes #695.
• Lucene.Net.Benchmark.Support.TagSoup.Parser::SetProperty(): Removed duplicate guard clause
• Lucene.Net.Analysis.Cjk.CJKBigramFilter: Changed the value of ALL to set all flags (was 0xff instead of 0xffff). Fixes #657.
• Order of precedence for translation of Remove() method args in FrenchStemmer.cs (#654)
• Fixed Infinite loop in HttpClientBase
• Fixed throw statement in BinaryDictionary
• Fixed use of insecure Path.GetTempFileName in ExternalRefSorter (#651)
• Lucene.Net.Search.package.md: Corrected link to TooManyClausesException
• Fix AssertionError in JapaneseTokenizer backtrace LUCENE-10059 (#777)
• Lucene.Net.Util.RandomizedContext: Create a separate instance of Randomizer() for each thread initialized with the same seed. Fixes #843.
• TestIndexWriterOnJRECrash: Removed using block to ensure that our original CheckIndex error bubbles up instead of any issue disposing (or double-disposing) the directory.
• Lucene.Net.Documents.DateTools: Convert DateTimeKind.Unspecified dates to UTC, otherwise they can produce ArgumentOutOfRangeException. Fixes #772.
• Lucene.Net.Store: Fixed several Dispose() methods so they are safe to be called multiple times (#854)
• Lucene.Net.QueryParsers.Classic.QueryParserTokenManager: Removed initialization code that caused writing a BOM to standard out upon creation. (#902)
• Lucene.Net.Search.FieldComparer.TermValComparer: Fixed sorting ambiguity between empty fields and missing fields (#912)
• Fix for DocumentsWriter concurrency (fixes #935, closes #886) (#940)
• Finalizer fix in IndexReader (#951)
• Fix for Lucene.Net.Util.SystemConsole throwing not supported exception in .NET MAUI app running on Android/iOS (#952)
• Fix TermStats.TermText access, add CLI comments and 1 CLI bug fix (#963)

Performance

• Lucene.Net.Support.Arrays::CopyOfRange(): Use the Copy() method rather than a for loop for a ~10x improvement in performance.
• Lucene.Net.Support.Arrays::CopyOf(): Use the Copy() method rather than a for loop for a ~10x improvement in performance.
• Lucene.Net.Support.Arrays::Fill(): Replaced for loop implementation with Array.Fill() or Span.Fill<T>() depending on platform.
• Lucene.Net.Support.Arrays: Added Copy() overloads that use the most efficient (known) copy method for the platform and data type based on benchmarks. Replaced all occurrences of Array.Copy() and Buffer.BlockCopy() with Arrays.Copy().
• Lucene.Net.Support.DictionaryExtensions: Reduced dependency on the Put() method and added documentation to indicate that it doesn't work with non-nullable value types. Also documented the PutAll() method and added guard clause.
• Lucene.Net.Analysis.Sinks.DateRecognizerSinkFilter: Prefer ReadOnlySpan<char> overloads of DateTime.TryParse() and DateTime.TryParseExact(), when available.
• Lucene.Net.Analysis.Util.HTMLStripCharFilter: Refactored to remove YyText property (method) which allocates a string every time it is called. Instead, we pass the underlying arra...

v4.8.0-beta00016

This release contains several important bug fixes and performance enhancements.

Breaking Index Changes

There are 2 breaking changes that may affect some users when reading indexes that were created from version 4.8.0-beta00015 and all prior 4.8.0 beta versions (not including 3.0.3).

1. A bug was fixed in the generation of segment file names. This only affects users with more than 10 segments in their index.
2. Lucene.Net.Documents.DateTools has been modified to return milliseconds since Unix epoch (that is, since Jan 1, 1970 at 00:00:00 UTC) by default to match Java Lucene. This only affects users who explicitly use Lucene.Net.Documents.DateTools in their application and store the result (in .NET ticks) in their index.

If you are affected by either of the above issues, it is recommended to regenerate your indexes during upgrading. However, if that is not feasible, we have provided the following workarounds.

1. If you have a large index with more than 10 segments, see #576 for details on how to enable legacy segment name support.
2. If you are storing the result of Lucene.Net.Documents.DateTools.StringToTime(string) or Lucene.Net.Documents.DateTools.Round(long) (a long) in your index, you are storing .NET ticks. There are now optional parameters inputRepresentation and outputRepresentation on these methods to specify whether the long value represents .NET ticks, .NET ticks as milliseconds, or millisenonds since the Unix epoch. To exactly match version 4.8.0-beta00015 and prior (including prior major versions):
   • Lucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation) should specify NumericRepresentation.TICKS for outputRepresentation.
   • Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) should specify NumericRepresentation.TICKS_AS_MILLISECONDS for inputRepresentation and NumericRepresentation.TICKS for outputRepresentation.

.NET Framework Recommendations

It is recommended that all .NET Framework users migrate as soon as possible.

1. In cases where Lucene.Net.Support.WeakDictionary<TKey, TValue> was used in .NET Framework and .NET Standard 2.0 due to missing APIs, but there is now a better solution using Prism.Core's weak events in combination with ConditionalWeakTable<TKey, TValue>, which means memory management is handled entirely by the GC in Lucene.Net.Index.IndexReader, Lucene.Net.Search.FieldCacheImpl, Lucene.Net.Search.CachingWrappingFilter and Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader. See #613.
2. All known issues with loss of floating-point precision on .NET Framework x86 have been fixed.

Change Log

Breaking Changes

• #547 - Lucene.Net.Util.StringHelper.GOOD_FAST_HASH_SEED: converted from a static field to a property and marked obsolete. Added a new property GoodFastHashSeed. Removed SystemProperties call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability.
• #547 - Lucene.Net.TestFramework: Added LuceneSetUpFixtureBuilder class to load either a subclass or our default instance of LuceneTestFrameworkInitializer. Also added LuceneTestCase.SetUpFixture to control initialization of LuceneTestFrameworkInitializer so it is only called on setup and teardown for the assembly. Added Initialize() method to LuceneTestFrameworkInitializer that must be used when setting factories during testing.
• #547 - Lucene.Net.TestFramework.Util.LuceneTestCase: Deprecated GetClassType() method and added TestType property
• #547 - Lucene.Net.TestFramework.Util.AbstractBeforeAfterRule``: Removed LuceneTestCaseparameter fromBefore()andAfter()` methods.
• #551 - Changed constructors of Lucene.Net.Util.NumberFormat and Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat to accept IFormatProvider rather than CultureInfo and changed Lucene.Net.Util.NumberFormat.Culture property to Lucene.Net.Util.NumberFormat.FormatProvider.
• #554 - Lucene.Net.Misc: Made DocFreqComparer and TotalTermFreqComparer into static singletons, only accessible by the Default property.
• #428, #429, #570 - Lucene.Net.Search.FieldComparer: Redesigned implementation to use reference types for numerics (from J2N) to avoid boxing.
• #570 - Lucene.Net.Search.FieldCache.IParser: Renamed method from TermsEnum() to GetTermsEnum() to match other APIs
• #570 - Lucene.Net.Queries: ObjectVal() returns a J2N.Numerics.Number-derived type rather than a value type cast to object. Direct casts to int, long, double, single, etc. will no longer work without first casting to the J2N.Numerics.Number-derived type. Alternatively, use the corresponding Convert.ToXXX() method for the type you wish to retrieve from the object.
• #574 - Lucene.Net.Suggest.Fst.FSTCompletionLookup/WFSTCompletionLookup: Changed Get() to return long? instead of object to eliminate boxing/unboxing
• #574 - Lucene.Net.Index.MergePolicy::FindForcedMerges(): Removed unnecessary nullable from FindForcedMerges() and all MergePolicy subclasses
• #574 - Lucene.Net.Replicator: Changed callback signature from Func<bool?> to Action, since the return value had no semantic meaning
• #575 - Lucene.Net.Index.DocValuesFieldUpdates: Refactored so the subclasses will handle getting the values from DocValuesFieldUpdatesIterator or DocValuesUpdate via a cast rather than boxing the value. Also marked internal (as well as all members of BufferedUpdates), since this was not supposed to be part of the public API.
• #573, #576 - Changed segment file names to match Lucene 4.8.0 and Lucene.NET 3.x
• #577 - Lucene.Net.Index.SegmentInfos: Changed Info() method to an indexer (.NET Convention)
• #580 - Lucene.Net.Documents.DateTools - Added NumericRepresentation enum to allow converting to/from long in the following formats:

  • Unix Epoch (default): Milliseconds since Jan 1, 1970 12:00:00 AM UTC.

  • Ticks: The raw ticks from DateTime or DateTimeOffset.

  • Ticks as Milliseconds: This is for compatibility with prior versions of Lucene.NET (3.0.3 and 4.8.0-beta00001 - 4.8.0-beta00015). The conversion done on input values is time * TimeSpan.TicksPerMillisecond and the conversion to output values is time / TimeSpan.TicksPerMillisecond.

    The long return value from Lucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation) has been changed from NumericRepresentation.TICKS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.

    The long input parameter provided to Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) has been changed from NumericRepresentation.TICKS_AS_MILLISECONDS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.

    The long return value from Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) has changed from NumericRepresentation.TICKS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.

• #580 - Lucene.Net.Documents.DateTools - De-nested Resolution enum and renamed DateResolution.
• #580 - Lucene.Net.QueryParsers.Flexible.Standard: Changed numeric nodes to accept and return J2N.Numerics.Number-derived types instead of object.
• #581 - SWEEP: Lucene.Net.Util.Fst: Changed API to use J2N.Numerics.Int64 instead of long? for generic closing type as it was designed to use reference equality comparison.
• #581 - SWEEP: Lucene.Net.Util.Fst: Added class constraints to each generic FST type and reverted to reference equality comparisons.
• #581, #279 - Lucene.Net.Util.Fst.Int32sRefFSTEnum: Added MoveNext() method and marked Next() method obsolete. This change had already been done to BytesRefFSTEnum, which made them inconsistent.
• #583 - Lucene.Net.QueryParsers.Flexible: Removed unnecessary nullable value types from ConfigurationKeys and configuration setters/getters in StandardQueryParser. Added AbstractQueryConfig.TryGetValue() method to allow retrieving value types so they can be defaulted properly.
• #583 - Lucene.Net.Queries.Function.ValueSources.EnumFieldSource::ctor() - changed enumIntToStringMap to accept IDictionary<int, string> instead of IDictionary<int?, string> (removed unnecessary nullable)
• #587 - Lucene.Net.TestFramework.Store.MockDirectoryWrapper: Renamed AssertNoUnreferencedFilesOnClose to AssertNoUnreferencedFilesOnDispose
• #619 - Lucene.Net.Spatial: Upgraded to new Spatial4n NuGet package that unifies the types from Spatial4n.Core and Spatial4n.Core.NTS
• #619 - Lucene.Net.Spatial.Prefix.Tree.Cell: Renamed m_outerInstance > m_spatialPrefixTree and constructor parameter outerInstance > spatialPrefixTree
• #619 - Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter.BaseTermsEnumTransverser: renamed m_outerInstance > m_filter, constructor parameter outerInstance > filter
• #619 - Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter: De-nested BaseTermsEnumTraverserclass
• #619 - Lucene.Net.Spatial.Prefix.Tree.GeohashPrefixTree.Factory: de-nested and renamed GeohashPrefixTreeFactory
• #619 - Lucene.Net.Spatial.Prefix.Tree.QuadPrefixTree.Factory: de-nested and renamed QuadPrefixTreeFactory
• #619 - Lucene.Net.Spatial.Prefix.AbstractVisitingPrefixTreeFilter: De-nested VisitorTemplate class and changed protected field m_prefixGridScanLevel to a public property named PrefixGridScanLevel.
• #619 - Lucene.Net.Spatial.Query: Renamed UnsupportedSpatialOperation > UnsupportedSpatialOperationException to match .NET conventions

Bugs

• #363, #534 - Lucene.Net.Replicator.Http.HttpReplicatorTest::TestBasic(). This was failing intermittently due to the Timeout value being set to 1 second instead of the 60 second valu...

v4.8.0-beta00015

This release contains important bug fixes, performance enhancements, concurrency improvements, and improved debugging support (full stack traces, consistent exception types, attributes for debug view, and structurally formattable lists).

Much of the exception handling has been changed so it is recommended to test thoroughly, especially if your application relies on catching exceptions from Lucene.NET for control flow. The full extent of the exception handling changes are not documented here, but can be viewed at https://github.com/apache/lucenenet/pull/476/files.

Known Issues

• Lucene.Net.Index.IndexWriter::Dispose(): Using Thread.Interrupt() to shutdown background threads in .NET is problematic because System.Threading.ThreadInterruptedException could be thrown on any lock statement with contention on it. This includes lock statements on code that we depend on or custom components that are engaged during a Commit() (such as a custom Directory implementation). These exceptions may cause Commit() to fail in unexpected ways during IndexWriter.Dispose(). While this affected all prior releases to a much larger degree, this release provides a partial solution using UninterruptableMonitor.Enter() to ensure these exceptions are ignored and the Thread.Interrupt() state restored, which greatly reduces the chance a Commit() could be broken or a deadlock can occur. This problem will not affect applications that do not call Thread.Interrupt() to shut down a thread. It is recommended never to use Thread.Interrupt() in conjunction with IndexWriter, ConcurrentMergeScheduler, or ControlledRealTimeReopenThread.

Change Log

Breaking Changes

• #455 - lucene-cli: Changed exit codes to well-defined constants to make testing simpler
• #407 - Moved all Document extensions to the Lucene.Net.Documents.Extensions namespace and added tests for DocumentExtensions in Lucene.Net.Tests._J-S, Lucene.Net.Tests.ICU and Lucene.Net.Tests.Facet. Added guard clauses and updated documentation of Document extension methods and some related fields.
• #474 - Lucene.Net.TestFramework.Util.TestUtil: Renamed method parameters from abbreviations to whole words to follow .NET API conventions and improved documentation.
• #475 - Lucene.Net.Grouping: Refactored and improved GroupingSearch Search API and added GroupByField() and GroupByFunction() methods.
• #479 - Moved Lucene.Net.Join types to Lucene.Net.Search.Join namespace
• Marked public exception constructors that were meant only for testing internal (affects only .NET Framework)
• #446, #476 - Redesigned exception handling to ensure that exception behavior is the same as in Lucene and so we consistently throw the closest .NET equivalent exception across all of the projects.
• #480 - Changed Cardinality() methods to Cardinality property. Added obsolete Cardinality() extension methods to the namespace of each of the pertinent types for backward compatibility.
  • Lucene.Net.Index.RandomAccessOrds
  • Lucene.Net.Util.FixedBitSet
  • Lucene.Net.Util.Int64BitSet
  • Lucene.Net.Util.OpenBitSet
  • Lucene.Net.Util.PForDeltaDocIdSet
  • Lucene.Net.Util.WAH8DocIdSet
• #481 - Lucene.Net.Index.Term: Changed Text() method into Text property. Added an obsolete Text() extension method to Lucene.Net.Index namespace for backward compatibility.
• #482 - Lucene.Net.BinaryDocValuesField: Changed fType static field to TYPE (as it was in Lucene) and added obsolete fType field for backward compatibility.
• #483 - Changed all GetFilePointer() methods into properties named Position to match FileStream. Types affected: Lucene.Net.Store.IndexInput (and subclasses), Lucene.Net.Store.IndexOutput (and subclasses). Added obsolete extension methods for each type in Lucene.Net.Store namespace for backward compatibility.
• #484 - Lucene.Net.QueryParser: Factored out NLS/IMessage/Message support and changed exceptions to use string messages so end users can elect whether or not to use .NET localization, as is possible with any other .NET exception type.
• #484 - Lucene.Net.QueryParsers.Flexible.Messages: Removed entire namespace, as we have refactored to use .NET localization rather than NLS
• #484 - Lucene.Net.Util: Removed BundleResourceManagerFactory and IResourceManagerFactory, as these were only to support NLS. The new approach to localizing messages can be achieved by registering QueryParserMessages.SetResourceProvider(SomeResource.ResourceManager, SomeOtherResource.ResourceManager) at application startup using any ResourceManager instance or designer-generated resource's ResourceManager property.
• #497, #507 - Factored out Lucene.Net.Support.Time in favor of J2N.Time. Replaced all calls (except Lucene.Net.Tests.Search.TestDateFilter) that were Environment.TickCount and Time.CurrentTimeMilliseconds() to use Time.NanoTime() / Time.MillisecondsPerNanosecond for more accurate results. This may break some concurrent applications that are synchronizing with Lucene.NET components using Environment.TickCount.
• #504 - Lucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer: Implemented singleton pattern so the class can only be used via the Default property.
• #502 - Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode: Added RemoveChildren() method from Lucene 8.8.1 to fix broken RemoveFromParent() method behavior (applies patch LUCENE-5805). This requires existing IQueryNode implementations to implement RemoveChildren() and TryGetTag().
• #502 - Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode: Added TryGetTag() method to simplify looking up a tag by name.
• #528 - Lucene.Net.Analysis.Stempel.Egothor.Stemmer.MultiTrie: Changed protected m_tries field from List<Trie> to IList<Trie>
• #528 - Lucene.Net.Search.BooleanQuery: Changed protected m_weights field from List<Weight> to IList<Weight>
• #528 - Lucene.Net.Search.DisjunctionMaxQuery: Changed protected m_weights field from List<Weight> to IList<Weight>

Bugs

• #461 - Lucene.Net.Grouping.GroupingSearch::GroupByFieldOrFunction<TGroupValue>(): Fixed casting bug of allGroupsCollector.Groups by changing the cast to ICollection instead of IList.
• #453, #455 - lucene-cli: Made appsettings.json file optional. This was causing a fatal FileNotFoundException after installing lucene-cli without adding an appsettings.json file.
• #464 - Lucene.Net.Codecs.SimpleText.SimpleTextStoredFieldsWriter + Lucene.Net.Codecs.SimpleText.SimpleTextTermVectorsWriter: Fixed Abort() methods to correctly swallow any exceptions thrown by Dispose() to match the behavior of Lucene 4.8.0.
• #394, #467 - Lucene.Net NuGet does not compile under Visual Studio 2017. Downgraded Lucene.Net.CodeAnalysis.CSharp and Lucene.Net.CodeAnalysis.VisualBasic from .NET Standard 2.0 to .NET Standard 1.3 to fix.
• #471 - Lucene.Net.Documents.FieldType: Corrected documentation to reflect the actual default of IsTokenaized as true and NumericType as NumericType.NONE, and to set to NumericType.NONE (rather than null) if the field has no numeric type.
• #476 - Lucene.Net.Analysis.Common.Util.CharArraySet: Throw NotSupportedException when the set is readonly, not InvalidOperationException to match .NET collection behavior
• #476 - Lucene.Net.Codecs.Bloom.BloomFilteringPostingsFormat::FieldsConsumer(): Throw NotSupportedException rather than InvalidOperationException
• #476 - Lucene.Net.Codecs.Lucene42.Lucene42DocValuesProducer::LoadNumeric(): Throw AssertionError rather than InvalidOperationException
• #476 - Lucene.Net.Store.CompoundFileDirectory::ReadEntries(): throw AssertionError rather than InvalidOperationException
• #476 - Lucene.Net.Util.Packed.DirectPackedReader::Get(): Throw AssertionError rather than InvalidOperationException
• #476 - Lucene.Net.Facet: Throw InvalidOperationException rather than ThreadStateException
• #476 - Lucene.Net.Grouping.BlockGroupingCollector: Throw NotSupportedException rather than InvalidOperationException
• #476 - Lucene.Net.Tests.Index.TestUniqueTermCount: Throw NotSupportedException rather than InvalidOperationException
• #486 - Changed all references that were float.MinValue and double.MinValue to float.Epsilon and double.Epsilon because those are the .NET equivalent constants to Float.MIN_VALUE and Double.MIN_VALUE in Java
• #492, #497 - Lucene.Net.Search.ControlledRealTimeReopenThread - Fixed time calculation issue that was causing wait to happen for unusually long time periods.
• Lucene.Net.Tests.Search.TestMultiThreadTermVectors: Removed stray [Test] attribute that was causing extra overhead with no benefit
• #509 - Lucene.Net.Support.WeakDictionary: Changed WeakKey to use WeakReference<T> instead of WeakReference to avoid problems with garbage collection
• #504 - Lucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer: Implemented singleton pattern so the class can only be used via the Default property.
• #506, #509 - Lucene.Net.Index.IndexReader: Use ConditionalWeakTable<TKey, TValue>/WeakDictionary<TKey, TValue> to ensure dead elements are pruned and garbage collected
• #525 - Fixed Lucene.Net.Index.TestIndexWriter::TestThreadInterruptDeadlock() and Lucene.Net.Index.TestIndexWriter::TestTwoThreadsInterruptDeadlock() that were failing due to a difference in .NET Thread.Interrupt() behavior. In Java, an InterruptedException is never thown from synchronized (this) (the equivalent of lock (this)), but .NET may throw `ThreadInterruptedExcepti...

v4.8.0-beta00014

This release contains bug fixes and minor performance improvements

Known Issues

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

Benchmarks (from #310)

Index Files

Click to expand

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
  [Host]          : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT

InvocationCount=1  IterationCount=15  LaunchCount=2  
UnrollFactor=1  WarmupCount=10  

Method	Job	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
IndexFiles	4.8.0-beta00005	905.6 ms	131.82 ms	197.30 ms	43000.0000	8000.0000	7000.0000	220.99 MB
IndexFiles	4.8.0-beta00006	707.1 ms	18.57 ms	26.04 ms	44000.0000	8000.0000	7000.0000	220.99 MB
IndexFiles	4.8.0-beta00007	712.2 ms	16.45 ms	23.06 ms	44000.0000	8000.0000	7000.0000	221.04 MB
IndexFiles	4.8.0-beta00008	785.7 ms	17.37 ms	25.46 ms	44000.0000	8000.0000	7000.0000	221.54 MB
IndexFiles	4.8.0-beta00009	824.9 ms	32.86 ms	48.17 ms	44000.0000	8000.0000	7000.0000	221.34 MB
IndexFiles	4.8.0-beta00010	789.6 ms	16.40 ms	24.04 ms	44000.0000	8000.0000	7000.0000	221.35 MB
IndexFiles	4.8.0-beta00011	805.4 ms	21.26 ms	31.82 ms	44000.0000	8000.0000	7000.0000	221.37 MB
IndexFiles	4.8.0-beta00012	827.8 ms	13.95 ms	20.89 ms	56000.0000	7000.0000	6000.0000	287.03 MB
IndexFiles	4.8.0-beta00013	793.6 ms	13.63 ms	19.55 ms	44000.0000	8000.0000	7000.0000	220.22 MB
IndexFiles	4.8.0-beta00014	812.0 ms	21.97 ms	30.79 ms	44000.0000	8000.0000	7000.0000	220.29 MB

Search Files

Click to expand

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
  [Host]          : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT

IterationCount=15  LaunchCount=2  WarmupCount=10  

Method	Job	Mean	Error	StdDev	Median	Gen 0	Gen 1	Gen 2	Allocated
SearchFiles	4.8.0-beta00005	421.1 ms	111.47 ms	163.38 ms	326.3 ms	18000.0000	1000.0000	-	82.11 MB
SearchFiles	4.8.0-beta00006	349.8 ms	24.03 ms	35.97 ms	338.9 ms	18000.0000	1000.0000	-	82.11 MB
SearchFiles	4.8.0-beta00007	333.6 ms	17.36 ms	25.98 ms	336.8 ms	18000.0000	1000.0000	-	81.9 MB
SearchFiles	4.8.0-beta00008	191.7 ms	7.17 ms	10.51 ms	187.9 ms	17000.0000	1000.0000	-	80.13 MB
SearchFiles	4.8.0-beta00009	186.6 ms	8.56 ms	12.55 ms	184.0 ms	17000.0000	1000.0000	-	80.13 MB
SearchFiles	4.8.0-beta00010	182.2 ms	6.69 ms	9.16 ms	181.6 ms	17000.0000	1000.0000	-	79.85 MB
SearchFiles	4.8.0-beta00011	208.9 ms	17.73 ms	26.54 ms	207.9 ms	17000.0000	1000.0000	-	79.85 MB
SearchFiles	4.8.0-beta00012	192.3 ms	10.99 ms	16.46 ms	187.8 ms	18000.0000	1000.0000	-	81.11 MB
SearchFiles	4.8.0-beta00013	177.4 ms	7.74 ms	11.59 ms	175.1 ms	14000.0000	1000.0000	-	65.78 MB
SearchFiles	4.8.0-beta00014	172.7 ms	5.93 ms	8.88 ms	168.9 ms	14000.0000	1000.0000	-	65.78 MB

Change Log

Breaking Changes

• #424 - Deprecated TaskMergeScheduler, a merge scheduler that was added to support .NET Standard 1.x
• #424 - Lucene.Net.TestFramework: Removed the public LuceneTestCase.ConcurrentMergeSchedulerFactories class

Bugs

• #405, #415 - Lucene.Net.Index.DocTermOrds: Fixed issue with enumerator (OrdWrappedTermsEnum) incorrectly returning true when the value is null.
• #427 - Lucene.Net.Analysis.Common: Fixed TestRollingCharBuffer::Test() to prevent out of memory exceptions when running with Verbose enabled
• #434, #418 - Hunspell affixes' file parsing corrupts some affixes' conditions
• #434, #419 - HunspellStemFilter does not work with zero affix
• #439 - Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader: Fixed synchronization issue between adding new items to the cache and reading RamBytesUsed method
• #439, #417, #319 - Lucene.Net.Spatial.Util.ShapeFieldCacheProvider: Fixed atomicity issue with loading the cache by using Lazy<T>.
• #441 - Lucene.Net.TestFramework.Support.Confguration.TestConfigurationFactory: Use Lazy<T> to ensure the configurationCache.GetOrAdd() factory is atomic.
• #441 - Lucene.Net.TestFramework.Search.ShardSearchingTestBase: Fixed possible KeyNotFoundExceptionwhen getting the value fromcollectionStatisticsCache`
• #441, #417, #319 - Lucene.Net.Spatial.Prefix.PrefixTreeFactory: Use Lazy<T> in ConcurrentDictionary to make the valueFactory atomic.
• #443 - Lucene.Net.Benchmark.ByTask.Feeds.SpatialDocMaker: Since Dictionary<TKey, TValue>.this[key]is not marked virtual in .NET, subclassingDictionary<string, string>is not a valid approach. So we implementIDictionary<string, string>` instead.
• #416 - CLI Documentation issue - environment variable token not replaced.
• #450 - Lucene.Net.Facet - Reverted locking in to the state it was in Lucene 4.8.1, however we are still making use of ReaderWriterLockSlim to improve read performance of caches. Also, removed the 1 second lock timeout from Cl2oTaxonomyWriterCache.

Improvements

• #269 - Added [AwaitsFix] attribute to known failing tests
• #391 - Improved plugins in DocFx when generating API docs
• #392 - Enabled GitHub Actions to Run Tests on Pull Request
• #395 - Improved performance of build pipeline by publishing the whole solution in one step instead of one project at a time
• #395 - Fixed dependency NuGet package version conflicts
• #395 - Added crash and hang detection to the test runs
• #395 - Upgraded to the latest dotnet CLI commands dotnet build and dotnet test rather than dotnet msbuild and dotnet vstest
• #41...

v4.8.0-beta00013

This release contains important bug fixes and performance enhancements.

Known Issues

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Benchmarks (from #310)

Index Files

Click to expand

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
  [Host]          : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT

InvocationCount=1  IterationCount=15  LaunchCount=2  
UnrollFactor=1  WarmupCount=10  

Method	Job	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
IndexFiles	4.8.0-beta00005	628.1 ms	8.41 ms	12.05 ms	43000.0000	8000.0000	7000.0000	220.82 MB
IndexFiles	4.8.0-beta00006	628.3 ms	13.19 ms	19.33 ms	44000.0000	8000.0000	7000.0000	220.67 MB
IndexFiles	4.8.0-beta00007	617.2 ms	8.44 ms	11.83 ms	44000.0000	8000.0000	7000.0000	220.73 MB
IndexFiles	4.8.0-beta00008	620.6 ms	5.62 ms	8.41 ms	44000.0000	8000.0000	7000.0000	221.06 MB
IndexFiles	4.8.0-beta00009	632.8 ms	12.57 ms	18.43 ms	44000.0000	8000.0000	7000.0000	220.95 MB
IndexFiles	4.8.0-beta00010	862.3 ms	51.13 ms	74.95 ms	44000.0000	8000.0000	7000.0000	221.22 MB
IndexFiles	4.8.0-beta00011	636.5 ms	11.06 ms	15.87 ms	44000.0000	8000.0000	7000.0000	221.09 MB
IndexFiles	4.8.0-beta00012	668.8 ms	14.78 ms	21.66 ms	56000.0000	7000.0000	6000.0000	286.63 MB
IndexFiles	4.8.0-beta00013	626.7 ms	7.78 ms	10.91 ms	43000.0000	8000.0000	7000.0000	219.8 MB

Search Files

Click to expand

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
  [Host]          : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT

IterationCount=15  LaunchCount=2  WarmupCount=10  

Method	Job	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
SearchFiles	4.8.0-beta00005	274.8 ms	7.01 ms	10.28 ms	18000.0000	1000.0000	-	82.12 MB
SearchFiles	4.8.0-beta00006	283.4 ms	7.78 ms	11.64 ms	18000.0000	1000.0000	-	82.13 MB
SearchFiles	4.8.0-beta00007	291.5 ms	8.91 ms	13.33 ms	18000.0000	1000.0000	-	81.9 MB
SearchFiles	4.8.0-beta00008	162.3 ms	5.50 ms	8.23 ms	17000.0000	1000.0000	-	80.13 MB
SearchFiles	4.8.0-beta00009	165.6 ms	2.61 ms	3.90 ms	17000.0000	-	-	80.13 MB
SearchFiles	4.8.0-beta00010	159.4 ms	2.84 ms	4.17 ms	17000.0000	1000.0000	-	79.85 MB
SearchFiles	4.8.0-beta00011	160.8 ms	1.93 ms	2.77 ms	17000.0000	1000.0000	-	79.85 MB
SearchFiles	4.8.0-beta00012	169.2 ms	6.48 ms	9.49 ms	18000.0000	1000.0000	-	81.11 MB
SearchFiles	4.8.0-beta00013	161.6 ms	3.28 ms	4.80 ms	14000.0000	1000.0000	-	65.78 MB

Change Log

Breaking Changes

• Lucene.Net.Search.FieldCache: Added interface ICreationPlaceholder and changed CreationPlaceholder class to CreationPlaceHolder<TValue>.

Bugs

• #356 - Lucene.Net.Store.NativeFSLockFactory: Modified options to allow read access on non-Windows operating systems. This caused the copy constructor of RAMDirectory to throw "The process cannot access the file 'file path' because it is being used by another process" excpetions.
• #296 - Lucene.Net.Util.Automaton.State: Removed Equals() implementation; it was intended to use reference equality as a unique key. This caused random IndexOperationExceptions to occur when using FuzzyTermsEnum/FuzzyQuery.
• #387 - Fixed formatting in ArgumentException message for all analyzer factories so it will display the dictionary contents
• #387 - Lucene.Net.Util.ExceptionExtensions.GetSuppressedAsList(): Use J2N.Collections.Generic.List<T> so the call to ToString() will automatically list the exception messages
• #387 - Lucene.Net.TestFramework.Analysis.MockTokenizer: Pass the AttributeFactory argument that is provided as per the documentation comment. Note this bug exists in Lucene 4.8.0, also.
• #387 - Lucene.Net.Analysis.Common.Tartarus.Snowball.Among: Fixed MethodObject property to return private field instead of itself
• #387 - Lucene.Net.Document.CompressionTools: Pass the offset and length to the underlying MemoryStream
• #388 - Downgraded minimum required Microsoft.Extensions.Configuration version to 2.0.0 on .NET Standard 2.0 and 2.1

Improvements

• Updated code examples on website home page
  1. Show cross-OS examples of building Directory paths
  2. Demonstrate where to put using statements
  3. Removed LinqPad's Dump() method and replaced with Console.WriteLine() for clarity
  4. Fixed syntax error in initialization example of MultiPhraseQuery
• Upgraded NuGet dependency J2N to 2.0.0-beta-0010
• Upgraded NuGet dependency ICU4N to 60.1.0-alpha.353
• Upgraded NuGet dependency Morfologik.Stemming to 2.1.7-beta-0001
• #344 - PERFORMANCE: Lucene.Net.Search.FieldCacheImpl: Removed unnecessary dictionary lookup
• #352 - Added Azure DevOps tests for x86 on all platforms
• #348 - PERFORMANCE: Reduced FieldCacheImpl casting/boxing
• #355 - Setup nightly build (https://dev.azure.com/lucene-net/Lucene.NET/_build?definitionId=4)
• PERFORMANCE: Lucene.Net.Util.Automaton.SortedInt32Set: Removed unnecessary IEquatable<T> implementations and converted FrozenInt32Set into a struct.
• PERFORMANCE: Lucene.Net.Util.Bits: Removed unnecessary GetHashCode() method from MatchAllBits and MatchNoBits (didn't exist in Lucene)
• Lucene.Net.Util.Counter: Changed Get() to Value property and added implicit operator.
• #361 - Make CreateDirectory() method virtual so that derived classes can provide their own Directory implementation, allowing for benchmarking of custom Directory providers (e.q LiteDB)
• #346, #383 - PERFORMANCE: Change del...

v4.8.0-beta00012

This release contains important bug fixes and performance enhancements.

Known Issues

• After installation, when upgrading from versions of Lucene.Net 4.8.0-beta00009 or higher may require a restart of all instances of Visual Studio in order to reload the code analysis analyzer.

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Change Log

Breaking Changes

• Lucene.Net.Facet: Renamed LRUHashMap > LruDictionary. Changed all members to be virtual to allow users to provide their own LRU cache.
• Lucene.Net.Facet.FacetsConfig: Removed ProcessSSDVFacetFields from public API (as was done in Lucene), avoid lock (this)
• Lucene.Net.Facet.TaxonomyReader: Changed DoClose() to Dispose(bool) and implemented proper dispose pattern. Avoid lock (this).
• Lucene.Net.Facet.WriterCache: Renamed NameInt32CacheLRU > NameIntCacheLru, NameHashInt32CacheLRU > NameHashInt32CacheLru. Refactored to utilize a generic type internally using composition to avoid boxing/unboxing without exposing the generic closing type publicly. Added public INameInt32CacheLru as a common interface between NameIntCacheLru and NameHashInt32CacheLru.
• Lucene.Net.Facet.Taxonomy.TaxonomyReader: Restructured ChildrenIterator into ChildrenEnumerator
• Lucene.Net.Facet.Taxonomy.CategoryPath: Changed FullPathLength from a method to a property
• Lucene.Net.Facet.DrillSideways: Changed ScoreSubDocsAtOnce from a method to a property
• Lucene.Net.Facet: Refactored OrdAndValue into a generic struct that can be used in both TopOrdAndSingleQueue and TopOrdAndInt32Queue. Added Insert method to Util.PriorityQueue<T> to allow adding value types without reading the previous value for reuse.
• Lucene.Net.Analysis.Common.Miscellaneous.CapitalizationFilter: Changed default behavior to use invariant culture instead of the current thread's culture to match Lucene, which seems more natural when using filters inside of analyzers. This also fits more in line with how other filters are selected.
• #279 - Lucene.Net.Analysis.Compound.Hyphenation.TernaryTree: Renamed Iterator > Enumerator, Keys() > GetEnumerator()
• #279 - Lucene.Net.Benchmarks.ByTask.Feeds.DirContentSource: Renamed Iterator > Enumerator

Bugs

• #269 - Removed cast from NGramTokenAnymousInnerClassHelper::IsTokenChar(int) that was causing surrogate pairs to fail in the TestUTF8FullRange() tests of NGramTokenizerTest and EdgeNGramTokenizerTest
• Fixed potential issue with ArgumentExceptions being thrown from char.ConvertToUtf32(string, int) by reverting back to CodePointAt() method in TestCharTokenizers.TestCrossPlaneNomalization().
• Lucene.Net.QueryParser.Surround.Query.ComposedQuery::MakeLuceneSubQueriesField(): Added missing using block on enumerator
• #296 - Fixed surrogate pair and culture-sensitivity issues with many analyzers.
• Lucene.Net.Analysis.Common: Fixed classes that were originally using invariant culture to do so again. J2N's Character class default is to use the current culture, which had changed from the prior Character class from Lucene.Net.Support that used invariant culture. Fixes TestICUFoldingFilter::TestRandomStrings().
• Lucene.Net.ICU: Fixed ThaiWordBreaker to account for surrogate pairs. Also added locking to help with thread safety. Note that the class is still not completely thread-safe, but this patch fixes the behavior.
• Lucene.Net.Spatial.Util.ShapeFieldCache: Removed unnecessary array allocation
• Lucene.Net.TestFramework: Fixed LineFileDocs to read byte by byte the same way that Lucene does, except using a BufferedStream to improve performance.
• Lucene.Net.TestFramework: Fixed NightlyAttribute, WeeklyAttribute, AwaitsFixAttribute, and SlowAttribute so they work at the class level
• Lucene.Net.Analysis.Icu.Segmentation.ICUTokenizer: Corrected call to ICU4N.UChar.IsWhiteSpace() rather than System.Char.IsWhiteSpace(), which may return different results.
• Lucene.Net.TestFramework.Search.SearchEquivalenceTestBase: Fixed exception when using OpenBitSet.FastGet() instead of OpenBitSet.Get(), since the size of the bit set is unknown.
• Lucene.Net.Index.DocumentsWriterFlushControl: Fixed issue due to misbehaving locking on Monitor.TryEnter(), the code was restructured to disallow any thread that doesn't have a lock into InternalTryCheckoutForFlush() so the threads do note compete for a lock.
• #274 - Lucene.Net.Facet: Fixed null reference exception in DrillSidewaysScorer from patch in Lucene 4.10.4 https://issues.apache.org/jira/browse/LUCENE-6001
• Lucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache: Fixed locking on Dispose() method and made it safe to call dispose multiple times
• Reviewed and added asserts that existed in Lucene and were missing in Lucene.NET. Effectively, this meant we were missing several test conditions that have now been put into place.
• Lucene.Net.ICU: Added locking to ICUTokenizer to only allow a single thread to manipulate the BreakIterator at a time. This is a temporary fix to get the tests to pass until a solution is found for making BreakIterator threadsafe.
• #332 - Lucene.Net.Replicator: Fixed an issue in IndexInputStream that meant the read method could return a number larger than what was passed in for read count or what the buffer could hold, it should instead return the total number of bytes that was read into the buffer, which logically can't be bigger than the buffer it self.
• Lucene.Net.Tests.Index.TestIndexWithThreads::TestRollbackAndCommitWithThreads(): Must catch and ignore AssertionException, as was done in Lucene
• Lucene.Net.Search.TopScoreDocCollector: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestSearchAfter::TestQueries(), TestTopDocsMerge::TestSort_1(), TestTopDocsMerge::TestSort_2().
• Lucene.Net.Sandbox.Queries.SlowFuzzyTermsEnum: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestTokenLengthOpt().
• Lucene.Net.Search.FuzzyTermsEnum: Disabled optimizations for Accept() method on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestTokenLengthOpt().
• Fixed several references to J2N.BitConversion that were calling the overload that normalizes NaN when they should have been calling the raw bit conversion instead (as was done in Lucene).
• #323 - Lucene.Net.Configuration: Removed the IConfigurationRoot interface from the ConfigurationRoot class when targeting a version of Microsoft.Extensions.Configuration less than 2.0. This will allow the end user to upgrade Microsoft.Extensions.Configuration seamlessly to versions 2.0 or higher.
• #286 - Lucene.Net.CodeAnalysis: Separated CSharp and VisualBasic into different assemblies to prevent cross-language dependency issues when using analyzers

Improvements

• #261 PERFORMANCE - Fixed FSTTester to delete while iterating forward instead of using .ElementAt() to iterate in reverse, which takes about 3x longer
• #261 PERFORMANCE - Lucene.Net.Facet.Taxonomy.WriterCache.NameInt32CacheLRU: Changed from Dictionary to ConcurrentDictionary so we can delete items from the cache while forward iterating through it.
• #261 PERFORMANCE - Lucene.Net.Index.FieldInfos: Changed Builder.FieldInfo() method to TryGetFieldInfo() to optimize check for value
• Upgraded NuGet dependency J2N to 2.0.0-beta-0009
• Upgraded NuGet dependency ICU4N to 60.1.0-alpha.352
• Upgraded NuGet dependency Morfologik.Stemming to 2.1.6-beta-0007
• #261 PERFORMANCE - Use J2N's ICollection<T>.ToArray() extension method that uses ICollection<T>.CopyTo(), which takes precedence over the LINQ IEnumerable<T>.ToArray() extension method. Benchmarks show about a 1/3 increase in performance.
• #261 PERFORMANCE - Lucene.Net.Support.IO.FileSupport::CreateTempFile(): Optimized the check for invalid characters
• Directory.Build.props: Disabled warnings for features that require .NET Standard 2.1
• #261 PERFORMANCE - Eliminated several calls to FirstOrDefault(), LastOrDefault(), Skip(), First(), and Last()
• Lucene.Net.Support.ListExtensions: Factored out BinarySearch in favor of implement...

v4.8.0-beta00011

This release contains a critical patch for .NET Framework users that use Microsoft.Extensions.Configuration higher than version 1.1.2. See #311 for details.

This release contains impactful performance enhancements.

Known Issues

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Change Log

Breaking Changes

• PERFORMANCE: Lucene.Net.Analysis.Compound: Changed protected m_tokens field from LinkedList<T> to Queue<T> for better throughput

Bugs

• #311 - Lucene.Net.Configuration: Removed IConfigurationBuilder implementation that prevents .NET Framework users from being able to upgrade to a higher version than 1.1.2.

Improvements

• PERFORMANCE: Lucene.Net.TestFramework: Reverted BaseTermVectorsFormatTestCase to use the original 5000 iterations instead of 500. Reverted TestUtil.RandomSimpleString(Random) to default to a maximum string length of 10 instead of 20, which was slowing down several tests.
• PERFORMANCE: Lucene.Net.TestFramework: Refactored Assert class to use custom comparisons for all members, since NUnit's Assert implementation uses very slow fluent expressions to do comparisons, which are not practical to use inside of tight loops.
• PERFORMANCE: Replaced LinkedList<T> with Queue` where there is a performance advantage
• PERFORMANCE: Reduced memory allocations of CaseInsensitiveComparer by using its singleton static property
• PERFORMANCE: Lucene.Net.Util.RamUsageEstimator: Switched back to System.Collections.Generic.Dictionary because indexer of J2N's dictionary is slower
• PERFORMANCE: Lucene.Net.TestFramework.Util.Fst.FSTTester: Use System.Collections.Generic.Dictionary for better performance
• PERFORMANCE: Switched all remaining tests from using NUnit.Framework.Assert to Lucene.Net.TestFramework.Assert, which are several orders of magnitude faster
• PERFORMANCE: Lucene.Net.Facet: Optimized DirectoryTaxonomyReader by reducing locking, removing unnecessary casts, and using LazyInitializer for the taxonomy array initialization
• PERFORMANCE: Lucene.Net.Tests.Analysis.Common: Changed Hunspell StemmerTestBase to use more optimized assert to compare arrays
• SWEEP: Consolidated empty array creation code so it is more DRY
• Upgraded to C# LangVersion 8.0
• Lucene.Net.Tests.Support.TestApiConsistency: Added regex filter to exclude public fields from the scan

v4.8.0-beta00010

This release contains impactful performance enhancements.

Known Issues

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Change Log

Breaking Changes

• PERFORMANCE: Lucene.Net.Suggest: Narrowed return type of Contexts property from IEnumerable<BytesRef> to ICollection<BytesRef> to improve performance for certain operations
• Lucene.Net.TestFramework.Util.LuceneTestCase: Refactored to correctly load static fields and converted them into static properties. The names were changed to match the conventions of .NET properties.
• #261 PERFORMANCE: Changed to use BitSet instead of BitArray so we don't have to resort to slow extension methods for certain operations

Bugs

• Lucene.Net.Support.Collections::ToString(): Fixed overloads to write "null" when the collection passed is null rather than throw an exception
• #267 - Lucene.Net.Codecs: Fixed testing condition for BaseTermVectorsFormatTestCase on TermVectorsReaders by throwing InvalidOperationException
• #301 - After upgrading NuGet package dependencies for NUnit to 3.12.0, NUnit3TestAdapter to 3.16.1, and Microsoft.NET.Test.Sdk to 16.6.1, several false positives were noted due to invalid try-catch logic in tests

Improvements

• Added missing documentation table of contents links for lucene-cli, Lucene.Net.Demo, Lucene.Net.Queries, Lucene.Net.QueryParser, Lucene.Net.Replicator, Lucene.Net.Sandbox, Lucene.Net.Spatial, and Lucene.Net.Suggest.
• #261 PERFORMANCE: - Removed all calls to Type.GetTypeInfo() and call properties/methods on the Type instance directly.
• #261 PERFORMANCE: - Lucene.Net.Util.AttributeSource: Create built-in attributes directly rather than using Activator.CreateInstance()
• #261 PERFORMANCE: - Lucene.Net.Util.AttributeSource: Optimized creation of string to identify attribute type based on attribute interface name
• #295 PERFORMANCE: - Lucene.Net.TestFramework: Added overloads of Assert.AreEqual for collections where J2N's aggressive mode can be switched off
• #295 PERFORMANCE: - Lucene.Net.TestFramework: Replaced overloads from NUnit.Framework.CollectionAssert with optimized implementations using J2N comparers
• #295 PERFORMANCE: - Lucene.Net.TestFramework: Compile expensive string concatenation out of the release build by using System.Diagnostics.Debug.Assert
• #295 PERFORMANCE: - Lucene.Net.Util.Automaton: Fixed State class to initialize and trim more efficiently.
  Fixes the performance of Lucene.Net.Util.Automaton.TestBasicOperations::TestEmptyLanguageConcatenate().
• #295, #261: PERFORMANCE: - Reduced zero length array allocations
• #295, #261: PERFORMANCE: - Reduced zero length collection allocations
• #295, #261: PERFORMANCE: - Lucene.Net.TestFramework.Search.RandomSimilarityProvider::ToString(): Use StringBuilder for better efficiency
• #295, #261: PERFORMANCE: - Lucene.Net.TestFramework.Util.LuceneTestCase: Cache codecType and similarityName as strings so they don't have to be regenerated for each test
• #295, #261: PERFORMANCE: - Lucene.Net.TestFramework: Use Assert.IsFalse() rather than Assert.That()
• #295, #261: PERFORMANCE: - Reduced calls to LINQ methods and expressions.
• #295, #261: PERFORMANCE: - Lucene.Net.Index.Term: Optimized equality checking
• #295, #261: PERFORMANCE: - Lucene.Net.Util (BytesRef + CharsRef): Implemented IEquatable<T>
• #295, #261: PERFORMANCE: - Lucene.Net.Tests.Index.TestDocumentsWriterDeleteQueue: Updated comparisons to reduce memory allocations
• #295, #261: PERFORMANCE: - Lucene.Net.TestFramework: Changed ConcurrentMergeSchedulerFactories.Values to only return the TaskMergeScheduler rarely, since it is no longer a default setting and is slowing down tests
• #295, #261: PERFORMANCE: - Added some aggressive inlining
• Lucene.Net.TestFramework: Added overloads of Assert.Throws to supply messages
• Lucene.Net.TestFramework: Use the Microsoft.Extensions.Configuration.EnvironmentVariables provider instead of our custom one.
• Lucene.Net.Analysis.Tokenizer: Allow enabling "asserts" for testing
• azure-pipelines.yml: Changed to use pipeline caching instead of build caching for better performance
• Lucene.Net.Tests (core) - Rearranged how test projects are split so parallel jobs can process them faster
• Fixed up leading whitespace to always use spaces instead of tabs
• Updated NuGet package dependency for J2N to 2.0.0-beta-0008
• Lucene.Net.Tests.Analysis.Phonetic.Language.Bm.PhoneticEnginePeformanceTest::Test(): Changed to use Stopwatch for more accurate timing
• Lucene.Net.Tests: Changed private/internal member variables to consistently use camelCase instead of PascalCase
• SWEEP: Changed all properties to use expression style syntax, reordered to put get before set, and changed all backing field names back to their original without the "_Renamed" suffix
• SWEEP: Removed fully-qualified exceptions and added using directives instead
• PERFORMANCE: Pre-compile and statically cache regular expressions
• PERFORMANCE: Lucene.Net.Analysis: Removed unnecessary allocations caused by calling ToString() rather than passing the ICharTermAttribute directly
• PERFORMANCE: Lucene.Net.Analysis.Common: Slight optimization of ToUpper and ToLower methods, which are used in several analyzers
• SWEEP: Removed .NET Standard 1.x/.NET Core 1.x support from all project files
• SWEEP: Removed unused dependencies for .NET Framework
• PERFORMANCE: Lucene.Net.Codecs.SimpleText: Using decimal is 30% faster than using BigInteger for addition and subtraction
• PERFORMANCE: Applied Slow and Nightly attributes where applicable for better testing performance
• Lucene.Net.TestFramework: Added logging of system properties to the output of tests
• PERFORMANCE: Lucene.Net.Util.ArrayUtil::GetNaturalComparer<T>(): Use statically cached singleton instance of comparer
• Upgraded NuGet dependency ICU4N to 60.1.0-alpha.351

New Features

• Lucene.Net.TestFramework.RandomExtensions: Added missing overload of NextInt64(long) to choose only max upper bound
• Added a global Lucene.Net.Diagnostics.Debugging.AssertsEnabled static property that can be used to toggle "asserts" on and off in the release build, similar to how it works in Java. The setting can be injected by end users with the "assert" system property (which is a boolean).
• Lucene.Net.TestFramework: Completed implementation of Nightly, Weekly, AwaitsFix and Slow attributes

v4.8.0-beta00009

Known Issues

• The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:

  F:\Projects\lucenenet>lucene
  Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The         physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
  

  Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.

  appsettings.json

  {
  }
  

  IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Change Log

Bugs

• #294 - Fixed the description for Lucene.NET 2.9.4 in the API documentation.

Improvements

• #260 - Changed culture-sensitivity of function-based queries to use the invariant culture when parsing/formatting values, to match the Lucene implementation.
• #266 - Added documentation to show example usage of injecting custom codecs, doc values formats, and postings formats with pure DI or using a dependency injection container. An demo project named Lucene.Net.Tests.TestFramework.DependencyInjection has also been provided as an example of using Microsoft.Extensions.DependencyInjection to provide custom codecs to the test environment so they can be used during testing of other components.
• #295 - PERFORMANCE Fixed boxing issue with NUnit asserts when passing common primitive types. This significantly improves the performance of running tests.
• Upgraded to ICU4N 60.1.0-alpha.350 and refactored the collation features of Lucene.Net.ICU to utilize the new UCultureInfo class.
• #282 - Restructured API documentation so each assembly has its own "mini-site" to fix x-ref linking issue, which fixes the TOC and breadcrumbs.
• #282 - Documented the docs and website building procedures.
• #282 - Fixed links on the download page and added missing packages (https://lucenenet.apache.org/download/version-4.8.0-beta00009.html).

New Features

• #254 - Implementation of "System Properties" which allow users to supply configuration values to the test framework and lucene-cli using configuration files or environment variables.

v4.8.0-beta00008

This release contains performance improvements and bug fixes.

The Lucene.Net.Support namespace has been phased out, and most of the components that previously existed there have been refactored and moved to J2N, a library to fill in gaps in functionality between the JDK and .NET.

Known Issues

• J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.

Change Log

Breaking Changes

• #239 - Changed Append() overloads of OpenStringBuilder to use startIndex/count rather than start/end to match the .NET convention

• #241 - Made public fields into properties in ArabicStemmer

• Lucene.Net.Collections: Factored out unmodifiable methods and related classes in favor of J2N's AsReadOnly() extension methods

• SWEEP: Lucene.Net.Support: Factored out Equatable, EquatableList, and EquatableSet, and replaced with collections from J2N

• SWEEP: Lucene.Net.Support: Factored out Collections.Equals() and Collections.GetHashCode()

• Lucene.Net.Support.Collections: Factored out Swap() and Shuffle() in favor of J2N's implementation

• Lucene.Net.Support: Factored out IdentityComparer, IdentityHashMap, and IdentityHashSet and used J2N.Runtime.CompilerServices.IdentityEqualityComparer in conjunction with standard Dictionary and HashSet types

• Factored out TreeSet and TreeDictionary from Lucene.Net.Support in favor of J2N.Collections.Generic.SortedSet and J2N.Collections.Generic.SortedDictionary

• Lucene.Net.Support.PriorityQueue: Factored out in favor of J2N's implementation

• Lucene.Net.Support: Refactored ConcurrentHashSet into ConcurrentSet - a wrapper class that can be used to synchronize any set object (ordered or not), similar to how it was done in Java. Changed ordered concurrent set types back to the original type from Lucene.

• Lucene.Net.Search.FieldComparer: Replaced CompareTo() calls with JCG.Comparer<T>.Default.Compare(), factoring out Lucene.Net.Support.SignedZeroComparer in the process.

• Factored out Character, ICharSequence, StringBuilderCharSequenceWrapper, StringBuilderExtensions, StringCharSequenceWrapper, and most StringExtensions methods in favor of J2N's implementation

• Changed semantics of CharTermAttribute.Append() overloads to act more like the .NET StringBuilder class.

  1. The 3rd parameter was changed from an exclusive end index to a count.
  2. A null parameter on a single parameter overload will be a no-op instead of appending the text "null".
  3. A null parameter on a 3 parameter overload will be a no-op if both startIndex and count are 0, otherwise it will throw an ArgumentNullException.

• Lucene.Net.Support: Factored out GeneralKeyedCollection and AttributeItem in favor of J2N.Collections.Generic.LinkedDictionary

• SWEEP: Upgraded to account for breaking changes (AsCharSequence() and BitOperation) in J2N.

• Lucene.Net.Support.Collections: Factored out ImplementsGenericInterface() in favor of J2N's implementation

• Lucene.Net.Support.Collections: Factored out Singleton() method and used collection initializers instead

• Lucene.Net.Support.Collections: Removed NewSetFromMap method and related SetFromMap class

• Lucene.Net.Support: Added ConcurrentHashSet class for internal use, and factored out the use ConcurrentSet throughout the project

• Removed Lucene.Net.Support.ConcurrentHashMapWrapper from the public API and renamed ConcurrentDictionary

• Marked ExceptionToClassNameConventionAttribute, ExceptionToNetNumericConventionAttribute, ExceptionToNullableEnumConventionAttribute, and WritableArrayAttribute internal

• Factored out StringExtensions

• Factored out DictionaryExtensions

• Factored out LurchTable in favor of J2N's implementation

• Lucene.Net.Support.Threading: Deleted ThreadLock, DisposableThreadLocalProfiler

• Factored out Lucene.Net.Support.AssemblyExtensions in favor of J2N's implementation

• Lucene.Net.Support: Marked all types internal

• Moved Lucene.Net.Support.SystemConsole to Lucene.Net.Util namespace

• Lucene.Net.Support: Moved ExceptionExtensions to Lucene.Net.Util namespace

• Lucene.Net.Support.ListExtensions: Moved AddRange, Sort, TimSort, and IntroSort extension methods to Lucene.Net.Util.ListExtensions.

• Lucene.Net.Support.NumberFormat: Moved to Lucene.Net.Util namespace

• Lucene.Net.TestFramework.Support.JavaCompatibility.AbstractBeforeAfterRule: Moved from Lucene.Net.Support namespace to Lucene.Net.Util

• Lucene.Net.TestFramework.Support: Changed namespace of ApiScanTestBase, CultureInfoSupport, and ExceptionSerializationTestBase to Lucene.Net.Util

• Lucene.Net.Util.NumberFormat, Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat: Changed protected locale field to private, made property named Culture, and changed constructors and methods to use "culture" rather than "locale" for parameter names

• Lucene.Net.Support: Moved SystemProperties class to Lucene.Net.Util namespace

• Lucene.Net.Benchmark.Support: Moved EnglishNumberFormatExtensions to Lucene.Net.Util namespace

Bugs

• #237 - Fix duplicate FragNum value on Highlight.TextFragment
• #235 - Fixed broken links on API documentation home page
• Lucene.Net.Search.BooleanClause::Equals(BooleanClause): Fixed potential null reference exception when Query is set to null
• LUCENENET-645 - Added missing call to FileStream::Flush() in FSIndexOutput::Flush() that was preventing persistence to disk from occuring at the necessary phase
• SWEEP: Pass Random instance to all Shuffle() method calls to ensure the same psuedo-random sequence is used based on the seed
• Lucene.Net.Index.DocumentsWriterDeleteQueue::DeleteSlice(): Removed extra Debug.Assert() statement that wasn't in Lucene which caused Lucene.Net.Index.TestIndexWriterUnicode::TestEmbeddedFFFF() test to fail when running in debug mode

Improvements

• #234 - Moved markdown converter to the src/docs folder
• #235 - Updated API documentation so each release exists on its own URL
• #235 - Updated documentation converter to be more robust when dealing with namespace matching
• #239 - Replaced all private IComparer<T> implementations with the built in .NET Comparer.Create() factory method
• #240 - Implemented IAppendable on OpenStringBuilder
• PERFORMANCE - Removed unnecessary memory copy operations from various IndexReader dependencies
• LUCENENET-640 - PERFORMANCE - Factored out WeakIdentityMap in favor of .NET's ConditionalWeakTable
• SWEEP: Factored out LinkedHashMap in favor of J2N's LinkedDictionary
• Lucene.Net.Tests.Search.TestFieldCache::Test(): Simplified expression with LINQ query
• Lucene.Net.Highlighter, Lucene.Net.Tests.Spatial: Swapped in LinkedHashSet from J2N, like in the original Lucene implementation
• SWEEP: Factored out Lucene.Net.Support.HashMap in favor of J2N.Collections.Generic.Dictionary
• SWEEP: Factored out System.Collections.Generic.SortedDictionary in favor of J2N.Collections.Generic.SortedDictionary
• Lucene.Net.Analysis.Kuromoji.Dict.UserDictionary: Swapped out C5 TreeDictionary for J2N.Collections.Generic.SortedDictionary
• SWEEP: Changed System.Collections.Generic.SortedSet to J2N.Collections.Generic.SortedSet
• SWEEP: Factored out C5's TreeSet
• SWEEP: Swapped out System.Collections.Generic.HashSet for J2N.Collections.Generic.HashSet
• SWEEP: Factored out Arrays.AsList(), which was causing both additional operational complexity and unnecessary memory allocations
• Lucene.Net.Util.Automaton.State: Implemented IEquatable<State>, changed enumerator to a struct
• Removed FEATURE_HASHSET_CAPACITY, since J2N now has the full .NET Core 3.x implementation with a capacity constructor
• Lucene.Net.Util.Fst.ListOfOutputs::Merge(): Streamlined so we don't have so many casts
• Lucene.Net.Util.Fst: Use J2N.Collections.List<T> for the closing type of Outputs<T> to ensure the outputs can be compared for structural equality
• Lucene.Net.TestFramework.Search.AssertingScorer: Changed to use ConditionalWeakTable/WeakDictionary
• LUCENENET-610, LUCENENET-640 - Reduced locking in FieldCache and its dependent classes
• Added cross-framework compatibility for nullable attributes
• Changed project files to automatically generate InternalsVisibleTo attributes based on ItemGroup/InternalsVisibleTo elements
• Lucene.Net.Support.DictionaryExtensions: Optimized Put() method, added guard clauses to Put and PutAll
• Lucene.Net.Support.DictionaryExtensions: Factored out Load() and Store() methods in favor of J2N's implementation
• LUCENENET-642 - Lucene.Net.Analysis.TokenStream: Removed Reflection code that is used to force the end user to make TokenStream subclasses or their IncrementToken() method sealed
• LUCENENET-643 - PERFORMANCE - Lucene.Net.Support.IO.FileStreamExtensions::Read(): Moved to StreamExtensions class and optimized to read bytes in bulk instead of one byte at a time
• Lucene.Net.Tests.Index (TestBagOfPositions + TestBagOfPostings): Fixed performance issue due to converting int to string and string to `...