Is there an existing issue for this?

• I have searched the existing issues

Task description

In BigramDictionary and WordDictionary of Lucene.Net.Analysis.SmartCn, the code that loads from files uses ByteBuffer to read int values as little endian. It was done this way in Java because the C-derived code uses little endian byte order but the default in Java is to use big endian byte order. However, in .NET, the default byte order is little endian so we can eliminate these extra allocations by wrapping the FileStream in a BinaryReader and calling BinaryReader.ReadInt32(). The intBuffer allocation can be eliminated in addition to the allocations caused by ByteBuffer.Wrap() in each loop. But the main benefit will be to decouple us from ByteBuffer.

In addition, in BigramDictionary, we can eliminate the calls to tmpword.ToCharArray() by using ReadOnlySpan<char>.

char[] carray = tmpword.ToCharArray();

can be changed to:

ReadOnlySpan<char> caarray = tmpword.AsSpan();

And we should change the caarray parameter to ReadOnlySpan<char> in the Hash1(), Hash2(), GetItemIndex(), GetBigramItemIndex() and GetFrequency() methods. No other changes to those methods should be required.

Note: the wordItem_charArrayTable in WordDictionary could be changed to use ReadOnlyMemory<char>, but I don't think it is worth the added complexity and will diverge significantly from upstream. The references would have to be maintained for all of the strings to keep them in scope.

Tests

Note that this code loads from disk and in the past has only been tested manually. There is some info here:

that contains info about how to test it.

It would be great if we could automate the tests, but it will require getting enough info about how to create a small dictionary file to load so we don't add multiple MB of data to the test project. A few KB is all that is required to ensure it loads from disk correctly. There would also need to be a way devised to determine that loading was successful, which will require some analysis and exploration.

Note also that the Kuromoji project loads dictionary data in a similar manner and also doesn't have automated tests.