Port performance improvements to Neovim/Lua

[NoahTheDuke]

Regarding Neovim, until Vim9 script support is added, the "New (legacy)" code will be used. If anyone would like to volunteer to write a Lua implementation of this code for Neovim please open an issue and we can discuss how to integrate it.

Originally posted by @axvr in #31 (comment)

I'd be willing to try my hand at it.

[SevereOverfl0w]

Fwiw, I believe the neovim team wrote a Vim9->Lua transpiler to handle this for all the Vim9 code in core.

[axvr]

@SevereOverfl0w, this? https://github.com/tjdevries/vim9jit

Interesting approach they've taken... It looks like they manually run that tool over Vim9 code which generates a Lua version and a Vim script stub for it. (As seen here: https://github.com/neovim/neovim/pull/21623/files)

("Interesting" in a bad way, as it seems it would make tracking and integrating upstream changes more challenging and doesn't work for plugins either.)

[axvr]

The main question on my mind currently is how to have 3 versions of the same code nicely integrated, to make maintenance easy but ensure that the following scenarios are met:

1. Use the new legacy Vimscript code if using this repo as a plugin on Vim 8.2 and below.
2. Use Vim9 script version when included in Vim.
3. Use Vim9 script version when this repos is used as a plugin on Vim 9+.
4. Use a Lua version of the code when this library is used as a plugin on Neovim.
5. Use a Lua version when included* in Neovim?

* I'm unsure how to get an optimised Lua version into Neovim. Previously I have only had to create a PR with Vim and a few days later, the Neovim team pick up the changes and copy them to their fork. So if a Vim9 version was included in Vim, would they compile that to Lua using Vim9jit?

Another unknown is the performance difference of a Vim9->Lua version, compiled with Vim9jit, vs. a hand-rolled Lua version. (The hand-rolled verison would definitely be more readable, but requires additional maintenance.)

[gpanders]

I'm unsure how to get an optimised Lua version into Neovim. Previously I have only had to create a PR with Vim and a few days later, the Neovim team pick up the changes and copy them to their fork. So if a Vim9 version was included in Vim, would they compile that to Lua using Vim9jit?

For now at least, the most likely answer is that if the ftplugin/indent files are converted to Vim9 upstream, they would simply not be ported to Neovim and we would continue to use the "legacy" version. This is obviously not ideal, but we have not yet started using the vim9jit tool in any systematic way (it's been used experimentally a few times, but that's it).

We are not (yet) accepting Neovim-specific implementations of ftplugins/indent files. Primarily because it has not been necessary, and tracking upstream both (1) benefits both projects and (2) minimizes maintenance burden. If filetype plugins (particularly those that are upstreamed to Vim) start moving to Vim9 then we will likely need to figure out a long term solution.