The Fanciest REPL History in the Land #59819

tecosaur · 2025-10-12T18:58:03Z

The Fanciest REPL History in the Land ✨

Do you dread typing out code for the second time? Are you a particular enjoyed of REPL history?

Well, I know I am, and for years I have yearned for something better than the current readline-style completion, better than OhMyREPL.jl's fzf-driven completion, better than any REPL history I've seen before!

It's not quite finished baking, but we're onto the final stretch 😀

repl_history_demo.webm

Thanks to @kdheepak, @jakobnissen, and @digital-carver for helping me design the UI and UX over on Zulip (#repl > Revamped REPL history).

Features

Zippy searching
- Event-driven asynchronous filtering UI
- Incremental, resumable searching with dynamic batch sizes
- Log-structured search checkpoints
Multi-selection
Faster histfile parsing (~2x)
Multiple search modes
A friendly help page
Syntax highlighting
Save multiple items to a file or your clipboard

TODO

Introduce annotation-preserving replace method
Thoroughly test the new replace method
Ask somebody more compiler-y about the performance pitfalls of the replace method (see: the REVIEW: ... code comments in annotated_io.jl)
Implement flashy REPL history
Restore up/down arrow history rotation in the REPL (collateral damage of over-zealous deleting)
Create a new test set for the new history
Make sure that enough is precompiled to be relatively snappy

This PR is on top of #59778, because I think I can safely assume that will be merged first.

DilumAluthge · 2025-10-12T21:20:03Z

Anyone else getting "video playback aborted due to a network error"?

KristofferC · 2025-10-13T10:53:06Z

If possible, it would be nice to cut this up into some orthogonal pieces. For example, the AnnotatedString perf improvements could be a separate PR that (with benchmarks) could be merged and would make the diff here smaller and easier to review.

stdlib/REPL/src/History/search.jl

kdheepak · 2025-10-13T15:09:26Z

stdlib/REPL/src/History/search.jl

+        filename = try
+            readline(term.in_stream)
+        catch err
+            if err isa InterruptException
+                ""
+            else
+                rethrow()
+            end
+        end
+        isempty(filename) && (println(out, S"\e[F\e[2K{light,grey:{bold:history>} {red:×} History selection aborted}\n"); return)
+        write(filename, "# Julia REPL history excerpt\n\n", content)


Maybe it should default write to a location? The prompt could default to ~/.julia/history.jl (or a more appropriate location).

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

I've had exactly the same thought. I just can't be bothered to make it happen 😅

tecosaur · 2025-10-13T16:06:22Z

If possible, it would be nice to cut this up into some orthogonal pieces.

I'm happy to split this up, but there are really just three parts to it:

A refactor to _replace_final (needed to reuse some code for annotation-preserving replacement)
Introducing support for annotation-preserving replacement (needed for changing multi-line annotated strings into single-line with linebreak symbols)
Replacing the old history system with the fancy new one

I could split 1-2 off into a separate PR to review, if that sounds like a nice idea?

the AnnotatedString perf improvements

There's only suspicious performance here, not perf improvements, unfortunately 😞. I say "suspicious" because if I remove the keyword argument from replace skip reconstructing a named tuple with a range shifed the total time of a small replace op drops from ~4400ns to ~200ns. Given that an equivalent annotation-less replace takes ~65ns a time of ~200ns would be decent I think, but ~4400ns is strange — I can't figure it out though. If I can interest anyone else in investigating, that would be great.

For now, I've left these REVIEW code comments:

julia/base/strings/annotated_io.jl

Lines 202 to 226 in cc7ed22

    
           # REVIEW: For some reason, construction of `newannot` 
        
           # can be a significant contributor to the overall runtime 
        
           # of this function. For instance, executing: 
        
           # 
        
           #     replace(AnnotatedIOBuffer(), S"apple", 
        
           #             'e' => S"{red:x}", 'p' => S"{green:y}") 
        
           # 
        
           # results in 3 calls to `_insert_annotations!`. It takes 
        
           # ~570ns in total, compared to ~200ns if we push `annot` 
        
           # instead of `newannot`. Commenting out the `_insert_annotations!` 
        
           # line reduces the runtime to ~170ns, from which we can infer 
        
           # that constructing `newannot` is somehow responsible for 
        
           # a ~30ns -> ~400ns (~13x) increase in runtime!! 
        
           # This also comes with a marginal increase in allocations 
        
           # (compared to the commented out version) of 2 -> 14 (250b -> 720b). 
        
           # 
        
           # This seems quite strange, but I haven't dug into the generated 
        
           # LLVM or ASM code. If anybody reading this is interested in checking 
        
           # this out, that would be brilliant 🙏. 
        
           # 
        
           # What I have done is found that "direct tuple reconstruction" 
        
           # (as below) is several times faster than using `setindex`. 
        
           newannot = (region = start+offset:stop+offset, 
        
                       label = annot.label, 
        
                       value = annot.value)

julia/base/strings/annotated_io.jl

Lines 236 to 240 in cc7ed22

    
           # REVIEW: For some reason the `Core.kwcall` indirection seems to cause a 
        
           # substantial slowdown here. If we remove `; count` from the signature 
        
           # and run the sample code above in `_insert_annotations!`, the runtime 
        
           # drops from ~4400ns to ~580ns (~7x faster). I cannot guess why this is. 
        
           function replace(out::AnnotatedIOBuffer, str::AnnotatedString, pat_f::Pair...; count = typemax(Int))

tecosaur · 2025-10-13T18:50:24Z

I've just added back ~400 loc of the original code for using up/down arrow keys to go through recent history without ^R.

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

I would like to add session-awareness. Maybe I'll try to drop that in this PR too?

tecosaur · 2025-10-14T13:43:20Z

Rebased to the latest REPL Syntax Highlighting HEAD.

tecosaur · 2025-10-14T13:45:35Z

Fixed some typos.

tecosaur · 2025-10-15T16:23:17Z

The precompilation situation is looking pretty decent, this is all I'm seeing with --trace-compile:

#=   19.4 ms =# precompile(Tuple{Type{Base.IOContext{IO_t} where IO_t<:IO}, Base.GenericIOBuffer{Memory{UInt8}}, Base.TTY})
#=   12.9 ms =# precompile(Tuple{typeof(Base.get), Base.Dict{Tuple{Symbol, Any}, Int64}, Tuple{Symbol, Symbol}, Int64})
#=   19.2 ms =# precompile(Tuple{typeof(Base.setindex!), Base.Dict{Tuple{Symbol, Any}, Int64}, Int64, Tuple{Symbol, Symbol}})
#=    2.5 ms =# precompile(Tuple{typeof(Base.print), Base.IOContext{Base.GenericIOBuffer{Memory{UInt8}}}, Base.AnnotatedString{String}})
#=    1.6 ms =# precompile(Tuple{typeof(Base._str_sizehint), UInt64})
#=    1.9 ms =# precompile(Tuple{typeof(Base.print), Base.GenericIOBuffer{Memory{UInt8}}, UInt64})
#=    2.5 ms =# precompile(Tuple{typeof(Base.AnnotatedDisplay.ansi_write), typeof(Base.write), Base.GenericIOBuffer{Memory{UInt8}}, Base.AnnotatedString{String}})
#=   14.6 ms =# precompile(Tuple{typeof(Base.getindex), Base.JuliaSyntax.GreenNode{Base.JuliaSyntax.SyntaxHead}, Int64})
#=    7.8 ms =# precompile(Tuple{typeof(Base.Terminals.cmove_up), Base.Terminals.TerminalBuffer})

tecosaur · 2025-10-15T16:28:36Z

Rebased now that #59778 has been merged.

tecosaur · 2025-10-15T16:50:25Z

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

Fixed.

tecosaur · 2025-10-16T18:22:49Z

Thanks to Miguel, Camillo, and Sundar for taking this for a test-drive and providing feedback 🙂

Changes:

Clarified that ; uses and logic in quick help
Strip meaningless whitespace around filter segments
Changed "negative" → "negated" search term
Arrow keys (and others) no longer accidentally abort save
Prevent runaway redisplays
More minimal display updates
Make mmap work on Windows

tecosaur · 2025-10-17T18:21:10Z

Rebase (get the REPL precompilation improvements)

tecosaur · 2025-10-20T17:29:26Z

Add a second in-REPL help page, with the ?? query
Move mention of backslash escaping to the second help page
Join adjacent space-separated matches
Add news and docs pages

tecosaur · 2025-10-20T17:30:10Z

On an unrelated note, I'm very sad that the diff is no longer +3333 loc. That was fun while it lasted.

tecosaur · 2025-10-21T12:09:04Z

Rebase after Pkg was bumped, and so we can also drop the Pkg bump commit

tecosaur · 2025-10-21T12:09:41Z

Drop mmap to see if that makes Windows CI happier

tecosaur · 2025-10-21T13:32:16Z

Fix a typo, tell topes that another "typo" isn't a typo at all (indexin)

stdlib/REPL/src/History/resumablefiltering.jl

tecosaur · 2025-10-21T13:53:19Z

Remove mmap as a dep
Fix example in docs
Change keybindings to be more consistent with readline defaults
Rebase with master, to resolve a conflict

tecosaur · 2025-10-21T17:02:22Z

Fix @async history_do_initialize(hp) call (arg1 was incorrectly hp.history)
Catch any errors in the @async history_do_initialize(hp) (use errormonitor)
Fix a potential bounds error in history file update!
Add a big explanatory comment for why replacing a pidfile lock with O_APPEND seems sane

tecosaur · 2025-10-21T17:16:01Z

Add the (previously) bounds-error causing incomplete history as a test case
Rebase to latest master

tecosaur · 2025-10-21T18:07:25Z

Remember to reset history state after the new test

rfourquet · 2025-10-22T14:16:33Z

That's truly excellent!! Thank you!

I just started using it, and noticed a couple of problems:

I use a kind of numbered prompt, which breaks as soon as history search is first used, or on the second session prompt, by loosing a reference to the first history entry in the current session (presumably the .start_idx field of REPLHistoryProvider is reset to 1 or similar). A side effect is that, even without using a numbered prompt, the Alt-< shortcut brings you back to the first history entry ever, not the first of the session. This can be seen with the official numbered prompt:

I get an error when entering history search if my prompt_prefix is set to a function. The error is at
runsearch(histfile::HistoryFile, term, prefix::String = "\e[90m"), whose prefix arg can only be a String.

Extract the replacement loop body into `_replace_once` to ease future annotation tracking during string replacement operations. The new function returns match information (pattern index, match range, bytes written) that will be needed to properly adjust annotation positions when replacements occur.

Implement `replace` function for `AnnotatedString` that properly handles annotation regions during pattern replacement operations. The function tracks which bytes are replaced versus preserved, maintaining annotations only on original content and adding new annotations from replacement text. - Supports AnnotatedChar, AnnotatedString, and SubString replacements - Drops, shifts, and splits existing annotations appropriately - Refactored `_insert_annotations!` to work with annotation vectors directly - Adjacent replacements with identical annotations are merged into single regions - Lots of tests (thanks Claude!) Performance is strangely poor. For the test case mentioned in the REVIEW comment within `_insert_annotations!` we should be able to perform the replacement in ~200ns (compared to ~70ns for the equivalent unannotated case). However, for two reasons that are beyond me instead it takes ~4400ns. See the REVIEW comments for more details, help would be much appreciated.

Discarding the annotations can come as a bit of a surprise best avoided.

Since the dawn of the Julia REPL, history completion has been limited to a readline-style interface. OhMyREPL improved the experience with fzf, but those who yearned for a delightful history completion experience (me) were left underwhelmed. With this overhaul, I now find myself spending more time looking through my history because it's just *so nice* to do so. The new history system is organised as a standalone module in stdlib/REPL/src/History with a clear separation of concerns: 1. History file management 2. Event-driven prompt/UI updating 3. Incremental filtering 4. UI display 5. Search coordination (prompt + display + filter) I've attempted to pull out all the (reasonable) stops to make history searching as fluid and snappy as possible. By memory mapping the history file in the initial read, and optimising the parser, we can read ~2 million history items per second. Result filtering is incremental and resumable, performed in dynamically sized batches to ensure responsiveness. Rapid user inputs are debouced. We store a log-structured record of previous search result, and compare search strictness to resume from prior partial results instead of filtering the history from scratch every time. Syncronisation between the interface and filtering is enabled via a Channel-based event loop. Enjoy! (I know I am)

tecosaur · 2025-10-22T17:38:01Z

Rebase to latest master
Fix a dodgy testcase
Show $mode> in the preview
Support prompt_prefix functions

kdheepak reviewed Oct 13, 2025

View reviewed changes

stdlib/REPL/src/History/search.jl Outdated Show resolved Hide resolved

kdheepak reviewed Oct 13, 2025

View reviewed changes

tecosaur force-pushed the fancy-repl-history branch from cc7ed22 to 75fd56d Compare October 13, 2025 18:48

tecosaur force-pushed the fancy-repl-history branch from 75fd56d to 221b4e5 Compare October 14, 2025 13:42

tecosaur force-pushed the fancy-repl-history branch from 221b4e5 to ee23322 Compare October 14, 2025 13:45

tecosaur mentioned this pull request Oct 15, 2025

Make banner size depend on terminal size #51811

Open

KristofferC mentioned this pull request Oct 15, 2025

add support for automatic bracket completion in REPL input #59847

Merged

tecosaur force-pushed the fancy-repl-history branch from ee23322 to 3a6d8d7 Compare October 15, 2025 16:28

tecosaur force-pushed the fancy-repl-history branch from 3a6d8d7 to 4bac24f Compare October 15, 2025 16:50

tecosaur force-pushed the fancy-repl-history branch from 4bac24f to c7a66cb Compare October 16, 2025 18:22

tecosaur force-pushed the fancy-repl-history branch 2 times, most recently from 7126dd7 to 0da2a86 Compare October 17, 2025 18:20

tecosaur force-pushed the fancy-repl-history branch from dade819 to 20658d5 Compare October 20, 2025 17:29

tecosaur force-pushed the fancy-repl-history branch from 20658d5 to 7e2228f Compare October 21, 2025 12:08

tecosaur force-pushed the fancy-repl-history branch from 7e2228f to 3761825 Compare October 21, 2025 12:09

tecosaur force-pushed the fancy-repl-history branch from 3761825 to c20074d Compare October 21, 2025 13:31

rfourquet reviewed Oct 21, 2025

View reviewed changes

stdlib/REPL/src/History/resumablefiltering.jl Outdated Show resolved Hide resolved

tecosaur force-pushed the fancy-repl-history branch 2 times, most recently from c97a251 to ba66c91 Compare October 21, 2025 13:53

tecosaur force-pushed the fancy-repl-history branch from ba66c91 to 0495021 Compare October 21, 2025 17:00

tecosaur force-pushed the fancy-repl-history branch 2 times, most recently from 73d36b0 to 9b7ceb7 Compare October 21, 2025 17:15

tecosaur force-pushed the fancy-repl-history branch from 9b7ceb7 to 29172d2 Compare October 21, 2025 18:07

tecosaur and others added 7 commits October 23, 2025 01:37

Preserve annots of case transformed annot substr

38c1e06

Discarding the annotations can come as a bit of a surprise best avoided.

Fix Mac precompile hang

ddf7bda

Add news and docs for the new history searcher

3aba8a9

Tell typos CI that "indexin" is valid

03ca9f0

tecosaur force-pushed the fancy-repl-history branch from 29172d2 to 03ca9f0 Compare October 22, 2025 17:37

Uh oh!

The Fanciest REPL History in the Land #59819

Are you sure you want to change the base?

The Fanciest REPL History in the Land #59819

Conversation

tecosaur commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!