Vimscript function for replacing the visually selected buffer in nvim

Question

I'm trying to create a neovim color picker plugin, I made a Python application for selecting color from a color wheel using Python's tkinter library, and used a vimscript function to print the output of the Python application in nvim's buffer window.

But the problem is when I visually select buffer in nvim and try to replace with the output of Python application, It appends the output in buffer window instead of replacing it.

How can I fix it?

Here is the vimscript function which I've tried:

function! s:open_color_picker()
    let lnum = line('.')
    let col = col('.')
    let current_line = getline(lnum)

    let is_visual_mode = (mode() ==# 'v' || mode() ==# 'V' || mode() ==# "\<C-v>")

    let insert_after = get(g:, 'NVIMColorPicker#InsertAfter#TheCursor', 0)
    let insert_before = get(g:, 'NVIMColorPicker#InsertBefore#TheCursor', 0)

    let color_picker_path = system('find ~/.local/share/nvim -type f -name color_picker.py')
    let color_picker_path = substitute(color_picker_path, '\n\+$', '', '')
    let color_picker_path = substitute(color_picker_path, '^\\n\+', '', '')

    let color = system('python3 ' . shellescape(color_picker_path))
    let color = substitute(color, '\n\+$', '', '')
    let color = substitute(color, '^\\n\+', '', '')

    if insert_after
        " insert after the cursor
        let hex_color = substitute(current_line, '\%' . (col + 1) . 'c', "\\0" . color, '')
        call setline(lnum, hex_color)

    elseif insert_before
        " insert before the cursor
        let hex_color = strpart(current_line, 0, col - 1) . color . strpart(current_line, col - 1) 
        call setline(lnum, hex_color)
    else
        " default behavior (insert after the cursor)
        let hex_color = substitute(current_line, '\%' . (col + 1) . 'c', "\\0" . color, '')
        call setline(lnum, hex_color)
    endif

        " for replacing the visually selected hex
    if is_visual_mode
        let old_reg = getreg('"')
        let old_regtype = getregtype('"')

        normal! gvy
        let selected_text = @"

        let new_output = color

        call setreg('"', new_output, 'v')

        let start_pos = getpos("'<")
        let end_pos = getpos("'>")

        let start_line = start_pos[1]
        let start_col = start_pos[2]
        let end_line = end_pos[1]
        let end_col = end_pos[2]

        if start_line == end_line
            let current_line = getline(start_line)
            let new_line = current_line[:start_col-2] . new_output . current_line[end_col-1:]
            call setline(start_line, new_line)
        else
            let first_line = getline(start_line)
            let last_line = getline(end_line)
            let new_first_line = first_line[:start_col-2] . new_output
            let new_last_line = last_line[end_col-1:]
            call setline(start_line, new_first_line)
            call setline(end_line, new_last_line)
            if end_line - start_line > 1
                call deletebufline('%', start_line + 1, end_line - 1)
            endif
        endif
        call setreg('"', old_reg, old_regtype)

    endif

endfunction

command! -range ColorPicker call s:open_color_picker()

Answer 1

Your question is not clear on what you want to do if a visual selection is present. This answer assumes you want to replace the entire selection with whatever it is that your color_picker.py returns.

I chose to retire the global variable g:NVIMColorPicker#Insert#After#TheCursor. There's really no point in having two variables that are mutually exclusive. Now, the default behavior is to insert after the cursor unless g:NVIMColorPicker#InsertBefore#TheCursor is set to a truthy value.

Another decision was to return a Vim command which can be run through :normal! instead of changing the lines through setline() but that's subject to debate. My solution uses execute 'normal!... a lot which is really something I like to avoid in general. One could debate about which approach is more pretty.

Note that I used Vim's built-in function findfile() instead of falling back to the external find command. For a finished plugin, I'd expect the paths to be reasonably defined. Knowing where it is is better than knowing where to search.

The whole range-management is now done within the command :ColorPicker which reliefs s:open_color_picker() of the burden of handling visual ranges.

Namely, if a <range> was present, we restore the last visual selection with gv (we're already in command-line mode and not in visual mode anymore) and shove it into the black-hole register with "_d. Bye bye.

The remark about command-line mode is by the way why your original function would not work. By the time we execute the function, visual mode is gone and mode() will have forgotten all about it.

After that, just execute whatever s:open_color_picker() returns as normal mode commands.

function! s:open_color_picker() abort
    let insert_before = get(g:, 'NVIMColorPicker#InsertBefore#TheCursor', 0)

    " Line below is untested as I did not have the script
    let color_picker_path = findfile('color_picker.py', expand('~')..'/.local/share/nvim/**')

    if ! executable('python3')
        " Python not in PATH? Log an error and bail out!
    endif

    let color = system('python3 ' . shellescape(color_picker_path))
    " I tested with the line below
    " let color = system('echo ff00ff')
    let color = substitute(color, '\n\+$', '', '')
    let color = substitute(color, '^\\n\+', '', '')

    if insert_before
        " insert before the cursor
        return 'i'..color
    else
        " default behavior (insert after the cursor)
        return 'a'..color
    endif
endfunction

command! -range ColorPicker if <range> | execute 'normal! gv"_d' | endif | execute 'normal! '..s:open_color_picker()

For reference, see the :help on the respective commands used.

One last thing, I wrote and tested all this in Vim 9.1. It should work in NeoVim as well.