Modular highlighting-by-filetype causes major slowdown. Please advise

Question

I would appreciate some help and/or tips adjusting some modular vim filetype-specific code in my vimrc. Thank you in advance.

My vim --version is 8.1.

I have extracted a minimal sample of the modular code below. Here is what I had in mind when I wrote it:

1. I have a function, LoadAllKeywords(), in which is defined N keyword lists (via syn keyword).
2. I have 1 function for each of M different filetypes (*.py, *.c, ...). Each of these M filetype-specific functions will load all keyword lists via a call to LoadAllKeywords(), and then choose a subset of those lists to perform highlighting on. This allows me to store all my keywords in a single place in code, while letting highlight rules vary per ft.
3. The modularity comes at the cost of instantaneous loading. The load time scales seemingly exponentially with the number of events. Events include loading a buffer (i.e. BufEnter) and accessing a buffer via regular old buffer switching (c-w c-h).

The code is below, as well as an excerpt of the function profiler output (:profile {}), which I discovered through this SO post.

Function definition for setting up N keyword lists

fun! LoadAllKeywords()
    syn keyword msgsHL containedin=.*Comment     msgs flash tick

    syn keyword mvpHL containedin=.*Comment      works_ dbugd_ depr_ collab_ delta_ aka_ restrict_ cisc_
    syn keyword pipeHL containedin=.*Comment     step_ 
    syn keyword warnHL containedin=.*Comment     warning_ gotcha_ sidefx_ improvement_ rbugd_ rename_ seq_

    syn keyword RecipeHL containedin=.*Comment      TITLE STEP
    syn keyword ToyHL containedin=.*Comment      src_ play_ risc_ overview_ idea_ mathidea_ csidea_ syntax_ question_

    syn keyword ContentHL IMAGE BLOOM PIPE ALGO GRID RE IXO SIMPLE DELTA 
    syn keyword StructureHL MVFR MVTO TOC RISC CISC TODO BKMK GOTO
endfu

A subset of M functions for filetype specific highlighting.

...
fun! PythonHighlights()
    call LoadAllKeywords() "load keywords to pick and choose from

    hi msgsHL ctermfg=Blue ctermbg=None cterm=None
    hi mvpHL ctermfg=DarkGreen ctermbg=None cterm=None
    hi pipeHL ctermfg=Grey ctermbg=None cterm=bold
    hi warnHL ctermfg=Yellow ctermbg=None cterm=None
endfu

fun! NotesHighlights()
    call LoadAllKeywords() "load keywords to pick and choose from
    hi ContentHL ctermfg=LightGrey ctermbg=None cterm=bold
    hi StructureHL ctermfg=Blue ctermbg=None cterm=None
endfu
...

functions are invoked from respective filetypes at .vim/ftplugin/*.

augroup python_autogroup "in ftplugin/python.vim
    au!
    autocmd BufEnter *.py call PythonHighlights() "moving this to _python.vim, where it will be called.If the function isn't defined, move _aesthetics to run before _python, _notes, and other filetypes.
augroup END

call NotesHighlights() "in ftplugin/notes.vim

autocmd BufEnter *.c call CLikeHighlights()
autocmd BufEnter *.h call CLikeHighlights()

This is the profiler, profiling nothing more than 1 opening of a .c file. Profiling was started after opening maybe 8-10 files, to induce significant slowdown. You can find the functions above as suspect in the sorted top 3 here.

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    1   1.599449   0.000926  <SNR>62_NetrwBrowseChgDir()
   98   1.130717   0.001972  CLikeHighlights()
   98   1.128745             LoadAllKeywords()
  102   0.200728   0.041670  <SNR>4_SynSet()
  102   0.105099   0.076247  <SNR>22_LoadFTPlugin()
    1   0.045744   0.001851  netrw#Explore()
    2   0.043202   0.000217  netrw#LocalBrowseCheck()
    1   0.042797   0.001287  <SNR>62_NetrwBrowse()
    1   0.035903   0.000969  <SNR>62_PerformListing()
  102   0.019905   0.017667  <SNR>23_LoadIndent()
    2   0.017507   0.001461  <SNR>62_NetrwSafeOptions()
    4   0.010980   0.001338  fugitive#detect()
    1   0.009971   0.000029  <SNR>28_record()
    1   0.009942   0.000298  <SNR>28_addtomrufs()
    1   0.009226   0.000012  <SNR>28_savetofile()
    1   0.009214   0.009204  ctrlp#utils#writecache()
    1   0.008837   0.008824  <SNR>29_persist()
    1   0.008067   0.004857  <SNR>62_LocalListing()
   17   0.005025             <SNR>38_Highlight_Matching_Pair()
    1   0.004706   0.000034  dist#ft#FTheader()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
   98              1.128745  LoadAllKeywords()
  102   0.105099   0.076247  <SNR>22_LoadFTPlugin()
  102   0.200728   0.041670  <SNR>4_SynSet()
  102   0.019905   0.017667  <SNR>23_LoadIndent()
    1   0.009214   0.009204  ctrlp#utils#writecache()
    1   0.008837   0.008824  <SNR>29_persist()
   17              0.005025  <SNR>38_Highlight_Matching_Pair()
    1   0.008067   0.004857  <SNR>62_LocalListing()
    1   0.002979   0.002905  <SNR>62_NetrwMaps()
    4              0.002700  <SNR>30_define_commands()
    1              0.002611  <SNR>62_NetrwSetSort()
   34              0.002496  <SNR>62_NetrwFile()
   98   1.130717   0.001972  CLikeHighlights()
    1   0.045744   0.001851  netrw#Explore()
    8              0.001737  <SNR>30_map()
    2   0.017507   0.001461  <SNR>62_NetrwSafeOptions()
    3              0.001410  <SNR>31_setup_vinegar()
    4   0.010980   0.001338  fugitive#detect()
    2              0.001306  <SNR>62_NetrwOptionSave()
    1   0.042797   0.001287  <SNR>62_NetrwBrowse()

Answer 1

:syn

:syn commands have no business being called outside of syntax scripts. If you want to add keyword to the existing roster for a given filetype, use the "after" mechanism:

" in after/syntax/<filetype>.vim
syn keyword msgsHL containedin=.*Comment msgs flash tick
[...]

If you want to store a bunch of keyword definitions in a central place, having them in a function is unnecessary. Instead, put them in a global syntax script:

" in syntax/mykeywords.vim
syn keyword msgsHL containedin=.*Comment msgs flash tick

syn keyword mvpHL containedin=.*Comment works_ dbugd_ depr_ collab_ delta_ aka_ restrict_ cisc_
syn keyword pipeHL containedin=.*Comment step_ 
syn keyword warnHL containedin=.*Comment warning_ gotcha_ sidefx_ improvement_ rbugd_ rename_ seq_

syn keyword RecipeHL containedin=.*Comment TITLE STEP
syn keyword ToyHL containedin=.*Comment src_ play_ risc_ overview_ idea_ mathidea_ csidea_ syntax_ question_

syn keyword ContentHL IMAGE BLOOM PIPE ALGO GRID RE IXO SIMPLE DELTA 
syn keyword StructureHL MVFR MVTO TOC RISC CISC TODO BKMK GOTO

that you can source in various local syntax script:

" in after/ftplugin/<filetype>.vim
runtime syntax/mykeywords.vim

:hi

Similarly, :hi commands generally have no business being called outside of a colorscheme.

If, for some reason, you can't add them to a proper colorscheme, then the next best thing to do is to execute those commands once, after a ColorScheme event:

" in your vimrc
function! MyHighlights() abort
    hi msgsHL ctermfg=Blue ctermbg=None cterm=None
    hi mvpHL ctermfg=DarkGreen ctermbg=None cterm=None
    hi pipeHL ctermfg=Grey ctermbg=None cterm=bold
    hi warnHL ctermfg=Yellow ctermbg=None cterm=None
    hi ContentHL ctermfg=LightGrey ctermbg=None cterm=bold
    hi StructureHL ctermfg=Blue ctermbg=None cterm=None
endfunction

augroup MyColors
    autocmd!
    autocmd ColorScheme * call MyHighlights()
augroup END

" the blocks above must appear before ANY :colorscheme command
colorscheme foobar

Conclusion

Vim's runtime is already modular enough and, if understood/used correctly, won't slow down exponentially. I know the urge to "modularise" one's config against arbitrary lines can be strong at some point in one's journey but it rarely delivers much benefit. If ever.

Case in point:

• A pretty efficient mechanism is already in place for defining filetype-specific highlight groups so there is no point in devising your own.
• It is already possible to source another syntax script from any syntax script so there is no point in complicating things with custom functions.
• The appearance of a highlight group can and should be set globally, via a proper colorscheme. Doing it at the ftplugin level or on BufEnter makes no sense at all.