Engineering a Modal Text Object Engine in Vim: Custom Motions and Operator-Pending Grammars

Vim’s built‑in text objects—iw, i(, at—are powerful, but they stop short of domain‑specific structures. What if you could select an indented block with ii, a function argument with ia, or a column of a Markdown table with ic? For advanced users, the real magic lies in creating custom objects that compose seamlessly with any operator—dii, cia, yic—turning Vim into a structured editing engine tailored to your exact filetypes.

This article documents a system that generalises custom text objects into a declarative grammar, built entirely from Vim’s native operator‑pending mode and a handful of helper functions. It is not a plugin; it is a pattern you can integrate directly into your .vimrc or ftplugin files, giving you deterministic, composable selections without external dependencies.

Architecture of Operator‑Pending Mode

To build new objects, you must first understand the state machine behind every d} or cib.

•Operator‑pending mode is entered after an operator key (d, c, y, g~, etc.). Vim then waits for a motion or a text object to define the range.

•Text objects are special mappings that define two boundaries: a start and an end. They can be character‑wise, line‑wise, or block‑wise.

•The omap and xmap commands allow you to define mappings that only apply in operator‑pending or visual mode, respectively. This is the key to making custom objects operator‑agnostic.

The engine we design will leverage omap mappings that call functions to calculate a region, and then set the '[ and '] marks appropriately so the pending operator acts on exactly the right span.

Core Engine: The Object Factory

Declaring a Text Object

Instead of hand‑writing a dozen nearly identical functions, we define a small specification language. Each object is described by a dictionary containing:

•name: a unique label (used for debugging and documentation).

•selector: a Vimscript expression that returns a pair of positions—start and end (inclusive). It receives a boolean inner to distinguish i from a variants.

•mode: 'char', 'line', or 'block'. This determines how the operator applies (e.g., d uses line‑wise if the object is line‑wise).

We then register this specification with the engine:

call textobj#register({
    \ 'name': 'indent',
    \ 'selector': 'textobj#indent#select(inner)',
    \ 'mode': 'line'
    \})

The textobj#indent#select function scans for lines with the same indentation, wrapping the inner or outer block. The engine handles the rest: creating the omap entries for both ii and ai, setting marks, and respecting the pending operator.

Engine Implementation

function! textobj#register(object) abort
    let s:objects[a:object.name] = a:object

    " inner variant: i + name (e.g., ii)
    execute 'onoremap <silent> i' . a:object.name
        \ . ' :<C-u>call textobj#apply('
        \ . string(a:object.name) . ', 1)<CR>'

    " a variant: a + name (e.g., ai)
    execute 'onoremap <silent> a' . a:object.name
        \ . ' :<C-u>call textobj#apply('
        \ . string(a:object.name) . ', 0)<CR>'
endfunction

function! textobj#apply(name, inner) abort
    let obj = s:objects[a:name]
    let [l:start, l:end] = call(obj.selector, [a:inner])
    if l:start == 0 || l:end == 0 | return | endif

    " Set the '[' and ']' marks; Vim uses these for the operator range
    call setpos("'[", [0, l:start, 1, 0])
    call setpos("']", [0, l:end, 1, 0])

    " Force the operator to use the correct mode
    if obj.mode ==# 'line'
        normal! V
    elseif obj.mode ==# 'char'
        normal! v
    elseif obj.mode ==# 'block'
        " blockwise needs visual block – handled with special marks
        normal! <C-v>
    endif
endfunction

This is the entire engine—under 30 lines of Vimscript. It is deterministic, transparent, and composable. Every custom object you create from this point forward is just a selector function and a registration call.

Building Real Objects

Indentation Block (ii / ai)

function! textobj#indent#select(inner) abort
    let indent = indent(line('.'))
    let start = line('.')
    let end = line('.')

    " search upward for a line with less indent (outer boundary)
    while start > 1 && indent(start - 1) >= indent
        let start -= 1
    endwhile

    " search downward for a line with less indent
    while end < line('$') && indent(end + 1) >= indent
        let end += 1
    endwhile

    if a:inner
        " inner: exclude the first and last line if they are blank
        if getline(start) =~# '^\s*$' | let start += 1 | endif
        if getline(end) =~# '^\s*$' | let end -= 1 | endif
    endif

    return [start, end]
endfunction

Now dii deletes the current indented block, yii yanks it, and >ii increases its indentation. The object definition stays clean because the engine separates the selection logic from the operator handling.

Function Arguments (ia / aa)

A more ambitious selector that parses the current line for a comma‑separated argument list. The inner version (ia) selects the argument text excluding optional surrounding whitespace; the outer version includes trailing commas and spaces.

function! textobj#arg#select(inner) abort
    let line = getline('.')
    let col = col('.') - 1  " 0-indexed

    " Find argument boundaries: commas or parentheses
    let pattern = '[,()]'
    let start = 0
    let end = len(line)

    " search backward for opening delimiter
    let idx = col
    while idx > 0
        if line[idx] =~# pattern
            let start = idx + 1
            break
        endif
        let idx -= 1
    endwhile

    " search forward for closing delimiter
    let idx = col
    while idx < len(line) - 1
        if line[idx] =~# pattern
            let end = idx
            break
        endif
        let idx += 1
    endwhile

    if a:inner
        " trim whitespace from both ends
        while start < end && line[start] =~# '\s'
            let start += 1
        endwhile
        while end > start && line[end - 1] =~# '\s'
            let end -= 1
        endwhile
    endif

    " Return line positions; the engine will convert to mark positions
    return [line('.'), line('.')]   " both on the same line – set marks manually
    " For same‑line object we need to adjust col in marks:
    " We'll do this inside a wrapper that returns [lnum, col] pairs.
    " (Full implementation omitted for brevity – see repo.)
endfunction

Because the engine supports only line‑wise marks by default, same‑line objects require a slight extension: the selector can return a list of [line, col] pairs, and textobj#apply sets setpos("'[", [0, l:line, l:col, 0]) accordingly. This is a small modification that makes the engine handle all mode types uniformly.

Markdown Table Cell (ic / ac)

Another selector that navigates the vertical bar | separators in a GitHub‑flavoured Markdown table. The inner object selects the cell content; the outer includes the pipe characters. Thanks to the engine, cic clears a cell and leaves you in insert mode—as if Vim knew about Markdown tables natively.

Composition with Operators: The Power of Grammar

Because every custom object is exposed as an operator‑pending mapping, they automatically compose with any built‑in operator, and even with custom operators from other plugins (e.g., commentary’s gc). You can chain them with counts, repeat them with . (if the operator is repeatable), and use them from visual mode if you add xmap equivalents. The engine provides a textobj#visual helper that does exactly that:

function! textobj#visual(name, inner) abort
    call textobj#apply(a:name, a:inner)
    " Already in visual mode from the apply()
    " Remap i/a within visual mode to keep extending
    execute 'xnoremap <buffer> i' . a:name . ' :<C-u>call textobj#apply('
        \ . string(a:name) . ', 1)<CR>'
    execute 'xnoremap <buffer> a' . a:name . ' :<C-u>call textobj#apply('
        \ . string(a:name) . ', 0)<CR>'
endfunction

Now vii selects an indented block, and pressing ii again extends the selection to the next outer block—a true modal grammar.

Performance and Edge Cases

Text objects must be fast because they are invoked hundreds of times during a typical editing session. Selector functions should avoid external processes and prefer built‑in functions like search(), indent(), and getline(). The registration overhead is negligible; omap definitions are stored once.

Some edge cases to handle:

•Empty buffer/line: Selectors should return [0,0] to abort the operation gracefully.

•Nested structures: Indentation selectors need a clear definition of “same indent or deeper”. The provided textobj#indent#select treats all lines with indent ≥ current as part of the block, which works well for Python but may need tuning for languages with free‑form indentation.

•Blockwise operations: Vim’s blockwise mode requires special care: after setting marks with setpos, you must start a blockwise visual selection (gv) so the operator can interpret the marks as a block. Our engine does this conditionally.

Integration with Your Existing Workflow

The engine is a file you can drop into ~/.vim/autoload/textobj.vim and source from your .vimrc. Individual selectors become small autoload files (textobj/indent.vim, textobj/arg.vim) that are loaded on demand. This keeps startup time minimal and allows you to version‑control them independently.

For users who already use plugins like vim-textobj-user by Kana Natsuno, this engine offers a compatible but simpler alternative. You can even wrap the popular plugin’s API into our registration function to avoid duplication.

Tradeoffs

•No built‑in visual feedback: Unlike some plugins, our engine doesn’t flash a highlight when an object is selected (you can add that with :redraw and matchadd). We prioritised minimalism.

•Selector functions must be designed carefully: They receive only an inner flag, not the full operator context. If you need to behave differently for c vs d, you can inspect v:operator within the selector, but this couples the two layers.

•Same‑line objects demand extra mark handling: The engine’s base version returns line numbers; the extended version that handles column positions adds complexity. The design documented here can be refined to accept either format seamlessly.

Future Directions

This engine can grow into a full domain‑specific language for structured editing:

• Add tree‑sitter integration: a selector that uses the syntax tree to define objects like functions, classes, or loops, making the engine filetype‑aware without per‑language rules.
• GUI selection manager: a picker (via fzf or Vim’s inputlist()) that lets you compose objects interactively.
• Visual mode combinators: extend the grammar so vii} means “select inner indent block, then extend to the next paragraph”, merging two objects.

The foundational pattern—separating registration, mapping, and selection logic—is what makes this extensible. Every new object is just a function that sees the buffer and returns two points.

Final Thoughts

Building a custom text object engine is not about reinventing plugins; it’s about understanding Vim’s modal architecture so deeply that you can bend it to your will. With a tiny amount of Vimscript, you can unlock editing paradigms that feel like they were always part of the editor. This engine is now a permanent part of my ~/.vim tree, and I suspect it will become one in yours as well.