← Blog

Parallel Go Engine  ·  September 1, 2026

Parallel Goroutine Indexing: Scanning Large Monorepos Fast

Cold-indexing a large repository has two distinct costs: walking the directory tree to find files, and reading + AST-parsing each one. codive treats these as two separate stages, because they parallelize very differently.

Stage 1: The Walk Is Serial — On Purpose

File discovery uses Go's standard filepath.Walk, which visits the tree in a single goroutine. This is deliberate, not an oversight: directory traversal is I/O-bound on metadata syscalls (stat, readdir) that don't benefit much from parallel walkers on most filesystems, and a single walker makes the ignore-list logic (skipping .git, node_modules, target, build, dist, .venv, .gradle, .mvn, and similar directories) trivial to reason about — one process, one exclusion set, no coordination needed.

Stage 2: Parsing Is Where Parallelism Pays Off

Reading file content and running it through an AST extractor is CPU-bound and embarrassingly parallel — one file's parse doesn't depend on another's. That's where codive spins up a bounded worker pool sized to the machine:

numWorkers := runtime.NumCPU() * 2
if numWorkers < 4 { numWorkers = 4 }
if numWorkers > 32 { numWorkers = 32 }

fileChan := make(chan db.FileRecord, totalFiles)
resultChan := make(chan parseResult, totalFiles)

var wg sync.WaitGroup
for w := 0; w < numWorkers; w++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for f := range fileChan {
            content, err := os.ReadFile(fullPath(f))
            // ... AST extraction happens here, per file ...
        }
    }()
}
close(fileChan)

go func() {
    wg.Wait()
    close(resultChan)
}()

The 2× CPU-core multiplier accounts for the fact that these goroutines spend part of their time blocked on file I/O rather than pure CPU work, so oversubscribing slightly keeps cores busy while some workers wait on disk reads. The clamp to [4, 32] exists for two reasons: a floor of 4 keeps small/underpowered machines from starving on a single worker, and a ceiling of 32 avoids goroutine and file-descriptor churn diminishing returns on very large machines where more workers just contend for the same disk.

Why the Split Matters

Because discovery and parsing are decoupled by a buffered channel, the walker can keep feeding fileChan while parser goroutines are still working through the backlog — there's no "walk everything, then parse everything" barrier. Results stream into resultChan as each file finishes, which is also what drives the live progress bar during codive init:

Indexing Codebase [████████████████████████████] 100% (171/171 files in 0.11s | 1,533 files/s)
Incremental re-indexing skips this entirely. codive update and the background watcher used by codive serve compare each file's on-disk ModTime against what's recorded in the files table and only re-parse what actually changed — so after the initial cold index, keeping the database current is a small delta operation, not a repeat full scan.