Parallel Go Engine · September 1, 2026
Parallel Goroutine Indexing: Scanning Large Monorepos Fast
Cold-indexing a large repository has two distinct costs: walking the directory tree to find files, and reading + AST-parsing each one. codive treats these as two separate stages, because they parallelize very differently.
Stage 1: The Walk Is Serial — On Purpose
File discovery uses Go's standard filepath.Walk, which visits the tree in a single goroutine. This is deliberate, not an oversight: directory traversal is I/O-bound on metadata syscalls (stat, readdir) that don't benefit much from parallel walkers on most filesystems, and a single walker makes the ignore-list logic (skipping .git, node_modules, target, build, dist, .venv, .gradle, .mvn, and similar directories) trivial to reason about — one process, one exclusion set, no coordination needed.
Stage 2: Parsing Is Where Parallelism Pays Off
Reading file content and running it through an AST extractor is CPU-bound and embarrassingly parallel — one file's parse doesn't depend on another's. That's where codive spins up a bounded worker pool sized to the machine:
numWorkers := runtime.NumCPU() * 2
if numWorkers < 4 { numWorkers = 4 }
if numWorkers > 32 { numWorkers = 32 }
fileChan := make(chan db.FileRecord, totalFiles)
resultChan := make(chan parseResult, totalFiles)
var wg sync.WaitGroup
for w := 0; w < numWorkers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for f := range fileChan {
content, err := os.ReadFile(fullPath(f))
// ... AST extraction happens here, per file ...
}
}()
}
close(fileChan)
go func() {
wg.Wait()
close(resultChan)
}()
The 2× CPU-core multiplier accounts for the fact that these goroutines spend part of their time blocked on file I/O rather than pure CPU work, so oversubscribing slightly keeps cores busy while some workers wait on disk reads. The clamp to [4, 32] exists for two reasons: a floor of 4 keeps small/underpowered machines from starving on a single worker, and a ceiling of 32 avoids goroutine and file-descriptor churn diminishing returns on very large machines where more workers just contend for the same disk.
Why the Split Matters
Because discovery and parsing are decoupled by a buffered channel, the walker can keep feeding fileChan while parser goroutines are still working through the backlog — there's no "walk everything, then parse everything" barrier. Results stream into resultChan as each file finishes, which is also what drives the live progress bar during codive init:
Indexing Codebase [████████████████████████████] 100% (171/171 files in 0.11s | 1,533 files/s)
codive update and the background watcher used by codive serve compare each file's on-disk ModTime against what's recorded in the files table and only re-parse what actually changed — so after the initial cold index, keeping the database current is a small delta operation, not a repeat full scan.