← all articles

OSSOpen-Source Contribution

Contribution to ProjectDiscovery's httpx

On interrupt, httpx saved a resume index based on dispatched targets, not completed ones, silently skipping 51 of 108 hosts on resume. PR #2393 makes shutdown drain in-flight work first.

projectdiscovery/httpx · PR #2393

TL;DR

httpx resumes interrupted scans from a small resume.cfg. The bug: it recorded how many targets were dispatched to the worker pool, not how many had finished. Press Ctrl+C and it wrote an index far ahead of real progress, then called os.Exit(1) while requests were still in flight. On -resume, every dispatched-but-unfinished target was skipped.

On a 108-host list that was 51 targets silently lost. After PR #2393: 0 lost.

Why the index lied

currentIndex advanced at dispatch time, but with 50 worker threads the dispatcher runs far ahead of the network. The SIGINT handler then snapshotted that inflated index and exited immediately, abandoning every in-flight goroutine.

The fix: two-stage graceful shutdown

Stop accepting new work, drain what is already running, then checkpoint honestly.

// runner.go
func (r *Runner) Interrupt()          { close(r.interruptCh) }
func (r *Runner) IsInterrupted() bool {
    select { case <-r.interruptCh: return true; default: return false }
}

for target := range inputTargets {
    if r.IsInterrupted() { break }  // stop dispatching; let outstanding work drain
    swg.Add(); go process(target)
}
swg.Wait()                          // only now is the index true

The first Ctrl+C requests a drain, a second forces exit. Crucially, SaveResumeConfig() moved to after RunEnumeration() returns, so the saved index reflects what actually completed. A regression test, TestRunner_resumeAfterInterrupt, locks the contract. Reviewed and merged by the maintainers (Mzack9999: "lgtm!").