Summary

Implement a persistent metadata cache to dramatically improve template loading performance when using filters (severity, tags, authors, protocols, etc.). By caching lightweight template metadata and filtering before parsing, subsequent runs can skip expensive YAML parsing for ~95% of templates.

Problem statement

Currently, nuclei parses all template YAML files on every run, even when filters would exclude most of them. For a typical scan with severity filtering. Ex:

# kill-when-loaded.sh
#!/usr/bin/env bash
set -euo pipefail

program="$@"
pattern="Templates loaded for current scan" # kill trigger
logfile="$(mktemp /tmp/nuclei-log.XXXXXX)"
start_ns=$(date +%s%N)

$program >"$logfile" 2>&1 &
pid=$!

tail -n +1 -F "$logfile" 2>/dev/null | while IFS= read -r line; do
    echo "$line"
    if [[ "$line" == *"$pattern"* ]]; then
        kill "$pid" 2>/dev/null || true
        if kill -0 "$pid" 2>/dev/null; then
            kill -9 "$pid" 2>/dev/null || true
        fi

        end_ns=$(date +%s%N)
        elapsed_ns=$((end_ns - start_ns))
        elapsed_s=$(printf "%d.%03d" $((elapsed_ns/1000000000)) $(((elapsed_ns/1000000)%1000)))

        echo "Elapsed: ${elapsed_s}s"

        break
    fi
done

wait "$pid" 2>/dev/null || true

$ bash kill-when-loaded.sh ./bin/nuclei -duc -u scanme.sh -s critical,high

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.5.1

		projectdiscovery.io

[INF] Current nuclei version: v3.5.1 (unknown) - remove '-duc' flag to enable update checks
[INF] Current nuclei-templates version: v10.3.2 (unknown) - remove '-duc' flag to enable update checks
[INF] New templates added in latest release: 130
[INF] Templates loaded for current scan: 3590
Elapsed: 7.625s

This currently requires:

1. Parsing ~11K+ template YAML files.
2. Extracting metadata (tags, severity, authors, etc.).
3. Applying filters.
4. Discarding ~90% of parsed templates that don't match filters.

Performance impact:

• Cold start: ~5-10 seconds just for template loading.
• Repeated scans: Same overhead every single run.
• CI/CD pipelines: Wasted time on every execution.