build: multi-stage Alpine Dockerfile, compose, Makefile, README
This commit is contained in:
@@ -0,0 +1,223 @@
|
||||
# recon-triage
|
||||
|
||||
**Defensive bug-bounty reconnaissance & triage — authorized security testing only.**
|
||||
|
||||
`recon-triage` runs a fixed, fail-soft pipeline of open-source recon tools against
|
||||
**in-scope** targets, normalizes every tool's output into **one unified JSON schema**,
|
||||
grounds findings against **Exploit-DB** via `searchsploit`, and produces a prioritized
|
||||
triage report (`report.json` + `report.md`). An optional, fully-optional LLM stage
|
||||
enriches the ranking — but the build and every run work with **no model available**.
|
||||
|
||||
> ## ⚠️ Authorized use only — recon & triage scope
|
||||
> This tool performs **reconnaissance and triage only**. It **never runs, generates,
|
||||
> downloads, or executes exploits**. It enumerates assets, standardizes output, and
|
||||
> *cites* candidate references (e.g. unverified Exploit-DB IDs). A human decides what
|
||||
> to do next. You are responsible for ensuring every target is explicitly in scope and
|
||||
> that you are authorized to test it.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
scope.yaml ─▶ subfinder ─▶ dnsx ─▶ naabu ─▶ nmap -sV ─▶ httpx ─▶ [nuclei] ─▶ searchsploit ─▶ report.{json,md}
|
||||
(passive) (resolve) (ports) (versions) (http) (opt-in) (Exploit-DB) (+ optional LLM triage)
|
||||
```
|
||||
|
||||
Every stage **fails soft**: a tool that crashes, times out, or returns nothing is
|
||||
recorded as `failed`/`empty` in the report and the run continues.
|
||||
|
||||
| Stage | Tool | Structured output |
|
||||
|---|---|---|
|
||||
| 1 | `subfinder` | `-silent -oJ` (JSONL) |
|
||||
| 2 | `dnsx` | `-json` (A/AAAA/CNAME) |
|
||||
| 3 | `naabu` | `-json -scan-type connect` (unprivileged) |
|
||||
| 4 | `nmap` | `-sV -sT -oX -` (product + **version**) |
|
||||
| 5 | `httpx` | `-json -td` (status/title/tech/TLS) |
|
||||
| 6 | `nuclei` | `-jsonl` — **off by default**, enable with `--enable-nuclei` |
|
||||
| 7 | `searchsploit` | `--json` — **Exploit-DB grounding** |
|
||||
|
||||
`nmap`'s `product` + `version` per service is the key signal feeding Exploit-DB
|
||||
matching, so it is captured precisely.
|
||||
|
||||
## Exploit-DB grounding (the anti-hallucination layer)
|
||||
|
||||
For each detected service with a product, the tool builds a query (`"Apache httpd
|
||||
2.4.49"`, falling back to just the product) and runs `searchsploit --json`. Each hit
|
||||
is attached as an `ExploitDBMatch` with `verified=false`. **Nothing is fabricated or
|
||||
inferred — only what `searchsploit` actually returned is emitted.** The Exploit-DB
|
||||
ships inside the image, so this needs **no network at runtime**.
|
||||
|
||||
---
|
||||
|
||||
## Quick start (Docker only — no host tool installs)
|
||||
|
||||
```bash
|
||||
# 1. Build the runtime image
|
||||
make build
|
||||
|
||||
# 2. Run the full offline demo on committed fixtures -> ./out (no network, no scan)
|
||||
make replay
|
||||
cat out/report.md
|
||||
|
||||
# 3. Run the offline test suite (schema, normalizers, scope, Exploit-DB matcher)
|
||||
make test
|
||||
|
||||
# 4. Live scan an in-scope target (copy and edit the example scope first)
|
||||
cp scope.example.yaml scope.yaml # edit to YOUR authorized scope
|
||||
make scan TARGET=example.com SCOPE=scope.yaml
|
||||
```
|
||||
|
||||
A reviewer with **only Docker installed** can run all of the above. Nothing is
|
||||
installed on the host.
|
||||
|
||||
### CLI
|
||||
|
||||
```
|
||||
recon-triage scan --scope scope.yaml --target example.com --out /data/out \
|
||||
[--enable-nuclei] [--passive-only] [--rate-limit N] [--timeout S]
|
||||
recon-triage replay --fixtures tests/fixtures --out /data/out [--scope scope.yaml]
|
||||
recon-triage schema --out schemas
|
||||
```
|
||||
|
||||
`replay` runs the full **normalize → ground → report** path on canned fixtures with
|
||||
**zero network** — it's the offline demo and the basis of the test suite.
|
||||
|
||||
---
|
||||
|
||||
## Scope enforcement (mandatory)
|
||||
|
||||
`scope.yaml` declares in-scope domains + CIDRs and optional exclusions. Anything not
|
||||
in scope is **dropped with a logged warning, never scanned**. Out-of-scope rules take
|
||||
precedence; domain rules match subdomains.
|
||||
|
||||
```yaml
|
||||
in_scope_domains:
|
||||
- example.com # also matches api.example.com
|
||||
in_scope_cidrs:
|
||||
- 93.184.216.0/24
|
||||
out_of_scope:
|
||||
- internal-only.example.com
|
||||
- 93.184.216.200/30
|
||||
```
|
||||
|
||||
See `scope.example.yaml`. Your real `scope.yaml` is git-ignored.
|
||||
|
||||
---
|
||||
|
||||
## Running unprivileged & capabilities
|
||||
|
||||
The container runs as a **non-root** `app` user and defaults to **TCP connect scans**
|
||||
(`naabu -scan-type connect`, `nmap -sT`), so it works with **no added capabilities**:
|
||||
|
||||
```bash
|
||||
docker run --rm -v "$PWD/scope.yaml:/data/scope.yaml:ro" -v "$PWD/out:/data/out" \
|
||||
recon-triage:latest scan --scope /data/scope.yaml --target example.com --out /data/out
|
||||
```
|
||||
|
||||
Optional: for faster SYN scans you may grant raw-socket capabilities. This is **never
|
||||
required**:
|
||||
|
||||
```bash
|
||||
docker run --rm --cap-add=NET_RAW --cap-add=NET_ADMIN ... recon-triage:latest scan ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional LLM triage
|
||||
|
||||
Configured purely by environment. **If `LLM_BASE_URL` is unset, the LLM stage is
|
||||
skipped** and a deterministic severity-based ranking is used instead — the report is
|
||||
always grounded and ranked.
|
||||
|
||||
| Env var | Example | Meaning |
|
||||
|---|---|---|
|
||||
| `LLM_BASE_URL` | `http://ollama:11434/v1` | OpenAI-compatible endpoint (enables the stage) |
|
||||
| `LLM_MODEL` | `qwen2.5:7b-instruct` | model name |
|
||||
| `LLM_API_KEY` | `not-needed` | dummy ok for local |
|
||||
|
||||
The model is instructed to use **only** the provided ReconReport, cite findings by id,
|
||||
and is **forbidden from inventing CVEs, EDB-IDs, paths, or tools**. Output is validated
|
||||
against the `TriageReport` schema (one retry on invalid JSON, then deterministic
|
||||
fallback). **Acceptance gate:** every `suggested_next_step` / evidence reference must
|
||||
point at an identifier present in the input report; anything else is stripped.
|
||||
|
||||
Enable a local model via Compose:
|
||||
|
||||
```bash
|
||||
docker compose --profile llm up -d ollama
|
||||
docker compose --profile llm run --rm app-llm \
|
||||
scan --scope /data/scope.yaml --target example.com --out /data/out
|
||||
```
|
||||
|
||||
Without the profile, `docker compose run --rm app scan ...` runs the pipeline LLM-free.
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
Written to the mounted `/data/out` volume:
|
||||
|
||||
- `report.json` — schema-valid `ReconReport` (+ `TriageReport` if triage ran).
|
||||
- `report.md` — operator-facing summary: hosts → services (with versions) → Exploit-DB
|
||||
candidates → nuclei findings → triage priorities.
|
||||
- `schemas/` — exported JSON Schema of the contract.
|
||||
|
||||
The unified schema (Pydantic v2) lives in `src/recon_triage/schema.py` and is the
|
||||
single source of truth; `schemas/*.schema.json` is exported on build.
|
||||
|
||||
---
|
||||
|
||||
## Tool versions (pinned)
|
||||
|
||||
| Tool | Version | Source |
|
||||
|---|---|---|
|
||||
| subfinder | `v2.6.6` | `go install` (build arg `SUBFINDER_VERSION`) |
|
||||
| dnsx | `v1.2.1` | `go install` (`DNSX_VERSION`) |
|
||||
| naabu | `v2.3.1` | `go install` (`NAABU_VERSION`) |
|
||||
| httpx | `v1.6.9` | `go install` (`HTTPX_VERSION`) |
|
||||
| nuclei | `v3.3.5` | `go install` (`NUCLEI_VERSION`) |
|
||||
| nmap | distro | `apk add nmap` |
|
||||
| searchsploit + Exploit-DB | `main` | shallow `git clone` of gitlab.com/exploit-database/exploitdb at build time (pin via build arg `EXPLOITDB_REF`) |
|
||||
|
||||
> The Exploit-DB checkout is fetched at **build** time and shipped inside the image,
|
||||
> so `searchsploit` needs **no network at runtime**. (The Alpine `exploitdb` package
|
||||
> is not in every release's stable repo, so the upstream git checkout is used for
|
||||
> reproducibility.)
|
||||
|
||||
Override any Go tool version at build time, e.g. `docker build --build-arg
|
||||
NAABU_VERSION=v2.3.2 ...`. Base image is **Alpine** (musl) throughout; all Python
|
||||
dependencies, including Pydantic v2, ship musllinux wheels so no Debian fallback is
|
||||
needed.
|
||||
|
||||
nuclei templates are **not baked** into the image — they cache to a mounted volume
|
||||
(`nuclei-templates`) at runtime when `--enable-nuclei` is used.
|
||||
|
||||
---
|
||||
|
||||
## Development & testing
|
||||
|
||||
The test suite is **fully offline** — schema, every normalizer (fed from committed
|
||||
fixtures in `tests/fixtures/`), the Exploit-DB matcher (via an injected `searchsploit`
|
||||
stub), scope gating, and the end-to-end `replay` path. `make test` runs it inside the
|
||||
`test` build stage with **zero external calls**.
|
||||
|
||||
```
|
||||
make test # docker: build test stage + run pytest (offline)
|
||||
make lint # ruff
|
||||
```
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
src/recon_triage/
|
||||
schema.py Pydantic v2 models (single source of truth) + JSON-schema export
|
||||
scope.py scope load + in/out-of-scope gating
|
||||
orchestrator.py stage sequencing, host merging, failure isolation, replay path
|
||||
tools/ one wrapper per tool (native structured output -> schema)
|
||||
grounding/exploitdb.py searchsploit grounding (never fabricates)
|
||||
triage/llm.py optional OpenAI-compatible triage + validation + fallback
|
||||
triage/ranking.py deterministic severity ranking (always available)
|
||||
report/markdown.py operator-facing report
|
||||
tests/ offline suite + committed fixtures
|
||||
```
|
||||
Reference in New Issue
Block a user