Evaluation & Optimization with Langfuse¶

ThreatForest integrates with Langfuse for tracing, SME review, and dataset export. This guide walks through the full evaluation pipeline — from connecting Langfuse to exporting scored datasets for prompt optimization.

Prerequisites¶

• ThreatForest installed (pipx install . or pip install .)
• A Langfuse account with API keys (self-hosted or cloud)
• strands-agents[otel] installed (included in ThreatForest dependencies)
• Sample applications or your own project to evaluate

Step 1: Configure Langfuse¶

Interactive Setup (CLI)¶

threatforest config langfuse

The wizard prompts for your public key, secret key, and host URL. On successful connection, score definitions are automatically registered with Langfuse.

Direct Setup¶

threatforest config langfuse \
  --enable \
  --public-key pk-lf-xxxx \
  --secret-key sk-lf-xxxx \
  --host https://cloud.langfuse.com \
  --test

The --test flag verifies the connection and auto-registers score configs.

Console UI Setup¶

Navigate to the Configure page in the ThreatForest web console. The Langfuse Tracing section lets you:

• Toggle tracing on/off
• Enter your public key, secret key, and host
• Test the connection
• Save settings

Disable Tracing¶

threatforest config langfuse --disable

When disabled, ThreatForest runs normally without any tracing overhead.

Step 2: Score Definitions¶

Score definitions are automatically registered when you configure Langfuse (via CLI, wizard, or --test). You can also register them manually:

threatforest config langfuse --register-scores

ThreatForest defines 16 evaluation dimensions across four capabilities:

Threat Statement Generation (5 dimensions)¶

Attack Tree Generation (6 dimensions)¶

TTP Matching (1 dimension)¶

Mitigation Quality (4 dimensions)¶

Scoring Scale¶

All dimensions (except TTP) use a 5-point categorical scale:

TTP mapping uses a specialized scale replacing "Unacceptable" with "No Mapping".

To sync with existing configs already in Langfuse:

threatforest config langfuse --sync-scores

Step 3: Run ThreatForest with Tracing¶

With Langfuse enabled, every ThreatForest run produces two types of traces:

OTEL Traces (Automatic)¶

Full graph execution traces captured via OpenTelemetry, including:

• Every LLM call with input/output, token counts, and latency
• Tool invocations (file reads, structural analysis)
• Agent event loop cycles
• All traces share a session ID (e.g., tf-a4162895b65c) for grouping

These are the detailed engineering traces for debugging and performance analysis.

Annotation Traces (Automatic)¶

Clean input/output pairs per subgraph, pushed via the Langfuse SDK for SME review:

These traces are designed for annotation queues — filter by the annotation tag in Langfuse.

Using Sample Applications¶

# S3 Multi-Tenant SaaS (S3, Access Points, Object Lambda, KMS)
threatforest run --project-path sample-applications/s3

# Healthcare Analytics (HIPAA, AWS, S3, Lambda, DynamoDB)
threatforest run --project-path sample-applications/hcls-example

# IoT Device Management (MQTT, Edge Computing, OTA)
threatforest run --project-path sample-applications/iot-device-management

# Connected Vehicle Platform (V2X, Telematics, OBD-II)
threatforest run --project-path sample-applications/vehicle-platform

Using Your Own Project¶

threatforest run --project-path /path/to/your/project

After each run, traces appear in your Langfuse dashboard under the session ID.

Step 4: SME Review in Langfuse¶

Create Annotation Queues¶

In the Langfuse UI, create queues for each capability:

1. Go to your Langfuse project → Annotation Queues
2. Click + New Queue and create:

Add Traces to Queues¶

1. Go to Traces → filter by tag annotation
2. Select traces and click Add to Annotation Queue
3. Choose the matching queue based on the trace name

Score Traces¶

SMEs open the annotation queue and work through each item:

1. Review the input context and generated output
2. Assign scores using the categorical scale
3. Optionally add free-text feedback

Step 5: Export to Datasets¶

Export scored traces to Langfuse Datasets for evaluation analysis or prompt optimization:

# Reviewed threat statement traces
threatforest export traces \
  --trace-type threat_statement \
  --status reviewed \
  --dataset-name threat-statements-v1

# Reviewed attack tree traces
threatforest export traces \
  --trace-type attack_tree \
  --status reviewed \
  --dataset-name attack-trees-v1

# Reviewed TTP matching traces
threatforest export traces \
  --trace-type ttp_matching \
  --status reviewed \
  --dataset-name ttp-matching-v1

Export Options Reference¶

Full Baseline Workflow¶

# 1. Configure Langfuse (auto-registers score configs)
threatforest config langfuse --test

# 2. Run sample applications
for domain in s3 hcls-example iot-device-management vehicle-platform; do
  threatforest run --project-path sample-applications/$domain
done

# 3. (Manual) SME review in Langfuse annotation queues

# 4. Export reviewed traces to datasets
threatforest export traces --trace-type threat_statement --status reviewed -d baseline-threats-v1
threatforest export traces --trace-type attack_tree --status reviewed -d baseline-attack-trees-v1
threatforest export traces --trace-type ttp_matching --status reviewed -d baseline-ttp-v1

Resilient Tracing¶

ThreatForest's tracing is designed to never block your workflow:

• If Langfuse is unreachable, tracing silently falls back to no-op mode
• All Langfuse API errors are caught and logged without interrupting execution
• When Langfuse is disabled (--disable), there is zero tracing overhead
• OTEL spans are flushed before the process exits to ensure delivery