Evaluation Guide

This guide covers running SageMaker Processing Jobs to evaluate deployed classification endpoints against labeled test sets.

Prerequisites

A deployed classification endpoint in InService status
A labeled test dataset in S3 organized by class folders
A SageMaker execution role with processing job permissions

Test Data Format

Organize your test data in S3 with one subfolder per class:

s3://your-bucket/test-data/
├── class_a/
│   ├── image001.dcm
│   ├── image002.dcm
│   └── image003.dcm
├── class_b/
│   ├── image004.dcm
│   └── image005.dcm
└── class_c/
    ├── image006.dcm
    └── image007.dcm

Each subfolder name is used as the ground-truth label.

Submit an Evaluation Job

Step 1 — Open the Evaluation tab

Navigate to http://localhost:8000 and click the Evaluation tab.

Step 2 — Fill out the evaluation form

Field	Required	Description	Example
Endpoint Name	Yes	The deployed endpoint to evaluate	`spine-classifier-v2`
Test Data S3 URI	Yes	S3 path to the labeled test dataset	`s3://bucket/test-data/`
Output S3 URI	Yes	S3 prefix where results are written	`s3://bucket/eval-output/`
Processing Image URI	Yes	ECR URI of the processing container	Same as inference image
Role ARN	Yes	SageMaker execution role	from `.env.local`
Instance Type	No	Processing instance type	`ml.m5.xlarge`

Step 3 — Submit

Click Run Evaluation. A SageMaker Processing Job is launched that:

Discovers class folders under the test data URI
Invokes the endpoint for every file
Collects predictions and probability scores
Computes metrics and generates output artifacts

Step 4 — Wait for completion

The job typically takes a few minutes depending on the dataset size. Status is polled from AWS.

Output Artifacts

The evaluation job writes four files to the output S3 URI:

File	Description
`metrics.json`	Accuracy, macro AUC, per-class precision/recall/F1, and support counts
`predictions.csv`	Per-file ground truth vs predicted class with per-class probabilities
`confusion_matrix.png`	Confusion matrix heatmap visualization
`roc_auc.png`	ROC-AUC curves (one per class, one-vs-rest)

`metrics.json` structure

{
  "accuracy": 0.8750,
  "macro_auc": 0.9412,
  "per_class": {
    "class_a": {
      "precision": 0.90,
      "recall": 0.85,
      "f1": 0.87,
      "support": 20
    },
    "class_b": {
      "precision": 0.82,
      "recall": 0.88,
      "f1": 0.85,
      "support": 25
    }
  }
}

`predictions.csv` structure

file,true_class,predicted_class,prob_class_a,prob_class_b,prob_class_c
image001.dcm,class_a,class_a,0.89,0.07,0.04
image004.dcm,class_b,class_c,0.12,0.35,0.53

Viewing Results

Once the evaluation job completes:

The results panel shows overall accuracy and macro AUC
Click on a job to view:
- Confusion matrix — visual heatmap of predictions vs ground truth
- ROC-AUC curves — per-class curves with AUC values
- Per-class metrics table — precision, recall, F1, support for each class
Download individual artifacts for further analysis

Endpoint Response Formats Handled

The evaluation script supports two endpoint response formats transparently:

Format	Structure
Batched arrays	`{"predictions": [[p0, p1, ...]], "predicted_class": [idx], "confidence": [c]}`
Named probabilities	`{"probabilities": {"class_a": 0.3, ...}, "predicted_class": "class_a"}`

Sync from AWS

Click Sync from AWS (or POST /api/evaluation/sync) to pull evaluation processing jobs from AWS. Use ?name_contains= to filter.

Tips

Use a representative test set — include edge cases and all class labels
Check class balance — imbalanced test sets can produce misleading accuracy; look at per-class metrics
Compare across runs — use the Report Generation feature to compile results from multiple jobs
Subgroup analysis — combine with Fairness Analysis to check if performance varies across demographic groups