# Deployment Guide

> **⚠️ Research Use Only** — Models trained and deployed through this platform are for research and development purposes. Clinical deployment requires regulatory review, validation against clinical standards, and human oversight. AI predictions must not be used as the sole basis for medical decisions.

This guide covers deploying trained models to SageMaker endpoints for real-time or asynchronous inference.

## Prerequisites

- A completed training job with a `model.tar.gz` artifact in S3
- An inference container image pushed to ECR (classification or segmentation)
- A SageMaker execution role with endpoint creation permissions

## Deploy a Model

### Step 1 — Open the Deployment tab

Navigate to `http://localhost:8000` and click the **Deployment** tab.

### Step 2 — Fill out the deploy form

| Field | Description | Example |
|---|---|---|
| **Endpoint Name** | Unique name for the endpoint | `spine-classifier-v2` |
| **Endpoint Type** | Radio button: **Real-time** or **Async** | `Real-time` |
| **Model Artifacts** | S3 URI of `model.tar.gz` | `s3://bucket/output/job/output/model.tar.gz` |
| **Inference Image URI** | ECR URI of the inference container | from `.env.local` |
| **Role ARN** | SageMaker execution role | from `.env.local` |
| **Instance Type** | Endpoint instance type | `ml.g5.xlarge` |
| **Output S3 URI** | S3 prefix for inference results (required for async, optional for real-time) | `s3://bucket/async-output/` |

### Step 3 — Submit

Click **Deploy**. The dashboard creates three AWS resources in sequence:
1. SageMaker Model
2. Endpoint Configuration
3. Endpoint

Status transitions: `Creating` → `InService` (or `Failed`).

### Step 4 — Wait for InService

The deployment list auto-refreshes status from AWS. Once the endpoint shows `InService`, it is ready to receive inference requests.

## Real-Time vs Async Endpoints

The deploy form presents a **radio button** to choose between endpoint types:

### Real-time

- Synchronous request/response
- Best for classification (fast, <5s) and pre-warmed segmentation
- **API gateway timeout: 60 seconds** — requests exceeding this fail regardless of container timeout
- Minimum 1 instance (cannot scale to zero)

> **Cold start warning:** The first request to a real-time endpoint after deployment (or after a period of inactivity) may time out. This is because the model loads into GPU memory on the first invocation. Subsequent requests are fast (<30s). To avoid cold-start timeouts:
> - Send a warm-up request immediately after the endpoint becomes InService
> - Or use async endpoints for segmentation, which have no timeout limit
>
> Both real-time and async endpoints use the same container and inference code. The only difference is the timeout behavior.

**Recommendation:**
- **Classification** → Real-time (fast inference, <5s)
- **Segmentation** → Async (no timeout limit, supports scale-to-zero for cost savings)

### Async endpoint

Select the **Async** radio button to create an asynchronous endpoint:

| Field | Required | Description |
|---|---|---|
| **Output S3 URI** | yes | S3 prefix where results are written |
| `max_concurrent` | no | Max concurrent invocations per instance (default `10`) |

Async endpoints are useful for:
- 3D segmentation volumes (inference takes 15-30s, first request may take longer)
- Batch inference workloads
- Cost optimization (scale to zero when idle — no billing when no requests)

## Endpoint Scaling

Each deployed endpoint card displays scaling controls:

| Button | Action | Availability |
|---|---|---|
| **Scale up (+)** | Adds one instance | All endpoints |
| **Scale down (-)** | Removes one instance (minimum 1 for real-time) | All endpoints |
| **Scale to 0** | Removes all instances (endpoint remains configured) | Async only |

Real-time endpoints require at least 1 instance and cannot be scaled to zero. Async endpoints support scaling to zero — they automatically wake up when new requests arrive.

## Active Endpoint Badge

The deployment tab header displays a count badge showing currently active endpoints. Hovering or clicking reveals endpoint names and their instance counts, providing a quick overview of deployed resources.

## Enable Data Capture at Deploy Time

Check **Enable Data Capture** before deploying to record all inference traffic from the first request:

| Field | Description |
|---|---|
| **Enable Data Capture** | Check to activate |
| **Capture S3 URI** | S3 prefix for captured traffic (e.g. `s3://bucket/capture/`) |
| **Capture Sampling %** | Percentage of requests to capture (1-100, default `100`) |

This is equivalent to calling the monitoring data-capture API separately, but happens atomically during deployment.

## Import an Existing Endpoint

To track an endpoint that was created outside the dashboard (e.g. via the AWS console or CLI):

1. Click **Import Endpoint** (or call `POST /api/deployment` with only the endpoint name)
2. Omit `model_artifacts` — the dashboard registers the endpoint without creating a new artifact
3. The endpoint appears in the deployment list and its status is polled from AWS

## Sync from AWS

Click **Sync from AWS** to pull existing endpoints from your AWS account and merge them into the local store:

- Optional `?name_contains=` filter to narrow results
- Already-tracked endpoints have their status updated
- New endpoints are added with minimal records

## Delete an Endpoint

### From the dashboard

Click the **Delete** button on any endpoint in the list. This:
1. Deletes the SageMaker endpoint
2. Deletes the endpoint configuration
3. Deletes the SageMaker model resource
4. Removes the local record from `data/deployments.json`

The local record is always cleaned up via `try/finally`, even if an AWS API call fails.

### External deletion handling

If an endpoint is deleted outside the dashboard (AWS console, CLI, etc.):
- The next status poll detects the missing endpoint
- The local record is marked as `"Deleted"`
- `Deleted` entries are filtered out of the list response

## Model Archive Requirements

SageMaker's PyTorch serving container expects inference code at `code/inference.py` inside `model.tar.gz`:

```
model.tar.gz
├── best_model.pth
├── result.json
└── code/
    └── inference.py
```

Training scripts automatically copy inference artifacts to `model_dir/code/` during training. If you have an older model archive with `inference.py` at the root, use the repackaging utility:

```bash
python tests/integration/repackage_model.py s3://your-bucket/output/job-name/output/model.tar.gz
```

### Security: Model Loading

All model loading in training and evaluation code uses `torch.load(path, weights_only=True)` to prevent arbitrary code execution from untrusted model files. The inference script (`code/inference.py`) uses `torch.load(path, map_location=device)` without `weights_only=True` because it loads models produced by the project's own training pipeline — this trust boundary is documented in the code.

## Troubleshooting

| Issue | Cause | Fix |
|---|---|---|
| Endpoint stuck in `Creating` | Container startup failure | Check CloudWatch logs for the endpoint |
| `Failed` status | Missing inference script or bad model | Verify `model.tar.gz` contains `code/inference.py` and `best_model.pth` (not `inference.py` at root) |
| Permission denied | Role ARN lacks permissions | Ensure the role has `sagemaker:CreateEndpoint`, `ecr:GetAuthorizationToken`, `s3:GetObject` |
| Timeout on invocation | Model too large or slow | Increase instance type or use async endpoint |