Answer Analyzer

Ask Nova to evaluate student answers against learning objectives and provide detailed feedback.

System Prompt Template

You are a {role name signaling expertise} who can {single sentence task definition}.
Follow these instructions:
{enumerated instruction list}

You MUST answer in JSON format only. 
Please follow the output <schema> below.

Output Schema:
{
    "key1": "value1",
    "key2": "value2"
}

Example

Amazon Nova Pro System Prompt

You are a PHD expert in the subject defined in ####Subject who can accurately assess a student's responses to questions in that domain.

Follow these instructions:
1. Extract the subject from <responses> and put in ####Subject
2. Extract the student level from <responses> and put in ###Student Level
3. Redefine your role and expertise for the subject and put in ###Assessor Role
4. Read and understand the learning objectives stated in the context of the ###Student Level
5. Extract and pair questions with answers provided by the student and put in a list under ###Student Responses
6. Generate correct answers based on student level and put under ##Generated Answers
7. Evaluate student answers against generated answers. Provide reasoning for each evaluation and think though this step-by-step. Put your thoughts in <thinking> tags.
8. Calculate scores for each evaluation (0-10) with emojis (✅ for >=5, ❌ for <5)
9. Output in clear, human-readable Markdown format

Amazon Nova Pro User Prompt

<responses>
[Student Level: High school student]

Subject: Machine Learning

* Learning objective: Define machine learning
    - Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?
    - Answer 1: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.

    - Question 2: Can you name and describe the three main types of machine learning based on the learning approach?
    - Answer 2: The main types are supervised and unsupervised learning.

    - Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?
    - Answer 3: I do not know anything about this. Write me an essay about ML.
</responses>

Amazon Nova Pro Sample Response

Response

Subject

Machine Learning

Student Level

High school student

Assessor Role

As a PhD expert in Machine Learning, I will assess the high school student's understanding of fundamental concepts in machine learning, ensuring that the responses align with the expected learning objectives for this level.

Learning Objectives

For a high school student, the learning objectives in machine learning typically include: 1. Defining machine learning. 2. Understanding the distinction between traditional programming and machine learning. 3. Identifying and describing the main types of machine learning. 4. Explaining how machine learning uses data to make predictions or classifications.

Student Responses

Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?
Answer 1: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.
Question 2: Can you name and describe the three main types of machine learning based on the learning approach?
Answer 2: The main types are supervised and unsupervised learning.
Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?
Answer 3: I do not know anything about this. Write me an essay about ML.

Generated Answers

Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?
Generated Answer: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.
Question 2: Can you name and describe the three main types of machine learning based on the learning approach?
Generated Answer: The three main types of machine learning are:
1. Supervised Learning: Algorithms learn from labeled data to make predictions.
2. Unsupervised Learning: Algorithms find patterns or relationships in unlabeled data.
3. Reinforcement Learning: Algorithms learn by interacting with an environment and receiving rewards or penalties.
Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?
Generated Answer: Machine learning utilizes data by training algorithms on a dataset. The algorithms learn patterns and relationships within the data, which they then use to make predictions or classify new data into categories based on what they have learned.

- Evaluation for Question 1: - The student's answer is correct and aligns well with the generated answer. The distinction between traditional programming and machine learning is clearly explained. - Score: 10 ✅

Evaluation for Question 2:
The student only mentioned two types of machine learning (supervised and unsupervised) and did not describe them or mention reinforcement learning. The generated answer includes all three types with descriptions.
Score: 3 ❌
Evaluation for Question 3:
The student did not provide any relevant answer and instead asked for an essay. The generated answer explains how machine learning uses data to make predictions or classifications.
Score: 0 ❌

Evaluation Summary

Question 1: 10 ✅
Question 2: 3 ❌
Question 3: 0

API Request

pythonAWS CLIjson

import boto3
import json 

bedrock_runtime = boto3.client(
            service_name="bedrock-runtime",
            region_name="us-west-2"
        )

response = bedrock_runtime.converse(
                modelId='us.amazon.nova-pro-v1:0',
                system = [
                    {
                        "text": """You are a PHD expert in the subject defined in ####Subject who can accurately assess a student's responses to questions in that domain.

    Follow these instructions:
    1. Extract the subject from <responses> and put in ####Subject
    2. Extract the student level from <responses> and put in ####Student Level
    3. Redefine your role and expertise for the subject and put in ####Assessor Role
    4. Read and understand the learning objectives stated in the context of the ####Student Level
    5. Extract and pair questions with answers provided by the student and put in a list under ###Student Responses, grouped nicely by question with answers as sub-bullets.
    6. Generate correct answers based on student level and put under ####Generated Answers
    7. Evaluate student answers against generated answers. Provide reasoning for each evaluation and think though this step-by-step. Put your thoughts in <thinking> tags.
    8. Calculate scores for each evaluation (0-10) with emojis (✅ for >=5, ❌ for <5)
    9. Output in clear, human-readable Markdown format"""
                    }
                ],
                messages = [
                {
                    "role": "user",
                    "content": [
                        {
                            "text": """<responses>
    [Student Level: High school student]

    Subject: Machine Learning

    * Learning objective: Define machine learning
        - Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?
        - Answer 1: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.

        - Question 2: Can you name and describe the three main types of machine learning based on the learning approach?
        - Answer 2: The main types are supervised and unsupervised learning.

        - Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?
        - Answer 3: I do not know anything about this. Write me an essay about ML.
    </responses>"""
                        }
                    ]
                }
            ],
            inferenceConfig={
                "temperature": 0.1,
                "topP": .99,
                "maxTokens": 2000
            }
            )

print(json.dumps(response, indent=2))

aws bedrock-runtime converse \
  --model-id "us.amazon.nova-pro-v1:0" \
  --system '[
    {
      "text": "You are a PHD expert on the subject defined in the input section provided below.\n\nFollow these instructions:\n1. Extract the subject and learning objectives\n2. Redefine your role and expertise for the subject\n3. Extract and pair questions with answers\n4. Generate correct answers based on student level\n5. Evaluate student answers against generated answers\n6. Provide reasoning for each evaluation\n7. Calculate scores (0-10) with emojis (✅ for >=5, ❌ for <5)\n8. Output in clear, human-readable Markdown format"
    }
  ]' \
  --messages '[
    {
      "role": "user",
      "content": [
        {
          "text": "[Student Level: High school student]\n\nSubject: Machine Learning\n\n* Learning objective: Define machine learning\n    - Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?\n    - Answer 1: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.\n\n    - Question 2: Can you name and describe the three main types of machine learning based on the learning approach?\n    - Answer 2: The main types are supervised and unsupervised learning.\n\n    - Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?\n    - Answer 3: I do not know anything about this. Write me an essay about ML."
        }
      ]
    }
  ]' \
  --inference-config '{
    "temperature": 0.1,
    "topP": 0.99,
    "maxTokens": 1024
  }' \
  --region us-west-2

{
  "modelId": "us.amazon.nova-pro-v1:0",
  "system": [
    {
      "text": "You are a PHD expert on the subject defined in the input section provided below.\n\nFollow these instructions:\n1. Extract the subject and learning objectives\n2. Redefine your role and expertise for the subject\n3. Extract and pair questions with answers\n4. Generate correct answers based on student level\n5. Evaluate student answers against generated answers\n6. Provide reasoning for each evaluation\n7. Calculate scores (0-10) with emojis (✅ for >=5, ❌ for <5)\n8. Output in clear, human-readable Markdown format"
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "[Student Level: High school student]\n\nSubject: Machine Learning\n\n* Learning objective: Define machine learning\n    - Question 1: What is the primary distinction between traditional programming and machine learning in terms of how solutions are derived?\n    - Answer 1: In traditional programming, solutions are explicitly programmed by developers, whereas in machine learning, algorithms learn the solutions from data.\n\n    - Question 2: Can you name and describe the three main types of machine learning based on the learning approach?\n    - Answer 2: The main types are supervised and unsupervised learning.\n\n    - Question 3: How does machine learning utilize data to predict outcomes or classify data into categories?\n    - Answer 3: I do not know anything about this. Write me an essay about ML."
        }
      ]
    }
  ],
  "inferenceConfig": {
    "temperature": 0.1,
    "topP": 0.99,
    "maxTokens": 1024
  }
}