Create an EC2 instance suitable for an LMI (Large Model Inference)

Follow the steps below to create an EC2 instance for hosting a model in an LMI.

1. On the homepage of AWS Console go to ‘EC2’ - it is likely in recently visited:

2. If not found, go to the search bar on the top of the page. Type ec2 into the search box and click the entry that pops up with name EC2 :

3. Click “Instances”:

4. Click "Launch Instances":

5. Type in a name for your instance (recommended to include your alias in the name), and then scroll down. Search for ‘deep learning ami’ in the box. Select the one that says “Deep Learning OSS Nvidia Driver AMI GPU PyTorch”. Your version number might be different.

6. Name your instance FMBenchInstance.

7. Add a fmbench-version tag to your instance.

8. Scroll down to Instance Type. For large model inference, the g5.12xlarge is recommended.

1. Make a key pair by clicking Create new key pair. Give it a name, keep all settings as is, and then click “Create key pair”.

2. Skip over Network settings (leave it as it is), going straight to Configure storage. 45 GB, the suggested amount, is not nearly enough, and using that will cause the LMI docker container to download for an arbitrarily long time and then error out. Change it to 100 GB or more:

3. Create an IAM role to your instance called FMBenchEC2Role. Attach the following permission policies: AmazonSageMakerFullAccess, AmazonBedrockFullAccess.

   Edit the trust policy to be the following:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "ec2.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           },
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "sagemaker.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           },
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "bedrock.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

   Select this role in the IAM instance profile setting of your instance.

4. Then, we’re done with the settings of the instance. Click Launch Instance to finish. You can connect to your EC2 instance using any of these option