Building the FMBench Python package

If you would like to build a dev version of FMBench for your own development and testing purposes, the following steps describe how to do that.

1. Clone the FMBench repo from GitHub, change directory to foundation-model-benchmarking-tool.

   cd foundation-model-benchmarking-tool
   

2. Install uv and create a virtual environment with all the dependencies that FMBench needs.

   curl -LsSf https://astral.sh/uv/install.sh | sh
   uv venv .fmbench_python312 && source .fmbench_python312/bin/activate && uv pip sync pyproject.toml
   export UV_PROJECT_ENVIRONMENT=.fmbench_python312
   uv add zmq
   python -m ipykernel install --user --name=.venv --display-name="Python (uv fmbench env)"
   sudo apt-get install tree
   

3. Make any code changes as needed. If you want edit any notebooks in the FMBench then select Python (uv fmbench env) conda kernel for your notebook.

4. Build the FMBench Python package.

   uv build
   

5. The .whl file is generated in the dist folder. Install the .whl in your current Python environment.

   uv pip install -U dist/*.whl
   

6. Run FMBench as usual through the FMBench CLI command.

7. You may have added new config files as part of your work, to make sure these files are called out in the manifest.txt run the following command. This command will overwrite the existing manifest.txt and manifest.md files. Both these files need to be committed to the repo. Reach out to the maintainers of this repo so that they can add new or modified config files to the blogs bucket (the CloudFormation stack would fail if a new file is added to the manifest but is not available for download through the S3 bucket).

   python create_manifest.py
   

8. To create updated documentation run the following command. You need to be added as a contributor to the FMBench repo to be able to publish to the website, so this command would not work for you if you are not added as a contributor to the repo.

   mkdocs gh-deploy