Welcome to the data and runtime repository for the SNOMED CT Entity Linking Benchmark. This repository contains a few things:

1. Submission template (examples/template/) — a template with the function signatures that you should implement in your submission.
2. Example submission (examples/submission/) — a submission with a simple demonstration solution. It will run successfully in the code execution runtime and outputs a valid submission.
3. FAISS + Qwen3 example (examples/faiss-qwen3/) — a more complete example using FAISS-based terminology search and an LLM (Qwen3-4B) for concept disambiguation. See its README for setup instructions.
4. Runtime environment specification (runtime/) — the definition of the environment where your code will run.

You can use this repository to:

💡 Get started: The example submission provides a demonstration solution that loads notes and uses them to generate valid span classification labels. Since it labels all of the text in the provided notes as 4596009 |Laryngeal structure (body structure)|, it won't produce a meaningful score, but you can use it as a guide for bringing in your own work and generating a real submission.

🔧 Test your submission: Test your submission using a locally running version of the benchmark runtime to discover errors before submitting.

📦 Request new packages in the official runtime: Since your submission will not have general access to the internet, all dependencies must be pre-installed. If you want to use a package that is not in the runtime environment, make a pull request to this repository.

Changes to the repository are documented in CHANGELOG.md.

• Prerequisites
• Setting up the data directory
• Running just commands
• Evaluating your annotations

• Code submission format
• Running your submission locally
• Logging and smoke tests
• Runtime network access

This quickstart guide will show you how to get the provided example solution running end-to-end on data and annotations from the training set. Once you get there, it's off to the races!

When you make a submission, we run your submission inside a Docker container, a virtual operating system that allows for a consistent software environment across machines. The best way to make sure your submission will run is to first run it successfully in the container on your local machine. For that, you'll need:

• A clone of this repository
• Docker
• At least 13 GB of free space for the Docker image
• just command runner (optional, but useful for running the commands in the justfile)

Additional requirements to run with GPU:

• NVIDIA drivers with CUDA 12.6
• NVIDIA container toolkit

In the official code execution platform, code_execution/data will contain data provided for the test set of clinical notes from MIMIC-IV-Note. The data format is a csv file (test_notes.csv) with two columns: "note_id", which contains the ID of the note, and "text", which contains the text of the note. Your code should read this file to obtain the text of each note and run model inference to generate annotated spans.

For local execution, you should simply copy over the set of train notes that you accessed from the benchmark training dataset into the data/ directory and name them test_notes.csv.

To test out the full execution pipeline, make sure Docker is running and then run the following commands in the terminal:

1. just pull pulls the latest official Docker image from the container registry (Azure). You'll need an internet connection for this.
2. just pack-example packages a code submission with the main.py contained in examples/submission/ that labels all text in the notes as "Larynx" and saves it as submission/submission.zip.
3. just test-submission will do a test run of your submission, simulating what happens during actual code execution. This command runs the Docker container with the requisite host directories mounted, and executes main.py to produce a submission.csv file containing your predicted annotations.

just pull
just pack-example
just test-submission

🎉 Congratulations! You've just completed your first test run for the SNOMED CT Entity Linking Benchmark. If everything worked as expected, you should see a new file submission/submission.csv has been generated.

If you were ready to make a real submission to the benchmark, you would upload the submission.zip file from step 2 above to the submissions page.

We also provide a script for you to evaluate your generated annotations. This script takes paths to the predicted annotations file and the corresponding ground truth annotations file and evaluates two metrics: the macro-averaged character-level IoU and the support-weighted character-level IoU.

python scripts/scoring.py submission/submission.csv data/train_annotations.csv
#> macro-averaged character IoU: 0.0000
#> support-weighted character IoU: 0.0000

You probably won't get a great score, but at least it's only up from here!

As you develop your own submission, you'll need to know a little bit more about how your submission will be unpacked for running inference. This section contains more complete documentation for developing and testing your own submission.

Your final submission should be a zip archive named with the extension .zip (for example, submission.zip). The root level of the submission.zip file must contain a main.py which generates a file called submission.csv that contains your predicted annotations for the notes. Your submission.csv file should have the same structure as the submission format.

A template for main.py is included at examples/template/main.py. For more detail, see the code submission documentation.

This section provides instructions on how to run your submission in the code execution container from your local machine. To simplify the steps, key processes have been defined in the justfile. Commands from the justfile are then run with just {command_name}. The basic steps are:

just pull
just pack-submission
just test-submission

Run just help for more information about the available commands as well as information on the official and built images that are available locally.

Here's the process in a bit more detail:

1. First, make sure you have set up the prerequisites.

2. Download the official Docker image:

   just pull

1. Save all of your submission files, including the required main.py script, in the submission_src folder of the runtime repository. Make sure any needed model weights and other assets are saved in submission_src as well.

2. Create a submission/submission.zip file containing your code and model assets:

   just pack-submission
   #> mkdir -p submission/
   #> cd submission_src; zip -r ../submission/submission.zip ./*
   #>   adding: main.py (deflated 73%)

3. Launch an instance of the Docker image, and run the same inference process that will take place in the official runtime:

   just test-submission

This runs the container entrypoint script. First, it unzips submission/submission.zip into /code_execution/src/ in the container. Then, it runs your submitted main.py. In the local testing setting, the final submission is saved out to submission/submission.csv on your local machine.

When you run just test-submission the logs will be printed to the terminal and written out to submission/log.txt. If you run into errors, use the log.txt to determine what changes you need to make for your code to execute successfully.

In order to prevent leakage of the IDs of notes in the test set, all logging is prohibited when running inference on the test set notes. When submitting on the platform, you will have the ability to submit "smoke tests". Smoke tests run with logging enabled on a reduced version of the training set notes in order to run more quickly. They will not be considered for evaluation and are intended to let you test your code for correctness. Smoke tests will be the only place you can view logs or output from your code and to debug. You should test your code locally as thoroughly as possible before submitting your code for smoke tests or for full evaluation.

The set of notes in the smoke test environment is a subset of the training set notes. We've made it easy to replicate the smoke test note environment locally - all you have to do is:

1. Copy the set of training notes into data/train_notes.csv
2. Copy the set of training annotations into data/train_annotations.csv
3. Run just smoke-test-data

You'll now have the smoke test notes in data/test_notes.csv and the corresponding annotations in data/smoke_test_annotations.csv. You can then run your submission using just test-submission and score your generated annotations by running

python scripts/scoring.py submission/submission.csv data/smoke_test_annotations.csv

If you've followed the above instructions, this score should match the one you receive from the smoke test environment on the platform.

If you want to use a package that is not in the environment, you are welcome to make a pull request to this repository. If you're new to the GitHub contribution workflow, check out this guide by GitHub.

The runtime manages dependencies using uv. The official runtime uses Python 3.12 environments.

To submit a pull request for a new package:

1. Fork this repository.

2. Edit runtime/pyproject.toml. There are two ways to add a requirement:

   • PyPI package (preferred): Add an entry to the [project.dependencies] section. This installs from PyPI using uv. uv performs robust dependency resolution with other packages, so we can avoid package version conflicts.
   • Direct URL or git: For packages not available on PyPI, you can add them as direct references in the dependencies.

3. Run just update-lockfile. This will resolve exact package versions and save the pinned environment to runtime/uv.lock.

4. Locally test that the Docker image builds successfully:

   just build

5. Commit the changes to your forked repository. Ensure that your branch includes updated versions of all of the following:

   • runtime/pyproject.toml
   • runtime/uv.lock

6. Open a pull request from your branch to the main branch of this repository. Navigate to the Pull requests tab in this repository, and click the "New pull request" button. For more detailed instructions, check out GitHub's help page.

7. Once you open the pull request, we will use GitHub Actions to build the Docker images with your changes and run the tests in runtime/tests. For security reasons, administrators may need to approve the workflow run before it happens. Once it starts, the process can take up to 30 minutes, and may take longer if your build is queued behind others. You will see a section on the pull request page that shows the status of the tests and links to the logs.

8. You may be asked to submit revisions to your pull request if the tests fail or if a maintainer has feedback. Pull requests won't be merged until all tests pass and the changes have been reviewed and approved.

A justfile with several helpful shell recipes is included in the repository. The runtime documentation above uses it extensively. Running just by itself in your shell will list relevant Docker images and provide you the following list of available commands:

Available recipes:
    help                                   # Show image status and available commands

    [* test submission locally]
    check-submission                       # Check contents of current submission.zip file
    interact-container                     # Open an interactive bash shell within the container
    pack-example example_name="submission" # Create submission.zip from examples folder (e.g., just pack-example submission)
    pack-submission                        # Create submission.zip from submission_src folder
    pull                                   # Pull the official container from Azure Container Registry
    smoke-test-data                        # Generate smoke test data from training notes and annotations
    test-submission                        # Test submission using submission.zip

    [development]
    build                                  # Build the runtime container (requires faiss-builder image)
    check-lock                             # Check that uv lockfile is in sync with pyproject.toml
    clean                                  # Delete temporary Python cache and bytecode files
    debug                                  # Print active variables for debugging
    format                                 # Format code with ruff
    update-lockfile *ARGS                  # Update uv lockfile

    [faiss]
    build-faiss                            # Build faiss-builder locally (slow, only if you can't pull from ACR)
    pull-faiss                             # Pull faiss-builder from ACR (fast, recommended)

    [tests]
    build-tests                            # Build the test container (requires faiss-builder image)
    interact-tests                         # Open an interactive bash shell in the test container
    run-tests                              # Run tests in the test container