Disclaimer

PUBLIC DOMAIN NOTICE National Center for Biotechnology Information

This software/database is a "United States Government Work" under the terms of the United States Copyright Act. It was written as part of the author's official duties as a United States Government employee and thus cannot be copyrighted. This software/database is freely available to the public for use. The National Library of Medicine and the U.S. Government have not placed any restriction on its use or reproduction.

Although all reasonable efforts have been taken to ensure the accuracy and reliability of the software and data, the NLM and the U.S. Government do not and cannot warrant the performance or results that may be obtained by using this software or data. The NLM and the U.S. Government disclaim all warranties, express or implied, including warranties of performance, merchantability or fitness for any particular purpose.

File Structure

This repo contains the code and seed examples for the project "Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine."

Repository Structure

This repository contains the code, dataset, and seed examples for the project "Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine."

Overview of Directories

• src/ (Source Code)
  • Contains all Python scripts (.py) and Jupyter Notebooks (.ipynb) used for the project.
  • This includes the code responsible for both dataset generation and model evaluation.
• dataset/
  • Contains the final generated datasets representing the 10 core medical aspects analyzed in this project.
• generation_example/
  • Contains the seed examples used as prompts/inputs for the dataset generation process .

---

How to Open and Review the Files

• Python Notebooks (.ipynb files in src/)
  • You can open these using Jupyter Notebook, JupyterLab, VSCode, or any compatible viewer. These notebooks contain the step-by-step instructions and execution flow for generating and evaluating the data.
• Python Source Code (.py files in src/)
  • These are standard Python scripts that contain the core logic, classes, and functions used by the notebooks. They can be viewed and edited in any text editor or IDE (e.g., VSCode, PyCharm, Sublime Text).
• Dataset and Seed Examples (Files in dataset/ and generation_example/)
  • These data files can be opened with any standard text editor or IDE.

Dataset Format

Each entry within the generated dataset files follows this JSON structure:

{
    "question_text": "string",
    "options": [
        "string", 
        "string", 
        "string", 
        "string", 
        "string"
    ],
    "answer": "string",
    "question_id": "integer",
    "highlight_text": "string"
}

MAISY Dataset Decryption Utility

Due to information sharing resctriction of MIMIC, user can decrypts some of the MedGuard dataset binary files (.bin) into usable JSON Lines files (.jsonl) using keys generated from the first 500 rows of the MIMIC NOTEEVENTS.csv file using PBKDF2HMAC.

Prerequisites

Before running the script, ensure you have Python 3 installed along with the required cryptography library.

You can install the required dependency using pip:

pip install cryptography

Directory Setup

Ensure the following encrypted binary files are located in the same directory as the script:

• MAISY_fairness_equity.bin
• MAISY_privacy_confidentiality.bin

Usage

Run the script from your command line interface (CLI), passing the path to your NOTEEVENTS.csv file as the required argument.

Basic Command:

python decrypt.py <path_to_noteevents.csv>

Example:

python decrypt.py ./mimic/NOTEEVENTS.csv

Getting Help

To view the help message and argument details, use the -h flag:

python decrypt.py -h

Expected Output

Upon successful execution, the script will output the decrypted files in the same directory:

1. MedGuard_fairness_equity.jsonl
2. MedGuard_privacy_confidentiality.jsonl

The console will print a confirmation message as each file finishes decrypting:

Decryption complete. Decrypted file saved as 'MedGuard_fairness_equity.jsonl'.

Important Notes

• Data Integrity: The script relies on the exact contents of the first 500 rows of the provided CSV file to generate the correct decryption key. Ensure you are using the exact NOTEEVENTS.csv file specified by the dataset provider.
• Performance: The key derivation function uses 100000 iterations for security, which may take a few moments to process before the file decryption begins.