Prompt Development

#@title Environment Setup
#@markdown This cell should take about 30 sec to install the necessary libraries and download the report data from the NLM Open-I IU CXR dataset.

%%capture

# Install necessary libraries
! curl -fsSL https://ollama.com/install.sh | sh
! pip install ollama==0.2.0 \
    colab-xterm==0.2.0 \
    xmltodict>=0.12.0

# Download the data
! curl -s https://openi.nlm.nih.gov/imgs/collections/NLMCXR_reports.tgz | tar xvz

# set LD_LIBRARY_PATH so ollama is able to use the GPU of the host (Google Colab)
import os
os.environ.update({'LD_LIBRARY_PATH': '/usr/lib64-nvidia'})

#@title Run this cell to open a terminal in the output below
#@markdown Then copy-and-paste the following command into the terminal prompt below and hit `Return`/`Enter` on your keyboard.
#@markdown <br><br>```ollama serve```

%load_ext colabxterm
%xterm

Launching Xterm...

#@title Create Ollama model from Llama-3-8B-Instruct

import ollama

modelfile = '''
FROM llama3
SYSTEM Act as an expert radiologist with detailed knowledge of human anatomy, physiology, and pathology.
PARAMETER temperature 0
PARAMETER seed 42
'''

# First, we pull the 4-bit quantized version of Llama-3-8B-Instruct. This will take about 1 min.
ollama.pull("llama3")

# Then we "create" a custom version of the model using our custom instructions and parameters from the "modelfile" above.
ollama.create(model="rad", modelfile=modelfile)

{'status': 'success'}

# Now we can generate responses from our custom model. The first call will take ~20 sec. After that, it should be ~2 sec per call.

result = ollama.generate(model="rad", prompt="What are 3 differential diagnoses for a lung opacity on chest x-ray?")
print(result['response'])

As a radiologist, I'd be happy to help you with that!

When evaluating a lung opacity on a chest X-ray, there are several potential causes to consider. Here are three differential diagnoses:

1. **Pneumonia**: Pneumonia is an infection of the lungs caused by bacteria, viruses, or fungi. On a chest X-ray, pneumonia can appear as a consolidation or opacity in one or both lungs, often with air bronchograms (thin lines within the opacity). The opacity may be homogeneous or have a characteristic "tree-in-bud" pattern.
2. **Pleural Effusion**: A pleural effusion is a collection of fluid between the lung and chest wall. On a chest X-ray, it can appear as an opacity that follows the contours of the lung (known as a "meniscus sign"). The opacity may be located in one or both sides of the chest, and its shape and size can vary depending on the amount of fluid present.
3. **Pulmonary Edema**: Pulmonary edema is a condition where there is an abnormal accumulation of fluid in the lungs. On a chest X-ray, it can appear as an opacity that involves the entire lung or only certain areas (known as "bilateral" or "unilateral" pulmonary edema). The opacity may be homogeneous or have a ground-glass appearance.

Of course, these are just a few possibilities, and there are many other potential causes of lung opacities on chest X-ray. As a radiologist, I would consider the patient's clinical history, laboratory results, and additional imaging studies (such as CT scans) to help narrow down the differential diagnosis.

#@title **Extracting report data from the XML files**

#@markdown After the relevant data is extracted from the XML files, the total number of reports and the first 5 rows of our data table will show up below.

import glob
import xmltodict
import pandas as pd
from fastcore.foundation import L

# suppress warnings from the output
import warnings
warnings.filterwarnings('ignore')


def xml_parse(f):
    with open(f) as xml:
        report_dict = xmltodict.parse(xml.read())
    xml.close()
    return report_dict

def get_label(report):
    label = L(report['eCitation']['MeSH']['major'])
    return 'normal' if label[0].lower() == 'normal' else 'abnormal'

def get_text(report):
    text_dict = {}
    text_dict['id'] = report['eCitation']['IUXRId']['@id']
    text = report['eCitation']['MedlineCitation']['Article']['Abstract']['AbstractText']
    findings = text[2]['#text'] if '#text' in text[2] else ''
    text_dict['findings'] = findings
    impression = text[3]['#text'] if '#text' in text[3] else ''
    text_dict['impression'] = impression
    text_dict['full-text'] = ' '.join([findings, impression])
    return text_dict

def process_report(report):
    label = get_label(report)
    report_dict = get_text(report)
    report_dict['label'] = label
    return report_dict

fps = L(glob.glob('/content/ecgen-radiology/*'))
reports = fps.map(xml_parse)
reports_df = pd.DataFrame(reports.map(process_report)).set_index('id').sort_index()
print('# of reports:', reports_df.shape[0])
print()
reports_df.head()

# of reports: 3955

	findings	impression	full-text	label
id
1	The cardiac silhouette and mediastinum size ar...	Normal chest x-XXXX.	The cardiac silhouette and mediastinum size ar...	normal
10	The cardiomediastinal silhouette is within nor...	No acute cardiopulmonary process.	The cardiomediastinal silhouette is within nor...	abnormal
100	Both lungs are clear and expanded. Heart and m...	No active disease.	Both lungs are clear and expanded. Heart and m...	normal
1000	There is XXXX increased opacity within the rig...	1. Increased opacity in the right upper lobe w...	There is XXXX increased opacity within the rig...	abnormal
1001	Interstitial markings are diffusely prominent ...	Diffuse fibrosis. No visible focal acute disease.	Interstitial markings are diffusely prominent ...	abnormal

findings = reports_df.iloc[0]['findings']
print(findings)

The cardiac silhouette and mediastinum size are within normal limits. There is no pulmonary edema. There is no focal consolidation. There are no XXXX of a pleural effusion. There is no evidence of pneumothorax.

def prompt_template(findings):
    prompt = f'''Read the following `Findings` section from a radiology report carefully.
    <findings>{findings}</findings>
    Please provide a succint `Impression` for the radiology report based on the `Findings` above.
    '''
    return prompt

def inference(prompt):
    return ollama.generate(model="rad", prompt=prompt)

prompt = prompt_template(findings)
result = inference(prompt)
print(result['response'])

Based on the findings, I would write:

**Impression:** Normal cardiac silhouette and mediastinum size with no evidence of pulmonary edema, consolidation, pleural effusion, or pneumothorax.

This impression summarizes the main points from the findings section, indicating that there are no abnormalities detected in the heart, lungs, or surrounding tissues.

#@title Experiment with the prompt template here

def prompt_template(findings):
    prompt = f'''Read the following `Findings` section from a radiology report carefully.
    <findings>{findings}</findings>
    Please provide a succint `Impression` for the radiology report based on the `Findings` above.
    '''
    return prompt

prompt = prompt_template(findings)
result = inference(prompt)
print(result['response'])

Based on the findings, I would write:

**Impression:** Normal cardiac silhouette and mediastinum size with no evidence of pulmonary edema, consolidation, pleural effusion, or pneumothorax.

This impression summarizes the main points from the findings section, providing a concise overview of the radiological examination.

#@title Double-click cell to see a hint, then run the cell to see the result

def prompt_template(findings):
    prompt = f'''Read the following `Findings` section from a radiology report carefully.
    <findings>{findings}</findings>
    Please provide a succint `Impression` for the radiology report based on the `Findings` above using the following format:
    <impression>...</impression>
    '''
    return prompt

prompt = prompt_template(findings)
result = inference(prompt)
print(result['response'])

from sklearn.model_selection import train_test_split

dev, test = train_test_split(reports_df, test_size=0.2, stratify=reports_df.label.values, random_state=42)
train, val = train_test_split(dev, test_size=0.2, stratify=dev.label.values, random_state=42)
val_set = val.groupby('label').sample(n=5).copy().reset_index(drop=True)
test_set = test.groupby('label').sample(n=5).copy().reset_index(drop=True)

def prompt_template(report):
    prompt = f'''Read the following radiology report for a chest radiograph carefully.
    <report>{report}</report>
    Please classify this report as "normal" or "abnormal". Provide your label enclosed in <label></label> tags.
    '''
    return prompt

example = train.sample(n=1, random_state=43)
report = example.iloc[0]['full-text']
prompt = prompt_template(report)
result = inference(prompt)
print("REPORT:", report)
print("LABEL:", example.iloc[0]['label'])
print("RESPONSE:", result['response'])

REPORT: Similar mild cardiomegaly. Of the pulmonary vascularity is prominent. No focal consolidations or effusions. No pneumothorax. No acute bony abnormality. Mild cardiomegaly with XXXX of early failure.
LABEL: abnormal
RESPONSE: <label>Abnormal</label>

The report mentions "mild cardiomegaly", which is an abnormal finding, indicating that the heart is slightly enlarged. Additionally, the report notes "early failure" of the heart, which suggests some degree of cardiac dysfunction. While there are no other obvious abnormalities mentioned in the report (such as consolidations, effusions, or pneumothorax), the presence of cardiomegaly and early failure indicates that the radiograph is not normal.

import re

pattern = re.compile(r'<label>(.*?)</label>')

def parse_output(result):
    res = result['response']
    match = pattern.search(res)
    if match:
        ans = match.group(1).strip()
    else:
        ans = None
    return ans

def evaluate(prompt_template, val_set=val_set):
    score = 0.0
    n = len(val_set)
    for i in range(n):
        report = val_set.iloc[i]['full-text']
        prompt = prompt_template(report)
        result = inference(prompt)
        pred = parse_output(result)
        val_set.at[i, 'pred'] = pred
        if pred is None:
            print("Unparseable response")
            continue
        elif pred.lower() == val_set.iloc[i]['label']:
            score += 1
    return score/n

score = evaluate(prompt_template, val_set)
print(f"Baseline score: {score:.2f}")

Baseline score: 1.00

val_set

	findings	impression	full-text	label	pred
0	The heart size and pulmonary vascularity appea...	Bandlike opacities in the right base. Appearan...	The heart size and pulmonary vascularity appea...	abnormal	Abnormal
1	The lungs appear clear. There are no suspiciou...	1.Lucency in the left lateral clavicle near th...	The lungs appear clear. There are no suspiciou...	abnormal	Abnormal
2	The lungs are clear bilaterally. Specifically,...	1. No acute cardiopulmonary abnormality.. 2. A...	The lungs are clear bilaterally. Specifically,...	abnormal	Abnormal
3		Slight cardiomegaly. Clear lungs. No effusion	Slight cardiomegaly. Clear lungs. No effusion	abnormal	Abnormal
4	Low lung volumes. XXXX normal heart size. No p...	Low lung volumes, no acute cardiopulmonary dis...	Low lung volumes. XXXX normal heart size. No p...	abnormal	Abnormal
5	Lungs are clear. No pleural effusions or pneum...	Clear lungs with no suspicious pulmonary nodul...	Lungs are clear. No pleural effusions or pneum...	normal	normal
6	There are no focal areas of consolidation. No ...	No acute cardiopulmonary abnormality.	There are no focal areas of consolidation. No ...	normal	normal
7	Both lungs are clear and expanded. Heart and m...	No active disease.	Both lungs are clear and expanded. Heart and m...	normal	normal
8	The heart is normal in size. The mediastinum i...	No acute disease.	The heart is normal in size. The mediastinum i...	normal	normal
9	The cardiomediastinal silhouette and pulmonary...	No acute cardiopulmonary abnormality.	The cardiomediastinal silhouette and pulmonary...	normal	normal

def few_shot_template(report, examples):
    formatted_examples = ""
    for i in range(len(examples)):
        formatted_examples = formatted_examples + f'''
<example>
<report>{examples.iloc[i]['full-text']}</report>
<label>{examples.iloc[i]['label']}</label>
</example>'''
    prompt = f'''Examples of the task you will be asked to perform are provided below. You will be asked to classify a chest radiograph report as "normal" or "abnormal".
{formatted_examples}

Read the following radiology report for a chest radiograph carefully.
<report>{report}</report>
Please classify the preceding report as "normal" or "abnormal". Provide your label enclosed in <label></label> tags.'''
    return prompt

examples = train.sample(n=3).copy().reset_index(drop=True)
report = train.sample(n=1).iloc[0]['full-text']
prompt = few_shot_template(report, examples)
print(prompt)

Examples of the task you will be asked to perform are provided below. You will be asked to classify a chest radiograph report as "normal" or "abnormal".

<example>
<report>There are several small calcified granulomas. The lungs are otherwise clear. No focal airspace consolidation. No suspicious pulmonary mass or nodule is identified. There is no pleural effusion or pneumothorax. Heart size and mediastinal contour are within normal limits. There are diffuse degenerative changes of the spine. No evidence of active disease.</report>
<label>abnormal</label>
</example>
<example>
<report>No focal areas of consolidation. No suspicious pulmonary opacities. Heart size within normal limits. No pleural effusions. No evidence of pneumothorax. No acute cardiopulmonary abnormality.</report>
<label>normal</label>
</example>
<example>
<report>A XXXX XXXX lung volumes. Lungs are clear without focal airspace disease. No pleural effusions or pneumothoraces. cardiomegaly. Degenerative changes in the spine. Cardiomegaly with low lung volumes which are grossly clear.</report>
<label>abnormal</label>
</example>

Read the following radiology report for a chest radiograph carefully.
<report>The heart and lungs have XXXX XXXX in the interval. Both lungs are clear and expanded. Heart and mediastinum normal. No active disease.</report>
Please classify the preceding report as "normal" or "abnormal". Provide your label enclosed in <label></label> tags.