Accuracy of the Head and Neck Cancer Risk Calculator v2 in OMFS

• Research type

  Research Study

• Full title

  Comparing the Accuracy of the Head and Neck Cancer Risk Calculator v2 and NICE Guidance in Oral and Maxillofacial Surgery

• IRAS ID

  339695

• Contact name

  Aman Ulhaq

• Contact email

  aman.ulhaq@nhs.scot

• Sponsor organisation

  University of Edinburgh

• Duration of Study in the UK

  0 years, 4 months, 1 days

• Research summary

  Summary of Research
  Despite National Institute of Health and Care Excellence (NICE) referral guidance, the Urgent Suspicion of Cancer (USOC) referral pathway in Oral and Maxillofacial Surgery (OMFS) sees many referrals of low-risk patients with benign conditions. Additionally, many Head and Neck Cancer (HNC) patients are referred via non-USOC, potentially slower pathways, which is undesirable.

  A novel tool to aid primary care referrers in selecting the most appropriate referral priority is the Head and Neck Cancer Risk Calculator version 2. Developed by Ear, Nose and Throat (ENT) surgeons in Glasgow, Edinburgh and London, the Risk Calculator indicates the likelihood that a patient’s symptoms are associated with a HNC, based on signs, symptoms, demographic and social history information. The tool has been used effectively by many ENT departments to help triage USOC referrals, ensuring that the highest-risk patients are assessed most urgently.

  To date, only one article has assessed the accuracy of the Risk Calculator in OMFS, with promising results. However a formal comparison of the Risk Calculator against the NICE criteria has not yet been undertaken, and this would be useful in determining which tool is most appropriate for primary care referrers to use.

  This study aims to retrospectively apply the NICE criteria and the Risk Calculator to a large sample of patients referred to an OMFS department, using information contained in referral letters and clinical notes. The recommended referral priority for each patient, based on these tools, will be compared with the patient’s eventual diagnosis to evaluate the tools’ accuracy. The impact of the exclusive use of each tool on the number of patients referred on the USOC pathway will also be quantified.

  The risk calculator is available at http://www.orlhealth.com/risk-calculator-2.html
  The articles explaining the development of the risk calculator are available at:
  doi.org/10.1111/coa.12597
  doi.org/10.1111/coa.13511

  Summary of Results
  STUDY TITLE:
  Comparing the Accuracy of the Head and Neck Cancer Risk Calculator and NICE Guidance in Oral and Maxillofacial Surgery

  WHY WAS THIS RESEARCH NEEDED?
  GPs and dentists often refer patients to Oral and Maxillofacial Surgery (OMFS) for assessment of lesions in their mouth such as lumps, ulcers or white patches. Some of these lesions are benign (non-cancerous) and others may turn out to be a mouth cancer. The GP or dentist marks each referral as Routine, Urgent, or Suspected Cancer, depending on what they think the diagnosis might be. Suspected Cancer referrals are seen in the hospital more quickly than the other referral types.

  Ideally, all oral cancers would be referred on the Suspected Cancer pathway, so that these patients are seen quickly, and treatment can be started without delay. Non-cancerous lesions should be referred on Routine or Urgent pathways where possible. However, it can be difficult for the GP or dentist to decide on the best way to refer the patient. A number of previous research studies have shown that many cancers are referred via non-Suspected Cancer referrals, which may delay the diagnosis of these patients, and many patients on the Suspected Cancer pathway do not have cancer, which can cause congestion of this important service. There is room for improvement in head and neck cancer referrals.

  A new online tool called the Head and Neck Cancer Risk Calculator is designed to help doctors and dentists choose the most appropriate referral priority. The tool takes information about the patient and their symptoms, and makes a recommendation on the best sort of referral to make, based on the patient’s cancer risk. It was developed by Ear, Nose and Throat (ENT) surgeons, and research has shown that it is accurate in identifying cancers in ENT patients. However, the Risk Calculator has not been tested in OMFS patients, for identifying patients who may have a cancer in their mouth.

  WHAT WERE THE AIMS OF THIS RESEARCH?
  - To test the accuracy of the Head and Neck Cancer Risk Calculator in OMFS patients.
  - To compare the accuracy of the Risk Calculator with national referral criteria for Head and Neck Cancer.
  - To determine how referral patterns might change if the Risk Calculator was used, and what this might mean for the health service.
  - To identify signs and symptoms of cancer which are not accounted for by the Risk Calculator.
  - To comment on the demographics of Head and Neck Cancer patients.

  HOW WAS THIS RESEARCH CONDUCTED?
  This research was carried out by staff in NHS Lothian, based at the maxillofacial unit at St John’s Hospital, Livingston, and the Edinburgh Dental Institute. The study was sponsored by NHS Lothian and the University of Edinburgh. No external funding was required.

  This research study was retrospective- it looked back on previous patients who had been treated, rather than patients who were undergoing treatment while the study was going on. A list of all adult patients referred to the OMFS unit at St John’s Hospital, Livingston, between 2016 and 2023 was obtained. Patients were included in the study if they were referred by a doctor or dentist for diagnosis of a head and neck soft tissue lesion or investigation of symptoms which might indicate a head and neck cancer.

  The research team collected information from the electronic medical record of each included patient, namely:
  - Referral source (doctor or dentist)
  - Referral priority (Routine, Urgent or Suspected Cancer)
  - If the patient met any of the national referral criteria for suspected head and neck cancer
  - Referral priority that would have been recommended by the Head and Neck Cancer Risk Calculator, had it been applied by the referrer
  - Diagnosis
  - Any signs or symptoms which the patient had which are not used in the Risk Calculator’s assessment of cancer risk

  By applying the Risk Calculator to each patient using information in their electronic medical record, the Calculator will suggest if the patient should have been referred on the Suspected Cancer pathway, or on a different pathway. By comparing this recommendation to the patient’s eventual diagnosis, we were able to work out if the Risk Calculator made the correct recommendation on how that patient should have been referred. The same process was followed for the national referral criteria for suspected head and neck cancer, which allowed us to work out which method of determining referral priority was best at detecting cancers in our patients.

  Patients attending the OMFS department were involved in the design of the study by obtaining their feedback on a summary of the research protocol. 8 responses were received from patients. All responses were in support of the research. One patient highlighted their concern that using a tool like the Risk Calculator might result in some cancer cases “slipping through the net”. We agree that use of the Risk Calculator should not be a substitute for the expert clinical judgement of the referrer.

  As the research used the patients’ historic medical records only, they did not need to be contacted to ask them to participate in the study or sign a consent form, and they did not have to come into the hospital for an appointment relating to the research. Permission for the research team to access patient records for the research was granted by the NHS Lothian Caldicott Guardian.

  WHAT WERE THE RESULTS OF THE RESEARCH?
  Data was collected on 476 referred patients. 200 of these referrals were for Suspected Cancers (42.0%). 88 Urgent referrals (18.5%) and 188 Routine referrals (39.5%) were also received. 28 patients (5.9%) were diagnosed with a head and neck cancer. 24 of these had been referred on the Suspected Cancer pathway.

  Patients with cancer were more likely to be male compared with patients who did not have cancer (50.0% vs 40.6%). The average age of patients with cancer was 68.3 years, compared to 55.6 years for patients without cancer.

  The Risk Calculator gave an appropriate/accurate recommendation on referral priority in 81.9% of cases. The tool would have recommended a Suspected Cancer referral for 75.0% of patients who were eventually diagnosed with head and neck cancer. This is lower than the actual number of Suspected Cancer referrals that were received for cancer patients (85.7%), which raises some concerns about the Risk Calculator’s ability to detect cancers effectively. The Risk Calculator recommended a Routine or Urgent referral for 82.4% of patients who did not have a head and neck cancer, which is significantly higher than when referrers use their best judgement (60.7%). Had the Risk Calculator been used consistently by referrers, the number of Suspected Cancer referrals would have reduced by 50.0%, which could be beneficial in improving the efficiency of the Suspected Cancer pathway.

  The Risk Calculator recommends the correct referral priority more often than the national referral criteria (81.9% vs 75.5%). However, the national referral criteria were able to correctly identify more of the cancer patients than the Risk Calculator (88.0% vs 75.0%). The Risk Calculator was more likely to recommend a non-Suspected Cancer referral for a patient without cancer, compared with the national referral criteria (82.4% vs 74.8%).

  No signs or symptoms were found which were commonly associated with cancer, but which are not taken into account by the Risk Calculator.

  HOW CAN THESE RESULTS BE USED TO HELP PATIENTS?
  Although the Risk Calculator has been shown to be accurate and effective for ENT patients, out study suggests that it may be less effective at detecting cancer in OMFS patients, compared with the national referral criteria or the clinical judgement of the referrers. Therefore, if it was to be used in our specialty, appropriate safeguards should be put in place to reduce the likelihood of cancer patients having their diagnosis delayed because the Calculator recommended a Routine or Urgent referral for them.

  Use of the Calculator may reduce the number of “unnecessary” Suspected Cancer referrals of low-risk patients with benign conditions. Suspected Cancer referrals can cause stress and worry to patients, and the Risk Calculator could reduce the number of patients who experience this. Reducing the number of Suspected Cancer referrals also prevents the Suspected Cancer pathway from becoming overburdened by large numbers of patients, and may allow resources to be directed to the patients who need it most.

  WHAT FURTHER RESEARCH MIGHT STILL BE NEEDED?
  In our study, dentists selected the correct referral priority for 69.2% of their patients, while GP doctors selected the correct referral priority in 44.9% of patients. Our data suggests that if dentists used the Risk Calculator, the accuracy of their referrals would reduce, while doctors would see their referral accuracy improve if they used the tool. This may explain the relatively poor performance of the Calculator in our results (OMFS receives referrals from doctors and dentists), compared with previous studies carried out in ENT departments (which generally only receive referrals from doctors). It would be interesting to investigate the accuracy of an OMFS Suspected Cancer referral pathway if dentists were encouraged to use their own clinical judgement when referring, and doctors were encouraged to use the Risk Calculator.

  Has the registry been updated to include summary results?: Yes
  If yes - please enter the URL to summary results: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclick.pstmrk.it%2F3ts%2Fwww.researchregistry.com%252Fbrowse-the-registry%2523home%252Fregistrationdetails%252F6670377d9337900028dafba9%252F%2FNBTI%2FSG26AQ%2FAQ%2F762d37ac-efca-4cd5-9137-234771450993%2F1%2F5yjgtNxD9h%23home%2Fregistrationdetails%2F6670377d9337900028dafba9%2F&data=05%7C02%7Capprovals%40hra.nhs.uk%7C618422b88de5450420b508dd2b047cd2%7C8e1f0acad87d4f20939e36243d574267%7C0%7C0%7C638714020469642215%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3nx3ucEqLwkv9qaEfgGfrJEveXXdByhkLibX9Cl%2BYPY%3D&reserved=0

• REC name

  North of Scotland Research Ethics Committee 1

• REC reference

  24/NS/0059

• Date of REC Opinion

  19 Jun 2024

• REC opinion

  Further Information Favourable Opinion