Journal of Rare Disorders: Diagnosis & Therapy

Abbreviations

AUC: Area under the curve; CI: Confidence interval; FAS: Full analysis set; NP–C Niemann-Pick disease Type C; ROC: Receiver operating characteristic; RPS: Risk prediction score; SI: Suspicion Index; ULR: Univariable logistic regression; VSGP: Vertical supranuclear gaze palsy

Introduction

Niemann-Pick disease Type C (NP–C; OMIM 257220 [NP–C1] and 607625 [NP–C2]) is a rare, autosomal recessive, neurodegenerative lysosomal lipid storage disorder. As a result of its heterogeneous, non-specific clinical presentation and variable age of onset, NP–C is often difficult to diagnose, resulting in delays in diagnosis [1] and subsequent treatment initiation.

The online NP–C Suspicion Index (SI) screening tool was developed in 2012, using data collected retrospectively from patients with NP–C and patients without NP–C, from the same specialist centers [2]. The NP–C SI tool provides a risk prediction score (RPS) to aid in screening for NP–C by assessing the presence of specific signs and symptoms at an individual patient level, and scoring these symptoms according to their relative association with a positive NP–C diagnosis [2].

A sub-analysis of the original patient cohort was performed to determine the sensitivity and discriminatory power of the NP–C SI screening tool by age and manifestation–associations [3]. This sub-analysis found the NP–C SI tool to be useful in identifying associations between manifestations across all symptom domains and all age ranges, and to display particularly strong predictive power for NP–C when used in patients > 4 years of age [3].

Despite its positive discriminatory power, the current online NP–C SI tool only assesses NP–C signs and symptoms in isolation. Combinations are taken into account only as symptom cluster combinations, not as individual symptom combinations. Clinical experts have recommended that, for more accurate detection of NP–C, an RPS should be based on the presence of individual signs and symptoms and the symptom combinations, rather than on isolated findings. These retrospective analyses were conducted to assess two important aspects in NP–C diagnosis: (1) the number of symptoms presented by a patient, and; (2) the strength of the relationship between these symptoms. These analyses led to the development of two new NP–C SI models designed to assess the combination of symptoms presented by each patient, the presence of individual symptoms in isolation, and the relationship between symptoms. This report provides an overview of the new NP–C SI models and how they can be used in clinical practice.

Methods

Patients

Retrospective, post hoc analyses were performed using data from the existing cohort of patients, previously used to develop the original online NP–C SI tool [2]. Details of this patient cohort have been published previously [2]. The full analysis set (FAS) consisted of 216 patients: 71 NP–C cases, confirmed by filipin staining and presence of NPC1/NPC2 gene mutations; 64 NP–C non-cases, suspected to have NP–C but confirmed negative by filipin staining; and 81 controls, not suspected to have NP–C, but all with at least one characteristic symptom of the disease [2]. An additional cohort of 42 patients was used to validate the new models [4]. All patients in the additional cohort had a diagnosis of NP–C, diagnosed in a routine clinical setting, confirmed by NPC1 and NPC2 gene sequencing, and complemented by filipin staining [4].

For the original patient cohort, analyses were performed using the FAS, and in a subpopulation excluding patients ≤4 years of age, across the following subgroups: NP–C cases versus NP–C noncases, and NP–C cases versus controls. For the additional patient cohort, analyses were performed with all patients and excluding patients ≤4 years of age.

Ethics Committee approval was not required for the original NP–C SI study [2], nor for the subsequent sub-analyses of the original patient cohort. For the additional patient cohort, subjects were enrolled after providing signed consent, in accordance with the IRB approval S-032/2012 from Heidelberg University. The study protocol conformed to the ethical guidelines of the Declaration of Helsinki [4].

Combination signs and symptoms model

Objectives and analyses: Analyses were performed on the original NP–C cohort FAS to identify the combinations of signs and symptoms that relate to a positive diagnosis of NP–C.

Combinations were created among the 21 items in the original online NP–C SI tool [2]; all possible combinations of two, three and four signs or symptoms were created and variables defined as ‘present’ when all signs and symptoms in that combination were present, or ‘not present’ when one of the items was absent. This model was designed to assess the descriptive predictive power of double symptom combinations, and their correlation with a positive diagnosis of NP–C.

Statistical methods: Frequency tables were produced that contained the number of counts per category. Results were expressed as a percentage of patients with available data. Data presented are based on the results of univariable logistic regression (ULR) or exact ULR analysis.

To establish the new RPS for each patient, individual prediction scores, without rounding (the original NP–C SI tool employed rounding up to 1, 5, 10, 20 or 40 points), were taken for each of the 21 items; if the value was negative, it was considered as zero (Online resource 1). Prediction values for double combinations of signs or symptoms were calculated and the sum of the individual scores of the respective items subtracted from the absolute value. All possible two-by-two combinations among all symptoms were analyzed; if the value was negative, it was considered as zero. Scores obtained for combinations of each symptom were sorted in descending order. The five highest values were included in the final score with the following weights, and the remaining values discarded: 100% for highest value; 50% for second highest value; 25% for third highest value; 12.5% for fourth highest value; 6.25% for fifth highest value.

Cut-offs were selected to achieve specificity close to 95% to classify NP–C cases versus controls. For different cut-offs, sensitivity and specificity statistics were computed. Using this method, individual RPS and combination prediction values were included to calculate the new total RPS.

The performance of the model and new total RPS were investigated using receiver operating characteristic (ROC) curves, and area under the curve (AUC). The relationship between a variable and likelihood of NP–C was modelled using ULR, with NP–C cases as the reference versus NP–C non-cases or controls, as described previously [2]. Modelling was performed using Proc Logistic in SAS version 9.2.

Internal validation of the RPS was performed using the bootstrap resampling method [5] to obtain a bootstrap-corrected estimate of the C-statistic. Summary statistics of the C statistic distribution from 1,000 bootstrap samples and the 95% confidence interval (CI) of the mean were computed.

Alternative prediction score model: the 2/7 Score

Objectives and analyses: Following assessment of the combination of signs and symptoms, the strong predictive power of double combinations of specific signs and symptoms led to identification of variables that could be investigated in isolation to improve classification. An alternative prediction score model, termed the 2/7 Score, was based on these variables, and dependent on the presence of only seven signs and symptoms. This model was defined as the sum of the following seven items: prolonged, unexplained neonatal jaundice or cholestasis; isolated unexplained splenomegaly (historical and/or current) with or without hepatomegaly; vertical supranuclear gaze palsy (VSGP); gelastic cataplexy; pre-senile cognitive decline and/or dementia; psychotic symptoms (hallucinations, paranoid delusions and/or thought disorder); parent, sibling or cousin with NP–C (two items combined), with one point added for each, and two points added for VSGP.

Statistical methods: The ability of the 2/7 Score to correctly classify NP–C cases was evaluated. Data were based on logistic regression models used for classification as parameter estimates, ROC curves and AUC. Results were presented as a probability analysis for NP–C.

Statistical analyses were performed and cut-offs selected as described for the combination signs and symptoms, and sensitivity and specificity were calculated. Sensitivity and specificity values were estimated by fitting the distribution of RPS for NP–C cases and controls with a zero truncated normal distribution.

Results

Assessment of the combination signs and symptoms model: the new NP–C SI tool

Double combinations of 21 items within the original SI tool were assessed, and points assigned for each combination (Table 1). Although combinations of two, three and four signs and symptoms were assessed, the addition of combinations of three or four signs and symptoms did not improve specificity of the model, and therefore only combinations of two signs and symptoms were included in development of the new NP–C SI tool.

Table 1: Prediction values of double combinations of NP-C signs and symptoms.

Original patient cohort: Frequency tables were created for combinations of signs and symptoms in all patients with available data, and total RPS calculated including points associated with each combination. Clear differences in mean total RPS were observed between NP–C cases and NP–C non-cases, and between NP–C cases and controls. Descriptive analysis revealed that the combination signs and symptoms model was capable of distinguishing between all subgroups in both analysis populations (FAS and excluding patients ≤4 years of age, both P<0.001; Table 2).

Table 2: Total RPS, raw scores plus weighted combinations (original patient cohort; FAS and excluding patients ≤4 years of age).

Sensitivity and specificity ROC curves were produced, and AUC calculated, to determine the performance of the combination signs and symptoms model in each subgroup in the two populations (FAS and excluding patients ≤4 years of age; Figure 1). Comparison with the original online NP–C SI tool [2] showed that the model is more sensitive for predicting a diagnosis of NP–C. In the FAS, NP–C cases versus controls, AUC was 0.923 (95% CI 0.877, 0.960) versus 0.941 (95% CI 0.906, 0.976) for the original tool and new model, respectively; and for NP–C cases versus noncases, AUC was 0.778 (95% CI 0.697, 0.860) versus 0.779 (95% CI 0.696, 0.862), respectively. In the population excluding patients ≤4 years of age, AUC for NP–C cases versus controls was 0.997 (95% CI 0.992, 1.000) versus 1.000 (95% CI 1.000, 1.000), and for NP–C cases versus non-cases AUC was 0.895 (95% CI 0.819, 0.971) versus 0.958 (95% CI 0.915, 1.000), for the original tool and new model, respectively.

Figure 1 ROC curves and AUC for the combination signs and symptoms model for all subgroups (a) FAS; (b) excluding patients ≤4 years of age
AUC: Area Under the Curve; FAS: Full Analysis Set; NP-C: Niemann-Pick disease Type C; ROC: Receiver Operating Characteristic

Sensitivity and specificity versus total RPS were calculated for all subgroups and for both populations (Figure 2). In the FAS, for NP–C cases versus controls a sensitivity of 86.4% and specificity of 94.4% were achieved with a total RPS of 40 (Figure 2a). A sensitivity of 95.0% and specificity of 65.7% were achieved with a total RPS of 20 (Figure 2a). When excluding patients ≤4 years of age, a sensitivity of 99.0% and specificity of 96.5% were obtained for NP–C cases versus controls with a total RPS of 40, and 99.9% and 61.2%, respectively with a total RPS of 20 (Figure 2c).

Figure 2 Sensitivity and specificity versus total RPS for all subgroups (a) FAS, NP‑C cases versus controls; (b) FAS, NP-C cases versus non-cases; (c) excluding patients ≤4 years of age, NP-C cases versus controls; (d) excluding patients ≤4 years of age, NP‑C cases versus non-cases FAS: Full Analysis Set; NP-C: Niemann-Pick disease Type C; RPS: Risk Prediction Score

Additional patient cohort: In order to further assess the predictive power of combination signs and symptoms, analyses were performed in the additional patient cohort. Descriptive analysis of all patients revealed that the combination signs and symptoms model was able to predict 83% of NP–C cases, versus 71% of NP–C cases with the original SI tool. Furthermore, when excluding patients ≤4 years of age, the combination assessment was able to predict 94% of cases, versus 85% of cases with the original SI tool.

Assessment of the alternative prediction score model: the 2/7 Score

Original patient cohort: Descriptive analysis across all subgroups of the original patient cohort demonstrated that the 2/7 Score model was able to distinguish between subgroups based on seven isolated signs and symptoms of NP–C, in both analysis populations (FAS and FAS excluding patients ≤4 years of age, both P<0.001; Table 3).

Table 3: 2/7 Score (original patient cohort; FAS and excluding patients ≤4 years of age).

For the 2/7 Score model, sensitivity and specificity ROC curves were produced and AUC calculated to evaluate and compare performance of the model in each subgroup in the two populations (FAS and excluding patients ≤4 years of age; Figure 3). For the FAS, the 2/7 Score AUC for NP–C cases versus controls was 0.947 (95% CI 0.920, 0.975; Figure 3a). When excluding patients ≤4 years of age, the 2/7 Score AUC for NP–C cases versus controls was 0.995 (95% CI 0.985, 1.000; Figure 3b).

Figure 3 ROC curves and AUC for 2/7 Score for all subgroups (a) FAS; (b) excluding patients ≤4 years of age AUC: Area Under the Curve; FAS: Full Analysis Set; NP-C: Niemann-Pick disease Type C; ROC: Receiver Operating Characteristic

Sensitivity and specificity versus 2/7 Score were calculated for all subgroups and for both populations. In the FAS, for NP–C cases versus controls, 93.8% specificity was obtained with a score of 2, and 100% specificity was obtained with a score of 3; 100% specificity was obtained for NP–C cases versus controls with a score of 2 when excluding patients ≤4 years of age.

Comparison with results from the original NP–C SI tool [2] showed that the 2/7 Score is generally more sensitive for predicting a diagnosis of NP–C. FAS NP–C cases versus controls AUC was 0.923 versus 0.947 for the original tool and 2/7 Score, respectively; NP–C cases versus non cases AUC was 0.778 versus 0.785 (95% CI 0.708, 0.862), respectively. In the FAS excluding patients ≤4 years of age, AUC for NP–C cases versus controls was 0.997 versus 0.995, and for NP–C cases versus non-cases was 0.895 versus 0.939 (95% CI 0.887, 0.991) for the original tool and 2/7 Score, respectively.

Additional patient cohort: Further analyses were conducted in the additional patient cohort to validate findings. Comparison with results from the original NP–C SI tool showed that the 2/7 Score was more sensitive than the original tool for predicting a diagnosis of NP–C (all patients: 71% versus 83%, respectively; excluding patients ≤4 years of age: 85% versus 94%, respectively).

Additional post hoc analysis

Although not intended for use in patient’s ≤4 years of age, the performance of the combination signs and symptoms model and 2/7 Score were assessed in the subpopulation of patient’s ≤4 years of age. With the combination signs and symptoms model, AUC was 0.828 (95% CI 0.730, 0.927) for NP–C cases versus controls and 0.414 (95% CI 0.251, 0.576) for NP–C cases versus non-cases. With the 2/7 Score, AUC was 0.842 (95% CI 0.757, 0.927) for NP–C cases versus controls and 0.464 (95% CI 0.315, 0.613) for NP–C cases versus non-cases.

Discussion

Increasing development and availability of gene sequencing assays, and advances in biochemical techniques, are anticipated to have a positive impact when diagnosing patients with NP–C [6]. However, due to the heterogeneous nature of the disease and lack of specific clinical presentation, identifying patients who require testing for NP–C relies on general awareness, understanding of signs and symptoms, and the ability of the diagnosing clinician to recognize these symptoms [7,8]. The analyses presented here were performed to update the online NP–C SI tool, increasing its reliability for identifying patients with a high suspicion of NP–C.

We have demonstrated that certain combinations of signs and symptoms can increase the predictive power of the NP–C SI tool. When assessed in isolation, the two new NP–C SI models were able to accurately predict patients with NP–C using the original patient population and the additional cohort of patients with confirmed NP–C. The models proved more sensitive than the original tool for detecting NP–C cases versus non-cases or controls. When patients ≤4 years of age were excluded from analyses, both models produced greater mean total scores for NP–C cases compared with RPS when all patients were analyzed; however, mean total scores for NP–C non-cases and controls remained similar in the two populations.

A platform for updating the online NP–C SI tool

With the development and validation of the two new models, the online NP–C SI tool will be updated with the improved total RPS calculations, obtained when symptom combinations are considered. In the new online NP–C SI tool, results will be presented as a probability analysis for NP–C. In this manner, individual calculations will be made for every patient and compared with patients in the database. This provides a quantitative assessment of the suspicion of NP–C. Outputs will be based on the newly defined cut-offs for RPS from the sensitivity and specificity analysis of the combination signs and symptoms model, and will provide an indication of high, moderate or low suspicion of NP–C. Based on these findings, a patient can be considered to have a high suspicion of NP–C with a total RPS of ≥40. Moderate suspicion and low suspicion are indicated with a total RPS of 20–39 and <20, respectively.

With the 2/7 Score, it is only necessary to assess the presence or absence of seven signs and symptoms of NP–C. Using the 2/7 Score, approximately 95% specificity is obtained with a score ≥2. Therefore, a patient can be considered to have a high suspicion of NP–C if they have two or more signs or symptoms or if they have VSGP (contributes 2 points). If a patient presents with a score <2, then this patient can be considered to have a low probability of NP–C. This model provides a qualitative assessment of suspicion of NP–C with positive discriminatory power, particularly in patients >4 years of age. The 2/7 Score will be included as an output of the new online NP–C SI tool, highlighting the seven key discriminatory signs and symptoms when they are selected, and prompting the user to look for additional symptoms.

Limitations of the analyses and future directions

The analyses for these models were conducted using data from the patient population previously used to develop the original NP–C SI tool [2]; however, due to the small sample size, this patient cohort may not be fully representative of the NP–C population as a whole. The possibility of a sampling bias was investigated by including a cohort of 42 previously unevaluated patients to corroborate findings [4]. Analysis of this additional population provided the unique opportunity to assess the sensitivity of the models in a group of patients with a known diagnosis of NP–C. This additional cohort is highly valuable, as assessing the reliability of the models with as yet unidentified patients is not feasible in such a rare disease. Although analysis of an unrelated population indicated that there is no sampling bias, comprehensive understanding of the suitability and reliability of the new online NP–C SI tool will only become apparent once the tool is in regular clinical use. Prospective examination of patients in clinical practice will provide additional information on the sensitivity and specificity of the new online NP–C SI tool. The adaptive design of the tool means that it will be possible to add to the patient dataset, increasing population size, and subsequently improving the reliability and predictive power of the online NP–C SI tool.

Despite its positive discriminatory power in NP–C, the 2/7 Score is not suitable as an educational tool in NP–C, nor is it intended to be used as an isolated screening tool in clinical practice. Because only seven signs and symptoms are assessed, the 2/7 Score does not reflect the comprehensive clinical picture of NP–C. For example ataxia, a very prominent, early and frequent symptom in NP–C, is not included within the 2/7 Score due to its low specificity. In general screening, the 2/7 Score should be used together with the combination signs and symptoms model to more accurately screen patients for NP–C. The 2/7 Score can be employed in clinical practice as an easy-to-use and easy-toremember tool for rapid screening on a daily basis.

As with the original NP–C SI tool [3], the combination signs and symptoms model and the 2/7 Score were not found to be reliable in patients younger than 4 years of age. Development of an infantile NP–C SI tool is currently underway to improve reliability and accuracy of screening in an infant population.

Conclusions

The new NP–C SI model demonstrates excellent classification performance. As a clinical tool, the new NP–C SI tool can provide more granularity and information about the severity of disease in individual patients, compared with the original NP–C SI tool. The new online NP–C SI tool is designed to improve NP–C screening in clinical practice by providing a more accurate RPS for individual patients, using combinations of signs and symptoms. Although not designed to provide a definitive diagnosis based on clinical symptoms alone, it is anticipated that application of the new online NP–C SI tool and use of the 2/7 Score in clinical practice will improve screening, identifying more patients with a high clinical suspicion of the disease who warrant further testing. With more widespread and extended use, the new online NP–C SI tool will add to the knowledge about NP–C signs and symptoms, and ultimately aid timely diagnosis and treatment of patients with NP–C.

Acknowledgements

The authors would like to thank all patients within the original NP–C Suspicion Index cohort, and the German Self Support Group and the European rare diseases registry for Niemann- Pick disease Type A, B and C (EU-NPDR initiative). Writing and editorial assistance in the preparation of this manuscript were provided by Rachel Kendrick at Fishawack Communications GmbH, Basel, Switzerland.

Funding

Medical writing support in the development of this manuscript was funded by Actelion Pharmaceuticals Ltd, Allschwil, Switzerland. Development of the NP–C Suspicion Index was supported by Actelion Pharmaceuticals Ltd, Allschwil, Switzerland.

Competing and conflicting interests

CJH has received consulting fees, honoraria and research grants from Actelion Pharmaceuticals Ltd.

MP has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

MF has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

MW has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

MS has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

HR is a full time employee of Merck & Co. and has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

MCP has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd, Agios Pharmaceuticals, Amicus Therapeutics, Inc., Orphazyme ApS, Shire, Synageva BioPharma Corp., and Vtesse, Inc.

JVT has received consulting fees or honoraria from Actelion Pharmaceuticals Ltd.

SAK is a full time employee of Actelion Pharmaceuticals Ltd.

References

1. Vanier MT (2010) Niemann-Pick disease type C. Orphanet J Rare 5: 16.
2. Wijburg FA, Sedel F, Pineda M, Hendriksz CJ, Fahey M, et al. (2012) Development of a Suspicion Index to aid diagnosis of Niemann-Pick disease type C. Neurology 8: 1560-1567.
3. Wraith JE, Sedel F, Pineda M, Wijburg FA, Hendriksz CJ, et al. (2014) Niemann-Pick type C Suspicion Index tool: analyses by age and association of manifestations. J Inherit Metabolic Dis 37: 93-101.
4. Stampfer M, Theiss S, Amraoui Y, Jiang X, Keller S, et al. (2013) Niemann-Pick disease type C clinical database: cognitive and coordination deficits are early disease indicators. Orphanet J Rare Dis 8: 35.
5. Iwi G, Millard RK, Palmer AM, Preece AW, Saunders M (1999) Bootstrap resampling: a powerful method of assessing confidence intervals for doses from experimental data. Phys Med Biol 44: N55-62.
6. Mengel E, Klunemann HH, Lourenco CM, Hendriksz CJ, Sedel F, et al. (2013) Niemann-Pick disease type C symptomatology: an expert-based clinical description. Orphanet J Rare Dis 8: 166.
7. Wraith JE, Baumgartner MR, Bembi B, Covanis A, Levade T, et al. (2009) Recommendations on the diagnosis and management of Niemann-Pick disease type C. Mol Genet Metab 98: 152-165.
8. Patterson MC, Hendriksz CJ, Walterfang M, Sedel F, Vanier MT, et al. (2012) Recommendations for the diagnosis and management of Niemann-Pick disease type C: an update. Mol Genet Metab 106: 330-344.