Validity of an Online Assessment of Attention-Deficit/Hyperactivity Disorder Among a Real-World Sample of Adults Seeking Web-Based Mental Health Care

Attention-deficit/hyperactivity disorder (ADHD) is a prevalent neurodevelopmental disorder characterized by persistent, maladaptive levels of inattention, hyperactivity, and impulsivity, and significant impairments in functioning.¹ While typically diagnosed in childhood, ADHD generally persists into adulthood,^2,3 with a prevalence of approximately 6% in children and up to 6% in adults.^3–6 Estimates of the incidence of adult ADHD range from 6.8 to 13.5 per 10,000 person years.^7,8 The disorder is more easily recognized in children⁶ and therefore may be less often screened for in adult populations. Additionally, adults with undiagnosed ADHD often present with comorbidities that may complicate diagnosis.^9,10

ADHD is typically chronic, although the impairments may fluctuate.³ Factors associated with persistence into adulthood include disease severity, failure to receive treatment, adverse childhood experiences, antisocial behavior, drug use,¹¹ and comorbid physical and mental health conditions.^12–15 ADHD negatively affects well-being, and the risk of premature death among adults with ADHD is significantly elevated,¹⁶ likely due to multiple factors including accidental injuries and suicide.^16–20 Both nonpharmacologic and pharmacologic treatments are effective at reducing symptoms.²⁰

The standard of care for diagnosing ADHD is a comprehensive clinical evaluation, including a detailed patient history conducted by a mental health professional.²¹ Unfortunately, access to behavioral health care in the US is variable and often inadequate, especially in nonmetropolitan areas. Recent statistics show dramatic disparities in behavioral health care access across the US, with poorer access in the most rural counties.²² Sex, age, mental health literacy,²³ and financial barriers²⁴ also contribute to reduced behavioral health care access. Moreover, diagnostic delays and undertreatment of adult ADHD are believed to be common^25,26 and likely contribute to poorer outcomes including morbidity and mortality.²⁵

The emergence of online ADHD assessments has significantly increased access to evaluation but has been accompanied by some controversy and skepticism regarding their validity and rigor.* Nevertheless, a recent U.S. Centers for Disease Control and Prevention (CDC) report found that 46% of adults with ADHD have used telehealth for ADHD services at least once,⁶ yet we know of no published work addressing the validity of online ADHD assessments,† especially those delivered asynchronously. This study sought to compare the results of a commercially available online ADHD assessment with virtual clinical interviews administered by licensed psychologists.

METHODS

Study Design and Procedures

A cross-sectional evaluation of agreement between the diagnoses assigned by clinicians applying DSM-5 criteria during virtual interviews and those assigned by a proprietary, online ADHD assessment was conducted. Ethical review and approval were obtained from the Advarra Institutional Review Board. Identifying information was kept confidential, and all data were de-identified for analysis and reporting. All assessments were completed by licensed, doctorate-level clinicians.

Participants were recruited from a volunteer sample of community-dwelling US adults seeking online resources to support ADHD diagnosis, treatment, or management. To participate in the study, participants responded to targeted advertisements and emails and completed an eligibility screener. Eligible individuals were fluent in English, aged 19 years or older, and a resident of one of the 50 US States or Washington, DC. Respondents were excluded if they had previously been diagnosed with or referred for assessment for ADHD. Furthermore, individuals with self-reported history of psychosis, dementia, or severe developmental disabilities or psychiatric conditions were excluded from participation.

After providing informed consent, participants completed a virtual clinical interview assessment for ADHD. Upon completion, clinicians rendered an ADHD-positive or ADHD-negative determination which was treated as the standard for evaluation of the online assessment. Participants completed the online assessment 1–4 weeks after the clinical interview. To measure the level of ADHD symptoms, the Adult ADHD Self-Report Scale (ASRS) was administered during the online assessment alongside other validated measures and DSM-5-aligned questions. Results of the online assessment were evaluated offline by a clinician blinded to the results of the interview. See Supplementary Materials for additional details on the clinical interview and online assessment. In cases where the results of the 2 assessments did not agree (ie, off-diagonal cases in the 2 × 2 table of results), a third clinician reviewed both the clinical interview notes and online assessment results and rendered a final “full data” diagnosis using all available patient information.

Data Management

A single dataset containing participant and rater characteristics, as well as results of the clinical interview, online assessment, and full-data adjudication process, was created and analyzed. Results were characterized based on their diagnostic result (Clinical Interview ADHD-positive, Clinical Interview ADHD-negative, Online Assessment ADHD-positive, Online Assessment ADHD-negative), the ADHD presentation type identified on both assessments (hyperactive, inattentive, combined‡), and agreement between assessment methods (true/false positive and negative).

Outcome Measures

True positive cases were defined as cases in which the result of both the interview and online assessment was ADHD-positive, indicating both results showed that DSM-5 criteria were met. True negative cases were defined as cases in which both the interview and online assessment were ADHD-negative. Given that the clinical interview was used to evaluate the online assessment, false-positive cases were defined as off-diagonal cases in which the clinical interview was ADHD-negative, and the online assessment was ADHD-positive. False negative cases were defined as off-diagonal cases in which the clinical interview was ADHD-positive, and the online assessment was ADHD-negative. Adjudicated full-data results were the results of the independent third-party review of both assessments to determine a final ADHD diagnosis for off-diagonal (false negative and false positive) cases.

Statistical Analysis

As a measure of reliability in ADHD symptom measurement, Cronbach’s alpha was calculated for the ASRS. Additionally, the distribution of assessments completed by clinicians, including the mean and median numbers of assessments by type, was calculated (Table 1). Chi-square, independent samples t tests, and Fisher exact tests were used to assess differences between clinical interview ADHD-negative and -positive groups.

Agreement between the clinical interview and online assessment was assessed using a 2×2 matrix of assessment results (Table 2). Cohen κ, total agreement, sensitivity, specificity, and positive and negative predictive value (PPV, NPV) were calculated.

To better understand factors contributing to assessment disagreement, follow-up analyses were conducted. Off-diagonal cases (see Table 2) were examined for their relationship with ADHD presentation using a 4 × 4 matrix (Table 3). This matrix was examined for sources of systematic differences between the 2 assessments (eg, consistent misclassification of a particular ADHD presentation). Next, alternate 2 × 2 crosstabulations compared the results of the online assessment (Table 4) and clinical interview (Table 5) to the adjudicated full-data diagnosis results for off-diagonal cases. Cohen κ, total agreement, sensitivity, specificity, PPV, and NPV were calculated.

Alpha levels were set at p < .05. All analyses were completed with SAS v. 9.4 (Cary, NC).

RESULTS

Study Attrition

A total of 2,810 participants responded to study advertisements. Of these, 1,164 (41%) were excluded based on their responses to the screening questionnaire. The most common exclusion reasons were a preexisting ADHD diagnosis (67.4%) and provider referral for assessment (20.7%). Respondents were also excluded for the following reasons: duplicate application submitted and previously excluded (7.8%), age 18 years or younger (2.0%), severe psychiatric comorbidity (1.9%), and not fluent in English (0.2%).§ After completing the screening questionnaire, 1,646 (59%) individuals were deemed eligible to participate and 1,231 (44%) completed informed consent. A total of 394 (14%) completed the clinical interview, and 345 (12%) completed the subsequent online assessment. The final sample comprised 345 individuals who completed both the virtual clinical interview and online assessment.

Demographics

The sample was predominantly female (80.9%), with an average age of approximately 35 years (Table 1). Overall, the sample had high levels of ADHD symptoms on the ASRS. Reliability of the ASRS total score was acceptable (α = .74). As expected, significantly higher ASRS scores were observed among the clinical interview ADHD-positive group.

ADHD Prevalence and Assessment Agreement

The online assessment was more conservative in rendering ADHD-positive diagnoses than the clinical interview (Table 2), with 78.8% of online assessments returning an ADHD-positive result, relative to 92.8% of clinical interviews. The overall accuracy of the online assessment was 78.0% (95% CI, 73.2–82.2%), with a PPV of 94.9% (95% CI, 92.8–96.3%), NPV of 15.1% (95% CI, 9.8–22.6%), sensitivity of 80.6% (95% CI, 75.9–84.8%), specificity of 44.0% (95% CI, 24.4–65.1%), and κ of 0.13 (SE = 0.06, 95% CI, 0.02–0.24, p < .01). An examination of off-diagonal cases further confirmed that the online assessment was more conservative than the clinical interview in rendering ADHD-positive diagnoses. Over 80% of off-diagonal cases (62/76) resulted from the online assessment not diagnosing ADHD when the clinical interview had.

Factors Associated With Disagreement

The most common ADHD presentations on both assessments were combined and inattentive (Table 3). Based on the prevalence of ADHD presentations among the total sample, none was consistently misclassified or over-represented among off-diagonal cases.

Assessments of agreement between the adjudicated full-data results and the online assessment (Table 4; κ = 0.49, SE = 0.06, 95% CI, 0.37–0.61, p < .001) and clinical interview (Table 5; κ = 0.46, SE = 0.05, 95% CI, 0.30–0.63, p < .001) revealed that agreement was equivalent for the 2 assessment types. The online assessment had significantly better PPV (online assessment: 98.5%, 95% CI, 96.4%–99.4%; clinical interview: 94.1%, 95% CI, 92.2%–95.5%, p < .05) and specificity (online assessment: 88.2%, 95% CI, 72.6%–96.7%; clinical interview: 44.1%, 95% CI, 27.2%–62.1%, p < .05) than the clinical interview. In contrast, the clinical interview showed significantly higher sensitivity (online assessment: 86.2%, 95% CI, 81.8%–89.8%; clinical interview: 96.8%, 95% CI, 94.2%–98.5%, p < .05). NPV was also higher for the clinical interview, but not significantly so (online assessment: 41.1%, 95% CI, 34.0%–48.6%; clinical interview: 60.0%, 95% CI, 42.3%–75.5%, p>.05). The false-positive rate (11.7%; 4/34) for the online assessment was significantly lower than the clinical interview (55.9%; 19/34). This further confirms the primary results based on overall rates of ADHD indicating that the online assessment was more conservative in rendering ADHD positive diagnoses.

DISCUSSION

This study measured the accuracy, reliability, and validity of an online assessment for adult ADHD by comparing the results of an online ADHD assessment with those of virtual clinical interviews. Among a population of adults seeking online ADHD assessment, the assessment was highly accurate at identifying ADHD-positive cases. The online assessment was also more conservative than the clinical interview in rendering ADHD-positive diagnoses, not significantly increasing the number of false-positive cases. Furthermore, in cases where an ADHD diagnosis was not made by the online assessment, further clinical evaluation was always recommended, as the participants were self-referred, presumably because they were experiencing symptoms and/or impairment which could be consistent with a mental disorder if not ADHD. These results suggest that this assessment will not contribute to overdiagnosis of adult ADHD and unnecessary prescription of psychostimulants.

A comparison of the results of the clinical interview and online assessment with the adjudicated full-data diagnoses revealed that the online assessment was significantly more accurate at identifying ADHD positive cases relative to the clinical interview. The clinical interview, on the other hand, may be better at distinguishing ADHD negative cases, as the false negative rate associated with the clinical interview was lower than the online assessment. This may have been because clinicians were more likely to diagnose subthreshold cases during the interview so patients could easily seek treatment, whereas the online assessment relied on more rigid diagnostic criteria in the absence of synchronous clinical interaction. The results related to ADHD presentation indicate that inattentive and combined ADHD are most common among this population, and that the online assessment did not systematically misclassify cases by subtype.

Strengths

This study has high external validity due to the utilization of a real-world population seeking online mental health care. Furthermore, results show that online ADHD assessments can be a reliable and accurate tool to improve access to mental health care globally. Our results also provide new information about individuals self-referring for online ADHD assessment. Remarkably, 81% of the sample was female. This may, in part, explain why recent CDC data shows increasing rates of ADHD in adult females that exceed those of males.⁶ Also notable is that nearly all these individuals (93%) met criteria for ADHD when interviewed by a psychologist. Although the online assessment was more conservative, both metrics show that most individuals had ADHD. Such high base rates in those seeking online help are reassuring because they lead to fewer false-positive diagnoses.

On the other hand, the high prevalence of ADHD makes it difficult to achieve a high κ coefficient, a measure of level of agreement beyond that expected by chance. Because the odds of chance agreement are high, achieving a high κ is difficult. Nevertheless, because our κ coefficients were significant, we can be assured that we have achieved a level of reliability that exceeds chance. Examining conditional probability metrics reveals why κ is understandably relatively low. Although sensitivity and PPV are very high, specificity and NPV are low. That these latter 2 metrics have led to a low κ is not concerning because they indicate that the ability to detect those without ADHD is low (low specificity), and when the online assessment assigns a negative diagnosis, it is usually wrong (low NPV). In contrast, high sensitivity shows that the online assessment can identify most self-referrals that have ADHD (high sensitivity), and when it assigns a diagnosis of ADHD, it is usually right (high PPV). These conclusions are, however, limited to self-referred samples with a high base rate of ADHD.

Limitations

The study population was drawn from individuals who were attracted to online adult ADHD assessment advertisements, and suspected they had ADHD, presumably because they were experiencing symptoms or impairment. Unsurprisingly, this sample had high levels of ADHD symptoms compared to published norms for the ASRS.²⁸ Because this sample was self-referred, it may be subject to selection and response bias and overrepresentation of individuals likely to seek mental health care, as well as those with higher socioeconomic status, education, and internet access. The sample was also mostly female. Adult ADHD clinical trials usually enroll either a small majority of males or an equal number of males and females.^29–31 As such, these results are likely not representative of the general adult population, and replication in a general population with a lower rate of ADHD symptoms may result in differing results related to sensitivity and specificity.

Due to feasibility issues, the clinical interview always preceded the online assessment because the online assessment automatically returned results to participants within 5 days of completion, and the research team determined that a minimum 7 day delay between assessments was necessary. Counterbalancing the 2 methods would have been preferable, although it seems unlikely that the assessment sequence that we used can explain the pattern of results given that the clinicians reviewing and rendering diagnoses for the online assessment were blind to the results of the clinical interviews. Additionally, interrater reliability for clinical assessment of adult ADHD, where reported, is fair to moderate but variable^32–35 and must be considered when evaluating these results. Further, the level of error in the clinical interview assessment was not measured and can only be assumed. Indeed, because the clinical interviews were completed online, it is possible that some relevant behavioral observations (eg, physical manifestations of ADHD symptoms) may have been missed. The virtual format was chosen in part due to feasibility and also in keeping with modern trends toward increased rates of online behavioral health care delivery.⁶ Finally, adjudication was only completed for cases in which the 2 assessment methods did not agree. As such, the degree to which adjudications would have agreed with cases in which the 2 assessment methods did agree is not certain.

Future Directions

Replication of this study among a sample of adults more representative of the general population (with a lower prevalence of ADHD) would expand our understanding of the performance of this online assessment. Additionally, this would better evaluate this assessment’s ability to distinguish between ADHD and other disorders. Further assessment of the population of adults seeking mental health care online would enhance the understanding of the patient journey and reasons for seeking assessment, as well other factors which may influence the accuracy and validity online mental health assessments. While there are many online ADHD assessments currently available, it is critical to distinguish between high- and low-quality assessments. While some platforms have drawn scrutiny for simplified assessments that may result in misdiagnosis or overdiagnosis, studied and validated assessments can be trusted to render reliable results without contributing to overdiagnosis and improper treatment. However, patients may not be able to determine which tools meet clinical standards, and thus, more transparency and regulation in this digital space are urgently needed. Finally, future studies should examine concordance rates between online assessments and clinical interviews for mental health conditions other than ADHD.

CONCLUSIONS

The online ADHD assessment performed well against the clinical interview in terms of accuracy, PPV, and sensitivity. Additionally, it was consistently more conservative in rendering an ADHD-positive diagnosis. In cases where individuals received an ADHD-negative result on the online assessment, further evaluation was always recommended. The results of this study can be used to improve online mental health assessments to more accurately discriminate and diagnose mental health conditions among the population of individuals seeking care online. Furthermore, this study provides information about the population of adults seeking mental health care online, a predominantly female group around age 35. Understanding the differences between this group and the general population is important to appropriately tailor online mental health care and diagnostic platforms to those that use them most. Finally, this study’s findings lend credibility to online mental health assessments as valid, accurate, and reliable. This approach to assessment has great potential to expand diagnostic accuracy in ADHD and could potentially be used to help address mental health care access barriers globally.

Article Information

Published Online: September 8, 2025. https://doi.org/10.4088/JCP.25m15846
© 2025 Physicians Postgraduate Press, Inc.
Submitted: February 19, 2025; accepted May 21, 2025.
To Cite: Herman BK, Faraone SV, Cutler AJ, et al. Validity of an online assessment of attention-deficit/hyperactivity disorder among a real-world sample of adults seeking web-based mental health care. J Clin Psychiatry 2025;86(3):25m15846.
Author Affiliations: Mentavi Health, Grand Rapids, Michigan (Herman, Ripper Lewis); Department of Psychiatry, Norton College of Medicine, SUNY Upstate Medical University, Syracuse, New York (Faraone, Cutler); Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York (Newcorn); Health Analytics, LLC, Clarksville, Maryland (LaFrance, Ruetsch).
Corresponding Author: Barry K. Herman, MD, MMM, Mentavi Health, 625 Kenmoor Ave SE, Ste. 301, Grand Rapids, MI 49546 ([email protected]).
Relevant Financial Relationships: Dr Herman is a full-time employee and shareholder of Mentavi Health and a shareholder of Thynk, Inc. In the past 24 months, Dr Faraone received income, potential income, travel expenses continuing education support and/or research support from Aardvark, Aardwolf, ADHD Online, AIMH, Akili, Attentiv, Axsome, Genomind, Ironshore, Johnson & Johnson/Kenvue, Kanjo, KemPharm/ Corium, Medice, Noven, Otsuka, Sandoz, Supernus, Sky Therapeutics, and Tris. With his institution, he has US patent US20130217707 A1 for the use of sodium-hydrogen exchange inhibitors in the treatment of ADHD. He also receives royalties from books published by Guilford Press: Straight Talk about Your Child’s Mental Health, Oxford University Press: Schizophrenia: The Facts and Elsevier: ADHD: Non-Pharmacologic Interventions. He is Program Director of www.ADHDEvidence.org and www.ADHDinAdults.com. Dr Faraone’s research is supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement 965381; NIH/NIMH grants U01AR076092, R01MH116037, 1R01NS128535, R01MH131685, 1R01MH130899, U01MH135970, Massachusetts General, Otsuka, Corium Pharmaceuticals, Supernus Pharmaceuticals, Tris Pharmaceuticals, Oregon Health & Science University. His continuing medical education programs are supported by The Upstate Foundation, Corium Pharmaceuticals, Noven, Tris, Pharmaceuticals, and Supernus Pharmaceuticals. Dr Cutler is a consultant and member of the following advisory boards: AbbVie, Acadia, Actinogen, Alfasigma, Alkermes, Anavex Life Sciences, Arrivo BioVentures, Autobahn Therapeutics, Axsome, Biogen, Biohaven, Boehringer Ingelheim, Brii Biosciences, Bristol Myers Squibb, Cerevel, Cognitive Research Corporation, Corium, Delpor, Evolution Research Group, 4M Therapeutics, Intra-Cellular Therapies, Ironshore Pharmaceuticals, J&J Innovative Medicine, Jazz Pharma, Karuna, Knight Therapeutics, LivoNova, Lundbeck, Luye Pharma, MapLight Therapeutics, MedAvante-ProPhase, Mentavi, Neumora, Neurocrine, Neuroscience Education Institute, NeuroSigma, Noven, Otsuka, PaxMedica, Relmada, Sage Therapeutics, Sirtsei Pharmaceuticals, Supernus, Teva, Thynk, Tris Pharma, Vanda Pharmaceuticals, and VistaGen. He is also a member of the speaker bureaus of AbbVie, Alfasigma, Alkermes, Axsome, Boehringer Ingelheim, Bristol Myers Squibb, Corium, Intra-Cellular Therapies, Ironshore Pharmaceuticals, J&J, Lundbeck, Neurocrine, Noven, Otsuka, Supernus, Teva, Tris Pharma, and Vanda Pharmaceuticals. Dr Cutler holds stock options/equity with 4M Therapeutics and is a member of the Data Safety Monitoring Boards of Alar Pharma, COMPASS Pathways, Freedom Biosciences, and Pain Therapeutics. Dr Newcorn is a consultant/advisory board member for AGB Pharma, Cingulate Therapeutics, Hippo T&C, Lumos, Medice, Mentavi Health, MindTension, Otsuka, Signant Health, and Supernus; received research support from Cingulate, MindTension, and Supernus; received honoraria for disease state lectures from Apsen and Otsuka; and served as a consultant for the US National Football League. Drs LaFrance and Ruetsch are full-time employees of Health Analytics, LLC and were contracted by Mentavi Health for this study. Ms Ripper Lewis is a full-time employee and shareholder of Mentavi Health.
Funding/Support: This study was funded by Mentavi Health, Grand Rapids, Michigan.
Previous Presentation: These data were presented at the 2025 ADHD World Congress; Prague, Czech Republic; May 8-11, 2025.
Acknowledgments: The authors thank Crystal Lamb and Eric White of Mentavi Health for their invaluable operational assistance in the study.
ORCID: Barry Herman: https://orcid.org/0000-0002-2875-5064; Stephen V. Faraone: https://orcid.org/0000-0002-9217-3982; Andrew Cutler: https://orcid.org/000-001-5800-0378; Jeffrey Newcorn: https://orcid.org/0000-0001-8993-9337; Emily LaFrance: https://orcid.org/0000-0002-5198-5722
Supplementary Material: Available at Psychiatrist.com.

Clinical Points

• Access to ADHD assessment and treatment in the US is variable and often inadequate. Online diagnostic evaluations may help address access barriers, but their accuracy and validity is unknown.
• These results lend credibility to online mental health assessments as reliable, accurate, and valid tools for diagnosing ADHD in adults and suggest that they hold significant potential to help address mental health care access barriers globally.

*See Rolison and Bloch (2023)27 for an example.

†In this article, the term assessment is used to describe a diagnostic process that combines structured self-report measures and descriptive, free text responses with clinical review. See Supplementary Material for additional information about assessment operationalization.

‡“ADHD-Combined” group includes “combined” and “other” presentations.

§Several participants attempted to complete the eligibility screening questionnaire multiple times, slightly skewing some of the study attrition counts. For this reason, only percentages are reported.