Primary Image

Rehabilitation Measures Database

MATRICS Consensus Cognitive Battery

Last Updated

Purpose

The MATRICS Consensus Cognitive Battery (MCCB) is intended to provide a relatively brief evaluation of key cognitive domains relevant to schizophrenia and related disorders.

Acronym MCCB

Area of Assessment

Attention & Working Memory
Cognition
Executive Functioning
Processing Speed
Reasoning/Problem Solving

Assessment Type

Performance Measure

Administration Mode

Paper & Pencil

Cost

Not Free

Actual Cost

$1275.00

Cost Description

? MCCB Kit (25 forms): $1,275.00
? MCCB Retest Packet-Additional Administrations (25 forms): $694.00
? MCCB Retest Packet-Alternate Form (25 forms): $694.00

CDE Status

Not a CDE--last searched 8/7/2024

Populations

Key Descriptions

  • Ten subtests are administered to evaluate 7 key cognitive domains, including speed of processing, attention/vigilance, working memory, verbal learning, visual learning, reasoning and problem-solving, and social cognition.
  • All raw scores for subtests are standardized to T scores (Mean = 59, SD = 10) based on a community sample.
  • The MCCB does not have minimum and maximum scores due to the use of T scores.
  • T scores are summed to calculate domain scores and an overall composite score for global cognition

Number of Items

MATRICS Consensus Cognitive Battery tests:
? Trail Making Test, Part A
? Brief Assessment of Cognition in Schizophrenia, symbol coding subtest
? Hopkins Verbal Learning Test—Revised, immediate recall subtest
? Wechsler Memory Scale, 3rd ed., spatial span task
? Letter-Number Span test
? Neuropsychological Assessment Battery, mazes subtest
? Brief Visuospatial Memory Test—Revised
? Category fluency test, animal naming
? Mayer-Salovey-Caruso Emotional Intelligence Test, managing emotions subtest
? Continuous Performance Test—Identical Pairs version

Supplemental tests:
? Neuropsychological Assessment Battery, daily living memory subtest
? Brief Assessment of Cognition in Schizophrenia, Tower of London subtest
? Neuropsychological Assessment Battery, shape learning subtest

Equipment Required

  • Paper and pencil
  • Table and chair
  • Computer and keyboard
  • Stopwatch
  • MCCB Kit

Time to Administer

1 to 1.5 hours

Required Training

Training Course

Required Training Description

? Qualification Level C
1. A degree from an accredited 4-year college or university in psychology, counseling, speech-language pathology, or a closely related field plus satisfactory completion of coursework in test interpretation, psychometrics and measurement theory, educational statistics, or a closely related area; or license or certification from an agency that requires appropriate training and experience in the ethical and competent use of psychological tests.
2. All qualifications for 1 plus an advanced professional degree that provides appropriate training in the administration and interpretation of psychological tests, or license or certification from an agency that requires appropriate training and experience in the ethical and competent use of psychological tests.
? One-day training
1. Didactic instruction and hands-on practice

Age Ranges

Adult

20 - 59

years

Instrument Reviewers

Kaiqi Zhou, MA, University of Wisconsin-Madison, rehabilitation psychology student under the direction of Timothy Tansey, PhD, Rehabilitation Psychology and Special Education Department, School of Education, University of Wisconsin-Madison

Kevin Fearn, MS, Shirley Ryan 香港六合彩即时开奖

 

ICF Domain

Body Function
Activity

Measurement Domain

Cognition

Professional Association Recommendation

None found—last searched on 8/7/2024

Considerations

  • The subtest for social cognition is not used to measure neurocognitive functioning
  • The MCCB may be better suited for clinically stable individuals with schizophrenia due to the characteristics of samples in existing study
  • Commercial version available in 44 languages, with language norms available for Simplified Chinese, German, Hindi, Italian, Japanese, Russian, Spanish (Central and South American), and Spanish (Spain)
  • Although the MCCB has been translated to other languages, its psychometric characteristics could be influenced by cultural differences
  • The MCCB could be utilized for adolescents and young adults (age 11-19) to identify specific abnormalities related to the development of schizophrenia, but clinicians need to refer to a standardized data set of healthy adolescent test performance for this assessment

Mental Health

back to Populations

Standard Error of Measurement (SEM)

Schizophrenia: (calculated from statistics in Keefe et al., 2011; n = 323; mean age = 43.1 (10.4) years; PANSS total score = 67.5 (11.7))

MCCB subscales

 SEM

Composite T-score

4.295

Speed of processing

5.545

Attention/vigilance

5.728

Working memory

5.132

Verbal learning

4.861

Visual learning

6.922

Reason/problem solving

4.460

Social cognition

6.300

 

Schizophrenia: (calculated from statistics in Georgiades et al., 2017; n = 2621; mean age = 42.7 (10.52) years)

MCCB subscales

 SEM

Neurocognitive composite

4.112

Cognitive composite

4.299

Speed of processing

5.318

Attention/vigilance

5.718

Working memory

5.147

Verbal learning

4.940

Visual learning

6.676

Reason/problem solving

4.561

Social cognition

6.418

Minimal Detectable Change (MDC)

Schizophrenia: (calculated from statistics in Keefe et al., 2011)

MCCB subscales

 MDC

Composite T-score

11.91

Speed of processing

15.37

Attention/vigilance

15.88

Working memory

14.23

Verbal learning

13.47

Visual learning

19.19

Reason/problem solving

12.36

Social cognition

17.46

 

Schizophrenia: (calculated from statistics in Georgiades et al., 2017)

MCCB subscales

 MDC

Neurocognitive composite

11.40

Cognitive composite

11.92

Speed of processing

14.74

Attention/vigilance

15.85

Working memory

14.27

Verbal learning

13.69

Visual learning

18.50

Reason/problem solving

12.64

Social cognition

17.79

Cut-Off Scores

Schizophrenia: (Kern et al., 2011; Schizophrenia or Schizoaffective Disorder: n = 176, mean age = 44.0 (11.2), male = 76%; Brief Psychiatric Rating Scale (BPRS) total score = 47.3 (13.6); Community Residents: = 300, mean age = 42.6 (11.6), male = 47%)

  • A T-score of ≤ 43.9 on Speed of Processing distinguishes persons with schizophrenia from community residents (sensitivity 66.4%; specificity 87.6%). Discrimination was further improved with the additional consideration of Social Cognition performance.
  • Among those with schizophrenia, a T-score of ≤ 30.5 on Speed of Processing distinguishes unemployed persons from employed persons (sensitivity 89.4%; specificity 34.5%). Discrimination was further improved with the additional consideration of Visual Learning and Attention/Vigilance.

Major Depressive Disorder: (Liang et al., 2020; n = 48; mean age = 39.7 (13.4); HDRS-17 total scores ≥ 18; Chinese sample and translation of MCCB)

Cut-off value, Sensitivity, and Sensitivity by domain for patients with major depressive disorder

Domain

Cut-off

Sensitivity

Specificity

Speed of processing

40

0.778

0.771

Attention/vigilance

37

0.667

0.743

Working memory

39

0.444

0.914

Verbal learning

40

0.622

0.800

Visual learning

45

0.644

0.857

Reason/problem solving

42

0.800

0.886

Social cognition

34

0.667

0.657

 

 

Normative Data

 

Major Depressive Disorder: (Liang et al., 2020)

Mean (SD) T-scores by MCCB domain and patient group

 

Domain

BD

(= 43)

HC

(= 35)

MDD

(= 48)

SCH

(= 32)

Speed of processing1

32.9 (12.8)

46.1 (12.3)

27 (14.2)

20 (14.6)

Attention/vigilance1

36 13.9)

43.9 (10.1)

31.7 (11.4)

28.3 (10.7)

Working memory1

41.4 (10)

47.6 (7.8)

39.7 (11.2)

36.3 (11)

Verbal learning1

39.8 (8)

49.4 (11.7)

37.2 (9.4)

32.6 (6.5)

Visual learning1

43 (10.2)

52.6 (8.6)

39.1 (12.2)

35.7 (12.6)

Reasoning/problem solving1

37.5 (7.5)

50 (7.9)

36.2 (6)

33.6 (4.9)

Social cognition2

37.5 (9.5)

37.6 (10.4)

30.9 (10.5)

29.5 (10.2)

狈辞迟别—BD: bipolar disorder in the euthymic phase; HC: healthy control; MDD: major depressive disorders; SCH: schizophrenia

1< 0.001

2p < 0.005

Major Depressive Disorder in Young Patients: (Liang et al., 2021; Y-MDD: = 50, mean age = 17.9 (2.7) years, female = 34 (66%); Y-BD: = 38, mean age = 18.9 (3.0) years, female = 20 (52.6%); HC: = 51, mean age = 17.7 (3.2) years; age range for all subjects = 13-24; Chinese sample and translation of MCCB)

Mean (SD) T-scores by MCCB domain and young patient group

Domain

Y-MDD

Y-BD

HC

Speed of processing1

35.3(12.6)

28.3 (13.7)

43.8 (10.1)

Attention/vigilance2

37.2 (13.0)

28.4 (14.2)

38.6 (12.7)

Working memory2

44.2 (12.8)

36.2 (8.4)

45.4 (10.0)

Verbal learning1

44.6 (9.2)

39.4 (9.3)

52.2 (11.5)

Visual learning1

42.0 (13.4)

40.1 (10.7)

47.4 (10.7)

Reasoning/problem solving1

43.2 (10.9)

35.6 (7.6)

49.3 (8.9)

Social cognition

46.7 (15.3)

41.3 (15.2)

44.3 (10.2)

狈辞迟别—Y-MDD: young patients with major depressive disorders; Y-BD: young patients with bipolar disorder in the euthymic phase; HC: healthy controls.

1< 0.001

2p < 0.005

 

Test/Retest Reliability

Schizophrenia: 

  • Excellent test-retest reliability for overall composite score (ICC = 0.90) (Nuechterlein et al., 2008; = 167; mean age = 44.0 (11.2); mean educational level = 12.4 (2.4) years; male = 76%; subjects diagnosed with schizophrenia or schizoaffective disorder, depressed type; tests at 4-week interval) 
  • Acceptable test-retest reliability for composite T-score (ICC = 0.88) (Keefe et al., 2011)
  • Acceptable test-retest reliability for cognitive composite T-score (ICC = 0.88) (Georgiades et al., 2017)
    • Acceptable test-retest reliability for neurocognitive composite T-score (all domains except for Social Cognition) (ICC = 0.88)

Major Depressive Disorder: (Liang et al., 2020)

  • Poor to Acceptable test-retest reliability (= 0.59 to 0.81)

 

Internal Consistency

Schizophrenia: (Burton et al., 2013; = 183, mean age = 44.45 (11.47), male = 127 (69.4%), community-dwelling outpatients diagnosed with schizophrenia or schizoaffective disorder) 

  • Adequate internal consistency for one-factor model (Cronbach's alpha = 0.76)
  • Poor to Adequate internal consistency for 3-factor model (Cronbach's alpha = 0.52 to 0.70)

Major Depressive Disorder: (Liang et al., 2020)

  • Adequate to excellent internal consistency of Total MCCB for MDD patients at baseline (α = 0.83) and one month later (α = 0.74)
  • Poor to excellent internal consistency for MDD patients at baseline for the four cognitive domains: Speed of processing (α = 0.69), Verbal learning (α = 0.88), Visual learning (α = 0.90), and Attention/vigilance (α = 0.90) (Working memory, Reason/problem solving, and Social cognition not tested)
  • Poor to excellent internal consistency for MDD patients for the four cognitive domains at one month: Speed of processing (α = 0.58), Verbal learning (α = 0.87), Visual learning (α = 0.89), and Attention/vigilance (α = 0.84) (Working memory, Reason/problem solving, and Social cognition not tested)

Major Depressive Disorder in Young Patients: (Liang et al., 2021)

  • Adequate to excellent internal consistency of Total MCCB in Y-MDD patients at baseline (α = 0.79) and two weeks later (α = 0.83)
  • Poor to excellent internal consistency in Y-MDD patients at baseline for the four cognitive domains: Speed of processing (α = 0.61), Verbal learning (α = 0.83), Visual learning (α = 0.86), and Attention/vigilance (α = 0.87) (Working memory, Reason/problem solving, and Social cognition not tested)  
  • Poor to excellent internal consistency in Y-MDD patients for the four cognitive domains two weeks later: Speed of processing (α = 0.58), Verbal learning (α = 0.87), Visual learning (α = 0.84), and Attention/vigilance (α = 0.90) (Working memory, Reason/problem solving, and Social cognition not tested)

 

Criterion Validity (Predictive/Concurrent)

Concurrent Validity:

Schizophrenia: 

  • Adequate correlations between the MCCB composite score with the UCSD Performance-Based Skills Assessment-2 (UPSA-2) total score (r = 0.42) (Silverstein et al., 2010; = 155; male = 102 (65%); outpatients or partial hospital patients with schizophrenia)
  • Excellent correlations with the UCSD Performance-Based Skills Assessment-Brief (UPSA-B) (r = 0.60) (Keefe et al., 2011)
  • Adequate correlations of all three MCCB factors with the UCSD Performance-Based Skills Assessment-Brief (UPSA-B) total score (Burton et al., 2013):
    • Speed of processing (= 0.461)
    • Attention and working memory (r  = 0.468)
    • Learning (= 0.465) 

Bipolar Disorder with Psychosis: (Sperry et al., 2015; Bipolar Disorder: n = 56, mean age = 30.5 (8.16), female = 57%; Schizophrenia: n = 37, mean age = 33.9 (11.12), female = 13.5%; Healthy Controls: n = 57, mean age = 25.7 (6,42), female = 59.6%)

  • Adequate correlation between the Positive and Negative Syndrome Scale (PANSS) and the Processing Speed domain (= -0.38, < 0.01)
  • Poor correlations between the PANSS and MCCB domains Working Memory (= -0.29, < 0.01), Verbal Memory (= -0.24, < 0.05), and the Overall Composite score (= -0.28, < 0.01)

Major Depressive Disorder: (Liang et al., 2020)

  • Adequate to excellent correlations between MCCB domains and total scores on the Montreal Cognitive Assessment (MoCA) at baseline (r = 0.46 to 0.74)
  • Adequate correlations between Speed of processing and verbal learning domains of the MCCB and total scores on the MoCA one month later (= 0.45 and 0.47, respectively)

 

Major Depressive Disorder in Young Patients: (Liang et al., 2021)

  • Adequate correlations between the recall domain of the MoCA and the T-scores of MCCB domains:
    • Speed of processing (r  = 0.47, < 0.05)
    • Reasoning and problem solving (= 0.53, < 0.01)
    • Social cognition (r  = 0.41, < 0.05)

 

 

Construct Validity

Convergent Validity:

Schizophrenia: (Kaneda et al., 2013; n = 37; mean age = 38.4 (11.2); BPRS score = 41.0 (7.1); Japanese sample and translation: MCCB-J)

  • Poor to excellent correlations between the MCCB-J and the Brief Assessment of Cognition in Schizophrenia (BACS-J) (r = 0.14 to 0.87)

Discriminant Validity:

Bipolar Disorder: (Van Rheenen et al., 2014; Bipolar Disorder: n = 50, mean age = 38, male = 16; Healthy Controls: n = 52, mean age = 34, male = 20; Australia sample)

  • There is a significant group effect for cognitive domain performance overall among MCCB domain scores (F (7, 90) = 2.47, p < 0.05, Wilk’s λ = 0.84)
  • Bipolar Disorder group showed significantly reduced performance on Speed of Processing (F (1, 96) = 7.33, p < 0.01), Working Memory (F (1, 96) = 9.78, p < 0.01), and Visual Learning (F (1, 96) = 8.19, p < 0.01)

Bipolar Disorder: (Russo et al., 2014; = 173 (64 patients with BD: mean age = 41.2 (10.5), male = 50% and 109 healthy controls: mean age = 37.9 (11.6), male = 53.2%)

  • Significantly worse performance for patients than healthy controls for all MCCB cognitive domains except for visual learning (= 0.178) and reasoning/problem solving (= 0.505)
  • Significant negative correlations for healthy controls between the TEMPS-A irritability temperament and the MCCB domains attention (= -0.315, = 0.035) and social cognition (= -0.386, = 0.009)
  • Significant positive correlations for BD patients between ratings on the cyclothymic temperament scale and MCCB scales processing speed (= 0.432, = 0.002), working memory (= 0.379, = 0.009), reasoning/problem solving (= 0.438, = 0.002), and global cognition as measured by an overall composite score (= 0.467, = 0.001)     

 

Major Depressive Disorder: (Liang et al., 2020; Major Depressive Disorder (MDD) n = 48; mean age = 39.7 (13.4); HDRS-17 total scores ≥ 18; Bipolar Disorder (BD) n = 43, mean age = 36.3 (11.9); Schizophrenia (SCH) n = 32, mean age = 42.0 (13.1); Healthy-control n = 35, mean age = 39.2 (14.2); Chinese sample and translation of MCCB)

  • Significant differences found in every domain among the four patient groups, including Speed of Processing (F = 10.93, p < 0.001), Attention/Vigilance (F = 10.31, p < 0.001), Working Memory (F = 5.78, p < 0.001), Verbal Learning (= 12.27, p < 0.001), Visual Learning (F = 6.61, p < 0.001), Reasoning and Problem Solving (F = 16.67, p < 0.001), and Social Cognition (F = 3.25, p = 0.002).
  • The scores for Attention/Vigilance (F = 4.16, p = 0.002), Verbal Learning (F = 3.37, p = 0.007), and Visual Learning (F = 2.33, p = 0.046) were lower in the MDD group than in the BD group, but higher than in the SCH group.

 

 

 

Content Validity

  • Subtests were selected based on discussion at the MATRICS consensus conference with 130 schizophrenia experts and professionals (Nuechterline et al., 2008, p. 203).
  • The resulting items were then tested through the MATRICS psychometric and standardization study using a stratified sample of 300 individuals from the general community at five sites in differing geographic regions (Kern et al., 2008, p. 214).

Face Validity

The MCCB is considered the FDA gold standard outcome measure for assessing cognitive treatment effects in schizophrenia clinical trials (Georgiades et al., 2017, p. 172).

Floor/Ceiling Effects

Schizophrenia: (Nuechterline et al., 2008)

  • Excellent: There were no noticeable ceiling effects or constrictions of variance at the second testing.

Responsiveness

Schizophrenia: (Keefe et al., 2011)

  • Small change for composite T-score (Cohen’s d = 0.18a) (Keefe et al., 2011)
  • Small changes for domains range from d = 0.20a for Speed of Processing to = 0.03a for Social Cognition (Keefe et al., 2011)
  • Small changes for overall cognitive composite (= 0.15b) and neurocognitive composite (= 0.17b) (Georgiades, et al., 2017)
  • Small changes for domains range from d = 0.18b for Speed of Processing to = 0.02b for Social Cognition (Georgiades, et al., 2017)

aEffect size for paired sample = (Baseline?Screening)/SD Screening.

bEffect size for paired sample = (Baseline?Screening)/pooled SD from Screening and Baseline.

 

Bibliography

Bo, Q., Mao, Z., Li, X., Wang, Z., Wang, C., & Ma, X. (2017). “Use of the MATRICS consensus cognitive battery (MCCB) to evaluate cognitive deficits in bipolar disorder: A systematic review and meta-analysis.” PLoS ONE, 12(4).

Burton, C. Z., Vella, L., Harvey, P. D., Patterson, T. L., Heaton, R. K., & Twamley, E. W. (2013). “Factor structure of the MATRICS Consensus Cognitive Battery (MCCB) in schizophrenia.” Schizophrenia Research, 146(1–3), 244–248.

Georgiades, A., Davis, V. G., Atkins, A. S., Khan, A., Walker, T. W., Loebel, A., Haig, G., Hilt, D. C., Dunayevich, E., Umbricht, D., Sand, M., & Keefe, R. S. E. (2017). “Psychometric characteristics of the MATRICS Consensus Cognitive Battery in a large pooled cohort of stable schizophrenia patients.” Schizophrenia Research, 190, 172–179.

Green, M. F., Nuechterlein, K. H., Kern, R. S., Baade, L. E., Fenton, W. S., Gold, J. M., ... & Marder, S. R. (2008). Functional co-primary measures for clinical trials in schizophrenia: results from the MATRICS Psychometric and Standardization Study. American Journal of Psychiatry, 165(2), 221-228.

Kaneda, Y., Ohmori, T., Okahisa, Y., Sumiyoshi, T., Pu, S., Ueoka, Y., Takaki, M., Nakagome, K., & Sora, I. (2013). “Measurement and treatment research to improve cognition in Schizophrenia Consensus Cognitive Battery: Validation of the Japanese version.” Psychiatry and Clinical Neurosciences, 67(3), 182–188.

Keefe, R. S. E., Fox, K. H., Harvey, P. D., Cucchiaro, J., Siu, C., & Loebel, A. (2011). “Characteristics of the MATRICS Consensus Cognitive Battery in a 29-site antipsychotic schizophrenia clinical trial.” Schizophrenia Research, 125(2–3), 161–168.

Kern, R. S., Gold, J. M., Dickinson, D., Green, M. F., Nuechterlein, K. H., Baade, L. E., ... & Marder, S. R. (2011). “The MCCB impairment profile for schizophrenia outpatients: results from the MATRICS psychometric and standardization study.” Schizophrenia Research, 126(1-3), 124-131.

Kern, R. S., Nuechterlein, K. H., Green, M. F., Baade, L. E., Fenton, W. S., Gold, J. M., Keefe, R. S. E., Mesholam-Gately, R., Mintz, J., Seidman, L. J., Stover, E., & Marder, S. R. (2008). “The MATRICS Consensus Cognitive Battery, part 2: Co-norming and standardization.” The American Journal of Psychiatry, 165(2), 214–220.

Kuo, S. S., Wojtalik, J. A., Mesholam-Gately, R. I., Keshavan, M. S., & Eack, S. M. (2020). “Transdiagnostic validity of the MATRICS Consensus Cognitive Battery across the autism-schizophrenia spectrum.” Psychological Medicine, 50(10), 1623–1632.

Liang, S., Yu, W., Ma, X., Luo, S., Zhang, J., Sun, X., Luo, X., & Zhang, Y. (2020). “Psychometric properties of the MATRICS Consensus Cognitive Battery (MCCB) in Chinese patients with major depressive disorder.” Journal of Affective Disorders, 265, 132–138.

Liang, S., Xing, X., Wang, M., Wei, D., Tian, T., Liu, J., & Sha, S. (2021). The MATRICS Consensus Cognitive Battery: Psychometric Properties of the Chinese Version in Young Patients With Major Depressive Disorder. Frontiers in Psychiatry, 12:745486.

Mohn, C., Lystad, J. U., Ueland, T., Falkum, E., & Rund, B. R. (2017). “Factor analyzing the Norwegian matrics consensus cognitive battery.” Psychiatry and Clinical Neurosciences, 71(5), 336–345.

Nitzburg, G. C., DeRosse, P., Burdick, K. E., Peters, B. D., Gopin, C. B., & Malhotra, A. K. (2014). “MATRICS cognitive consensus battery (MCCB) performance in children, adolescents, and young adults.” Schizophrenia Research, 152(1), 223–228.

Nuechterlein, K. H., Green, M. F., Kern, R. S., Baade, L. E., Barch, D. M., Cohen, J. D., Essock, S., Fenton, W. S., Frese, F. J., III, Gold, J. M., Goldberg, T., Heaton, R. K., Keefe, R. S. E., Kraemer, H., Mesholam-Gately, R., Seidman, L. J., Stover, E., Weinberger, D. R., Young, A. S., … Marder, S. R. (2008). “The MATRICS consensus cognitive battery, part 1: Test selection, reliability, and validity.” The American Journal of Psychiatry, 165(2), 203–213.

Russo, M., Mahon, K., Shanahan, M., Ramjas, E., Solon, C., Braga, R. J., Burdick, K. E. (2014). Affective temperaments and neurocognitive functioning in bipolar disorder. Journal of Affective Disorders, 169, 51-56.

Silverstein, S., Jaeger, J., Donovan-Lepore, A.-M., Wilkniss, S., Savitz, A., Malinovsky, I., Hawthorne, D., Raines, S., Carson, S., Marcello, S., Zukin, S., Furlong, S., & Dent, G. (2010). “A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries.” Journal of Clinical & Experimental Neuropsychology, 32(9), 937.

Sperry, S. H., O’Connor, L. K., Ongur, D., Cohen, B. M., Keshavan, M. S., & Lewandowski, K. E. (2015). Measuring cognition in bipolar disorder with psychosis using the MATRICS Consensus Cognitive Battery. Journal of the International Neuropsychological Society, 21, 468-472.

Van Rheenen, T. E., & Rossell, S. L. (2014). “An empirical evaluation of the MATRICS Consensus Cognitive Battery in bipolar disorder.” Bipolar Disorders, 16(3), 318–325.