Cross-Cultural Adaptation and Validation of a Generic Acceptability Questionnaire to Spanish

Introduction

The successful implementation of new health interventions depends on multiple factors, including the relative advantages of the new intervention over existing ones, its ease of use, and users' health literacy [1]. Other relevant factors include those associated with health professionals and the social, political, legal, and economic context [2,3]. These elements collectively determine the implementability of an intervention, which refers to its likelihood of adoption in routine practice and user behaviors across different settings over time [4].

The conceptual framework proposed by Klaic et al. for evaluating the implementability of health interventions encompasses five critical components: acceptability, fidelity, feasibility, scalability, and sustainability [4]. Acceptability is the first component, which allows the rest to be achieved hierarchically and consecutively. Therefore, acceptability is an essential prerequisite for successfully implementing any health intervention [4,5].

Acceptability is a multifaceted construct that reflects the extent to which individuals, delivering or receiving a specific health intervention, consider it appropriate based on cognitive and emotional responses [5]. It is an emergent property of a complex and adaptive system of interactive elements such as knowledge, beliefs, and attitudes [6]. These elements are deeply contextualized and interconnected with social and cultural norms [6].

Significant outcomes of a health intervention, such as patient engagement and effectiveness, are determined by acceptability. Patients who perceive an intervention as acceptable are likelier to adhere to recommendations, potentially enhancing the benefits obtained [6]. It has been hypothesized that acceptability may also indirectly influence outcomes through pre-existing attitudes [7]. Conversely, interventions not perceived as acceptable are less likely to succeed, even if they are clinically effective [4].

Conducting acceptability studies is essential to identify factors that may facilitate or hinder the implementation of health interventions, thereby enabling necessary adaptations and improvements [8]. These assessments also provide insights into the preferences and needs of the target population, ensuring that interventions are relevant and context-specific [9]. For these reasons, the guidelines for the design and evaluation of health interventions by the UK Medical Research Council have included, as a recommendation, the assessment of acceptability [10,11]. However, these guidelines do not explain how it should be assessed.

In response to the need to evaluate the acceptability of health interventions, Sekhon et al. developed a generic questionnaire based on the theoretical framework of acceptability (TFA) [12]. This instrument, consisting of 7 Likert-type items, is designed to evaluate the acceptability constructs of the TFA: affective attitude, burden, ethicality, perceived effectiveness, intervention coherence, self-efficacy, and opportunity costs (Table 1) [5]. Additionally, the questionnaire incorporates an extra item that measures general acceptability. As a generic instrument, the wording of the items includes sections that can be replaced with the name of the intervention being investigated and related specific actions. Thus, the instrument can be used to study a wide range of interventions.

Table 1. Definitions of the component constructs of the Theoretical Framework of Acceptability (TFA) proposed by Sekhon et al. [5].

To date, there are no instruments in Spanish to evaluate the acceptability of health interventions. Spanish-speaking and developing countries in Latin America, which have limited resources, could particularly benefit from such tools. Instruments such as the generic TFA questionnaire could help direct efforts and resources towards more acceptable interventions and, therefore, with greater implementability. Studying the acceptability of health interventions in terms of each country's social norms and specific cultural contexts would allow for opportune and relevant adaptations to increase the chances of successful implementation. Therefore, this study aimed to adapt and validate the generic TFA questionnaire to Spanish.

Methods

A prospective observational two-phase study was conducted for the cross-cultural adaptation of the questionnaire and its subsequent validation. The ethical-scientific committees of the Servicio de Salud Valparaíso Antonio and the Universidad Santo Tomás approved the study protocol. All participants were required to sign an informed consent form. DeepL Translate and Write softwares [DeepL.com] were used for generating English translations and identifying potential improvements in the writing style of this manuscript.

Cross-cultural adaptation

The generic TFA questionnaire was initially translated independently by two bilingual English-Spanish speakers (native Spanish speakers). A healthcare professional synthesized the translations, resolving discrepancies between both documents to obtain a preliminary version of the questionnaire in Spanish. Minor differences related to the use of synonymous words were noted, and there were no grammatical discrepancies. This document was then back-translated into English by a bilingual Spanish-English speaker (native English speaker). Subsequently, an expert committee comprised a professional translator, a linguist, and a bilingual Spanish-English (native Spanish speaker) healthcare professional. This committee reviewed the direct and back translations and the original English version of the generic TFA questionnaire and developed a preliminary Spanish version of the questionnaire (Table 2).

Table 2. Adaptation to Spanish of the items from the generic TFA questionnaire proposed by Sekhon et al. [5].

The original version of the generic TFA questionnaire includes two alternative questions for the items of affective attitude and ethicality constructs because the authors proposed to make a flexible and adaptable instrument based on the specific intervention being evaluated and the target population to ensure its comprehension. Thus, the two questions for these items can be used interchangeably [12]. However, the expert committee determined that, in Spanish, the question alternatives explore different aspects. For this reason, they referenced the respective conceptual definitions according to the TFA and proposed that the item on affective attitude should be evaluated through the question “¿Cuán cómodo(a) se sintió [un comportamiento; por ejemplo, participar en] [la intervención]?” (How comfortable were you [did you feel] with [behavior; i.e., participate in engaging with] [the intervention]?), and the ethicality item through the question “Existen consecuencias morales o éticas debido a [un comportamiento; por ejemplo, participar en] [la intervención]” (There are moral or ethical consequences due to [behavior; i.e. participate in] [the intervention]?).

This preliminary version was piloted in a pilot group of 50 older adults who participated in a group physical activity workshop organized by the Municipality of Algarrobo. Their opinions were collected regarding their understanding of the questions. Practically all subjects (n = 48) stated they adequately understood the questions, so no further adjustments or modifications were made.

Questionnaire validation

The final version of the generic TFA questionnaire was administered to a sample of 194 older adults aged between 68 and 76 years. A non-probabilistic convenience sampling was used across four primary healthcare centers (PHCs) in the Valparaíso Region, Chile. Inclusion criteria required participants to have a normal cognitive status. This was verified through scores equal to or higher than 13 points on the abbreviated version of the Mini-Mental State Examination (MMSE), a tool routinely used in Chilean PHCs for dementia screening [13]. Detailed descriptive statistics of the sample are presented in Table 3.

Table 3. Descriptive statistics of the sample (n=194).

The questionnaire was administered after the patients participated in two clinical procedures at the PHCs, one implemented as part of a pilot study and the other implemented as a standard practice. The former was a mHealth-based auditory screening evaluation using the uHear app [14]. This application allows for auditory evaluation using mobile devices with iOS operating systems. For this purpose, 10.2-inch 64 GB iPads with Sony ZX110 headphones were used. The assessment was carried out by a professionally trained nurse for these purposes within the facilities of the PHCs.

The second procedure was the Preventive Medicine Examination for Older Adults - EMPAM (from Spanish: Examen de Medicina Preventiva del Adulto Mayor). This examination is conducted annually and is one of the standard healthcare procedures carried out in PHCs for older adults in Chile [15]. It was implemented in 1999 to assess health and functionality, identify and manage risk factors for loss of functionality, and serve as input for developing care and follow-up plans for the elderly population. The examination lasts approximately 60 minutes and includes measurements of vital signs, anthropometry, physical activity history, diet, pharmacological treatments, pathologies, vaccinations, functional, cognitive, depression, dependency, and fall risk assessments. A nursing professional also carried out this procedure.

Finally, since the generic TFA questionnaire is designed to be used in various health interventions, it contains predefined sections (enclosed in “[]”) in the question statements that must be replaced with the specific actions of the intervention being studied, the medical condition it targets, and the name of the intervention. For the questions directed at uHear, the action was completed with “haber sido evaluado” (have been assessed), the condition was “pérdida auditiva” (hearing loss) and, and the name used for the intervention was “evaluación auditiva” (hearing assessment). For the questions directed at EMPAM, the action was “haber sido evaluado” (have been assessed), the condition was “adultos mayores” (older adults), and, as this is an examination with which this population is highly familiar, the name EMPAM was used as the name of the intervention.

Statistical Analysis

An Exploratory Factor Analysis (EFA) was conducted to assess construct validity. The suitability of the data was verified through the Kaiser-Meyer-Olkin (KMO) measure and Bartlett's test of sphericity. Spearman's rank correlation coefficient (rho) was used to analyze item correlations. The assumption of multivariate normality was verified through Mardia's tests of skewness (mSkewness) and kurtosis (mKurtosis). To determine the number of factors to extract, the Kaiser-Guttman criterion, visual inspection of scree plots, and an estimation of a parallel analysis (10.000 replications) were utilized. Oblique oblimin rotation was used for factor interpretation. Factor loadings greater than 0.3 were considered significant [16]. Finally, McDonald's omega (ω) was used as a measure of internal consistency, as it is a more sensitive and less biased indicator than Cronbach's alpha [17]. The analyses were performed using the base code and the "psych" library of RStudio software and STATA software v16.1.

Results

Appropriateness of factor analysis

Table 4 shows the correlation matrix for the items of the generic TFA questionnaire, both for uHear and EMPAM. In both cases, significant correlations of weak to moderate magnitude were evident. The pattern of correlations was similar for both procedures. The frequency of responses for each option is presented in Appendix 1.

Table 4. Correlation matrix for the items of the generic TFA questionnaire.

The items obtained a KMO=0.500 for uHear and KMO=0.653 for EMPAM, indicating that, in both cases, there is a proportion of variance among all observed variables that could be explained by underlying factors, which suggests that a factorial analysis could be useful with the data. Bartlett's test, for both uHear (χ² ₍₂₁₎=149.105; p<0.001) and EMPAM (χ²(21)=200.146; p<0.001), indicates that the respective correlation matrices are not identity matrices and, therefore, that a factorial analysis could be useful. The uHear data did not exhibit multivariate normality with an mSkewness = 59.720 (χ² ₍₈₄₎ = 1968.390; p<0.001) and an mKurtosis = 146.753 (χ² ₍₁₎ = 2700.061; p<0.001). The EMPAM data also did not present multivariate normality with an mSkewness = 55.034 (χ2₍₈₄₎=1813.910; p<0.001) and an mKurtosis = 137.931 (χ² ₍₁₎=2161.210; p<0.001). The general acceptability item was not considered in this or further analysis, as it does not respond to a specific TFA construct.

Exploratory factor analysis

Regarding factor extraction, the Kaiser-Guttman criterion was initially used, which suggests determining the number of factors to extract by considering those with eigenvalues greater than 1 [18]. Secondly, a visual inspection of the scree plot was utilized, according to which extraction should stop at the point where the slope tends to "level off" or when there are no more "abrupt drops" [19]. Finally, a parallel analysis was used, in which a correlation matrix was calculated from a random selection of the dataset, and the eigenvalues of the correlation matrix were computed [20]. This criterion retains factors from the observed data with eigenvalues greater than those generated by their counterparts in the parallel analysis.

In Figure 1, the result of the three analysis criteria can be observed. For uHear, three factors with eigenvalues greater than one are observed; however, both the scree plot slope and the parallel analysis suggest the presence of two factors. Regarding EMPAM, it is observable that only two factors have eigenvalues greater than 1, and both the scree plot slope analysis and the parallel analysis suggest the same. Considering that most analyses support it, the decision was made to extract two factors. The detailed results of the parallel analysis can be found in Appendix 2.

Figure 1. Screeplot and parallel analysis of generic TFA questionnaire with 10.000 replications.

Given that the dataset violated the assumption of multivariate normality, the least squares method, known for its robustness against asymmetric distributions, was used for factor extraction. An oblique oblimin rotation was conducted for the interpretation of the factor loadings. For the uHear data, it is observable that the items for affective attitude, perceived effectiveness, intervention coherence, and self-efficacy had significant factor loadings for the first factor, while the items for ethicality and opportunity costs had significant factor loadings for the second factor. The item for burden did not have a relevant loading for any of the factors, and none of the other items had significant shared loadings for both factors. The same pattern of factor loadings was obtained for the EMPAM dataset. In Table 5, the detail of the factor loadings can be observed. For uHear, factor 1 explained 15.73%, and factor 2 explained 14.40% of the variance, and both factors had a negligible correlation of rho = 0.159. For EMPAM, factor 1 explained 22.64%, and factor 2 explained 15.13% of the variance, and both factors had a negligible correlation of rho=0.057.

Table 5. Factor loadings after oblique rotation for a 2-factor solution, with and without burden item.

The burden item did not have significant loadings for any factor. Therefore, following standard recommendations in this type of analysis, it was decided to eliminate it and re-estimate the EFA [16]. The rest of the items maintained the distribution of their factor loadings for the uHear and EMPAM data. The items for affective attitude, perceived effectiveness, intervention coherence, and self-efficacy obtained significant loadings for the first factor. The items for ethicality and opportunity costs obtained significant loadings for the second factor. For uHear, factor 1 explained 18.96%, and factor 2 explained 14.97% of the variance, and both factors had a negligible correlation of rho = 0.157. For EMPAM, factor 1 explained 26.28%, and factor 2 explained 20.44% of the variance, and both factors had a negligible correlation of rho = 0.090. The final version of the questionnaire can be found in the Appendix 3.

Internal consistency

For the uHear dataset, the Spanish version of the generic TFA questionnaire achieved an acceptable reliability (ω = 0.60), as did the second factor (ω = 0.63). In contrast, the first factor exhibited poor reliability (ω = 0.56). Removing the burden item, these results showed virtually no changes, both for the complete instrument (ω = 0.61) and for the first (ω = 0.57) and second factor (ω = 0.60). For the EMPAM dataset, the Spanish version of the instrument obtained an acceptable reliability (ω = 0.65), as did the first factor (ω = 0.72), whereas the second factor exhibited poor reliability (ω = 0.54). Upon removing the burden item, the complete instrument (ω = 0.73), as well as the first (ω = 0.72) and second factor (ω = 0.74), achieved acceptable reliability.

Discussion

This study aimed to adapt and validate the generic TFA questionnaire to Spanish. The instrument was applied following two interventions at PHCs: a pilot auditory screening procedure and a standard preventive medicine procedure for older adults. Although independent EFAs were conducted for each procedure, the same items were retained, and the same two-factor solution was produced. This indicates that the underlying factors are stable despite referring to different procedures, suggesting a high external reliability.

The first factor includes affective attitude, perceived effectiveness, intervention coherence, and self-efficacy, reflecting aspects related to attitudes toward the intervention and confidence in it and in one's capacity to carry out the required actions. This could be named "knowledge and attitudes toward the intervention." On the other hand, the second factor, comprising the items of ethicality and opportunity costs, refers to costs in terms of benefits, gains, or values forgone by participating in the intervention. For this reason, it could be termed "cost of the intervention".

These results are similar to those reported by Haydon et al., who utilized the TFA to develop the Digital Health Acceptability Questionnaire (DHAQ), an instrument designed to assess the acceptability of telemedicine services via phone and video conferencing [21]. Like in our study, the authors found that the DHAQ underlying structure is comprised of two factors: "attitude towards the service as a means to meet healthcare needs and affective attitude" and "individual's capacity and effort to use the service." The first factor includes items that assess the constructs of affective attitude, ethicality, and perceived effectiveness. In contrast, the second factor is composed of items that assess the constructs of burden, self-efficacy, and opportunity costs. These factors retained their structure across both types of interventions, thereby also demonstrating good external reliability.

On the other hand, Timm et al. utilized the TFA as a basis for developing a specific instrument to evaluate a telephonic coaching intervention for preventing and managing type 2 diabetes [22]. This questionnaire was found to be composed of three factors: "affective attitude," "coherence and understanding," and "perceived burden." The first was made up of questions that probed into the constructs of affective attitude, perceived effectiveness, intervention coherence, self-efficacy, and ethicality. The second factor consisted of questions related to intervention coherence and burden. Finally, the third factor was composed of questions designed to assess the construct of opportunity costs.

Our findings, along with those from the previously mentioned questionnaires, suggest that the constructs defined by the TFA are not necessarily independent of one another. The knowledge, beliefs, and attitudes forming the complex and adaptive system from which acceptability emerges are interactive [6]. In this interaction, some of these components may overlap or be interrelated [23]. Furthermore, it has been noted that the response formats of instruments probing latent variables could encourage this overlap [24,25].

We decided to remove the “burden” item due to its lack of significant factor loading on any of the retained factors. This decision is similar to what was observed for the DHAQ, in which items related to “intervention coherence” were removed [21]. At first glance, both instruments omitted one of the central constructs of acceptability. However, the interaction between the components suggests that, even though their specific items were removed, the underlying constructs might still be represented by a higher-order latent variable determined by the identified factors. This interpretation is supported by the observation in the instrument by Timm et al., where items intended to assess the same construct demonstrated significant factor loadings across different factors [22]. This phenomenon reflects the inherent complexity of measuring multidimensional constructs and the importance of considering the interrelation among different aspects of acceptability.

Removing certain items from the TFA constructs in our study highlights that acceptability is not a universal phenomenon. Its components are closely linked to social and cultural norms and, therefore, can vary significantly according to the specific characteristics of the studied population [6]. Given that the original TFA was developed primarily from biomedical studies in populations of developed countries, its applicability might only be somewhat suitable for other contexts [5]. Casale et al. suggested a conceptual framework and an exploratory model for the acceptability of interventions among African adolescents and youth, including nine constructs, of which only some coincide with those of the TFA [5]. This discrepancy emphasizes the possibility that the constructs eliminated in our study did not represent socially and culturally relevant aspects of the specific population under investigation.

Recommendations and Limitations

The versatility of the generic TFA questionnaire for assessing various health interventions and facilitating their comparison is undoubtedly one of its main strengths. However, given its generic nature, it is essential to carefully adapt the sections of the instrument that refer to specific actions, the health condition under study, and the name of the intervention in question. Inappropriate modifications of these sections could significantly alter the wording of the items and, consequently, impact the construct validity of the instrument [26]. Therefore, it is recommended to handle these modifications cautiously to preserve the questionnaire's psychometric properties and ensure its effectiveness and applicability in different health research contexts.

One of the main limitations of this study was the small sample size. Although the number of subjects studied was sufficient to perform an Exploratory Factor Analysis (EFA), it did not allow for a Confirmatory Factor Analysis (CFA). While EFA helps identify empirical factor structures, which are highly dependent on the sample and the number of items [16], CFA validates these structures, provides fit indices to compare different theoretical models, and tests factorial invariance across different populations. This is the standard next step for the adaptation and validation of instruments [27]. It is suggested that future studies using this instrument to assess the acceptability of health procedures and interventions should estimate a CFA as a first analysis to test the underlying factor structure proposed in the present study.

Conclusion

The current study conducted the cross-cultural adaptation and validation of the generic TFA questionnaire's Spanish version. The final instrument consists of 6 items focused on assessing the acceptability constructs defined in the TFA, along with one additional item dedicated to measuring general acceptability. The structure of the questionnaire is organized into two factors that demonstrate good external reliability: "knowledge and attitudes toward the intervention" and "cost of the intervention." With acceptable internal consistency, this adapted version of the instrument offers a valuable tool for Spanish-speaking countries to evaluate and compare the acceptability of various health interventions.

References

1. Sung M, He J, Zhou Q, Chen Y, Ji JS, Chen H, et al. Using an Integrated Framework to Investigate the Facilitators and Barriers of Health Information Technology Implementation in Noncommunicable Disease Management: Systematic Review. J Med Internet Res [Internet]. 2022;24(7):e37338. doi: https://doi.org/10.2196/37338

2. Carlfjord S, Lindberg M, Bendtsen P, Nilsen P, Andersson A. Key factors influencing adoption of an innovation in primary health care: a qualitative study based on implementation theory. BMC Fam Pract [Internet]. 2010;11:60. doi: https://doi.org/10.1186/1471-2296-11-60

3. Watson DP, Adams EL, Shue S, Coates H, McGuire A, Chesher J, et al. Defining the external implementation context: An integrative systematic literature review. BMC Health Serv Res [Internet]. 2018;18(1):209. doi: https://doi.org/10.1186/s12913-018-3046-5

4. Klaic M, Kapp S, Hudson P, Chapman W, Denehy L, Story D, et al. Implementability of healthcare interventions: an overview of reviews and development of a conceptual framework. Implement Sci [Internet]. 2022;17(1):10. doi: https://doi.org/10.1186/s13012-021-01171-7

5. Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: An overview of reviews and development of a theoretical framework. BMC Health Serv Res [Internet]. 2017;17(1):88. doi: https://doi.org/10.1186/s12913-017-2031-8

6. Perski O, Short CE. Acceptability of digital health interventions: embracing the complexity. Transl Behav Med [Internet]. 2021;11(7):1473-80. doi: https://doi.org/10.1093/tbm/ibab048

7. Casale M, Somefun O, Haupt Ronnie G, Desmond C, Sherr L, Cluver L. A conceptual framework and exploratory model for health and social intervention acceptability among African adolescents and youth. Soc Sci Med [Internet]. 2023;326:115899. doi: https://doi.org/10.1016/j.socscimed.2023.115899

8. Park SY, Nicksic Sigmon C, Boeldt D. A Framework for the Implementation of Digital Mental Health Interventions: The Importance of Feasibility and Acceptability Research. Cureus [Internet]. 2022;14(9):e29329. doi: https://doi.org/10.7759/cureus.29329

9. Hosking R, O’connor SY, Wangdi K, Kurscheid J, Lal A. Acceptability measures of water, sanitation and hygiene interventions in low-and middle-income countries, a systematic review. PLoS Negl Trop Dis [Internet]. 2022;16(9):e0010702. doi: https://doi.org/10.1371/JOURNAL.PNTD.0010702

10. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ [Internet]. 2015;350:h1258. doi: https://doi.org/10.1136/bmj.h1258

11. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ [Internet]. 2021;374:n2061. doi: https://doi.org/10.1136/bmj.n2061

12. Sekhon M, Cartwright M, Francis JJ. Development of a theory-informed questionnaire to assess the acceptability of healthcare interventions. BMC Health Serv Res [Internet]. 2022;22(1):279. doi: https://doi.org/10.1186/s12913-022-07577-3

13. Jiménez D, Lavados M, Rojas P, Henríquez C, Silva F, Guillón M. Evaluación del minimental abreviado de la evaluación funcional del adulto mayor (EFAM) como screening para la detección de demencia en la atención primaria. Rev Med Chil [Internet]. 2017;145(7):862-8. doi: https://doi.org/10.4067/s0034-98872017000700862

14. Unitron Hearing Limited. uHear Kiosk [Mobile App]. 2016 [cited 2023 Dec 31]. Available from: https://apps.apple.com/co/app/uhear-kiosk/id1097960307

15. Ministerio de Salud. Orientación técnica para la atención de salud de las personas adultas mayores en atención primaria [Internet]. Ministerio de Salud, Gobierno de Chile; 2014. 108 p.

16. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 8th ed. Hampshire: Cengage Learning; 2019. 813 p.

17. Dunn TJ, Baguley T, Brunsden V. From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. Br J Psychol [Internet]. 2014;105(3):399-412. doi: https://doi.org/10.1111/bjop.12046

18. Kaiser HF. The application of electronic computers to factor analysis. Educational and psychological measurement [Internet]. 1960;20(1):141-51. doi: https://doi.org/10.1177/001316446002000116

19. Cattell RB. The Scree Test For The Number Of Factors. Multivariate Behav Res [Internet]. 1966;1(2):245-76. doi: https://doi.org/10.1207/s15327906mbr0102_10

20. Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika [Internet]. 1965;30:179-85. doi: https://doi.org/10.1007/BF02289447

21. Haydon HM, Major T, Kelly JT, Catapan SC, Caffery LJ, Smith AC, et al. Development and validation of the Digital Health Acceptability Questionnaire. J Telemed Telecare [Internet]. 2023;29(10-Suppl):8S-15S. doi: https://doi.org/10.1177/1357633X231202279

22. Timm L, Annerstedt KS, Ahlgren JÁ, Absetz P, Alvesson HM, Forsberg BC, et al. Application of the Theoretical Framework of Acceptability to assess a telephone-facilitated health coaching intervention for the prevention and management of type 2 diabetes. PLoS One [Internet]. 2022;17(10):e0275576. doi: https://doi.org/10.1371/journal.pone.0275576

23. Lamb R, Firestone JB, Ardasheva Y. A computational modeling of rapid attitude formation during surveys about immigrants and immigration. Computers in Human Behavior [Internet]. 2016;63:179-88. doi: https://doi.org/10.1016/j.chb.2016.05.031

24. Höhne JK, Schlosser S, Krebs D. Investigating Cognitive Effort and Response Quality of Question Formats in Web Surveys Using Paradata. Field Methods [Internet]. 2017;29(4):365-82. doi: https://doi.org/10.1177/1525822X17710640

25. Dolnicar S, Grün B. Does one size fit all? The suitability of answer formats for different constructs measured. Australasian Marketing Journal [Internet]. 2009;17(1):58-64. doi: https://doi.org/10.1016/j.ausmj.2009.04.001

26. Clark LA, Watson D. Constructing validity: New developments in creating objective measuring instruments. Psychol Assess [Internet]. 2019;31(12):1412-27. doi: https://doi.org/10.1037/pas0000626

27. Tabachnik BG, Fidell LS. Using multivariate statistics. 7th ed. New York: Pearson; 2019. 848 p.

Appendices

Appendix 1 Frequency of responses for the generic TFA questionnaire items.

Appendix 2 Eigenvalues obtained from the data, from the parallel analysis and their difference.

Appendix 3 Spanish version of the generic TFA questionnaire.