Validity and reliability of the theory of planned behavior questionnaire on the balanced dietary behavior of adolescents in a post-disaster area

Introduction: The assessment of mediator components of behavior change based on the Theory of Planned Behavior (TPB) can be carried out with a questionnaire instrument. Meanwhile, the questionnaire to assess the mediator of adolescents’ balanced dietary behavior in post-disaster areas has not yet been developed. Failure to recognize nutritional problems in this period can increase morbidity and mortality. Therefore, this present study attempts to provide the overview of the implementation of content and face validity as well as reliability of the TPB questionnaire on the balanced dietary behavior of adolescents in the post-disaster area by taking a study case in Palu City, Central Sulawesi, Indonesia. Methods: By involving nine expert panelists, the content validity was conducted qualitatively and quantitatively by calculating the Content Validity Index (CVI) and Kappa statistics scores. Face validity was conducted qualitatively by interviewing ten adolescents’. Meanwhile, Cronbach’s Coefficient Alpha was calculated to measure the reliability by interviewing 30 adolescents aged 15–17 years old living in the disaster site with their mothers during the massive disaster in 2018. Results: The results showed that the Item CVI and the kappa statistics scores were between 0.75 and 1 and 0.72 and 1, respectively. The final version of the questionnaire was resolved based on the panelists’ recommendation, the face valid-ity’s findings, and the results of the questionnaire reliability assessment. There were 14 items of attitude, 12 items of subjective norms, 17 items of behavioral control, and eight items of intention components with acceptable questionnaire reliabilities. Conclusion: Overall, the TPB questionnaire on the balanced dietary behavior of adolescents has excellent validity and reliability to measure the mediator of balanced dietary behavior changes of adolescents.


INTRODUCTION
Adolescence is a critical period in the human life cycle (1). Adolescents' nutritional and health conditions will affect their health in adulthood and old age (2,3), and one of the factors affecting the conditions is dietary behavior (4,5). However, the dietary behavior of Indonesian adolescents is yet to follow balanced nutrition guidelines (6). According to the recommended portion, the prevalence of adolescents who do not consume vegetables and fruits is 95%; 44% of them are fatty food consumers, 45% regularly consume sweet food and drinks, and 31% like to eat salty food (6). Meanwhile, almost a quarter of Indonesian adolescents are stunted, with the prevalence of overweight increasing to 11% (7). In the post-disaster area, the adolescents' dietary behavior can be worsened due to socio-economic changes (8). The lack of nutritional awareness and the failure to recognize nutritional problems during post-disaster periods can increase morbidity and mortality levels (9). Food insecurity and other priority targets, such as nutritional problems in toddlers and pregnant women in post-disaster areas, also contribute to the worse dietary behavior of adolescents (10,11).
Changes in behavior should be implemented by comprehending the mediator of behavior change. Understanding the mediator makes it easier to design the appropriate behavior change intervention strategy as required (12). The behavior change theory that is mainly employed to assess the mediator of dietary behavior is the Theory of Planned Behavior (TPB) (13). This theory further assumes that the decision made with reasonable ways, namely, attitude, subjective norms, and behavioral control, will affect one's intention to carry out specific dietary behavior (14). Attitude is either a positive or negative evaluation of an individual in performing a behavior. Then, subjective norms are individuals' notions that believe that other people who significantly influence them contribute to supporting those behaviors; meanwhile, behavioral control is the perception where an individual believes that they have complete control to perform certain behaviors (14). Moreover, adolescents' intentions to undergo balanced dietary behavior are affected by attitudes, subjective norms accepted by the family, peers, environment, and behavioral control, among other external factors (13,14).
A questionnaire instrument can be used to assess attitude, subjective norms, behavioral control, and intention as behavior shaper (15). The TPB Questionnaire (hereafter, the TPB Questionnaire) comprises the dimensions of attitude, subjective norms, behavioral control, and intention answered by choices based on the Likert scale (15). The development of dietary behavior mediator questionnaire based on the TPB has been widely conducted in many countries with adolescents from various socio-economic levels of the family as the subject (16,17). In Indonesia, the assessment of dietary behavior mediator with the TPB includes consumption of vegetables and fruit, snacks, and sweet and fatty foods (18). However, to the author's knowledge, no questionnaire has ever been designed to assess the mediators of balanced dietary behavior, especially in post-disaster areas.
A good questionnaire must have good validity in measuring what needs to be measured (19). Moreover, the items included in the questionnaire also need to be consistent in measuring such concepts and, thereby, the results are reliable as measurements (20). This study aims to describe the implementation of content and face validity and reliability on the TPB questionnaire. The questionnaire in this present study focuses on the balanced dietary behavior of adolescents in the post-disaster area of Palu City, Central Sulawesi, Indonesia. This study is expected to be beneficial to assess the balanced dietary behavior mediator in post-disaster areas that have never been done before. Thus, the behavioral change in nutrition intervention can be carried out in accordance with the behavioral mediators behind it.

METHODS
Nine panelists, comprising four community nutritionists and five nutrition education and behavior change specialists, were involved in the content validity assessment. The determination of the number of the panelists was based on a minimum of five recommended people who can control the coincidence of arguments (21). Meanwhile, the increase in the number of panelists would decrease the probability of the chance agreement (21).
The face validity was performed qualitatively involving ten adolescents with the inclusion criteria who were staying in Palu, Central Sulawesi, Indonesia, during the massive natural disaster in 2018, 15-17 years old, and lived with their mothers. Meanwhile, the reliability assessment was carried out to 30 adolescents with the same criteria as the respondents of the face validity. The determination of the number of the sample in the face validity was based on the recommended number to perform a cognitive interview as the questionnaire evaluation, that is, five to 15 people (22). On the other hand, the determination of the sample to assess a questionnaire's reliability was based on the sufficient minimum size of the sample as unbiased estimators of the alpha coefficient (23).
Every participant stated their willingness by filling in the informed consent after receiving an explanation before approval. The informed consent for the adolescents was obtained through the consent of their parents, while the adolescents themselves signed the informed assent.
Validity and reliability assessments were performed from August to September 2021. The TPB questionnaire instrument in this present study was the modified result of a questionnaire used to assess urban Native American youth (24). The previous questionnaire had undergone a development following the procedures proposed by Ajzen and Fishbein (25). Due to a copyright issue, the complete questionnaire is not displayed in the article. However, the questionnaire is available under requested.
An initial translation by a translator was the starting procedure in adopting the questionnaire (26). The translated version was then clarified by researchers with a nutritional educational background and formal English education. After that, the translated version of the questionnaire was adjusted to the components of attitude, subjective norms, behavioral control, and intentions, based on the guidelines for the balanced diet in Indonesia through a literature study adapted to the local culture and eating habits.
The authors conducted the literature study by searching similar articles and instruments. Hence, the determination of the questionnaire items was based on the questionnaire that had been developed previously (24). However, the modified questionnaire did not list the self-efficacy and barrier components as separated components like in the original questionnaire (24); both items were merged into the behavioral control component instead (14). The specific themes of each TPB questionnaire component expected to be assessed through a questionnaire, moreover, are shown in Table 1.
The questionnaire scoring was based on the five options in each question, namely, "strongly agree," "agree," "unsure," "disagree," and "strongly disagree." The response to each positive statement was given a score from 5 to 1 (from strongly agree to strongly disagree), and to each negative statement was given a score from 1 to 5 (from strongly agree to strongly disagree). Attitudes, subjective norms, behavioral control, and intentions were determined based on the total score obtained.
In addition, the panelists carried out the content validity assessment quantitatively and qualitatively. The former assessment was conducted by measuring the content validity index (CVI) score for relevance and clarity of each statement item (I-CVI) and calculating the value of the multirater kappa statistic. The latter, moreover, was completed by leaving comments and suggestions on each questionnaire item.
The panelists provided general inputs and comments to the TPB questionnaire, then assessed the clarity and relevance of each statement with the TPB components (14) on the balanced dietary behavior in adolescents, containing attitude, subjective norms, behavioral control, and intention. -Belief in having behavioral control (Personal trust in adolescents' self-ability to do balanced dietary behavior) -Power perceived in facing obstacles (Perceived obstacles including personal resources and external obstacles) Intention How likely a person is to do or engage in balanced dietary behavior.

State of mind in the form of commitment to action
The clarity assessment included 1 = unclear, 2 = rather unclear (item needs revision), 3 = quite clear (item is clear but needs minor revision), and 4 = very clear (item is understandable). Meanwhile, the relevance assessment comprised 1 = irrelevant, 2 = rather irrelevant (item needs revision), 3 = quite relevant (item is relevant but needs minor revision), and 4 = very relevant (item is proper). Furthermore, adjustments to the statements were made based on input and suggestions from the panelists at the qualitative content validity step.
The face validity was performed qualitatively, involving ten adolescents. The qualitative assessment was conducted with interviews to test their understanding and language complexity on the question items. An interviewer read aloud the statement items on the questionnaire to the adolescents.
They were asked to answer the questions, and then the interviewer asked whether they understood the meaning of the questions and whether the language used was simple and easy to understand.
The reliability assessment was done on the revised items based on the suggestions given during the face validity. Those items were then analyzed further. Moreover, the reliability assessment was carried out on the items that had a correlation with the significant total score with a significance level of 5% (A Spearman rank correlation) (27). Next, Cronbach's Coefficient Alpha (CCA) was utilized to calculate the reliability by measuring the correlation among the significant items (28).
On the quantitative content validity assessment, I-CVI was obtained by adding up the number of panelists scoring 3 or 4 for relevance and clarity on each question item then divided by the total number of experts. The CVI score was then adjusted with the multi-rater kappa statistics to eliminate the possibility of an increase in value due to coincidental agreement in the assessment process of each expert (29).
The kappa statistic calculation was carried out by first calculating the value of the Probability of chance agreement (Pc), which was counted by N is the total number of experts, and A is the number of experts who agree that the items are clear or relevant. Once the Pc was obtained, the Kappa score was attained by utilizing K = (I-CVI-Pc)/(1-Pc).
Furthermore, the CVI and Kappa statistic assessments were completed using Microsoft Excel, while the Spearman rank correlation test and the CCA for the reliability assessment were done by utilizing the SAS program version 6.12. This present study received approval from IPB University Ethics Commission no. 464/IT3.KEPMSM-IPB/SK/2021.

RESULTS
The total number of statements in the TPB questionnaire included in content validity was 26 items of attitude, 28 items of subjective norms, 25 items of behavioral control, and ten items of intention. The statements in the attitude section related to the understanding related to the impact of eating vegetables, fruits, sources of animal and plantbased proteins, sweet, salty, and fried foods, and the participants' point of view on the effects of those foods. For example, "Sweet food is delicious" and "It is important for me to eat or drink something delicious." Subjective norms were measured through the responses to the statement that the adolescents' parents, peers, teachers, or idol celebrities influence them to consume well-balanced foods every day and how much they wanted to follow these influences. For instance, "My friends do not like eating vegetables" and "It is important for me to eat what my friends like." Perceptions of behavioral control were then assessed based on adolescents' belief in their ability to consume well-balanced diets and their perceptions on personal resources or external influences as difficulties to face, such as "I cannot eat fruits every day because it is unavailable" and "it is hard for me not to eat fried foods." The last component was adolescents' intentions to consume well-balanced diets, which was measured through the statements regarding their plans to eat more vegetables, fruits, and sources of animal and plant-based proteins and to reduce eating sweet, salty, and fried foods. For example, "I plan to eat vegetables every day starting next week" and "I plan to reduce eating sweet foods starting next week." The results of the CVI assessment during the content validity process quantitatively revealed that the I-CVI on all items depicted a range score between 0.75 and 1. Three items in the subjective norm component had the I-CVI score of relevance and clarity of 0.75. These three items also obtained the kappa statistics score of 0.72 on relevance and clarity. The I-CVI and kappa statistics scores for each statement item are presented in Table 2.
Besides providing quantitative assessment, the panelists also provided comments and inputs to the statements. For instance, the place of the statements that need to be made sequentially, such as, "Sweet drinks taste better than lesssugar drinks," was placed after "Sweet drinks are delicious." Another input was the statements that emphasize the taste of animal protein sources should be distinguished since some protein sources are preferred over others, like chicken and eggs over fish. Furthermore, the panelists thought that passive sentences should be changed into simple active sentences, such as, "Eating healthy food every day is hard to prepare," into "It is hard for me or my family to prepare  healthy food every day." In the behavioral control component items, the panelists also suggested that the term "healthy food" in the statement, "The school canteen provides healthy food," was more specified by including some examples of the referred healthy food, such as vegetables and fruits. Thus, this statement was then split into "The school canteen provides vegetables" and "The school provides fruits." By this change, the behavioral control component items became 26 statement items.
The respondents in the face validity process were two adolescent boys and eight adolescent girls. They had mothers aged 37-54 years with the educational background of elementary of one person, high school of three people, and higher education of six people.
According to the respondents, five statements were considered not simple and difficult to understand, comprising two items of attitude, two items of behavioral control, and one item of intention, so these statements needed revision. For instance, the change occurred to the statement, "It is important for me to eat what makes me feel full," into "It is important for me to eat filling food." The statement "I find it difficult to concentrate on studying although I eat healthy food" was also confusing and not simply for the respondents, so that it was changed into "Eating healthy food makes it easier for me to concentrate on studying." The revision of the statement items of attitude, behavioral control, and intention components is presented in Table 3. Eventually, the total item of each indicator after the face validity was 26 items for the attitudes, 28 for subjective norms, 25 for behavioral control, and ten for intentions.
The item analysis further indicated that there were valid items, consisting 14 items of attitudes, 12 items of subjective norms, 17 items of behavioral control, and eight items of intentions. These valid items were then tested for their reliability and produced Cronbach's Alpha scores of 0.73, 0.75, 0.84, and 0.82, respectively, for each component (Table 4). There are then five items to be eliminated to increase the questionnaire's reliability.

DISCUSSION
Content validity aims to decide whether the instrument developed can assess the content components of the phenomenon to be measured (30). It is crucial to create an instrument since the instrument needs to correlate the concept and indicators to be measured (29). The TPB questionnaire in this present study was thus expected to measure the specific theme on each TPB component on the balanced dietary behavior of adolescents in a post-disaster area.
The statement items in the questionnaire were the modified version of the questionnaire that was developed for native American adolescents (24). Different adolescents' characteristics and different cultural backgrounds required certain statements to be adjusted to local customs. Specific statements regarding the disaster on eating behavior were not included in the questionnaire, given that this research was conducted 2 years after the earthquake, tsunami, and liquefaction disasters in 2018. However, some statements about healthy food availability at home, such as, "I do not eat healthy food because it is not available at my house," could depict the food condition in the house and environment. Food insecurity in the household that influenced adolescents' dietary behavior was argued as one significant impact, although disasters had ended, especially during the rehabilitation and construction phases (9). Content validity can be conducted qualitatively and quantitatively (31). Qualitative content validity is an approach that is widely employed despite its subjectivity (32). Expert panelists conducted the qualitative content validity of this present study by providing general comments and the indicators' relevance with the TPB concept of the actual balanced dietary behavior to be measured.
Meanwhile, quantitative content validity is developed as needed to determine the content validity that can be quantified, and it is considered the more objective approach (29). CVI is the method that is often utilized in quantitative content validity (21,33). The recommended minimum score of I-CVI is not less than 0.78 if the number of panelists is six or more (34). There were three statement items in the TPB questionnaire that had I-CVI that needed to be eliminated with the number below 0.78 with the kappa statistics scores 0.72. The kappa statistics score of 0.72 was still interpreted to have good content validity (35). Thus, the item decided to remain in the instrument. Although CVI is often employed in the content validity process, each panelist's coincidental agreement in the assessment process is very likely in the CVI calculation (29). It is suggested to assess multi-rater kappa statistics to eliminate the possibility of an increase in value due to this agreement.
Face validity is performed so that the respondents can easily accept the developed instrument. The sentence complexity It is important for me to eat filling food.
19 I find it difficult to concentrate on studying although I eat healthy food.
The statement is not simple and not easy to understand.
Eating healthy food makes it easier for me to concentrate on studying. Behavioral control 21 I want to eat healthy food every day but don't want to give up on my favorite food.
The statement is not simple. I want to eat healthy food every day but don't want to stop eating my favorite food. 22 My school canteen provides fruits and vegetables.
The statement for fruits and vegetables should be separated.
• My school canteen provides vegetables.
• My school canteen provides fruits. Intention 5 I plan to eat legumes such as tempeh, tofu, or beans starting next week.
The statement is confusing; the word "every day" should be added.
I plan to eat legumes such as tempeh, tofu, or beans every day starting next week.  and easiness are vital parts mainly for the target respondent where a phenomenon wants to be discovered (36). The adolescents involved in the face validity process aged between 15 and 17 years old, middle-aged teenagers, which could be assumed that they could think abstractly and understand the long-term implications and consequences of the activities carried out (37). In the process, five statements still had a complex structure in terms of clarity and formulation, according to the respondents. These statements were then revised according to the comments given.
The item analysis revealed that each item could measure what it wanted to be measured based on the expected concepts. In the TPB questionnaire test, it was found that the total of 52 items and the Cronbach's Alpha score ranged between 0.73 and 0.85. In other words, the reliability of the TPB questionnaire in this study was good; it even obtained a higher score than the previous questionnaire (24). Therefore, a reliable questionnaire was used to measure the components of subjective norms, behavioral control, and intentions as the dietary behavior shaper of adolescents in a post-disaster area.
The TPB questionnaire has been developed to measure nutritional behavior in adolescents (17,24,(38)(39)(40)(41). Some TPB questionnaires were developed for disadvantaged groups such as low social level or special groups such as native population (24,39). The validity of this questionnaire is almost as good as the previous questionnaire which was developed in the low socioeconomic status group with Cronbach's Alpha values of 0.7-0.8 on the item's value. Compared to the questionnaire developed in Native American group, Cronbach's Alpha score on this questionnaire was higher for the attitude and subjective norm components with a score of more than 0.7 (24).
Eventually, this present study provided an overview of validity and reliability processes on the TPB instrument on the balanced dietary behavior of adolescents in a post-disaster area. This study implies that a valid and reliable questionnaire can be used to direct behavior change interventions in  This study involved samples that can carry out content validity, face validity, and reliability; yet, it was not enough to carry out construct validity. The COVID-19 pandemic, however, affected the face-to-face validation process. It was challenging to perform a validation process that involved a bigger-scale sample. Even though the questionnaire was addressed to the adolescents living in a post-disaster area, the disaster-related contents could not be further and deeper elaborated, given that this study was conducted 2 years after the disaster. This validation process could provide a big picture that the questionnaire was valid and reliable to measure the behavior change of adolescents related to balanced dietary behavior in a post-disaster area, especially during the rehabilitation and construction periods.

CONCLUSION
The TPB questionnaire on the balanced dietary behavior of adolescents in a post-disaster area presented in this study retained good content validity and reliability. The final version of the questionnaire was determined based on the panelists' assessments on the content validity process, the findings of the face validity process, and the results of the reliability assessment. Furthermore, the questionnaire could be used to assess adolescents' dietary behavior in other post-disaster areas by undergoing a psychometric validity process on a more significant sample.