How well do 100W predict our motif? - Our Study of Motif
Implicit motivation - i.e. motives that guide human thinking and actions - still attract attention and interest (Schultheiss & Pang, 2007). The prognostic validity of implicit motives has also been shown in the prediction of career success factors, such as job satisfaction or position within a company (e.g. Collins, Hanges & Locke, 2004). Various tests can be used to capture implicit motivation. Traditionally, test subjects are presented with ambiguous images that are intended to stimulate them to produce stories. It is assumed that implicit motivations influence people's thinking and behavior and are therefore reflected in the text or writing samples. These text/writing samples are then manually captured by coding forces using coding rules. A big disadvantage of this procedure is obvious: The great effort that has to be made to identify the individual's motivations.
Schultheiss (2013) already poses the question of whether motifs can be identified automatically from language. Ultimately, it is also written scripts that form the basis for traditional coding. Moreover, motifs should be represented in single indicative words or combinations of words. Similar to a human being, a program would code specific words or word combinations as belonging to a motif. This is because certain words, such as "famous" (for power), "developed" (for performance), and "being together" (for relationship) suggest underlying motives.
What to clarify with this motif study
It is, therefore, necessary to clarify whether and how well automatic text analysis programs can predict the results of traditionally coded motifs. Already Schultheiss (2013) studied how highly categories of the Linguistic Inquiry and Word Count (LIWC, version 2001) correlate motifs that were collected with the classical coding manual. He found hypothesis-compliant correlations whose effect strengths were mostly in the middle range. His results are summarized as follows:
"Assessment of implicit motives with a word-count approach yields scores that converge with content-coded motive measures, that predict well-documented validation criteria of implicit motive measures, and that respond sensitively to experimental arousal of motivation. (Schultheiss, 2013, p.18)
The Aim of 100W's motif study
The present motif study aims to investigate the validity of the 100W motif categories. For this purpose, we check whether there is convergence to classically coded motifs. In addition to testing the convergent validity, we want to investigate whether a frequently made observation of classically coded motifs can also be replicated with 100W text analysis (=criteria validity). In studies, a gender effect has often been observed for the relationship/connection motive, such that women have a higher relationship motive than men. We also expect this correlation in the motives measured by the 100W text analysis. In doing so, we orientate ourselves on his approach.
In the following, we explain which data set we selected for the current motif study. Furthermore, we will explain how the 100W motif categories are composed and how the external criterion serving as a validation feature was created.
Schultheiss already studied in 2013 how well the results of classically coded motifs can be predicted by text analysis programs. The data analyzed for this purpose were kindly made available to us for our motif study (validation study). We would like to take this opportunity to thank him and his colleagues.
This data set was created in 2009 as part of the study "Activity inhibition: a predictor of lateralized brain function during stress". 100 students of the Friedrich-Alexander University of Erlangen participated in this study. They completed a six-picture "picture-story exercise" (Picture-Story-Exercise, PSE; McClelland et al., 1989), which largely reflects the Thematic Perception Test, but is somewhat reduced in scope. The participants were instructed to think and write down a story based on each of the six ambiguous pictures. The stories were then evaluated by two experienced coders using the Winter (1994) coding manual. The two coding achieved average inter-rater reliability of 0.80 which was considered sufficiently high.
Our motif study - The 100W motif Corpora
The 100W text analysis contains dictionaries to capture implicit motives. According to McClelland's definition, these are divided into dictionaries that capture the power motive, the performance motive, and the relationship motive. Each of these three motif categories is divided into three subcategories. These subcategories represent an approach and avoidance tendency for each motive, as well as a neutral category. This category contains words that are associated with the underlying ad spec but do not have a valence. All motif dictionaries contain 7438 words.
It should be noted that the 100W text analysis does not use stemming. This means our analysis does not reduce words to a stem. It, therefore, contains significantly more words than text analysis programs that use stemming (e.g. LIWC with 979 words in the three motif categories in the English version of 2015). The 100-word text analysis does not use stemming, as this can lead to errors in the recognition of word meanings, as Schwartz et al (2013) point out.
The external criterion for the Motif Study
To investigate the convergent validity of the 100W's motif categories, an external criterion was needed that reliably measures implicit motives. Due to the difficulty of measuring implicit motives, only a few methods can be considered from the outset. In addition to linguistic/written methods, there are also reaction-time based methods such as the Motive Superiority Measure (MSM; Eichstaedt, 2003). In the end, we decided on the Picture Story Exercise (PSE). It has already been used as a criterion for the validation of text analysis programs. Therefore we can compare our results with them.
Motif Study – Results
First, internal consistencies were calculated for the nine motif categories and the distribution functions were analyzed. It was found that the three subcategories had too low a probability of occurrence. Therefore, we combined the three subcategories into one ad spec category each for all further calculations. Thus, only the dictionaries' power, performance, and relationship were further investigated.
The results for the three aggregated dictionaries are shown in Table 1. The Kolmogorov-Smirnov test shows a deviation from the expected normal distribution. At a significance level of alpha=0.05 and 612 observations, the largest absolute deviation must not exceed 0.06. All three motive categories exceed this value (achievement motive: 0.15; power motive: 0.14; relationship motive: 0.12).
Schultheiss also observed a deviation from the normal distribution in his analysis of the LIWC dictionaries. He therefore recommended a log transformation (new score = ln (1 + old score)). For categories with a too low probability of occurrence (= zero scores> 20% of the test subjects), the variable was dichotomized, i.e. a zero was assigned if the character was not observed and a 1 if the characteristic was observed. We follow this proposal and perform the transformations according to the observed distribution. Table 2 shows which transformations were performed.
The Word Recognition Rate
The word recognition rate, i.e. the ratio of (with the 100W motif categories) words recognized in relation to the total number of words, is 64%. Recognition rates of 75% (New York Times texts) and 92% (Expressive Writing Exercises) are reported for the new version of the LIWC (2015) and 63% for the German version of the LIWC (2001) (Wolf et al., 2008). Correlations were calculated to obtain information on convergent validity. Significant correlations between the results of the 100W motif categories and the coded motifs were found. All three motif categories were positively correlated with the coded motifs to a medium to a high degree (Table 2).
Table 1: Descriptive Statistics (M= Mean Value, SD= Standard Deviation) and internal consistant (=Cronbachs Alpha) for traditional coded motif and 100W motif category.
Note on Our Motive Study
All results of the motive study are significant at the 0.001 level. When testing the independence of the 100W motif categories in the prediction of traditionally coded motifs, a not to be sneeze at overlap was found. These ranged from 0.07 (for 100W of power on n power) to -0.37 (for 100W of power on n relationship). This finding is examined in more detail in the section "Conclusion".
In addition to testing the convergent validity, the connection of gender with the relationship motive was also examined in order to obtain indications of the criterion validity. It was expected that a positive relationship exists. This correlation could be observed. Gender correlated positively with the 100W relationship motif (t(612)=0.12, p<0.005), but not with the other motif categories. This correlation is also somewhat weaker with the classically coded relationship motif (t(612)=0.11, p<0.005).
Table 2: correlation between traditional code and the 100W Motif categories:
Conclusion of Our Motif Study
The objective of our motif study was to show whether the automated detention of words and word combinations using 100 words text analysis can provide information about classically coded motifs. For this purpose, the texts provided in the Schultheiss dataset with the 100W motif categories were examined. Subsequently, convergent and criterion validity were calculated. It was assumed that there are significant positive correlations between the 100W motif categories and the coded motifs. It was also assumed that gender was also positively correlated with the relationship motif.
The coefficients in the study of convergent validity ranged from medium to high and in the low range for criterion validity. This was slightly higher than the correlation between gender and the classically coded relationship motif. It should be noted here that correlations with an external criterion (such as gender) are often smaller than correlations between different tests.
The Convergent Validity of Data
The convergent validity of data observed in the present study is higher than those reported by Pennebaker and King (1999) and Schultheiss (2013). To determine the quality of the 100W motif categories, a comparison with results from Winter (1991) also seems to make sense. There he examined convergences of two versions of his motif-coding manual. The correlation between the two versions ranges from 0.45 to 0.72. The effect strengths found here are in a similar range. They show that even under traditional coding manuals there is no perfect convergence.
Generally, the independence of the motifs from each other is demanded. This was not the case in the present study, nor Schultheiss (between 0.06 and 0.49). In the present motif study, the 100W motive power and performance were negatively associated with n relationship. To a lesser extent, the 100W motive relationship was associated with n performance and n power. It should be noted that the traditionally coded motives in the data set under investigation also share variance. Thus n power and n relationship covariate to -0.34 and n power and n relationship to -0.17. It appears that the motives power and relationship are the opposite. To a smaller extent, this also applies to the relationship between the motives of power and relationship. The recognition rate of 100W of text analysis is 64%, which is comparable to LIWC's values.
Overall, it can be said that the 100W motif categories are well suited to predict classically coded motifs. Difficulties with the reliability as well as the independence of the categories do not only exist with the 100W text analysis, but also with LIWC. Therefore, they should be considered in the further development and improved if possible.
Collins, C.J., Hanges, P.J. & Locke, E.A. (2004). The relationship of achievement motivation to entrepreneurial behaviour: A meta-analysis. Human Performance, 17, 95-117.
Pennebaker, J. W., & King, L. A. (1999). Linguistic styles: language use as an individual difference. J. Pers. Soc. Psychol. 77, 1296–1312.
Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic Inquiry and Word Count (LIWC): LIWC2001. Mahwah, NJ: Lawrence Erlbaum Associates.
Schultheiss, O.C. & Pang, J.S. (2007). Measuring implicit motives. In R.W. Robins, R.C. Fraley & R.F. Krueger (Hrsg.), Handbook of research methods in personality psychology (S. 322-344). New York: Guilford Press.
Winter, D. G. (1991). Measuring personality at a distance: Development of an integrated system for scoring motives in running text. In J.M. Stewart, J.M. Healy, Jr., & D.J. Ozer (Eds.), Perspectives in personality, Vol. 3: Teil B: Approaches to understanding lives. (S. 59-89). Bristol: Jessica Kingsley Publishers.
Winter, D.G. (1994). Manual for scoring motive imagery in running text (Version 4.2). University of Michigan.
Wolf, M., Horn, A., Mehl, M., Haug, S., Pennebaker, J. W., & Kordy, H. (2008). Computergestützte quantitative Textanalyse: Äquivalenz und Robustheit der deutschen Version des Linguistic Inquiry and Word Count. Diagnostica 2, 85–98.