Table 1 Summary of results

Baek, Cappella and Bindman* [29]

To explore the usefulness and reliability of automated content analysis of answers to open-ended survey questions from a survey on bioethical issues in genetics research.

Effective, efficient, useful, simple, reliable, high face validity, systematic, versatile, flexible, systematic, consistent, superior to manual coding.

Open-ended cross-sectional survey responses (n = 1961) to questions on bioethical issues.

Appropriate, effective for coding purposes and efficient for extracting concepts from large texts. Also effective for analysis of short, informal survey responses. Word usage in reference and target texts needs to be similar for optimal analysis.

Choose reference text samples carefully to ensure comparability.

Baumann, Debus and Müller [41]

To evaluate policy positions on abortion legislation using automated content analysis.

Effective, useful, versatile, flexible, appropriate, strong validity, a promising method that helps inform a rich (nuanced) analysis.

Policy opinion surveys among constituents and speeches on abortion policy in the Irish parliament, 2001–2013, speeches by legislators and advocacy groups.

Effective and reliable method for policy analysis of health-related issues. Suitable for analysis of speeches, debates, policy drafts, advocacy group statements, and constituent surveys.

Consider all stages of the legislative process when selecting a scope of texts for analysis for a richer account.

Bernauer and Bräuninger* [38]

To estimate policy positions of legislators on the left-right scale using Wordscores and to explore links between intra-party faction membership and policy positions.

Effective, reliable, useful, convincing, strong validity, and language-blind (can be used to analyse in any language).

Plenary speeches of legislators (n = 453) of the German Bundestag, 2002–2010. Policy positions related to economy and social issues including abortion and euthanasia.

Ensure that ‘virgin’ and reference texts are of similar length and type for optimal comparison.

Comparative research across document types and political actors is recommended. Requires careful selection of reference texts to ensure context requirements are met and analysis is reliable.

Budge and Pennings* [37]

Assesses the Wordscores method for reliability and validity in policy position analysis.

Efficient, simple, systematic, innovative, consistent, promising method.

British party manifestos, 1979–1997.

Promising method for repeated use over time. Results dependent on initial document sets selected for analysis.

Aggregate texts within each time period to create reference sets for pairwise comparisons.

Coffé and Da Roit [39]

To explore changes in party positioning on social and economic issues after a major political event.

Useful, appropriate, reliable.

Party programs for 2006–2008 Italian coalitions: Casa delle Liberta’ and Unione on economic and social policies.

Standard errors accompany each score estimation. No validity issues when rescaling raw score estimates.

Suitable method for analysis of reference texts of varying length and context.

Costa, Gilmore, Peeters, McKee and Stuckler [28]

To determine the influence of the tobacco industry on EU Tobacco Products Directive.

Reliable, simple, objective, innovative, superior to manual coding.

EU Tobacco Products Directive policy drafts, tobacco and health lobby group position papers (n = 20).

Requires some prior technical knowledge. Enables visualisations. Clustering effects of virgin text scores: reference text scores tend to be more extreme than in virgin texts.

Efficient for rapid analysis of changes in health policies through different draft stages.

Debus [35]

To explore various methods for analysing policy preferences of political actors.

Robust, reliable, and superior to human coding.

German party programs.

Estimation was left solely to computer algorithms to remove human error. Assumes the systematic use of certain words by policymakers.

Enables the researcher to identify policy or program positions and directions based on automated content analysis.

Hug and Schulz [40]

To assess changes in policy positions over time using various content analysis methods including Wordscores.

Effective, efficient, reliable, strong validity, suitable for retrospective analysis.

Swiss party manifestos, roll call data from the Swiss parliament and voting recommendations, 1991–2003. Range of policy positions including health and social policies.

Wordscores produces most reliable and consistent policy estimates when compared to other content analysis approaches in time series analysis.

Reference texts must have enduring relevance over time period being examined to produce correct measures for policy position changes over time.

Klemmensen, Hobolt and Hansen* [30]

Assesses the Wordscores method for usefulness, reliability and validity in policy position analysis.

Efficient, cost-effective, valid, easy to use, systematic, innovative, flexible, and versatile.

Danish election manifestos from 1945 to 2005 and speeches in parliament.

Supplies time series of policy positions with high face validity. Can be used with Stata/Java. May work best with longer texts.

Can be used for retrospective analysis of policy positions.

Laver, Benoit and Garry* [25]

To explore the usefulness of the Wordscores method for analysing policy positions of legislators.

Effective, efficient, simple, easy to use, quick, reliable, systematic, strong validity, inexpensive, flexible, innovative, versatile, language blind.

Party manifestos and speeches of legislators in the British, Irish and German parliaments on economic and social policies (incl. abortion), expert surveys on policy positions, 1990–1997.

Efficient and rapid method of text analysis with large number of potential applications.

Ensure policy assumptions in reference texts are valid prior to analysis. Researcher not required to understand text.

Applicable across languages and contexts but comparisons must be made between texts of similar context and format to ensure validity.

Lowe* [31]

To explore the strengths and weaknesses of Wordscores.

Effective, simple, easy to use, versatile, flexible, and empirically successful.

Other studies conducted using Wordscores method.

Lack of functional and distributional assumptions. Choose reference texts of a similar nature to the texts under investigation.

When using Wordscores, analyse research to select texts suitable for comparison.

Volkens* [36]

To evaluate strengths and weaknesses of three approaches to measuring party policy positions.

Effective, simple, easy to use, quick, reliable, versatile, flexible, promising, high validity, suitable for retrospective analysis, innovative, superior to manual coding, and has useful internal checks and controls.

Other studies conducted on party policy assessment methodologies including Wordscores.

Analysis at specific time points enables the creation of a time line between cause and effect. Focus on text rather than meaning not considered as problematic.

Comparable to expert analysis, but more reliable than manual coding. Wordscores requires skilled researchers to make efficient coding decisions.