From Alternative conceptions of honesty to alternative facts in communications by U.S. politicians
Jana Lasser | Graz University of Technology | Complexity Science Hub Vienna
It's a team effort!
Outline
Study 1: How trustworthy is the information shared by political elites on social media platforms?
Article "Social media sharing of low quality news sources by political elites", Lasser et al. 2022, in PNAS nexus.
Study 2: Can the sharing of untrustworthy information be explained by a changing "ontology of truth"?
Article "From Alternative conceptions of honesty to alternative facts in communications by U.S. politicians", Lasser et al. 2023, arXiv.
Study 1: Motivation
Political elites have agenda-setting powers [1, 2].
Relatively little attention to information sharing practices of politicians in the literature [3].
Aim: Quantify the trustworthiness of information shared by politicians on social media.
[1] Chong D., Druckman JN., A theory of framing and opinion formation in competitive elite environments, Journal of Communication (2017)
[2] Lewandowsky S., Jetter M., Ecker UKH., Using the president’s tweets to understand political diversion in the age of social media, Nature Communications (2020)
[3] Mosleh M., Rand D., Measuring exposure to misinformation from political elites on Twitter, Nat. Comm. (2022)
Study 1: Research questions
RQ 1: How does the trustworthiness of the information shared on social media differ between parties?
RQ 2: How does the trustworthiness of the shared information differ over time?
Study design
(1) Collect tweets by politicians in three major western democracies.
U.S: two-party system, negiglible pubic broadcasting
U.K: (almost) two-party system, one public broadcaster
Germany: many-party system, multiple public broadcasters
(2) Extract all links from the posts. Follow shortened links.
(3) Assess the trustworthiness of the information that was linked to.
Tweet corpus
Tweet corpus
Assessing information trustworthiness
We assess trustworthiness of the domain using journalistic criteria – not of the content (articles) themselves.
Validation with independently compiled super-list of academic lists and fact-checking sites.
Freely available list: "High level of agreement across different news domain quality ratings", Lin et al. 2022, arXiv
NewsGuard rating criteria
(-) Does not repeatedly publish false content (22)
(-) Gathers and presents information responsibly (18)
(-) Regularly corrects or clarifies errors (12.5)
(-) Handles the difference between news and opinion responsibly (12.5)
(-) Avoids deceptive headlines (10)
(-) Website discloses ownership and financing (7.5)
(-) Clearly labels advertising (7.5)
(-) Reveals who is in charge, including any possible conflicts of interest (5)
(-) The site provides names of content creators, along with contact/bio info (5)
See also NewsGuard's rating criteria.
NewsGuard rating distribution
NewsGuard is not a blacklist.
Data base coverage & validation (NewsGuard)
Manual validation of all domains that contribute > 0.1% of link volume shows that no major news sites are missing.
Results: differences between parties
Parties to the right overall share more links to untrustworthy sites (< 60 points).
Results: trustworthiness in Germany over time
Trustworthiness of sources stays relatively constant across parties.
Results: trustworthiness in the U.K. over time
Trustworthiness of sources stays relatively constant across parties.
Results: trustworthiness in the U.S. over time
Trustworthiness of sources shared by Republican Congress Members drops.
Study 1: summary
RQ 1: Parties on the right share more links to untrustworthy sites in the U.S., U.K. and Germany.
RQ 2: The steep drop in trustworthiness of links shared by Republicans is unique to the U.S.
"Social media sharing of low quality news sources by political elites", Lasser et al. 2022, PNAS nexus.
Study 2: Motivation
Following Lewandowsky (2020) "Willful construction of ignorance: A tale of two ontologies":
Could the increase in untrustworthy information have its root cause in a new ontology of truth?
Belief-speaking: relates only to the speaker's beliefs, thoughts, and feelings, without regard to factual accuracy.
Truth-seeking: relates to the search for accurate information and an updating of one's beliefs based on that information.
Study 2: Research questions
RQ 1: Can we identify belief-speaking and truth-seeking in public-facing statements by politicians?
RQ 2: How does the usage of the two honesty components in public-facing statements differ between parties and over time?
RQ 3: Is there a link between honesty components and the trustworthiness of information?
Measuring honesty components in text
Distributed Dictionary Representation (DDR) approach [1]:
(1) Create a dictionary of words for each honesty component
(2) Embed each word in the dictionary to obtain a vector representation & average all words.
(3) Embed each word in the document of interest & average all words.
(4) Calculate the cosine similarity between the averaged embeddings.
[1] Garten, J. et al.,"Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis" in Behaviour Research Methods, 2018.
Step 1: Dictionary creation
[1] Di Natale. et al.,"LEXpander: applying colexification networks to automated lexicon expansion" in arXiv, 2022.
Step 1: Dictionary refinement
Step 1: Dictionary validation
Step 1: Dictionary validation
Step 1: Dictionary validation
Step 1: Final dictionaries
Dictionaries available at https://osf.io/mq6kz.
Step 2: Word embeddings
Step 2: Word embeddings
Step 2: embedding the dictionaries
Step 3: embedding the tweets
Step 4: calculating the similarity
Bonus: document level validation
Recap
We now have measurement instruments that allow us to measure belief-speaking similarity Db and truth-seeking similarity Dt in texts.
Db < 0 means a text is dissimilar to belief-speaking.
Db > 0 means a text is similar to belief-speaking.
Dt < 0 means a text is dissimilar to truth-seeking.
Dt > 0 means a text is similar to truth-seeking.
Belief-speaking and truth-seeking in Tweets of U.S. politicians
Words associated with honesty components
Produced with scattertext https://github.com/JasonKessler/scattertext.
Example tweets
Produced with scattertext https://github.com/JasonKessler/scattertext.
Development of honesty components over time
Both belief speaking and truth seeking increased after Trump's election.
Development of honesty components over time
Democrats lead, Republicans follow.
How do the honesty components relate to sharing of untrustworthy information?
Belief-speaking and untrustworthy information
An increase in belief-speaking similarity of tweets of 10% predicts a drop in average NewsGuard score of 12.8 points for Republicans.
Truth-seeking and untrustworthy information
An increase in truth-seeking similarity of tweets of 10% predicts an increase in average NG score of 10.6 points for Republicans and 2.1 points for Democrats.
Honesty components in linked articles
We collect the text of 154,000 news articles linked in the tweets.
We confirm the effects and also find a relationship for belief-speaking and Democrats.
Correlation with NewsGuard scores
We reproduce these correlations with an independently compiled trustworthiness data base.
Study 2: summary
RQ 1: We develop a new instrument to measure the honesty components belief-speaking and truth-seeking in political speech using DDR.
RQ 2: There is significant variance of honesty components over time and between parties.
RQ 3a: There is a strong negative correlation between belief-speaking and untrustworthy information for Republicans.
RQ 3b: There is a strong positive correlation between truth-seeking and trustworthy information for both parties.
"From Alternative conceptions of honesty to alternative facts in communications by U.S. politicians", Lasser et al. 2023, arXiv.
Discussion
(-) Both Democrats and Republicans increasingly use both belief-speaking and truth-seeking in public-facing communication on social media.
(-) Republicans increasingly share untrustworthy information, Democrats do not.
(-) Belief-speaking and low information trustworthiness are strongly correlated mostly for Republicans.
Belief-speaking might be necessary but not sufficient to engage in sharing of untrustworthy information.