Central principles of measurement

Adapted from Lazer et al. 2021: Meaningul measures of human society in the twenty-first century

Central principles of measurement

Adapted from Lazer et al. 2021: Meaningul measures of human society in the twenty-first century

Central principles of measurement

Adapted from Lazer et al. 2021: Meaningul measures of human society in the twenty-first century

Central principles of measurement

Adapted from Lazer et al. 2021: Meaningul measures of human society in the twenty-first century

Central principles of measurement

Adapted from Lazer et al. 2021: Meaningul measures of human society in the twenty-first century

Issues with measurement: Reflexivity

Imagine this poll is posted shortly before an election:

How would you expect voters of black to react to the poll?

How voters of red? How voters of green?

Issues with measurement: Reflexivity

Imagine this poll is posted shortly before an election:

Simon 1954: Bandwagon and Underdog Effects and the Possibility of Election Predictions

Reflexivity

Humans actively change the world they are observing by acting on the knowledge gained   →   self-fulfilling prophecy.

Measurements can distort the phenomena they are designed to measure.

Observer effects

Davis 1997: The Direction of Race of Interviewer Effects among African-Americans: Donning the Black Mask.

Observer effects

Observer effects

Observer effects

Observer effects

Observer effects

Algorithmic confounding

Systematic algorithmic biases
Example: The performativity of networks

Sudden changes in algorithms
Example: Changes in a recommender system

Algorithm "reflexivity"
Example: The Parable of Google Flu

Further reading: Wagner et al. 2021: Measuring algorithmically infused societies

The ambiguity of human expressions

Li et al. 2020: A brief history of risk.

The ambiguity of human expressions

Human language is drifting: Words might change meaning, become dog whistles and new words might be created.

There are advances in automated sarcasm detection (see for example Hazarika et al. 2018) but detecting sarcasm, irony and hyperbole in text remains notoriously hard.

The context of a text might completely change it's meaning.
What contexts can you think of that are relevant for social media?

The meaning of measures

The meaning of measures

Goal: measure a theoretical concept (infection risk, media consumption, physical activity, ...).

To approximate the concept we need to extract meaning from the raw data available to us. We do this by constructing measures.

There will be slippage between construct and concept.

Constructs and even concepts will be unstable in space and time.

Ways forward: Triangulating measures

Garcia et al. 2021: Social media emotion macroscopes reflect emotional experiences in society at large.

Ways forward: Linking data sources

Hughes et al. 2021: Using Administrative Records and Survey Data to Construct Samples of Tweeters and Tweets.

Ways forward: Updating models and measures

Established constructs might have drastically changed and need to be updated to match current social realities (example: voter interview questionnaire).

Many data sources offer continuous data streams. This gives the opportunity for continuous evaluation.