Better Metrics for a Healthier Society
We need to incentivize technologists and tech companies to do more than growth hack. One solution is to invest in public measures of long term value. Read to the end for opportunities to be involved.
Technically speaking, I can improve the economy by breaking your car window—economic activity increases when you pay to get it fixed. But it’s obvious that our world is not better with more vandalism. As many have noted, a single-minded focus on a metric like gross domestic product can miss the broader picture of what is important. This type of single-metric focus is also pervasive in tech, where startup culture preaches rapid experimentation and “growth hacking” with sometimes perverse results for longer term value to users and society.
It is entirely possible for companies - especially more mature companies - to provide more value when they are not growing. I recently left a job at Facebook (now Meta) in part because the mission of connecting people seemed to be taking a back seat to beating TikTok on time spent in the app. Instead of figuring out how to connect people in the ways they find most meaningful, meaningfulness gets defined in terms of business friendly engagement and entertaining strangers start to fill our feeds. Is a back and forth interaction between friends about a funny video really as connecting as the same interaction about something that happened in their lives? If users wanted short form video in addition to pictures of their friends’ kids, do they really need it in the same app? TikTok already exists. Is making a clone of it in Instagram really serving anyone other than the business?
Once upon a time, Facebook was a place where I could connect with all my friends and see their kids grow up. It still fills that function for many and it could innovate new and interesting ways for people to connect with the people they know, rather than creators and influencers who they don’t. But the market’s obsession with growth and market share make that impossible. What if we could actually measure the value a product brings to people’s lives rather than the business results that well meaning but hardly neutral executives use as a proxy? In that world, Facebook might spend its time making deeper interactions rather than more of them.
It’s easy to pick on Facebook but the tyranny of short term metrics goes beyond any company or industry. Almost every ad on the internet optimizes for clicks and also contributes to ad blindness by being awful and clickbait-y. Ranker, which I helped start before Facebook, has grown into a very successful ad supported business with over 100 employees. But back when the company was trying to get to profitability, we had a hard time getting truly beneficial crowdsourced rankings like the most important life goals or bucket lists for teens to “go viral”, and ended up finding success by steering most of our resources into content that would get more clicks in social feeds.
In a similar way, when Obama was elected, people lauded the data driven insight that you made more money if you didn’t worry about turning people off with more donation appeals. The short term metric of money made trumped any concern about the long term relationship that Obama wanted to have with those receiving his emails or their experience receiving such a volume of email. Politicians now routinely talk about how they a/b test content for performance and end up generating more outrageous and polarizing content as a result, which drives up their online popularity, but turns people off to politics as a whole. Given the need for collective action on issues like climate change and pandemics, one could argue that we are literally optimizing ourselves to death.
We need a new culture of metrics that understands the tyranny of short term metric optimization. We all agree that we should measure what matters, but when measuring engagement, users and revenue is easy, but measuring long term user value is hard, the engineer that wants to succeed is going to aim for the former, not the later. Robert Kennedy once said that GDP “measures everything except that which is worthwhile”. The tech industry is on the verge of developing the world’s most advanced AI driven systems that optimize for how long we spend on our phones, how much we click, and how much we consume.
There is an alternative. What if Amazon focused on recommending the best product for long term satisfaction rather than the one that was most profitable? What if Google measured when a search result provided new insight rather than just a link to a website we spent time on? What if social media companies measured when we reconnected with a friend and gave or received emotional support? What if TikTok and YouTube measured when someone learned something truly useful and competed on that metric instead of time spent? Business metrics are not inherently evil but they also aren’t always aligned with long term user value and I’d much rather invest in a business that is measurably both profitable AND is providing such long term value to its users. Providing better experiences to users often also leads to better outcomes for society. For example, the changes detailed in a recent Wall St. Journal article showed how providing users with content that was more “worth their time”, as opposed to content that was shared or commented on, led to reductions in bullying, misinformation, and graphic content.
Measuring long term value is hard but not impossible. Many companies run surveys of user experience that are part of the answer, but in my experience, these metrics often are treated as second class citizens in product decision making. Society could help by creating standardized metrics of where people are learning new things or connecting with others that could motivate companies to compete in the realm of user value rather than the realm of time spent, by giving them credit where credit is due. It’s infinitely harder than measuring engagement and also infinitely more important. Society often debates whether platforms are getting better or worse based on anecdotes, because that is often all we have.
At USC’s Neely Center, we are hoping to provide a critical piece of this puzzle. Leveraging previous work on the effects of bad experiences on users, the signal available in user perceptions of content, and the importance of feeling safe, learning new things and connecting with others in people’s online experiences, we recently launched a nationally representative tracking survey of user experiences across common online communications platforms. In addition to measuring usage and positive and negative experiences, we plan to longitudinally measure variables connected to platform usage, including affective polarization, subjective well-being, and trust in institutions - to determine if these variables change over time in relation to usage and experiences. The survey leverages USC’s Understanding America Study panel and is scheduled to be run monthly for as long as we have funding. We would welcome any help from this community, both in terms of potential funding leads and in terms of interest in working with us directly (see our job announcements at the bottom of this newsletter).
A Dall-E Generated image for “Better Metrics”. In addition to tracking experiences with social media, we will be tracking experiences with generative AI.
Nothing worth doing is easy. AI can help us optimize for anything we want but what we really want is not so easily measured. Measuring long term value is possible but only if we are willing to go the extra mile to accept the cost and complexity of metrics that go beyond usage and profit. As regulators take renewed aim at tech companies, failing to do this may just delay the inevitable. As I sometimes said to my former colleagues, the world has a way of getting companies to do the right thing - it’s just a question of how much pain we are all willing to endure to get there. I’m hopeful that building public metrics of long term value can be part of that path.
—-
Below are a few announcements from our Psychology of Technology Institute community:
We are hiring for two positions at USC’s Neely Center that will help with our measurement efforts, as well as with many other initiatives. See these job ads for a Research Program Manager and a Project Assistant.
Sarah Oh, one of our community members who previously worked at both Facebook and Twitter on issues of user safety, recently published this article about her experiences within Myanmar and was also featured in this Wired article about her new Twitter alternative, T2
We often write about regulating design, and the transparency needed to effectively do that, in this newsletter. Some readers have made the point that regulating transparency can also be seen, by some, as constitutionally problematic and I wanted to share this article by Eric Goldman at Santa Clara University as a perspective that all of us working in this area should take into account.
Gloria Mark, author of Attention Span and a professor at the University of California-Irvine, recently published this op-ed in the Wall Street Journal about how individuals can take control of their digital focus and restore their attention spans.
Adam Watyz, a psychologist and professor at the Kellogg School of Management at Northwestern, recently published this Harvard Business Review article on how the conflation of activity with achievement can be counter-productive for both individuals and companies.
In line with many of the themes we discuss in this newsletter, William Brady, who is also a professor at Northwestern’s Kellogg School of Management, recently shared this pre-print on how algorithms affect social learning online.
Rosanna Guadagno, a professor at the University of Oulu, is helping to hire for multiple positions at her university. This includes a senior research fellow in AI Ethics, a professor in augemented, mixed, and/or VR technology, a professor in learning process data analytics, and a professor in AI based computer vision and machine learning.
We will be presenting a paper on social media’s role in polarization and conflict, in collaboration with Jonathan Stray from UC-Berkeley and Helena Puig Larrauri of Build Up, at the Columbia Knight First Amendment Institute’s conference on algorithmic amplification on April 28th and 29th. The event is open for RSVPs for people interested in attending.
We will also be chairing a panel at the University of Michigan’s Social Media and Society in India conference, which may be of interest for people who are especially focused on technology’s impact in the global south. The event is April 7-8 and both in-person and online participation options are available.