Topic on Research talk:ORES paper

As currently written, it sounds like we are setting up a straw man argument here, and I don't yet see what this review of previous interventions motivates. Where are we going with this?

OK so my thoughts are here that "we've tried the obvious solutions" that are dominant in the literature. This is why I point to "nudge". I want to use that as a placeholder for a set of strategies that are dominant in the HCI literature:

1. Identify a desired behavioral change
2. Redesign interface to "nudge" behavior
3. Run an A/B test to see if nudge is effective
4. Iterate.

In nearly all cases, we have shown a lack of substantial (and maybe even significant) effect from these UI nudges (see Growth Experiments). Jtmorgan and I know that the Teahouse is a notable exception, but in a lot of ways, this is the point. The Teahouse isn't a nudge. It's a self-sustaining sub-system of Wikipedia built to empower Wikipedians to solve their own problems. It's far less removed than an interface change intended to direct behavior.

Agreed, and I think the Teahouse is a great example of a nudge vs. something else I don't have a word for yet. I don't want to make it a strawman, but I do think that there is something pretty different.

This might be a side-rant, but I have really come to find the "nudge" approach problematic and would be happy to challenge that in this paper. And I certainly got caught up in that mindset years ago, wanting to make a few small changes that would fix everything forever. It is a really appealing approach for a lot of reasons. There are a few nudges that do genuinely work, and those get really high-profile publications and TED talks. But we rarely hear about all the nudges people try that don't work (publication bias, etc.). So there is this powerful belief that we can achieve social progress primarily through small, non-controversial technological changes. It's great when you find a nudge that works, but if your goal is changing long-established patterns of behavior, then a designer/developer should probably expect that 95% of their nudges won't work, rather than the opposite.

I see where you're coming from, but if we're going to critique nudges, we should engage Thaler and Sunstein more directly, and be clearer about how what we're proposing is different from a nudge. Because while Snuggle/Teahouse may not be nudges, both systems employ choice architecture in their design. And I expect that tools built on top of ORES will as well, at least based on some of the (probably outdated by now) ReviewStream concept mocks I've seen.

I like the point that @Jtmorganis making, while a (proposed language in quotes) 'systems-level intervention' might be the only approach shown to be effective, there are lower/'nudge-level' dimensions that matter _in that systems-level design_

An analogy that's difficult to delineate, but feels intuitive, is ecology. It's common to talk about systems-level change (e.g. The role of a swamp in cleaning water nearby), but have that outcome fail if one of the 'nudge-level details is left out (e.g. The correct form of bacteria may not be able to live in certain climates, and thus won't clean the water properly).

That styling is bizarre, and air can't figure out how to fix it.

I don't think it's fair to refer to the Teahouse as a designed thing in the way that "choice architecture" imagines designed things. I see the Teahouse as 5% designed things that formed a bedrock and conveyed a specific set of ideas and 95% what people chose to do with those ideas. The 5% of designed things affect very little direct change while the 95% of what the Teahouse hosts have made the Teahouse into is critical. The behaviors of Teahouse hosts may have been intended but they were not really designed. Instead they emerged. By relying on emergence, the founders of the Teahouse were taking a risk and better on "hearing to speech" -- if we design a space that is explicitly for a certain type of behavior (with the right nudges), then from that we might see a sustainable community turn our designed things into something that fits their view of newcomer socialization and support.

Also, the Teahouse nudges seem to be once-removed. Teahouse designers aren't nudging the newcomer (except maybe with what questions to ask -- not really nudging newcomers to stick around Wikipedia though). They were lightly nudging the Teahouse hosts maybe. But it seems to me that a more apt metaphor is that the Teahouse designer is that of founders. They made fertile ground for the Teahouse to grow, but what the Teahouse became was largely up to the Hosts who would take over running it.

Jtmorgan, does this jibe? I'm not sure my knowledge of the Teahouse's history is complete enough.

@EpochFailI'm going to stop using the word "nudge" for a moment in order to draw attention to a variety of small design choices that we made when we created the Teahouse with the goal of fostering setting particular expectations, communicating particular messages, and suggesting particular courses of action for hosts and guests:

1. the Teahouse welcome message - a "nicer message" intended to contrast with other, less nice messages new editors might receive on their talkpages.
2. the 5 host expectations - a short list of !rules that communicate the way hosts should interact with guests at the Teahouse
3. Ask a question OR create a profile - two equally-weighted calls to action on the Teahouse landing page, communicating that guests are still welcome to participate (by creating a profile) even if they don't currently have a question to ask
4. Host profiles - an auto-sorted list of recently active Teahouse hosts who are willing to share a little about themselves, and will help out if contacted directly
5. "sign your post with five tidles" prompt - a prompt in the Teahouse Q&A gadget that teaches new editors how to sign their posts on talkpages
6. Host badges - a series of badges (basically Teahouse-specific barnstars) related to desirable behaviors that hosts can give to one another, and place on their profiles (pretty popular, for a while)

There are a lot more examples, but this makes my point I think. The point is that the Teahouse is in some ways a very designed thing, and that like most designed things it's full of nudges. Thaler and Sunstein didn't invent nudges, they just developed a theoretical framework to describe the phenomenon. The framework may make it seem like behavioral change is easier than it actually is, but that doesn't mean that small-scale interventions can't work.

I think what you're trying to get at in the criticism of nudges is that you can't expect any old small design tweak to work the way you want it to. You have to take a system/ecological perspective and understand the potential impact of that change in context, and the way people other than you will understand the change. You can't just add a "like" button to any old forum and expect that people will use it like they do on Facebook. So if we want to critique past WMF new editor engagement initiatives or any other unsuccessful design interventions in an honest way, we need to talk about how these specific changes were and were not contextually appropriate.

I agree with you that ultimately in the case of the Teahouse and pretty much any other successful, self-sustaining designed system the end users need to be able to appropriate/reshape/reinterpret the system according to their own needs and desires. But initial conditions often have a big impact on that process, and small design decisions make a big difference. If, when we created the Teahouse, we hadn't made "Welcome everyone" !rule #1, it would probably not be the friendly place it is today.

I think we're talking about two things here. The Teahouse itself is not a nudge, but design decisions (even those that are part of the Teahouse!) can cause nudges. Sure. It seems that maybe this hit a design-matters nerve? I'm certainly not trying to make the argument that design doesn't matter. Instead, I'm trying to make the argument that nudges/minor-design-changes alone are the wrong strategy for addressing a problematic cultural state like the dominant quality control culture in Wikipedia. We've tried many simple "nudges" directed at newcomers with little effect on retention (see the history of the Growth team). I think that addressing a problem like reduced/biased retention requires more than nudges to encourage newcomers to create profiles for themselves or to make copy edits rather than big contributions. It requires a culture shift. I think your "5 host expectations" is a good example of something that is totally not a nudge, but more of a purposeful, culture statement. By making "Welcome everyone" !rule #1, you weren't implementing a nudge at all. You were implementing a cultural norm.