Selected Publications

To engage or not to engage AI for critical judgments: The importance of ambiguity in professionals’ judgment process

Lebovitz S., Lifshitz-Assaf H., and Levina N. 2022

Organization Science. 33(1):126-14

Artificial intelligence (AI) technologies promise to transform how professionals conduct knowledge work by augmenting their capabilities for making professional judgment. We know little, however, about how human-AI augmentation takes place in practice. Yet gaining this understanding is particularly important when professionals use AI tools to form judgments on critical decisions. We conducted an in-depth field study in a major US hospital where AI tools were used in three departments by diagnostic radiologists making breast cancer, lung cancer, and bone age determinations. The study illustrates the hindering effects of opacity that professionals experienced when using AI tools and explores how these professionals grappled with it in practice. In all three departments, this opacity resulted in professionals experiencing increased uncertainty because AI tool results often diverged from their initial judgment without providing underlying reasoning. Only in one department (of the three), did professionals consistently incorporate AI results into their final judgments, achieving what we call engaged augmentation. These professionals invested in AI interrogation practices – practices enacted by human experts to relate their own knowledge claims to AI knowledge claims. Professionals in the other two departments did not enact such practices and did not incorporate AI inputs into their final decisions, which we call un-engaged “augmentation.” Our study unpacks the challenges involved in augmenting professional judgment with powerful, yet opaque, technologies and contributes to literatures on AI adoption in knowledge work.

Is AI ground truth really “true”? The dangers of training and evaluating AI tools based on experts’ know-what

Lebovitz S., and Levina N. Lifshitz-Assaf H. (2021)

Management Information Systems Quarterly, , 45(3), 1501-1526

Organizational decision-makers need to evaluate AI tools in light of increasing claims that such tools outperform human experts. Yet, measuring the quality of knowledge work is challenging, raising the question of how to evaluate AI performance in such contexts. We investigate this question through a field study of a major US hospital, observing how managers evaluated five different machine-learning (ML) based AI tools. Each tool reported high performance according to standard AI accuracy measures, which were based on ground truth labels provided by qualified experts. Trying these tools out in practice, however, revealed that none of them met expectations. Searching for explanations, managers began confronting the high uncertainty of experts’ know-what knowledge captured in ground truth labels used to train and validate ML models. In practice, experts address this uncertainty by drawing on rich know-how practices, which were not incorporated into these ML-based tools. Discovering the disconnect between AI’s know-what and experts’ know-how enabled managers to better understand the risks and benefits of each tool. This study shows dangers of treating ground truth labels used in ML models objectively when the underlying knowledge is uncertain. We outline implications of our study for developing, training, and evaluating AI for knowledge work.

Minimal and Adaptive Coordination: How Hackathons’ Projects Accelerate Innovation without Killing it

Lifshitz-Assaf H., Lebovitz S., and Zalmanson L. (2020)

Academy of Management Journal, in- press.

The innovation journey of new product development processes often spans weeks or months. Recently, hackathons have turned the journey into an ad hoc sprint of only a couple of days using new tools and technologies. Existing research predicts such conditions would result in failure to produce new working products, yet hackathons often lead to functioning innovative products. To investigate this puzzle, we closely studied the product development process of 13 comparable projects in assistive technology hackathons. We find that accelerating innovation created temporal ambiguity, as it was unclear how to coordinate the challenging work within such an extremely limited and ad hoc time frame. Multiple projects worked to reduce this ambiguity, importing temporal structures from organizational innovation processes and compressing them to fit the extremely limited and ad-hoc time frame. They worked in full coordination to build a new product. They all failed. Only projects that sustained the temporal ambiguity – by working with merely a minimal basis for coordination and let new temporal structures emerge – were able to produce functioning new products under the intense time pressure. This study contributes to theories on innovation processes, coordination, and temporality.

Scaling Up Analogical Innovation with Crowds and AI

Kittur A., Yu L., Hope T., Chan J., Lifshitz-Assaf H., Gilon K., Ng F., Kraut R.E., and Shahaf D. (2019)

Proceedings of the National Academy of Sciences, 116 (6), 1870-1877.

Analogy—the ability to find and apply deep structural patterns across domains—has been fundamental to human innovation in science and technology. Today there is a growing opportunity to accelerate innovation by moving analogy out of a single person’s mind and distributing it across many information processors, both human and machine. Doing so has the potential to overcome cognitive fixation, scale to large idea repositories, and support complex problems with multiple constraints. Here we lay out a perspective on the future of scalable analogical innovation and first steps using crowds and artificial intelligence (AI) to augment creativity that quantitatively demonstrate the promise of the approach, as well as core challenges critical to realizing this vision.

Dismantling Knowledge Boundaries at NASA: The Critical Role of Professional Identity in Open Innovation

Lifshitz-Assaf H. (2018)

Administrative Science Quarterly, 63(4), 746–782.

Using a longitudinal in-depth field study at NASA, I investigate how the open, or peer-production, innovation model affects R&D professionals, their work, and the locus of innovation. R&D professionals are known for keeping their knowledge work within clearly defined boundaries, protecting it from individuals outside those boundaries, and rejecting meritorious innovation that is created outside disciplinary boundaries. The open innovation model challenges these boundaries and opens the knowledge work to be conducted by anyone who chooses to contribute. At NASA, the open model led to a scientific breakthrough at unprecedented speed using unusually limited resources; yet it challenged not only the knowledge-work boundaries but also the professional identity of the R&D professionals. This led to divergent reactions from R&D professionals, as adopting the open model required them to go through a multifaceted transformation. Only R&D professionals who underwent identity refocusing work dismantled their boundaries, truly adopting the knowledge from outside and sharing their internal knowledge. Others who did not go through that identity work failed to incorporate the solutions the open model produced. Adopting open innovation without a change in R&D professionals’ identity resulted in no real change in the R&D process. This paper reveals how such processes unfold and illustrates the critical role of professional identity work in changing knowledge-work boundaries and shifting the locus of innovation.

Neither a Bazaar nor a Cathedral: the Interplay between Structure and Agency in Wikipedia’s Role System

Arazy O., Lifshitz-Assaf H., and Balila A. 2018.

Journal of the Association for Information Science and Technology, 70(1), 3-15.

Roles provide a key coordination mechanism in peer-production. Whereas one stream in the literature has focused on the structural responsibilities associated with roles, the another has stressed the emergent nature of work. To date, these streams have proceeded largely in parallel. In seeking to enhance our understanding of the tension between structure and agency in peerproduction, we investigate the interplay between structural and emergent roles. Our study explored the breadth of structural roles in Wikipedia (English version) and their linkage to various forms of activities. Our analyses show that despite the latitude in selecting their mode of participation, participants’ structural and emergent roles are tightly coupled. Our discussion highlights that: (I) participants often stay close to the “production ground floor” despite the assignment into structural roles; and (II) there are typical modifications in activity patterns associated with role-assignment, namely: functional specialization, multi-specialization, defunctionalization, changes in communication patterns, management of identity, and role definition. We contribute to theory of coordination and roles, as well as provide some practical implications.

The Art of Balancing Autonomy and Control: What Managers Can Learn from Hackathon Organizers about Spurring Innovation

Lifshitz-Assaf H., Lebovitz S., and Zalmanson L. (2018)

MIT Sloan Management Review, 60(2), 1–6

Today, managers recognize that innovation requires a high level of work autonomy for their employees. This encourages curiosity, enables independent thinking, and provides an environment in which employees can experiment and test new problem-solving approaches with minimal fear of failure. At the same time, top-level management and shareholders expect managers to innovate at an increasingly demanding pace, putting top-down pressure on employees to channel this autonomy into productivity. The challenge for managers becomes figuring out how to balance autonomy and control in order to achieve organizational goals without jeopardizing innovation.

Turbulent Stability of Emergent Roles: The Dualistic Nature of Self-Organizing Knowledge Co-Production

Arazy O., Daxenberg J., Lifshitz-Assaf H., Nov O., and Gurevych I. (2016)

Information Systems Research, 27(4), 792–812.

Increasingly, new forms of organizing for knowledge production are built around self-organizing co-production community models with ambiguous role definitions. Current theories struggle to explain how high-quality knowledge is developed in these settings and how participants self-organize in the absence of role definitions, traditional organizational controls, or formal coordination mechanisms. In this article, we engage the puzzle by investigating the temporal dynamics underlying emergent roles on individual and organizational levels. Comprised of a multi-level large-scale empirical study of Wikipedia stretching over a decade, our study investigates emergent roles in terms of prototypical activity patterns that organically emerge from individuals’ knowledge production actions. Employing a stratified sample of a thousand Wikipedia articles, we tracked two hundred thousand distinct participants and seven hundred thousand coproduction activities, and recorded each activity’s type. We found that participants’ role taking behavior is turbulent across roles, with substantial flow in and out of co-production work. Our findings at the organizational level, however, show that work is organized around a highly stable set of emergent roles, despite the absence of traditional stabilizing mechanisms such as pre-defined work procedures or role expectations. This dualism in emergent work is conceptualized as “Turbulent Stability”. We attribute the stabilizing factor to the artifact-centric production process and present evidence to illustrate the mutual adjustment of role taking according to the artifact’s needs and stage. We discuss the importance of the a↵ordances of Wikipedia in enabling such tacit coordination. This study advances our theoretical understanding of the nature of emergent roles and self-organizing knowledge coproduction. We discuss the implications for custodians of online communities, as well as for managers of firms engaging in selforganized knowledge collaboration.

“Open Innovation and Organization Design.

Tushman, M., Lifshitz-Assaf H., and Lakhani K. (2012)

Special Issue on Future of Organizational Design, Journal of Organization Design, no. 1: 24–27. (SSRN’s Top Ten download list for Organizational Structural Designs, Innovation & Product Development)

Abernathy’s (1978) empirical work on the automotive industry investigated relationships among an organization’s boundary (all manufacturing plants), its organizational design (fluid vs. specific), and its ability to execute product and/or process innovations. Abernathy’s ideas of dominant designs and the locus of innovation have been central to scholars of innovation, R&D, and strategic management. Similarly, building on March and Simon’s (1958) concept of organizations as decision making systems, Woodward (1965), Burns and Stalker (1966), and Lawrence and Lorsch (1967) examined relationships among organizational boundaries, organization structure, and innovation in a set of industries that varied by technology and environmental uncertainty. These and other early empirical works have led a diverse group of scholars to develop theories about firm boundaries, organization design, and the ability to innovate.