Selected Publications

To engage or not to engage AI for critical judgments: The importance of ambiguity in professionals’ judgment process

Lebovitz S., Lifshitz H., and Levina N. 2022

Organization Science. 33(1):126-14

Artificial intelligence (AI) technologies promise to transform how professionals conduct knowledge work by augmenting their capabilities for making professional judgment. We know little, however, about how human-AI augmentation takes place in practice. Yet gaining this understanding is particularly important when professionals use AI tools to form judgments on critical decisions. We conducted an in-depth field study in a major US hospital where AI tools were used in three departments by diagnostic radiologists making breast cancer, lung cancer, and bone age determinations. The study illustrates the hindering effects of opacity that professionals experienced when using AI tools and explores how these professionals grappled with it in practice. In all three departments, this opacity resulted in professionals experiencing increased uncertainty because AI tool results often diverged from their initial judgment without providing underlying reasoning. Only in one department (of the three), did professionals consistently incorporate AI results into their final judgments, achieving what we call engaged augmentation. These professionals invested in AI interrogation practices – practices enacted by human experts to relate their own knowledge claims to AI knowledge claims. Professionals in the other two departments did not enact such practices and did not incorporate AI inputs into their final decisions, which we call un-engaged “augmentation.” Our study unpacks the challenges involved in augmenting professional judgment with powerful, yet opaque, technologies and contributes to literatures on AI adoption in knowledge work.

Is AI ground truth really “true”? The dangers of training and evaluating AI tools based on experts’ know-what

Lebovitz S., and Levina N. Lifshitz H. (2021)

Management Information Systems Quarterly, , 45(3), 1501-1526

Organizational decision-makers need to evaluate AI tools in light of increasing claims that such tools outperform human experts. Yet, measuring the quality of knowledge work is challenging, raising the question of how to evaluate AI performance in such contexts. We investigate this question through a field study of a major US hospital, observing how managers evaluated five different machine-learning (ML) based AI tools. Each tool reported high performance according to standard AI accuracy measures, which were based on ground truth labels provided by qualified experts. Trying these tools out in practice, however, revealed that none of them met expectations. Searching for explanations, managers began confronting the high uncertainty of experts’ know-what knowledge captured in ground truth labels used to train and validate ML models. In practice, experts address this uncertainty by drawing on rich know-how practices, which were not incorporated into these ML-based tools. Discovering the disconnect between AI’s know-what and experts’ know-how enabled managers to better understand the risks and benefits of each tool. This study shows dangers of treating ground truth labels used in ML models objectively when the underlying knowledge is uncertain. We outline implications of our study for developing, training, and evaluating AI for knowledge work.

Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality

F Dell’Acqua, E McFowland III, ER Mollick, H Lifshitz, K Kellogg, … (2023)

Harvard Business School Technology & Operations Mgt. Unit Working Paper

The public release of Large Language Models (LLMs) has sparked tremendous interest in how humans will use Artificial Intelligence (AI) to accomplish a variety of tasks. In our study conducted with Boston Consulting Group, a global management consulting firm, we examine the performance implications of AI on realistic, complex, and knowledge-intensive tasks. The pre-registered experiment involved 758 consultants comprising about 7% of the individual contributor-level consultants at the company.

Open innovation and organizational boundaries: task decomposition, knowledge distribution and the locus of innovation

KR Lakhani, H Lifshitz, M Tushman (2013)

Handbook of economic organization: Integrating economic and organizational …

This chapter contrasts traditional, organization-centered models of innovation with more recent work on open innovation. These fundamentally different and inconsistent innovation logics are associated with contrasting organizational boundaries and organizational designs. We suggest that when critical tasks can be modularized and when problem-solving knowledge is widely distributed and available, open innovation complements traditional innovation logics. We induce these ideas from the literature and with extended examples from Apple, the National Aeronautics and Astronomical Agency (NASA) and LEGO. We suggest that task decomposition and problem-solving knowledge distribution are not deterministic but are strategic choices. If dynamic capabilities are associated with innovation streams, and if different innovation types are rooted in contrasting innovation logics, there are important implications for firm boundaries, design and identity. 

The Open Innovation in Science research field: a collaborative conceptualisation approach

S Beck, C Bergenholtz, M Bogers, TM Brasseur, ML Conradsen, Hila Lifshitz, … (2022)

Industry and Innovation 29 (2), 136-185

Openness and collaboration in scientific research are attracting increasing attention from scholars and practitioners alike. However, a common understanding of these phenomena is hindered by disciplinary boundaries and disconnected research streams. We link dispersed knowledge on Open Innovation, Open Science, and related concepts such as Responsible Research and Innovation by proposing a unifying Open Innovation in Science (OIS) Research Framework. This framework captures the antecedents, contingencies, and consequences of open and collaborative practices along the entire process of generating and disseminating scientific insights and translating them into innovation. Moreover, it elucidates individual-, team-, organisation-, field-, and society‐level factors shaping OIS practices. To conceptualise the framework, we employed a collaborative approach involving 47 scholars from multiple …

Crisis as opportunity, disruption and exposure: Exploring emergent responses to crisis through digital technology

M Gkeredakis, H Lifshitz, M Barrett (2021)

Information and Organization 31 (1), 100344

We live in a technologically advanced era with a recent and marked dependence on digital technologies while also facing increasingly frequent extreme and global crises. Crises, like the COVID-19 pandemic, are significantly impacting our societies, organizations and individuals and dramatically shifting the use of, and dependence on, digital technology. The way digital technology is used to cope with crises is novel and not well understood theoretically. To explore the varied uses and impact of digital technologies during crises, we propose to view crisis as (1) opportunity, (2) disruption, and (3) exposure. Examining crisis as opportunity reveals how digital technologies enable experimentation and accelerate innovation while raising coordination challenges and risky implementation. Viewing crisis as disruption highlights how digital technologies enable the rapid shifting of organizational and occupational practices …

On the” how” and” why” of emergent role behaviors in Wikipedia

O Arazy, H Lifshitz, O Nov, J Daxenberger, M Balestra, C Cheshire (2017)

Proceedings of the 2017 ACM conference on computer supported cooperative …

Research on peer-production suggests that as participants choose what actions to perform, prototypical activity patterns emerge. Recent work characterized these patterns and demonstrated that informal emergent roles are highly stable. Nonetheless, we know little about the ways in which contributors take on and shed emergent roles. The objectives of this study are to: (a) delineate the temporal dynamics of participants’ emergent role taking behaviors, and (b) identify the motivations driving role-transition behaviors. Our study links motivation to role-transition behaviors within Wikipedia. Our first sample covered eleven years and 222,119 contributors, and was used to identify four categories of temporal role-taking behaviors, that differ in their mobility between emergent roles and across Wikipedia articles. Our second examination linked the motivations of 175 new participants to their subsequent role-taking activity …

Intelligence augmentation: Human factors in ai and future of work

S Paul, L Yuan, HK Jain, LP Robert Jr, J Spohrer, H Lifshitz (2022)

AIS Transactions on Human-Computer Interaction 14 (3), 426-445

The availability of parallel and distributed processing at a reasonable cost and the diversity of data sources have contributed to advanced developments in artificial intelligence (AI). These developments in the AI computing environment are not concomitant with changes in the social, legal, and political environment. While considering deploying AI, the deployment context and the end goal of human intelligence augmentation for that specific context have surfaced as significant factors for professionals, organizations, and society. In this research commentary, we highlight some important socio-technical aspects associated with recent growth in AI systems. We elaborate on the intricacies of human-machine interaction that form the foundation of augmented intelligence. We also highlight the ethical considerations that relate to these interactions and explain how augmented intelligence can play a key role in shaping the future of human work.

Analogy Mining for Specific Design Needs

SD Gilon K., Chan J., Ng F., Lifshitz H., Kittur A. (2018)

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems 121

Using qualitative research for “how” questions

M Anteby, H Lifshitz, M Tushman (2014)

Strategic Management Journal 3

Open innovation and organizational boundaries: the impact of task decomposition and knowledge distribution on the locus of innovation in Handbook of Economic Organization …

K Lakhani, H Lifshitz, M Tushman (2013)

Handbook of Economic Organization: Integrating Economic and Organization …

This paper contrasts traditional, internal organization‐centered models of innovation with more recent work on open innovation. These fundamentally different and inconsistent innovation logics are associated with contrasting organizational boundaries and organizational designs. We suggest that when critical tasks can be modularized and when problem‐solving knowledge is widely distributed and available, open innovation complements traditional innovation logics. We induce these ideas from the literature and with extended examples from Apple, NASA, and LEGO. We suggest that task decomposition and problem‐solving knowledge distribution are not deterministic but are strategic choices. If dynamic capabilities are associated with innovation streams, and if innovation types are rooted in contrasting innovation logics, there are important implications for the firm, and its boundaries, design, and identity.

Report on the 2020 foss contributor survey

F Nagle, DA Wheeler, H Lifshitz, H Ham, J Hoffman (2020)

The Linux Foundation Core Infrastructure Initiative

Doubting the diagnosis: How artificial intelligence increases ambiguity during professional decision making

S Lebovitz, H Lifshitz, N Levina (2019)

New York University

The Digital Economy Runs on Open Source. Here’s How to Protect It

H Lifshitz, F Nagle (2021)

Harvard Business Review

Experts Innovating with the Crowd: Towards a Solver-Aware Systems Architecting Framework for the Design of Complex Systems

Szajnfarber Z., Topcu T., and Lifshitz, H. (2022)

Design Science, Vol 8, e10, DOI: https://doi.org/10.1017/dsj.2022.7

This article proposes the solver-aware system architecting framework for leveraging the combined strengths of experts, crowds and specialists to design innovative complex systems. Although system architecting theory has extensively explored the relationship between alternative architecture forms and performance under operational uncertainty, limited attention has been paid to differences due to who generates the solutions. The recent rise in alternative solving methods, from gig workers to crowdsourcing to novel contracting structures emphasises the need for deeper consideration of the link between architecting and solver-capability in the context of complex system innovation. We investigate these interactions through an abstract problem-solving simulation, representing alternative decompositions and solver archetypes of varying expertise, engaged through contractual structures that match their solving type. We find that the preferred architecture changes depending on which combinations of solvers are assigned. In addition, the best hybrid decomposition-solver combinations simultaneously improve performance and cost, while reducing expert reliance. To operationalise this new solver-aware framework, we induce two heuristics for decomposition-assignment pairs and demonstrate the scale of their value in the simulation. We also apply these two heuristics to reason about an example of a robotic manipulator design problem to demonstrate their relevance in realistic complex system settings. 

Minimal and Adaptive Coordination: How Hackathons’ Projects Accelerate Innovation without Killing it

Lifshitz H., Lebovitz S., and Zalmanson L. (2020)

Academy of Management Journal, in- press.

The innovation journey of new product development processes often spans weeks or months. Recently, hackathons have turned the journey into an ad hoc sprint of only a couple of days using new tools and technologies. Existing research predicts such conditions would result in failure to produce new working products, yet hackathons often lead to functioning innovative products. To investigate this puzzle, we closely studied the product development process of 13 comparable projects in assistive technology hackathons. We find that accelerating innovation created temporal ambiguity, as it was unclear how to coordinate the challenging work within such an extremely limited and ad hoc time frame. Multiple projects worked to reduce this ambiguity, importing temporal structures from organizational innovation processes and compressing them to fit the extremely limited and ad-hoc time frame. They worked in full coordination to build a new product. They all failed. Only projects that sustained the temporal ambiguity – by working with merely a minimal basis for coordination and let new temporal structures emerge – were able to produce functioning new products under the intense time pressure. This study contributes to theories on innovation processes, coordination, and temporality.

Scaling Up Analogical Innovation with Crowds and AI

Kittur A., Yu L., Hope T., Chan J., Lifshitz H., Gilon K., Ng F., Kraut R.E., and Shahaf D. (2019)

Proceedings of the National Academy of Sciences, 116 (6), 1870-1877.

Analogy—the ability to find and apply deep structural patterns across domains—has been fundamental to human innovation in science and technology. Today there is a growing opportunity to accelerate innovation by moving analogy out of a single person’s mind and distributing it across many information processors, both human and machine. Doing so has the potential to overcome cognitive fixation, scale to large idea repositories, and support complex problems with multiple constraints. Here we lay out a perspective on the future of scalable analogical innovation and first steps using crowds and artificial intelligence (AI) to augment creativity that quantitatively demonstrate the promise of the approach, as well as core challenges critical to realizing this vision.

Dismantling Knowledge Boundaries at NASA: The Critical Role of Professional Identity in Open Innovation

Lifshitz H. (2018)

Administrative Science Quarterly, 63(4), 746–782.

Using a longitudinal in-depth field study at NASA, I investigate how the open, or peer-production, innovation model affects R&D professionals, their work, and the locus of innovation. R&D professionals are known for keeping their knowledge work within clearly defined boundaries, protecting it from individuals outside those boundaries, and rejecting meritorious innovation that is created outside disciplinary boundaries. The open innovation model challenges these boundaries and opens the knowledge work to be conducted by anyone who chooses to contribute. At NASA, the open model led to a scientific breakthrough at unprecedented speed using unusually limited resources; yet it challenged not only the knowledge-work boundaries but also the professional identity of the R&D professionals. This led to divergent reactions from R&D professionals, as adopting the open model required them to go through a multifaceted transformation. Only R&D professionals who underwent identity refocusing work dismantled their boundaries, truly adopting the knowledge from outside and sharing their internal knowledge. Others who did not go through that identity work failed to incorporate the solutions the open model produced. Adopting open innovation without a change in R&D professionals’ identity resulted in no real change in the R&D process. This paper reveals how such processes unfold and illustrates the critical role of professional identity work in changing knowledge-work boundaries and shifting the locus of innovation.

Neither a Bazaar nor a Cathedral: the Interplay between Structure and Agency in Wikipedia’s Role System

Arazy O., Lifshitz H., and Balila A. 2018.

Journal of the Association for Information Science and Technology, 70(1), 3-15.

Roles provide a key coordination mechanism in peer-production. Whereas one stream in the literature has focused on the structural responsibilities associated with roles, the another has stressed the emergent nature of work. To date, these streams have proceeded largely in parallel. In seeking to enhance our understanding of the tension between structure and agency in peerproduction, we investigate the interplay between structural and emergent roles. Our study explored the breadth of structural roles in Wikipedia (English version) and their linkage to various forms of activities. Our analyses show that despite the latitude in selecting their mode of participation, participants’ structural and emergent roles are tightly coupled. Our discussion highlights that: (I) participants often stay close to the “production ground floor” despite the assignment into structural roles; and (II) there are typical modifications in activity patterns associated with role-assignment, namely: functional specialization, multi-specialization, defunctionalization, changes in communication patterns, management of identity, and role definition. We contribute to theory of coordination and roles, as well as provide some practical implications.

The Art of Balancing Autonomy and Control: What Managers Can Learn from Hackathon Organizers about Spurring Innovation

Lifshitz H., Lebovitz S., and Zalmanson L. (2018)

MIT Sloan Management Review, 60(2), 1–6

Today, managers recognize that innovation requires a high level of work autonomy for their employees. This encourages curiosity, enables independent thinking, and provides an environment in which employees can experiment and test new problem-solving approaches with minimal fear of failure. At the same time, top-level management and shareholders expect managers to innovate at an increasingly demanding pace, putting top-down pressure on employees to channel this autonomy into productivity. The challenge for managers becomes figuring out how to balance autonomy and control in order to achieve organizational goals without jeopardizing innovation.

Turbulent Stability of Emergent Roles: The Dualistic Nature of Self-Organizing Knowledge Co-Production

Arazy O., Daxenberg J., Lifshitz H., Nov O., and Gurevych I. (2016)

Information Systems Research, 27(4), 792–812.

Increasingly, new forms of organizing for knowledge production are built around self-organizing co-production community models with ambiguous role definitions. Current theories struggle to explain how high-quality knowledge is developed in these settings and how participants self-organize in the absence of role definitions, traditional organizational controls, or formal coordination mechanisms. In this article, we engage the puzzle by investigating the temporal dynamics underlying emergent roles on individual and organizational levels. Comprised of a multi-level large-scale empirical study of Wikipedia stretching over a decade, our study investigates emergent roles in terms of prototypical activity patterns that organically emerge from individuals’ knowledge production actions. Employing a stratified sample of a thousand Wikipedia articles, we tracked two hundred thousand distinct participants and seven hundred thousand coproduction activities, and recorded each activity’s type. We found that participants’ role taking behavior is turbulent across roles, with substantial flow in and out of co-production work. Our findings at the organizational level, however, show that work is organized around a highly stable set of emergent roles, despite the absence of traditional stabilizing mechanisms such as pre-defined work procedures or role expectations. This dualism in emergent work is conceptualized as “Turbulent Stability”. We attribute the stabilizing factor to the artifact-centric production process and present evidence to illustrate the mutual adjustment of role taking according to the artifact’s needs and stage. We discuss the importance of the a↵ordances of Wikipedia in enabling such tacit coordination. This study advances our theoretical understanding of the nature of emergent roles and self-organizing knowledge coproduction. We discuss the implications for custodians of online communities, as well as for managers of firms engaging in selforganized knowledge collaboration.

“Open Innovation and Organization Design.

Tushman, M., Lifshitz H., and Lakhani K. (2012)

Special Issue on Future of Organizational Design, Journal of Organization Design, no. 1: 24–27. (SSRN’s Top Ten download list for Organizational Structural Designs, Innovation & Product Development)

Abernathy’s (1978) empirical work on the automotive industry investigated relationships among an organization’s boundary (all manufacturing plants), its organizational design (fluid vs. specific), and its ability to execute product and/or process innovations. Abernathy’s ideas of dominant designs and the locus of innovation have been central to scholars of innovation, R&D, and strategic management. Similarly, building on March and Simon’s (1958) concept of organizations as decision making systems, Woodward (1965), Burns and Stalker (1966), and Lawrence and Lorsch (1967) examined relationships among organizational boundaries, organization structure, and innovation in a set of industries that varied by technology and environmental uncertainty. These and other early empirical works have led a diverse group of scholars to develop theories about firm boundaries, organization design, and the ability to innovate.