Selected Publications

To engage or not to engage with AI for critical judgments: How professionals deal with opacity when using AI for medical diagnosis

Lebovitz S., Lifshitz-Assaf H. and Levina N.(2022)

Organization Science. 33(1):126-14 (Special Issue on Theorizing Emerging Technologies)

Augmenting their capabilities for making professional judgment. We know little, however, about how human-AI augmentation takes place in practice. Yet gaining this understanding is particularly important when professionals use AI tools to form judgments on critical decisions. We conducted an in-depth field study in a major US hospital where AI tools were used in three departments by diagnostic radiologists making breast cancer, lung cancer, and bone age determinations. The study illustrates the hindering effects of opacity that professionals experienced when using AI tools and explores how these professionals grappled with it in practice. In all three departments, this opacity resulted in professionals experiencing increased uncertainty because AI tool results often diverged from their initial judgment without providing underlying reasoning. Only in one department (of the three), did professionals consistently incorporate AI results into their final judgments, achieving what we call engaged augmentation. These professionals invested in AI interrogation practices – practices enacted by human experts to relate their own knowledge claims to AI knowledge claims. Professionals in the other two departments did not enact such practices and did not incorporate AI inputs into their final decisions, which we call un-engaged “augmentation.” Our study unpacks the challenges involved in augmenting professional judgment with powerful, yet opaque, technologies and contributes to literatures on AI adoption in knowledge work.

Experts Innovating with the Crowd: Towards a Solver-Aware Systems Architecting Framework for the Design of Complex Systems

Szajnfarber Z., Topcu T., and Lifshitz-Assaf, H. (2022)

Design Science, Vol 8, e10, DOI: https://doi.org/10.1017/dsj.2022.7

This article proposes the solver-aware system architecting framework for leveraging the combined strengths of experts, crowds and specialists to design innovative complex systems. Although system architecting theory has extensively explored the relationship between alternative architecture forms and performance under operational uncertainty, limited attention has been paid to differences due to who generates the solutions. The recent rise in alternative solving methods, from gig workers to crowdsourcing to novel contracting structures emphasises the need for deeper consideration of the link between architecting and solver-capability in the context of complex system innovation. We investigate these interactions through an abstract problem-solving simulation, representing alternative decompositions and solver archetypes of varying expertise, engaged through contractual structures that match their solving type. We find that the preferred architecture changes depending on which combinations of solvers are assigned. In addition, the best hybrid decomposition-solver combinations simultaneously improve performance and cost, while reducing expert reliance. To operationalise this new solver-aware framework, we induce two heuristics for decomposition-assignment pairs and demonstrate the scale of their value in the simulation. We also apply these two heuristics to reason about an example of a robotic manipulator design problem to demonstrate their relevance in realistic complex system settings. 

Is AI ground truth really “true”? The dangers of training and evaluating AI tools based on experts’ know-what

Lebovitz S., and Levina N. Lifshitz-Assaf H. (2021)

Management Information Systems Quarterly (special issue on Managing AI), 45(3), 1501-1526; DOI: 10.25300/MISQ/2021/16564.

Organizational decision-makers need to evaluate AI tools in light of increasing claims that such tools outperform human experts. Yet, measuring the quality of knowledge work is challenging, raising the question of how to evaluate AI performance in such contexts. We investigate this question through a field study of a major US hospital, observing how managers evaluated five different machine-learning (ML) based AI tools. Each tool reported high performance according to standard AI accuracy measures, which were based on ground truth labels provided by qualified experts. Trying these tools out in practice, however, revealed that none of them met expectations. Searching for explanations, managers began confronting the high uncertainty of experts’ know-what knowledge captured in ground truth labels used to train and validate ML models. In practice, experts address this uncertainty by drawing on rich know-how practices, which were not incorporated into these ML-based tools. Discovering the disconnect between AI’s know-what and experts’ know-how enabled managers to better understand the risks and benefits of each tool. This study shows dangers of treating ground truth labels used in ML models objectively when the underlying knowledge is uncertain. We outline implications of our study for developing, training, and evaluating AI for knowledge work.

Crisis as opportunity, disruption and exposure: Exploring emergent responses to crisis through digital technology

Gkeredakis M., Lifshitz-Assaf H., and Barret M. (2021)

Information and Organization, 31(1)

We live in a technologically advanced era with a recent and marked dependence on digital technologies while also facing increasingly frequent extreme and global crises. Crises, like the COVID-19 pandemic, are significantly impacting our societies, organizations and individuals and dramatically shifting the use of, and dependence on, digital technology. The way digital technology is used to cope with crises is novel and not well understood theoretically. To explore the varied uses and impact of digital technologies during crises, we propose to view crisis as (1) opportunity, (2) disruption, and (3) exposure. Examining crisis as opportunity reveals how digital technologies enable experimentation and accelerate innovation while raising coordination challenges and risky implementation. Viewing crisis as disruption highlights how digital technologies enable the rapid shifting of organizational and occupational practices to new digital spaces, allowing work continuity, yet potentially distorting work practices and raising challenges of over-dependence. Finally, crisis exposes the societal implications in making visible and exposing digital inequalities and producing moral dilemmas for us all. We use these three perspectives to shed light on the varied uses of digital technologies in the COVID-19 crisis and suggest new avenues for research on crises more broadly.

Minimal and Adaptive Coordination: How Hackathons’ Projects Accelerate Innovation without Killing it

Lifshitz-Assaf H., Lebovitz S., and Zalmanson L. (2020)

Academy of Management Journal, in- press.

The innovation journey of new product development processes often spans weeks or months. Recently, hackathons have turned the journey into an ad hoc sprint of only a couple of days using new tools and technologies. Existing research predicts such conditions would result in failure to produce new working products, yet hackathons often lead to functioning innovative products. To investigate this puzzle, we closely studied the product development process of 13 comparable projects in assistive technology hackathons. We find that accelerating innovation created temporal ambiguity, as it was unclear how to coordinate the challenging work within such an extremely limited and ad hoc time frame. Multiple projects worked to reduce this ambiguity, importing temporal structures from organizational innovation processes and compressing them to fit the extremely limited and ad-hoc time frame. They worked in full coordination to build a new product. They all failed. Only projects that sustained the temporal ambiguity – by working with merely a minimal basis for coordination and let new temporal structures emerge – were able to produce functioning new products under the intense time pressure. This study contributes to theories on innovation processes, coordination, and temporality.

Scaling Up Analogical Innovation with Crowds and AI

Kittur A., Yu L., Hope T., Chan J., Lifshitz-Assaf H., Gilon K., Ng F., Kraut R.E., and Shahaf D. (2019)

Proceedings of the National Academy of Sciences, 116 (6), 1870-1877.

Analogy—the ability to find and apply deep structural patterns across domains—has been fundamental to human innovation in science and technology. Today there is a growing opportunity to accelerate innovation by moving analogy out of a single person’s mind and distributing it across many information processors, both human and machine. Doing so has the potential to overcome cognitive fixation, scale to large idea repositories, and support complex problems with multiple constraints. Here we lay out a perspective on the future of scalable analogical innovation and first steps using crowds and artificial intelligence (AI) to augment creativity that quantitatively demonstrate the promise of the approach, as well as core challenges critical to realizing this vision.

Dismantling Knowledge Boundaries at NASA: The Critical Role of Professional Identity in Open Innovation

Lifshitz-Assaf H. (2018)

Administrative Science Quarterly, 63(4), 746–782.

Using a longitudinal in-depth field study at NASA, I investigate how the open, or peer-production, innovation model affects R&D professionals, their work, and the locus of innovation. R&D professionals are known for keeping their knowledge work within clearly defined boundaries, protecting it from individuals outside those boundaries, and rejecting meritorious innovation that is created outside disciplinary boundaries. The open innovation model challenges these boundaries and opens the knowledge work to be conducted by anyone who chooses to contribute. At NASA, the open model led to a scientific breakthrough at unprecedented speed using unusually limited resources; yet it challenged not only the knowledge-work boundaries but also the professional identity of the R&D professionals. This led to divergent reactions from R&D professionals, as adopting the open model required them to go through a multifaceted transformation. Only R&D professionals who underwent identity refocusing work dismantled their boundaries, truly adopting the knowledge from outside and sharing their internal knowledge. Others who did not go through that identity work failed to incorporate the solutions the open model produced. Adopting open innovation without a change in R&D professionals’ identity resulted in no real change in the R&D process. This paper reveals how such processes unfold and illustrates the critical role of professional identity work in changing knowledge-work boundaries and shifting the locus of innovation.

Neither a Bazaar nor a Cathedral: the Interplay between Structure and Agency in Wikipedia’s Role System

Arazy O., Lifshitz-Assaf H., and Balila A. 2018.

Journal of the Association for Information Science and Technology, 70(1), 3-15.

Roles provide a key coordination mechanism in peer-production. Whereas one stream in the literature has focused on the structural responsibilities associated with roles, the another has stressed the emergent nature of work. To date, these streams have proceeded largely in parallel. In seeking to enhance our understanding of the tension between structure and agency in peerproduction, we investigate the interplay between structural and emergent roles. Our study explored the breadth of structural roles in Wikipedia (English version) and their linkage to various forms of activities. Our analyses show that despite the latitude in selecting their mode of participation, participants’ structural and emergent roles are tightly coupled. Our discussion highlights that: (I) participants often stay close to the “production ground floor” despite the assignment into structural roles; and (II) there are typical modifications in activity patterns associated with role-assignment, namely: functional specialization, multi-specialization, defunctionalization, changes in communication patterns, management of identity, and role definition. We contribute to theory of coordination and roles, as well as provide some practical implications.

The Art of Balancing Autonomy and Control: What Managers Can Learn from Hackathon Organizers about Spurring Innovation

Lifshitz-Assaf H., Lebovitz S., and Zalmanson L. (2018)

MIT Sloan Management Review, 60(2), 1–6

Today, managers recognize that innovation requires a high level of work autonomy for their employees. This encourages curiosity, enables independent thinking, and provides an environment in which employees can experiment and test new problem-solving approaches with minimal fear of failure. At the same time, top-level management and shareholders expect managers to innovate at an increasingly demanding pace, putting top-down pressure on employees to channel this autonomy into productivity. The challenge for managers becomes figuring out how to balance autonomy and control in order to achieve organizational goals without jeopardizing innovation.

Turbulent Stability of Emergent Roles: The Dualistic Nature of Self-Organizing Knowledge Co-Production

Arazy O., Daxenberg J., Lifshitz-Assaf H., Nov O., and Gurevych I. (2016)

Information Systems Research, 27(4), 792–812.

Increasingly, new forms of organizing for knowledge production are built around self-organizing co-production community models with ambiguous role definitions. Current theories struggle to explain how high-quality knowledge is developed in these settings and how participants self-organize in the absence of role definitions, traditional organizational controls, or formal coordination mechanisms. In this article, we engage the puzzle by investigating the temporal dynamics underlying emergent roles on individual and organizational levels. Comprised of a multi-level large-scale empirical study of Wikipedia stretching over a decade, our study investigates emergent roles in terms of prototypical activity patterns that organically emerge from individuals’ knowledge production actions. Employing a stratified sample of a thousand Wikipedia articles, we tracked two hundred thousand distinct participants and seven hundred thousand coproduction activities, and recorded each activity’s type. We found that participants’ role taking behavior is turbulent across roles, with substantial flow in and out of co-production work. Our findings at the organizational level, however, show that work is organized around a highly stable set of emergent roles, despite the absence of traditional stabilizing mechanisms such as pre-defined work procedures or role expectations. This dualism in emergent work is conceptualized as “Turbulent Stability”. We attribute the stabilizing factor to the artifact-centric production process and present evidence to illustrate the mutual adjustment of role taking according to the artifact’s needs and stage. We discuss the importance of the a↵ordances of Wikipedia in enabling such tacit coordination. This study advances our theoretical understanding of the nature of emergent roles and self-organizing knowledge coproduction. We discuss the implications for custodians of online communities, as well as for managers of firms engaging in selforganized knowledge collaboration.

“Open Innovation and Organization Design.

Tushman, M., Lifshitz-Assaf H., and Lakhani K. (2012)

Special Issue on Future of Organizational Design, Journal of Organization Design, no. 1: 24–27. (SSRN’s Top Ten download list for Organizational Structural Designs, Innovation & Product Development)

Abernathy’s (1978) empirical work on the automotive industry investigated relationships among an organization’s boundary (all manufacturing plants), its organizational design (fluid vs. specific), and its ability to execute product and/or process innovations. Abernathy’s ideas of dominant designs and the locus of innovation have been central to scholars of innovation, R&D, and strategic management. Similarly, building on March and Simon’s (1958) concept of organizations as decision making systems, Woodward (1965), Burns and Stalker (1966), and Lawrence and Lorsch (1967) examined relationships among organizational boundaries, organization structure, and innovation in a set of industries that varied by technology and environmental uncertainty. These and other early empirical works have led a diverse group of scholars to develop theories about firm boundaries, organization design, and the ability to innovate.