Developing the skills to be successful at the integration of AI, software development, and IT operations has challenged everyone. The demands placed on IT to provide robust production systems for business-critical AI workloads while managing additional environments for development of new initiatives have severely taxed already limited resources. The pressure to advance from concept, through experimentation, and to production has frequently resulted in data scientists and developers abandoning collaboration with IT. Instead, they attempt to proceed faster alone. This situation, which surfaced during the years of rapid change in the business intelligence and microservice-oriented applications development eras, further strains the already challenged relationships between IT, the developer community, and the business management communities.

The experimental nature of data science work makes collaboration and planning challenging. Allocation of IT resources in an environment in which resources are required for “development labs” is difficult to predict. The uncertainty of time to value makes budgeting and workload management nearly impossible. The IT department can feel that it lacks sufficient information to allocate resources effectively, and developers often feel that there is a lack of priority in response to changes in a previously agreed-to plan.

These factors have produced incentives for groups that are involved in the rush to implement AI workloads to behave in ways that are not cost effective for their organizations. The most common ways that groups attempt to “go faster” is to use unmonitored public cloud usage, reuse of equipment for an unintended effort, or acquisition of business unit funding for siloed development initiatives that are outside the official IT capital budgeting process. These types of information systems, which exist largely hidden from managers and official IT units, create the ‘‘shadow IT’’ problem with which all larger organizations must deal. The circumstances that motivate groups to choose the shadow IT route are particularly acute in the AI application and machine intelligence research areas.

This focus on AI, once the interest of only a small community of researchers and computer scientists, is now so intense that many organizations feel that despite considerable investment, they are falling further behind their competitors. The rate of new data management and AI being brought to market increases the cost of system planning and increases the risk of having to change course more frequently.