Managing Complexity: Clarity in Roles

Managing through the growing complexity of AI/ML projects is an essential component to gaining clarity and making better project choices.

First, we need to make sure we have the right resources and that the right people are at the table.

An AI or Machine Learning system has three distinct development roles.

Types of developers needed for AI/ML systems

Oversimplifying the different types of development leads to failure. Here's how it happens and why you can't afford to ignore it:

Organizational Silos

The first barrier to these projects is a structural one.

Depending on your organization, each role may report up to not only a different team but they could be in totally different departments.

It is common to have Data Scientists in the business area, while Developers have their own team in IT or on some Digital team. And finally, your Engineers might be in a data warehousing team that sits in either IT or is in various parts of the business. I've seen every variation - but I've never seen an organizational structure that had these three types of roles under the same manager.

Because of it, it isn't uncommon for a project to be months into development before anyone even consults the right type of developer. The worst outcomes are when it never happens. The worst part is that it's not out of malice, it's out of a genuine misunderstanding of the parts of the work that aren't in your wheelhouse. You've implemented plenty of new software - why should this AI/ML project be any different? The code used by a data scientist looks simple enough!

Language Barriers

The next barrier is language.

They don't 'speak' the same language - literally, they have industry terms which are identical in name but are entirely different in their meaning. They don't 'use' the same languages - literally they all code in different languages and even if they did (ie. Python), the use of it is so different that they might as well be different languages.

Python can be used for all three - but the packages and structure used is entirely different. Even if they 'know' python, that is not enough to turn a software developer into a data scientist. Using the wrong type of developer at the wrong stage of the project leads to an inefficient code base. Plain and simple.

Make sure you have the right resources assigned to the right portions of the project. The days of the unicorn who can do it all - are over.