Emotional Intelligence for Data Scientists

3 reasons why you need it

Photo by redcharlie on Unsplash

Data science is unlike any other industry. It crashed into the world of work like a giant meteorite. It was sudden, felt like it came out of nowhere, but most of all, it completely changed the way we do things.

In the blink of an eye, if a company wasn’t doing data science, they were seen as irrelevant. Data scientists became overnight superstars, the reasons for which many people didn’t know. Doesn’t matter if you don’t know what they do, you just need one. Can you think of any other industry like that?

And as data science continued its meteoric rise into stardom, it became one of the most challenging jobs in the market. And the top skill you need as a data scientist is the one that people aren’t talking about.

Emotional Intelligence (EQ).

Yes, the irony that a job combining Statistics and Computer Science requires so much emotional intelligence does not escape me, given that these two disciplines are known for… Well, literally the opposite of emotional intelligence.

But there is a massive chasm between the people who know data science and the people who don’t. Knowledge of data science feels black or white, and isn’t showing any signs of improvement. So what’s the result? Here are a few scenarios you might face, and if you’re going to navigate them properly you’ll require more skills with people than programming.

1. Manager doesn’t know data science

This is super common, and isn’t always a bad thing. But it could be problematic if you’re relatively new to the industry. And to make matters worse, it’s an easy trap to fall into, especially if you’re fresh out of education and wanting to get into a job as fast as possible. You might be lured in thinking you’ll have a blank canvas on which to paint your machine learning masterpieces. You may, however, be sadly mistaken.

The problem. A new employee needs direction to begin their job, but a manger with no data science knowledge can only suggest things that they have a grasp on. They might not know about the tools you require; the normal best practise in starting a data science role; the kinds of problems data scientists should tackle and when; the operations of a data scientist that need to be set up.

See it from their perspective: as a manager they know a new employee needs stuff to do, and as long as you appear happy and are getting on with the work they gave you, there’s no reason to believe they’re doing anything wrong.

Where EQ comes in. It is crucial you set up a dialogue early. Address the problem head on: ask them how they plan to get you settled in over the next few weeks and months. Use this as an opportunity to also mention the things that you expect to be doing.

Let them know that you need to set up your environment and install necessary software ASAP. This is a good opportunity to educate them in the tools you use and what you use them for, if they didn’t know already.

Listen to their domain expertise and their guidance on how the company is structured. Find out where the data come from, and the key stakeholders in the whole process. Ask them what the high-level expectations of the role are.

Give them an idea of what you expect to achieve, realistically. If you’re the only data scientist, tell them that building models can take time, and that a proper workflow should be discussed between the two of you. This might be a good place to double check that they actually know what machine learning is, and start educating them if not.

Ideally, having this continuous dialogue right from the beginning will enable you find a good middle-ground going forward. You can get some space to experiment with building models, and you can become aware of the business-as-usual tasks they’ll want you to work with. Listen, and be honest with them. That’s the key.

2. Company thinks they’re data driven but aren’t.

They think you will be the cherry on top of the sundae, but really there is no sundae. Similar to the first scenario, but much, much worse because this is a systemic issue.

You may be surprised to find out that this happens a lot more frequently than it seems. Many companies label themselves as “data driven” or a “Fintech” purely because of their intention to act like one. They usually aren’t actually doing anything to deserve the title yet. You know, I’m pretty sure some companies just flat out lie about being data driven just because they want to appear cool and fit in. I’m not even joking.

The problem. Companies like this often have no intention of doing any actual data science, because they have already set up the processes and operations that they believe is data science-y. So if you come in and start suggesting things, you’ll likely be met with serious blockers. They might say that they can’t disrupt the workflow they currently have; or that they don’t have time to try new ideas right now; or that one day they’ll do what you’re saying but for now they need to keep it the same.

Who can blame them though? They don’t know any better. Nobody has told them otherwise. And having an in house data scientist earns them major bragging rights (oh don’t get me started — being used as a mascot deserves its own post entirely). As far as they’re concerned, everything is fine.

Where EQ comes in. You need to pick your battles. Don’t suggest single-handedly revamping a huge part of what the company or team does. Don’t try to dazzle them with the most awesome things you could do. Don’t tell them that a machine learning algorithm can replace a really long and costly process they have. Trust me, it doesn’t work like you’d expect.

Instead, find a small aspect that you could improve or optimise without too much difficulty — automating a report that is currently produced using Excel, is a good example.

Doing these smaller jobs will help to educate your colleagues about what you can do, and earn you points that you can cash in for a more large-scale project.

Once again though, this strategy needs to be combined with communication. Talk to your manager about what you’re working on, so they don’t think you’re trying to undermine them. Reassure them that the business-as-usual tasks are being handled still. Tell them the results you achieved.

The ultimate goal is to get all these mini projects you’ve completed to filter out to the rest of the company and, more importantly, the senior management team. You want them to hear about what you’ve done and say “let’s do more of that!”.

It may take time, but if you can pull it off, it’ll be so much more rewarding than if you just quit.

3. Co-workers think you’re going to steal their jobs

It’s hard to know for sure if and when this is the case because people will rarely say it out loud. But believe me, some people will definitely be thinking it.

As with all disruptive technologies, machine learning will make certain activities obsolete — that is a reality. So if that’s all they know, it’s understandable that people might not like the idea of a data scientist coming in.

The problem. Nobody will want to help you if they think your purpose is to push them out of a job. They won’t make it obvious, they might not even realise they’re doing it, but it can make things very difficult for you. Data science requires synergy to function well, and if people are dragging their feet when you need their help, or not responding well to ideas you have, then you’ll feel pretty disenchanted.

What they might think though is quite the opposite of the reality. Data science is often about automating the boring parts of a process, so people can focus on the more interesting parts. Automating data manipulation to free up more time to do analysis, for example. It’s up to you to convince them of that.

Where EQ comes in. You need to make people realise that you’re not there to make them redundant, but in order for that to happen, they need to trust you.

And it’s not hard to get people to trust you — at least within the limits of the work environment. Show them that you’re interested in how they do their jobs by asking them about it. Show them that you’re aware that they know more about the company than you if you’re a new starter. Ask them about the problems they face in their role, and how they go about solving them.

Then if you can think of a way to fix a problem of theirs using your skills in data science, don’t feel the need to tell them straight away. It’s likely that they’ve been dealing with it for some time and have developed their own work-arounds or solutions to it. So they won’t thank you for solving their problem in a matter of minutes (yes, humans are very proud creatures, myself included).

Whatever you do, you need to demonstrate that you want to and are listening to them. That’s the only way to get them to trust you, and in return support your role.

There are so many reasons why you might require EQ as a data scientist, just like most jobs. But the ones listed in this post owe themselves to the serious separation of data science knowledge that exists in the business world, and are therefore quite unique to data science.

Ask yourself if you’ve ever been faced with one of these scenarios. How did you deal with it? Could you have communicated better? Did you just find another job? If so, take a moment to reflect. Maybe next time you could try what has been discussed here.

Data science is still very young, and we’re all ambassadors. So it’s our responsibility to make sure that knowledge is spread amongst colleagues and businesses. The sooner we do that, the sooner we’ll see data science and business reaching entirely new levels.