The Challenges of Obtaining Data: It’s a People Problem


Before you can even think about putting your data to work, you have to figure out how you’re going to obtain all the data you need.

When used as a true enterprise asset, data can support or inform nearly every function of your organization: tens of thousands of decisions every day, from the most junior analyst or investigator to the CEO. At Enigma, we work with financial institutions using data in new ways to catch criminals and keep client data safe, with city governments drawing on public data to improve public service delivery and save lives, and with pharmaceutical companies tapping into data to keep harmful drugs off the shelf and ensure patient safety. Together, we have learned much not only through working deep in the data, but also through experiences with the organizational and human reality that that data purports to reflect.

In every one of these data management and intelligence scenarios, the success of the project hinges on obtaining and applying the data. We’ve identified four challenges core to doing so: legal, cultural, political, and technological.

As it turns out, the first three are often the most difficult. Organizational challenges tend to outweigh the tech barrier to entry in becoming a data-driven enterprise. The human decision-making and adoption components often move at a glacial pace, making the engineering and integration parts the most simple in the equation. But without greasing the so-called political, cultural and legal skids, any technology solution is moot. When the first three steps are done well, the tech piece falls into place quite easily.

You may be surprised to hear this from a company that builds enterprise data technology. But don’t be. What should raise eyebrows is a technology vendor that is not having this conversation with you.


It’s a people problem

Every organization is a political, cultural, and legal entity because every organization is made up of humans. Like data, human relationships are messy. There are all kinds of reasons people do things in the way we do, be it historical, habitual, or just gut feelings. While processes may not always seem logical, people tend to think their way works and grow attached to it. 

Whether you’re dealing with the simplest widget or a potentially game-changing tech solution, you need to be cognizant that entire cultures can grow up around certain processes.

Over time, people adapt to doing things a certain way, especially if the initial change agent disrupted established routines or workflows. Take the introduction of email, for example. An entire workplace culture was formed around using email to advance our ability to communicate. This was a huge departure from the interoffice mail circulated in manila envelopes. As such, the style in which we communicate—and the things we say, for that matter—has adapted with our ability to exchange words digitally in real time.

 As in the manila envelope scenario, the way things have always been done isn’t always the best way. But it’s going to take some finesse to reorient the culture and get stakeholders on board to determine legitimate use cases for whatever your data intelligence project may be. By and large, technology is meant to be a decision support mechanism. You can’t forget that humans are still the ones making the decisions, for now anyway.

As I wrote in my recent post, the best approach to driving effective change is to put your data to work alongside your people. To get true value out of your data, people and credibility are your greatest assets. Nowhere is this more true than in the case of obtaining data.


It’s also a people solution

People may be part of the problem, but they’re also the source of solution. Effectively obtaining and applying data requires a people-first approach to breaking down political, cultural, and legal barriers. Here are some tried and tested strategies to support you in that effort:


1. Don’t start with legal. Or, bring stakeholders in first.

The job of the lawyer is to tell you what’s appropriate vs. inappropriate, what’s possible vs. impossible. Of course you need legal involved, but timing is key. By nature, lawyers are risk averse. That’s their job. That often means thinking in terms of black and white instead of finding a feasible solution in the gray. It’s a “this seems risky, we shouldn’t do it” conversation versus a “we can make this work if we do x, y and z” dialogue. If you bring legal in too early, before you have all the important stakeholders on board with the project, they’ll likely slow your effort to a halt before it has even taken off. That said, if you ignore the lawyers altogether, you are likely risking not just your efforts but the health and safety of your company. So, I’m not suggesting that, either. It’s a question of when, not whether.

The first step is to start socializing the project with whoever is in charge of risk as it relates to data —maybe it’s the CTO, the Chief Data Officer, the Chief Risk Officer; or sometimes it’s even the head of business. Having leadership buy-in before you attempt legal sign-off will help make sure the lawyer in the room is helping to make something work (or finding the best solution) versus nipping something in the bud just because it breaks with the norms. A legal team open to fully serving your organization — sometimes in new or unexpected ways — is an incredibly valuable asset. A separate but related aside: while it’s good to line up your business leaders, you still need the CIO on board early; you’ll be writing a check from the CIO’s bank account.



2. Secure data access

This is a big one. The most important thing you can do is secure data access from whoever owns the data. Without this step, you’ll sentence everyone involved to a big waste of time and money.

Most people don’t see it as their first order of business to make data available to other departments. After all, they may not understand (or even be thinking about) how it will help them do their job. Not to mention it costs time and money to allocate finite resources to another actor.

It’s important to have an internal change agent own data awareness – someone who can explain the value of sharing siloed data for both individual departments and for the organization as a whole.

With a healthy amount of reconnaissance to determine the lift, articulate who will be affected, assess what data is involved and how it will be used in a cross-departmental way, that “data evangelist” will be a big factor in laying the foundation for a culture shift.



3. Be prepared to surface uncomfortable truths

If you’re doing this correctly, chances are you’ll surface some uncomfortable truths. For example, as NPR recently reported, cities may not be allocating limited emergency resources in the best way possible. In many cities, there’s a good chance that a fire truck (instead of an ambulance) is nearby and will race to your door after a 911 call, but they may not be able to deliver the emergency care and transport you need.

Someone owns these uncomfortable truths. Shining a light on the fact that a program isn’t working optimally can cause a lot of disruption, especially when it goes against historical wisdom or political clout, as may be the case in reallocating emergency services. While the initial proposal may cause a backlash, clearly spelling out the benefits and providing a transition/migration plan can go a long way. At the end of the day, if leadership doesn’t really want to know what’s going on throughout the organization, don’t bother. Perhaps that seems harsh, but I’ve seen it happen enough to put it down as a valuable lesson learned.



4. Deliver a project with early return

Organizations are constantly making the same mistake — someone spends $10M (or more) on a 5 year project to build one ring to rule them all. Please. Stop. Doing. That.

 It doesn’t pay off to start too big. Rather, apply a technology solution to a discrete problem inside of the organization that data can give an immediate boost to. For example, with the City of New Orleans, we started with two smaller projects: streamlining the housing abatement program and optimizing delivery of smoke alarms to the city’s residents. With both of these projects we were able to put data to work to improve delivery of the city’s services. In the case of the smoke detectors project, that meant leveraging data from both the American Housing Survey (AHS) and American Community Survey (ACS) to identify homes with the greatest fire fatality risk. The fire department was then able to target those homes first during their outreach program.

Through these smaller projects we were able to get a sense of the city’s business processes and operations and better understand the factors that ultimately led to political buy in. We conducted both projects in a non-invasive manner that would add significant value to the organization if successful, but would not damage the org should they not be effective. After delivering value through both use cases — and garnering a better understanding of the political, cultural, and legal hurdles — we had both the buy-in and the know-how to deploy technology on a larger scale.

To support a use case with a quick return, use a sandbox environment to test it while creating the path for iterative product deployment. The success of your small project can become a very powerful catalyst for change.

When it works, making lives easier and alleviating pain, more people will be keen to jump on board. This initial buy-in, coupled with the project’s overall potential will help to garner support and enthusiasm for ongoing work much faster. 

When you take the steps to first address the political, cultural, and legal challenges, you’ll be better prepared to fully realize the value of data – and any technology solutions you choose to implement to surface the right data to the right people for decision support.