Data ‘cleaning’ chewing up time for BI professionals

A new study has shown that integrating data from different platforms is the biggest challenge for BI professionals in making data ‘analytics ready’.

The study, conducted by big data integration platform Xplenty, focused on several areas of the ETL (Extract, Transform and Load) process, including preferences for on-premise or cloud-based solutions, perceived challenges, and the amount of time spent on ETL.

Respondents to the survey were united in saying that ETL was critical for their business intelligence efforts, with 97 per cent agreeing that it was.

When it comes to their biggest ETL challenges, 55 per cent cited integrating data from different platforms, 39 per cent said transforming, cleansing and formatting incoming data, 32 per cent said integrating relational and non-relational data (32 per cent), and 21 per cent said it is the sheer volume of data that needs to be managed at any given time.

“Reformatting, cleansing and consolidating large volumes of data from multiple sources can be overwhelming,” said Yaniv Mor, CEO and co-founder, Xplenty. “BI professionals are still struggling with the best approach to shorten the time between integration and analytics. As a result, businesses are often slow to unlock their data’s true potential for revenue or operational improvements.”

It’s also clear many businesses are spending a large amount of time on ETL, with 30 per cent of the BI professionals polled saying they spend between 50 and 90 per cent of their time on ETL alone.

“BI professionals should be spending the majority of their time evaluating data and deciphering patterns gleaned through the analytics process – not readying data for analytics,” said Mor. “The more time they spend making raw data analytics usable, the less time they have to generate real value from it. We have to accelerate big data’s ‘time-to-insight,’ boosting efficiency and bringing more immediate answers to an organisation so that they can more quickly take advantage of them.”

Just over half (51 per cent) of BI professionals polled said they currently use on-premise ETL solutions, versus 49 per cent cloud-based. Of those who currently use on-premise tools, 51 per cent said they were “strongly considering” moving all ETL processes to the cloud.

“Cloud ETL offers a host of benefits over on-premise, from increased agility in resource deployment to reduced costs. As such, the cloud is an increasingly attractive option from both a performance and operational perspective.”