Enterprises are increasingly using real-time data analytics to make faster decisions and stay ahead of competitors. Abdul Montaqim talks to some of the companies making the most of the new technology and asks if real-time data has become an essential requirement for business leaders.
Past, present and future – big data is providing valuable insights into all three time frames. So far, most analytics engines have been used for data that has already been collected and stored, only to be analysed in retrospect – after the event, often a day or so after – providing a snapshot of the past, so to speak.
But increasingly, companies are choosing real-time data analytics solutions that look into the present – at any given moment.
Real-time data systems can provide information about activity on a network within seconds – depending on how sophisticated or involving the analysis part is.
The potential applications of real-time data analytics are far-reaching, says Ciaran Dynes, VP of products at open-source software company Talend. He says public services and private enterprise could use real-time data in many ways, from ensuring public safety to detecting fraud as it happens.
“The ability to analyse data in real-time to drive actionable insight is increasingly critical,” says Dynes.
Many of the analytics engines are being built on massive cluster computing systems such as Hadoop MapReduce, Apache Storm, and Apache Spark. Google, IBM and Amazon are among others who also distribute computing platforms to offer real-time analytics. However, much of the infrastructure for real-time deployment is still being developed and many of the services are relatively new.
“The City of Chicago has built a real-time data analytics platform using MongoDB that pulls together 7 million different pieces of data from city departments every day.”
The American Dream, in real-time
In the US, real-time data analytics is increasingly becoming the norm in many sectors, with government organisations taking something of a lead. For example, the City of Chicago has built a real-time data analytics platform using MongoDB that pulls together 7 million different pieces of data from city departments every day.
Chicago has called its system “WindyGrid”, in reference to its nickname as the “Windy City”, and has connected what it considers to be 15 of its most crucial departments, including its emergency services, to provide real-time data analysis to city managers so they can better predict and allocate resources, respond quickly to emergencies and uncover trends that would have otherwise been invisible.
Meanwhile, in the enterprise space, social media company BuzzFeed uses MongoDB to pinpoint when content is viewed, where it’s shared and how it’s being consumed by its website visitors, estimated at more than 400 million a month. The system enables BuzzFeed’s 800 employees to analyse, track and display these metrics to writers and editors in real-time.
Another company using the MongoDB platform is Germany’s largest retailer, Otto, which has annual turnover in excess of €2.2 billion. Otto’s website generates some 10,000 events per second. Every click and hover of every mouse is stored in the database, and real-time data analytics is used to provide unique and personalised web experiences to individual visitors.
MongoDB’s director, Mat Keep, says: “To me there are two broad consumers of real-time analytics: machine and human.
“The first category, machine, is where the real-time data is read and analysed by an algorithm which then takes an instant action based on the result. Otto.com is an example: the platform analyses data about the specific customer as soon as they land on the site, looking at data points such as age, gender, preferences or location.
“The other type of real-time analytics is what we see at BuzzFeed. That’s where people are reviewing the data as it is generated and taking their own actions based on that. Perhaps the editors at BuzzFeed see that a particular story isn’t performing as well as they would like. Based on the data and their experience as journalists, they re-write the headline and share it again across their social channels. Re-writing a headline is the type of creative thinking that computers are still not good at.”
“Real-time responses are critical in the fast paced world of sport.”
Common or garden variety real-time data
Already commonplace in the US, real-time data analytics is also becoming increasingly popular in the UK, with many companies either beginning to use or planning to introduce real-time data analytics as part of their daily operations.
Business leaders are now less concerned with how easy or difficult the real-time data analytics solutions are to use and more interested in what competitive edge they can gain from using it.
“If an organisation can capture an accurate picture of what their customers or clients are doing in real-time, they can leverage this information to gain a commercial advantage,” says David Leyland, head of the next generation data centres for Dimension Data in UK and Ireland.
One of the most common uses of real-time data analytics, according to Leyland, is IT security. Organisations are increasingly deploying behavioural monitoring solutions in order to visualise and spot potential security breaches or unexpected incidents the very moment they happen.
Retailers and banks are also showing strong interest in real-time analytics. If they can understand exactly what their customers are interested in and when, they would be in a strong position to deliver value-added services and advice on the fly.
And the many millions of people who watch sports on television will be familiar with how real-time data can provide a deeper insight into an event.
“One of the most notable examples of real-time data analytics that Dimension Data is involved with is the Tour de France,” says Leyland. “By articulating up-to-the-second rider information to viewers around the world, we are helping to enrich the TV experience. In addition, this information is allowing teams to understand how to benchmark their riders against their team-mates and other competitors.”
Leyland says real-time data analytics is already proving “useful and valuable” to companies and organisations, and is reaching the point where it will become “absolutely necessary” for some enterprises. “If organisations don’t understand the technology’s capabilities, while their competitors do, they will find themselves disadvantaged pretty quickly,” he adds
Diversity of real-time data
Real-time data analytics solutions are finding their way into an increasingly diverse range of sectors.
Software company Tibco’s real-time data analytics customers include high street retail brand The North Face and vehicle breakdown-recovery service the Automobile Association (AA). The company also has a partnership with the Sports Alliance, which works with 65 professional sports clubs.
“Real time responses are critical in the fast paced world of sport,” says Tibco CTO, EMEA, Maurizio Canton, and commented on how new data analytics solutions are “now poised to be the future of fan marketing”.
“Tibco’s technology can produce an integrated and accurate picture of a driver’s performance with sensors that track speed, braking, steering and mileage and collate the data into one definitive bundle.”
Canton says, “Tibco Business Works and Tibco Enterprise Message Service combine to support real-time fan marketing applications, such as welcome text messages on arrival to the stadium and mobile club apps all of which depend on real-time analysis of fans’ behaviour.”
Tibco’s work with high street retail outdoor brand The North Face uses real-time data to manage customer loyalty and reward initiatives. The solution includes a reward platform which provides “a 360 degree view” of the customer journey and enables them to decide their own reward offerings and preferences in real-time.
For the AA, Tibco has provided a solution that enables the organisation to respond in real-time to events in context rather than simply collecting data for later analysis, missing the opportunity for action when it matters most. This enables them to respond in an environment where a high rate of change is expected and performance and availability are of paramount importance.
Real-time data analytics, combined with predictive analytics, could also bring greater transparency and equality to how car insurance premiums are assessed.
“Traditionally, assumptions about age and driving style have routinely seen young and elderly drivers be unfairly penalised – not anymore,” says Canton. “Tibco’s technology can produce an integrated and accurate picture of a driver’s performance with sensors that track speed, braking, steering and mileage and collate the data into one definitive bundle.
“This insight – based on habits, history, and degree – provides an accurate picture and hard evidence from which to base the premium on.
“Furthermore, predictive analytics come into play, enabling the driver and insurer to better forecast certain occurrences to pre-empt breakdowns and the risk of accident.”
“We are expecting an increasing proportion of the analysis to be done in real-time, using machine learning as well as more sophisticated analytics tools.”
Mind real-time learning gap While the adoption of real-time data analytics solutions is increasing, there is still a learning gap to be overcome by enterprises. That’s one of the reasons why Unicom held a seminar recently at the Hilton London.
“The conference looked at Big Data as well as real-time analytics and predictive analytics,” says Irene Moncada, researcher and events commissioner at Unicom.
Among the speakers at the event was Richard Veryard of Glue Reply, who told us, “Over the next couple of years, as the technology gets better, the data scientists get even smarter, and the marketing people get more sophisticated, we are expecting an increasing proportion of the analysis to be done in real-time, using machine learning as well as more sophisticated analytics tools.”