You might be surprised to learn that, according to IBM, the rise in data levels has reached such a pitch that 90% of the data in the world today has actually only been created in the past two years, writes Billy MacInnes.
That's a pretty spectacular increase, all things considered, but when you factor in the proliferation of transactional data from companies, web logs, RFID, sensor networks, social networks, internet text and documents, internet search indexing, medical records, video archives, photography archives, it's perhaps not quite as astonishing as it first appears.
This brings us to the phenomenon known as "big data", data sets so large that existing database management tools, relational databases and desktop statistics/visualisation packages, struggle to capture, store, search, share, analyse or visualise them.
How big is big data? The consensus appears to be "very big, but we can't tell you just how big" on the basis that big data sets are a constantly moving target.
It's not just big, it's important. According to a study entitled Big data: The next frontier for innovation, competition and productivity, published in May by McKinsey Global Institute, big data "can play a significant economic role to the benefit not only of private commerce but also of national economies and their citizens".
It suggested that if US healthcare could use big data creatively and effectively, the potential value from data in the sector "could be more than $300 billion in value every year".
Furthermore, a retailer using big data to the full could potential "increase its operating margin by more than 60 percent".
According to the report, Tesco has already used big data to capture market share from local competitors and McKinsey expects "to see value accruing to leading users of big data at the expense of laggards, a trend for which the emerging evidence is growing stronger".
This month, Sybase published a guide on big data analytics entitled Intelligence for Everyone: Transforming Business Analytics Across the Enterprise, which was designed to illustrate that there were tools and methods to make sense of the new, massive data sets.
"Big data is here - and it is to be embraced and exploited," said Sybase vice president of database products, Brian Vink.
"Organisations that are prepared for the data explosion have a tremendous chance to achieve significant advantages in their competitive marketplace... The opportunity is there, ready for the taking."
At first sight, this may appear a little bit above the pay grade of many companies in the channel but vendors active in the big data space are keen to stress there is a role for partners to play.
Take Kurt Bilafer, global vice president, analytics & channel partners at SAP, who is adamant that the vendor's channel partners "are an integral part in bringing innovative solutions to market", adding that the go to market strategy for SAP's In-Memory HANA solution "is to enable, include and leverage our VAR channel partners to help customers quickly bring about business benefits".
Suzanne Hoffman, director of channel at data discovery tool specialist Tableau Software, says the channel "is just recognising the huge opportunity big data represents" with enterprises able to aggregate their data warehouses and federated sources into a single view that is visualised for the user.
"The channel, whether it was implementation partners or strategists, could never do this before," she argues.
Sean Farrington, regional vice president for Northern Europe and UK managing director at QlikTech, believes there are "huge opportunities for the channel around supporting big data applications".
He says channel companies can shine by helping organisations of all sizes "address any issues they face when facilitating a big data application such as integration, master data management, data quality and migration".
Data analysis in many corporate systems is complicated by the need to integrate information from multiple sources in multiple formats. Channel partners can help to make the process efficient and manageable, transforming something "very demanding that requires detailed technical understanding and tools, into a seamless operation that suits both customers and technology providers.
Channel partners can stand out by offering an all-encompassing, balanced solution which, when done well, can be exceptional for all parties involved".
Turning big data into actionable information is an area of potential major difficulty for businesses, according to Keith Goldstein, vice president for indirect channels at open source data management specialist Talend.
This has given rise to demand for big data applications which, in turn, "creates a huge amount of opportunity for channel partners to reinforce their position in the market by providing applications that leverage big data.
By offering users a package that addresses big data, the specialist relationship between resellers and user remains intact," he claims.
Judging by the responses from those operating in the big data application market, the consensus is that the channel's traditional role in integrating multiple technologies and platforms is the one that stands it in best stead for playing in this space.
Adam Chapman, marketing manager for ReadSoft UK, says that the expertise required to bring together multiple elements of a big data solution, such as input devices, scanners, software, hardware, storage and network elements, is something that no single vendor can do on its own.
"The channel can bring all of this together with a deep understanding of the customer's needs, project management and a really personal touch," he adds.
Hoffman at Tableau is quick to stress that channel partners are no longer just a sales and distribution network, but a significant part of the technology community and they need to be able to address the tactical and strategic side of their customers' business.
"In fact, the channel is more important as the vendors cannot possibly be engaged in the larger projects with all the software and hardware components that need to interact. As we sell software, we need to rely on the expertise of our partners to engage with the customer at all levels of decision-making".
All of this sounds pretty positive but in order to fulfil that role, channel partners need to be prepared to get in on the big data act.
Something which, Hitachi Data Systems regional UK and Ireland sales director, Harvey Smith says they are not rushing to do. "Many partners are wary of approaching big data applications because of the scale and resources that are required to fit this type of project," he observes.
"As a result, the channel is missing out on a valuable opportunity to support vendors as they look to implement this type of solution." What do they need to do to take advantage of the opportunity? Pretty much what they've always done. "Partners and resellers need to create a role for themselves and act as a point in the market for different technology and skill sets to meet," Harvey suggests.
"The channel typically has access to unique products and solutions that large vendors may not be able to reach. As a result, they can bring together multiple technologies and vendors and provide unique solutions that meet vendors unique sets of requirements."
And as Chapman points out, most vendors would love to work with an "easy or clean installation" but the reality is very different. A lot of customers have a complex environment and legacy systems that need specific integration work and expertise to make them work.
"The value from the channel comes in handling those legacy, bespoke or customised systems - they can't just be forgotten about," he states.
Jeremy Wallis, systems engineering director at NetApp UK, also sees value in partners helping to provide customers with an integrated solution.
"To deploy solutions to big data challenges, we need channel partners," he says, "there's no question about that". He also believes that while it might not be very relevant to smaller partners at the moment, as big data becomes more relevant to more customers "it will be more applicable to a wider range of channel partners".
He believes the big data trend will have "a huge impact on existing relational database suppliers and their partners", citing the example of none bank where his contact suggested it could be completely rid of relational databases in five years and reliant on some kind of big data business analytics instead.
"We're starting to see volumes of data [being generated] in certain customers that mean traditional relational databases don't scale enough," Wallis adds, "and we'll probably see partners that have traditionally been selling databases and database consultancy starting to look at business analytics and skill up in those areas.
The shift to business analytics is likely to lead to the vendor selling to different kinds of partner, especially those with expertise in business analytics, he argues.
It may well be that existing partners need to change to take on expertise beyond their storage capabilities or that NetApp will need to develop additional channels. It's not going to be an overnight phenomenon.
Business analytics "represents adjacent markets", it is not a matter of cannibalising existing markets, he argues. Tom Cahill, EMEA vice president at Jaspersoft, says there are definite opportunities for the channel around supporting big data applications. "We're seeing them every day," he claims, citing the example a solution that combines IBM's Hadoop-based InfoSphere BigInsights product with Jaspersoft's BI (business intelligence) suite to help partners get the most out of big data apps.
He reveals that System Integrator partners are working with BI builders in enterprise customers to build reporting and BI solutions using low cost, highly interoperable open standard software technologies.
"These SIs co-sell or resell our BI software and form our channel," although he adds they are "highly dependent on our expertise, services and support". Goldstein at Talend says that as software vendors and SaaS providers incorporate embeddable big data management technology into their own product stacks, they can provide reseller partners "with the perfect tool for their customers.
By focusing on their core competency and leveraging other technologies, resellers can equip themselves with the perfect big data package for their customers, maintaining their place in the channel".
Bilafer at SAP says the opportunities are already there for its channel partners, with the ability to participate in sales cycles and enable their organisations to better understand and exploit "the power of HANA. In fact, we are leveraging the same enablement tools that we provide our direct sales teams to ensure that the channel is ready and capable to deliver real business benefits that our customers have come to expect from SAP and our ecosystem".
It appears to be working, he adds, citing comments by Co-CEO Bill McDermott in the company's most recent earnings call when he revealed the HANA pipeline had increased to over €600m since its release in June: "Demand for this solution requires all of SAP, direct and indirect to be aligned, selling and most importantly addressing the needs of our customers."
Smith at HDS says it offers selected resellers specialised training to equip them with the skills to work on enterprise projects and deliver virtualisation, migration and application integration which they can apply across multiple verticals to position themselves as industry experts well equipped to work with vendors, understand where their skills lie and where they need extra support.
He believes this gives the channel the opportunity to negotiate with the best partners and consultancies to deliver projects supporting big data applications which meet customers' expectation and enhance their own offerings.
Wallis at NetApp cautions that big data "is a very generic term that encompasses so much". He says it's unlikely there will be such a thing as a "big data reseller", more likely there will be people with expertise in particular requirements for a particular category.
"We could very well see a number of niche partners that are small but very specialised. That's what happened with our channel initially and we've got a much broader partner base now."
So how big is the opportunity for the channel in terms of supporting big data applications? Well, to use a phrase that appeared at the beginning of this article: Big, very big even, but we can't tell you just how big.
Images courtesy: John Foxx/Thinkstock