Neeraj Nathani SmartBridge Trading Solutions Pvt Ltd

(Neeraj Nathani SmartBridge Trading Solutions Pvt Ltd)

The Big Value In Big Data: Seeing Customer Buying Patterns

Is Big Data simply a popular catchphrase or the launch of a new era? Overuse doesn’t automatically transform a buzzword into a best business practice, and several factors, such as measurable results, focus and sustainability, determine whether an idea is much more than just that. The core question is this: does big data actually solve real-world business problems? The short answer is yes – and here’s a real world example of how leveraging Big Data can solve the complexity around product proliferation by helping companies align product offering and supply chain based on customer-buying patterns.

Big Data: More Than Just A Trend

According to Gartner, unstructured and structured data held by enterprises continues to grow at explosive rates. However, volume and velocity of data – what the business world is beginning to understand as the “Big Data Problem” – are becoming less of an issue than the variety of data. Each silo within the enterprise – operations, supply management, sales, marketing – faces its own data variety challenges, where bits exist in a multitude of formats and types.

Neeraj Nathani:

Due to the variability of data across silos, systems can’t “speak” to one another, and gaining an accurate, enterprise-wide view of demand and performance seems impossible. In fact, most business and IT managers accept the lack of intersystem collaboration as a given, an inevitable limit that must be worked around. As a result, what we know is being increasingly outpaced by the things we don’t know. Performance within individual silos is clear, but this view does little to inform effective strategic direction for the organization.

There is a better way to tackle this challenge in variety and capture the opportunity posed by Big Data.

Patterns And Connections
Traditionally business silos have individually sorted, stored and managed data that is relevant to the needs within the silo. This has resulted in enterprise-related data being disrupted into disconnected pieces across the silos. The Big Data challenge requires aggregating the data across these silos for a single view of the organization. The purpose of the aggregation is to reveal the intelligence and causality across the business functions for strategic insight. This Big Data challenge requires solutions that can harness the intelligence from the data and deliver actionable intelligence to the business user. Conventional business intelligence and data warehouse tools aren’t designed to analyze, identify and surface critical data linkages and causality. As a result, insights, contexts and market opportunities remain hidden from view because users don’t know what they don’t know.
These critical connections and causalities are the key to managing big data and allowing companies to see a more comprehensive picture of their product and product variants based on actual customer-buying patterns. The connection patterns reveal valuable information quickly, accurately and allow for faster, more relevant decision-making.
Freeing the data to reveal connections and causation through pattern-based analytics solutions will paint a bigger picture – one that can better manage product variants and streamline sales by shedding light on what customers are buying, when, where and how. Currently companies pour their non-standard data into spreadsheets that then require teams of data analysts to interpret and derive meaning from it. This is not scalable and often misses the mark. Big data demands applications that can interpret and deliver immediate actionable intelligence to business users

Leveraging Data To Address Product Proliferation

One of the major business problems companies are facing today is the complexity caused by product proliferation – one of the biggest drivers of material cost and inventory levels. This is a complex issue that isn’t easily resolved with traditional approaches, but can be addressed with a systematic and enterprise-wide pattern-based analytics approach to leveraging the data across the silos within the organization.

For high value manufacturing, product variety is the single biggest driver of cost as it dictates the material requirements and inventory, which is typically over 80 percent of the total cost. Chasing diverse customer demands with an explosion of product variants dramatically increases total cost and causes volatility across the supply chain. But what are customers really buying? Are customers really buying all that companies are offering? Is there a way to satisfy them with fewer variations? Or alternate variations that improve supply chain efficiency? This is crucial knowledge for not only satisfying the demand, but also reducing supply chain costs and increasing margins.

As product proliferation has increased, so have operational complexity and cost structure; “what-if” scenarios with alternate product portfolios that meet the majority of customer demand have been an underutilized lever and represent the next frontier of business process improvement. A poor product mix will drive complexity throughout the value chain – impacting supply chain, marketing and sales, and service and support. In the same vein, a good product mix can dramatically improve delivery costs and increase profits.

The product portfolio also impacts the sales efficiency and top line revenue. To understand the opportunity cost with sales, consider the typical OEM salesperson: Let’s say he or she has a $4 million quota and spends 10.5% of his or her time defining a customer solution, configuring and pricing it, and then tracking delivery. Reducing that time by just 1 percent for a 100-person sales force represents an opportunity cost of $42 million.

Pattern-based analytics solutions are already being leveraged by some companies, giving them a much-needed competitive advantage in a high-risk environment by providing insights into customer-buying patterns to guide the product offering, supply chain planning and execution. For example, when NCR needed to optimize product configurations across their ATM line, they implemented pattern-based analytics solutions to analyze customer-buying patterns. With insight into what’s selling where, to whom, when and how often, NCR optimized the product line, defined customer segments and then seamlessly pushed this to the sales team to help reduce lead times.

At NCR, this resulted in a dramatic improvement in sales efficiency and supply chain performance. Pattern-based analytics can reveal deep insight into what customers are buying, and leverages that to offer and guide customers to the best choices based on availability and product margin. This approach uses customer-buying patterns to create the best product offering, and simultaneously guides customers to the best choices based on what is on hand. This is a win-win as the supply chain builds what is being bought and the sales reps sell what is in stock.

Source: Wikipedia.

Neeraj Nathani SmartBridge Trading Solutions Pvt Ltd

(Neeraj Nathani SmartBridge Trading Solutions Pvt Ltd)

Trade Promotion Forecasting

Trade Promotion Forecasting (TPF) is the process that attempts to discover multiple correlations between trade promotion characteristics and historic demand in order to provide accurate demand forecasting for future campaigns. The ability to distinguish the uplift or demand due to the impact of the trade promotion as opposed to baseline demand is fundamental to model promotion behavior. Model determination enables what-if analysis to evaluate different campaign scenarios with the goal of improving promotion effectiveness and ROI at the product-channel level by selecting the best scenario.

Trade promotion forecasting challenges

Trade Promotion spending is one of the consumer goods industry’s largest expenses with costs for major manufacturers ranging from 10 percent to 20 percent of gross sales. Understandably, 67 percent of respondents to a recent survey said they were concerned about the return on investment (ROI) gained from such spending. Quantifying ROI depends heavily on the ability to accurately identify the “baseline” demand (the demand that would exist without the impact of the trade promotion) and the uplift.^[1]

In fact, forecast accuracy plays a critical role in the success of consumer goods companies. Aberdeen Group research found that best-in-class companies (with an average forecast accuracy of 72 percent) have an average promotion gross margin uplift of 28 percent, while laggard companies (with an average forecasting accuracy of only 42 percent) have a gross margin uplift of less than 7 percent.^[2]

A bottom-up sales forecast at the SKU-account/POS level requires taking into account product attributes, historical sales levels and store specifics. A complicating factor is that the large number of different variables which describe the product, the store and the promotion attributes, both quantitative and qualitative, could potentially have many different values. Selecting the most important variables and incorporating them into a prediction model is a challenging task.^[3]

Despite these challenges, two-thirds of companies in the consumer supply chain consider forecast accuracy a high business priority. 74 percent said it would be helpful to develop a bottom-up forecast based on stock-keeping unit (SkU) by key customer.^[4]

Traditional trade promotion forecasting methods

Many companies forecast the impact of trade promotions primarily through a human expert approach. Human experts are unable to take into account all the variables involved and also cannot provide an analytic prediction of campaign behavior and trends. A recent survey by Aberdeen Group showed that 78 percent of companies used Microsoft Excel spreadsheets as their primary trade promotion technology tool. The limitations of relying upon spreadsheets for trade promotion planning and forecasting include lack of visibility, ineffectiveness and difficulty in tracking deductions.^[5]

Specialized applications have been developed and become more common. 35 percent of companies now use legacy systems, 30 percent use Sales and Operations Planning (S&OP) applications, 26 percent use integrated Enterprise Resource Planning (ERP) modules and 17percent use home grown trade promotion solutions. These applications support the planning process, while still primarily relying on human knowledge and intuition for forecasting. One problem with this approach is that humans tend to make optimistic assumptions when forecasting and planning. The result is that forecasts most commonly err on the optimistic side and that human forecasters also tend to underestimate the amount of uncertainty in their forecasts.^[6]

A further issue is that the majority of manufacturers use legacy trade promotion systems that contribute to internal fragmentation of trade marketing data. Many of these companies are currently using assumption-based forecasts with limited accuracy.^[7]

Analytic approaches to trade promotion forecasting

TPF is complicated by the fact that campaigns are described by both quantitative (such as price and discount) and qualitative (such as display space and support by sales representatives) variables. New approaches are being developed to address this and other challenges. Most of these approaches attempt to incorporate large amounts of heterogeneous data in the forecasting process. One researcher validated the ability of multivariate regression models to forecast the impact on sales of a product of many variables including price, discount, visual merchandizing, etc.^[8]

The term Big Data describes the increasing volume and velocity of heterogeneous data that is coming into the enterprise. The challenge is to combine this data across all of the silos within the organization for a single view. The data can be used to improve trade promotion forecast accuracy because it usually contains real connections and causation that can help to better understand what customers are buying, where they are buying it, why they are buying and how they are buying.^[9]

Traditional methods are insufficient to assimilate and process such a large volume of data. Therefore more sophisticated modeling and algorithms have been developed to address the problem. Some companies have begun using machine learning methods to utilize the massive volumes of unstructured and structured data they already hold to better understand these connections and causality.^[10]

Machine learning can make it possible to recognize the shared characteristics of promotional events and identify their effect on normal sales. Learning machines use universal approximations of nonlinear functions to model complex nonlinear phenomena. Learning machines process sets of input and output data and develop a model of their relationship. Based on this model, learning machines forecast outputs associated with new sets of input data.^[10]

Intelligible Machine Learning (IML) is an implementation of Switching Neural Networks that has been applied to TPF. Starting from a collection of promotional characteristics, IML is able to identify and present in intelligible form existing correlations between relevant attributes and uplift. This approach is designed to automatically select the most suitable uplift model in order to describe the future impact of a planned promotion. In addition, new promotions are automatically classified using the previously trained model, thus providing a simple way of studying different what-if scenarios.^[11]

TPF systems should be capable of correlating and analyzing vast amounts of raw data in different formats such as corporate sales histories and online data from social media. The analysis should be able to be performed very quickly so planners can respond quickly to demand signals.^[12]

Groupe Danone used machine learning technology for trade promotion forecasting of a range of fresh products characterized by dynamic demand and short shelf life. The project increased forecast accuracy to 92 percent resulting in an improvement in service levels to 98.6 percent, a 30 percent reduction in lost sales and a 30 percent reduction in product obselecense.^[13

Source: Wikipedia.

Picking Winners In Big Data Neeraj-Nathani SmartBridge

Picking Winners In Big Data

Big data solutions are picking up speed in the IT industry. There’s a Cambrian explosion of interesting start-ups, and all the database and business intelligence incumbents have moved to create big data offerings, or rebrand into the new universe.

The key to seeing the value of big data is understanding that it’s a business problem, not a matter of picking the right tools and waiting for the magic to happen. That said, technology choices still need to be made in an increasingly crowded and confused market.

The biggest question for anybody wanting to invest or adopt technology in this area is how to pick the winner? Glancing through marketing materials will do little to help you: everybody claims relevance to big data.

As I am often asked my opinion about big data companies, I thought I’d share some of the principles I use to help me think about the industry.

Where’s the value?

We need to understand where the actual value is in the data world. For the most part, this value doesn’t lie purely in the software. Over the past decade we have seen a rising tide of commoditization of the software stack: from operating system, to relational databases, to Hadoop itself. In fact, without this, we wouldn’t have the big data revolution as we know it.

As Hadoop has become a de facto standard, so has the notion of building on top of it with open source. There is some advantage in software innovation, but it is momentary. Once something is known to be possible, there are enough smart programmers out there that reproducing it becomes straightforward. (Because of this, I gloomily predict no shortage of patent battles in the not-too-distant future.)

Despite the flux in the software world, two things about big data remain constant: the need for compute and storage, and data itself. It’s ultimately to ownership of one or both of these factors that IT industry value will gravitate.

Compute and storage

The ever growing need for computing power and storage bodes well for companies providing the basics. These fall into two categories: hardware manufacturers, and cloud infrastructure providers. Not that these two markets are immune to their own fluctuations, thanks to standardization and commoditization, but fundamentally, getting paid for use of metal is the name of the game.

In this respect, it’s not too much of a mystery why storage company EMC has plowed so early and so deep into the big data world. Neither is it hard to see the reasoning behind Intel creating its own optimized Hadoop distribution.

Data

Value resting in data is the more subtle of the two axes of big data success.

Big data is ultimately about the smart use of data to drive a business. There are two kinds of data: data about your business, and data external to your business that you can create value from.

It’s easy to see who might get success from the latter, external data. We can expect that massive data owners such as Google, Facebook, Thomson Reuters, Bloomberg will experience ongoing success for as long as they are able to create product from their data.

Who owns your data, though? The obvious answer, you, isn’t the only answer. In fact, your data is locked up inside the platform choices you make, at both the hardware and software level. If your systems are based on Oracle, Microsoft, you are very unlikely to move in a hurry. Data likes to stay where it is, and tends to attract more data as you build systems around it. Production systems are expensive to replace.

So, the vendors of your software platforms of choice also get long term value from your data. For this reason, it’s hard to bet against existing enterprise application platform incumbents in the big data world. Big data, for most of today’s organizations, is an additive phenomenon, not a challenge to the core of IT.

We’ll see more large platforms ensuring nobody needs to move away. Examples include SAP adding HANA, in order to enable their existing customers for big data, or Amazon Web Services’ addition of their data warehouse Redshift, to ensure that the entire data needs of a company can be met on their platform.

Finally, it would be lunacy to bet against either Oracle, who have been portentously quiet in the big data world over the last year, or Microsoft, who in Excel have the world’s most popular data manipulation environment.

It is all business as usual?

I’m not saying there is no new opportunity in the big data industry. What I am saying is that, as an additive technology, big data is unlikely to enable anybody to challenge Oracle or Microsoft for the throne.

There will be change, though, and it will bring both winners and losers.

The area of value we haven’t yet looked at yet is the point where data interacts with the actual mechanism of a business. That’s where it transfers value to you and enables you to leverage data to get ahead. This breaks down into several areas of opportunity for big data innovators.

• Domain specific: tools that enable the manipulation and exploitation of data in a way that’s specific to a business segment. We see initial evidence of this market opportunity in the evolution of web and customer analytics products.
• Machine learning: more data means more to understand, and the only way an organization can realistically do this is with the aid of computers. Machine learning helps automate many parts of the data wrangling process. Furthermore, cross-company data sharing can significantly boost the effectiveness of machine-learning, creating the opportunity for companies gaining early market share. A recent example of this is Sift Science, a fraud detection application.
• Tools for exploration: human interaction with data is a requirement that’s hard to abstract away. Tableau has a head start in this market for big data, filling the role of “Microsoft Office for Data”, but the field is ripe for new innovation, especially with the increasing power of graphical capabilities and new device formats.
• Data agility: speed-to-decision is a critical factor in business competitiveness. Any solution that removes laborious steps has an advantage: a particularly problematic area here is data integration, the loading of both internal and external data sources ready for analytics. Most of today’s big data solutions are frankensteined combinations of layers: there’s a great opportunity for vertically integrated solutions that removes needless impedance to data manipulation.

Move up http://i.forbesimg.com t Move down

Areas of risk

What are those riskier big data options? If a solution isn’t scoring high in the categories above, it’s not likely to be around for the long run.

The biggest risk is with solutions that address only a single horizontal part of the data architecture. Time is against companies in this game. By selling just part of a complete solution, they’re working against the rising tide of commoditization. Customers will demand standardization (e.g. Hadoop compatibility) in order to feel safe adopting such solutions, but that prevents the lock-in that will protect that business. It’s not Oracle’s relational database that cements their position: it’s their vertical position up and down the application stack.

Therefore, it’s not surprising to see the pure Hadoop distribution companies making partnerships, and branching out into other vertical layers. In the long term, it’s a tough road they’ve chosen.

One company working actively to solve this problem is DataStax, who have pivoted from being seen as the corporate backer for the Cassandra NoSQL database—a risky horizontal play—to selling an integrated stack of Cassandra, Hadoop and Solr, intended as a complete platform for building enterprise applications. They’re going after some of the platform business: being involved in the actual use of data to drive business value.

For some start-ups, not having long term big data viability might be just fine as a strategy. As the bigger enterprise companies lumber into the arena, they’ll prove handy acquisitions. But given the crowded space and progressive commoditization, this isn’t a certain future. And certainly for their customers, the risks are growing.

More controversially, another area at risk is that of traditional ETL. Or in its broader sense, data integration. This is a hard, hard, problem. It’s not easy to integrate data retrospectively, and it’s not all certain that the integration players from the data warehouse world will be able to translate that success into the big data world. Many early adopters of Hadoop were motivated by the fact that existing ETL solutions couldn’t meet their needs.

In the long term, data integration is likely to be best served by entire new architectures that don’t try to separate and tame the data in the first place. It’s a lot easier to build truly integrated data infrastructure as greenfield.

For those who crack the integration problem, the potential rewards are high. But so is the risk.

Conclusion

In the long term, the additive nature of big data, combined with inertia, makes it a safe bet that current enterprise IT incumbents will continue their reign, as long as they move to embrace big data in their architectures.

There is plenty of opportunity though, especially in greenfield and cloud scenarios. Expect to see increasing returns for those who provide integrated solutions and do a good job of equipping human decision makers.

There’s long term value in metal, and value in data. About everything else, it’s worth thinking carefully.

Source: Wikipedia./Forbes