Big Data’s Management Revolution
by Erik Brynjolfsson and Andrew McAfee | 10:05 AM September 11, 2012
Big data has the potential to revolutionize management. Simply put, because of big data, managers can measure, and hence know, radically more about their businesses, and directly translate that knowledge into improved decision making and performance. Of course, companies such as Google and Amazon are already doing this. After all, we expect companies that were born digital to accomplish things that business executives could only dream of a generation ago. But in fact the use of big data has the potential to transform traditional businesses as well.
We’ve seen big data used in supply chain management to understand why a carmaker’s defect rates in the field suddenly increased, in customer service to continually scan and intervene in the health care practices of millions of people, in planning and forecasting to better anticipate online sales on the basis of a data set of product characteristics, and so on.
Here’s how two companies, both far from Silicon Valley upstarts, used new flows of information to radically improve performance.
Case #1: Using Big Data to Improve Predictions
Minutes matter in airports. So does accurate information about flight arrival times: If a plane lands before the ground staff is ready for it, the passengers and crew are effectively trapped, and if it shows up later than expected, the staff sits idle, driving up costs. So when a major U.S. airline learned from an internal study that about 10% of the flights into its major hub had at least a 10-minute gap between the estimated time of arrival and the actual arrival time — and 30% had a gap of at least five minutes — it decided to take action.
At the time, the airline was relying on the aviation industry’s long-standing practice of using the ETAs provided by pilots. The pilots made these estimates during their final approach to the airport, when they had many other demands on their time and attention. In search of a better solution, the airline turned to PASSUR Aerospace, a provider of decision-support technologies for the aviation industry.
In 2001 PASSUR began offering its own arrival estimates as a service called RightETA. It calculated these times by combining publicly available data about weather, flight schedules, and other factors with proprietary data the company itself collected, including feeds from a network of passive radar stations it had installed near airports to gather data about every plane in the local sky.
PASSUR started with just a few of these installations, but by 2012 it had more than 155. Every 4.6 seconds it collects a wide range of information about every plane that it “sees.” This yields a huge and constant flood of digital data. What’s more, the company keeps all the data it has gathered over time, so it has an immense body of multidimensional information spanning more than a decade. RightETA essentially works by asking itself “What happened all the previous times a plane approached this airport under these conditions? When did it actually land?”
After switching to RightETA, the airline virtually eliminated gaps between estimated and actual arrival times. PASSUR believes that enabling an airline to know when its planes are going to land and plan accordingly is worth several million dollars a year at each airport. It’s a simple formula: Using big data leads to better predictions, and better predictions yield better decisions.
Case #2: Using Big Data to Drive Sales
A couple of years ago, Sears Holdings came to the conclusion that it needed to generate greater value from the huge amounts of customer, product, and promotion data it collected from its Sears, Craftsman, and Lands’ End brands. Obviously, it would be valuable to combine and make use of all these data to tailor promotions and other offerings to customers, and to personalize the offers to take advantage of local conditions.
Valuable, but difficult: Sears required about eight weeks to generate personalized promotions, at which point many of them were no longer optimal for the company. It took so long mainly because the data required for these large-scale analyses were both voluminous and highly fragmented — housed in many databases and “data warehouses” maintained by the various brands.
In search of a faster, cheaper way, Sears Holdings turned to the technologies and practices of big data. As one of its first steps, it set up a Hadoop cluster. This is simply a group of inexpensive commodity servers whose activities are coordinated by an emerging software framework called Hadoop (named after a toy elephant in the household of Doug Cutting, one of its developers).
Sears started using the cluster to store incoming data from all its brands and to hold data from existing data warehouses. It then conducted analyses on the cluster directly, avoiding the time-consuming complexities of pulling data from various sources and combining them so that they can be analyzed. This change allowed the company to be much faster and more precise with its promotions.
According to the company’s CTO, Phil Shelley, the time needed to generate a comprehensive set of promotions dropped from eight weeks to one, and is still dropping. And these promotions are of higher quality, because they’re more timely, more granular, and more personalized. Sears’s Hadoop cluster stores and processes several petabytes of data at a fraction of the cost of a comparable standard data warehouse.
These aren’t just a few flashy examples. We believe there is a more fundamental transformation of the economy happening. We’ve become convinced that almost no sphere of business activity will remain untouched by this movement.
Without question, many barriers to success remain. There are too few data scientists to go around. The technologies are new and in some cases exotic. It’s too easy to mistake correlation for causation and to find misleading patterns in the data. The cultural challenges are enormous, and, of course, privacy concerns are only going to become more significant. But the underlying trends, both in the technology and in the business payoff, are unmistakable.
The evidence is clear: Data-driven decisions tend to be better decisions. In sector after sector, companies that embrace this fact will pull away from their rivals. We can’t say that all the winners will be harnessing big data to transform decision making. But the data tell us that’s the surest bet.
This blog post was excerpted from the authors’ upcoming article “Big Data: The Management Revolution,” which will appear in the October issue of Harvard Business Review.
Disaster Recovery is no longer a “nice to have” option for anyone.
By Peter Sardis, InSync Management Consulting LLC, Managing Principle Consultant, 5:00 PM November 2nd, 2012
As I write these lines, I am sitting at my personal DR (Disaster Recovery) location, which is no other than my brother in law’s house in upstate New York that luckily was not hit as hard by Sandy. This is because we have been out of our home in Edgewater New Jersey for four days now while waiting for the power company to restore service in our area. I have been lucky to have an alternative for myself and my family other than a dark cold home with no hot water and with sporadic communication via an old fashion copper wire landline phone hanging on my kitchen wall, which by the way and surprisingly is the only system that worked without interruption throughout this disaster.
I also have been lucky to have a job that allows me to work remotely without having to deal with long commutes and long lines at the gas stations for now.
The question for individuals, small and large businesses and governments going forward will be how much is the real cost of unintended disruption to normal operations due to a natural or manmade disasters such as war, terrorism, nuclear power plan accidents, economic melt downs, cyber terrorism etc . What can be done to mitigate the huge risks of such disruptions that can threaten the local and global economy and the wellbeing of millions around the globe is a debate we must have. In such an unpredictable and complex world the focus of smart businesses and local and federal governments should be on how to build supply chains, power grids, communication, financial, health care and transportation networks systems, emergency response procedures and processes that are resilient, adaptable, flexible and agile. The transformation from systems that served the public for the past century or so will not be easy, however it is necessary if we want to have economically vital businesses going forward that are better prepared to absorb the shocks caused by such massive global operations disruptions.
This past week the resilience of all the systems around the tri-state area has been tested and unfortunately do not get a passing grade. Sandy has come to demonstrate the vulnerability of transportation and communication networks and the shortcomings of the local supply chain as we can see with the gasoline shortages we are experiencing here in New York and New Jersey. We see the devastating effect of creating complex mega cities that are vulnerable due to lack of integrated planning. Sandy has also attenuated our increased dependency on consumer products like our ipads, iphones and all sorts of android devices that proved to be unreliable for the most part since the underline communications network infrastructure and the lack of power sources to charge such devises when disaster strikes renders them inoperable.
“…We can no longer ignore the economic impact and the potential loss of life that is caused by a combination of outdated systems, major weather pattern changes and manmade disasters. …”
At the enterprise level, because of the dependency enterprises have to the same communication, supply chain, transportation networks and power grid, it proved extremely difficult or impossible to get information out to employees affected by the disaster and coordinate a response to the emergency. In many cases even for businesses that had a well thought out DR plans proved to be difficult or impossible to act on these plans as people had no way to commute to and from the work sites due to the damage caused by the flood waters and the disruption to the transportation network. The gasoline supply shortages and the wide spread power outages has added to the misery and has demonstrated that even an enterprise’s well thought out plan is not adequate due to its dependency to much larger and complex systems for an operation to continue to function. Millions of work hours are lost and the real total economic impact to the enterprises affected by this disaster will not be known for a while but it looks like it will be in the billions.
After the massive earthquake, tsunami and nuclear meltdown in Japan, the tsunami that hit Indonesia earlier with quarter million dead, the violent weather patterns and wild fires experienced in the past few years all over US and North Europe, it is clear that the global economy experiences ripple effects from major disruptions that cannot be predicted, they are random and extremely difficult to prepare for. Never before all over this planet, were countries and businesses more integrated and dependent on each other. At a macro level the global supply chain is getting interrupted much more frequent than in the past due to natural and manmade disasters and the impact can be felt on every level of the societies affected.
We can no longer ignore the economic impact and the potential loss of life that is caused by a combination of outdated systems, major weather pattern changes and manmade disasters. There is an urgent need to invent and invest in infrastructure that builds smarter sustainable cities and enterprises. We need to rethink risk management and create adaptable and resilient Disaster Recovery plans for individuals, businesses and governments that provide alternatives to assure access to goods and services at all times and especially during emergency situations. We need to re-engineer flexible emergency response procedures and processes that allow local and federal agencies and first responders to assist swiftly and with efficiency.
“…We are vulnerable to weather patterns and natural disasters, but this is not new. What is new is the dependency we have for survival on technology and systems that if not available can be disastrous for societies…”
There is an opportunity in the aftermath of Sandy, and before its devastating effects fades from our memories to collaborate and take preventative action in developing and implementing new technologies that can be utilized locally, nationally and globally in new infrastructure, systems, processes and procedures that will meet the needs of the citizens of the 21st century and beyond. Further hesitation at all levels of the influence spectrum; political squandering and choosing the status quo instead of taking steps to prepare ourselves for the next disaster are beyond foolish. We are vulnerable to weather patterns and natural disasters, but this is not new. What is new is the dependency we have for survival on technology and systems that if not available can be disastrous for societies. We are also vulnerable to ill intentions from groups and countries that if they choose may try to attack this antiquated infrastructure sometime in the future. The time to act is now so we have an opportunity to build a new future for all of us and in the process help create new jobs and economic growth.
How Hacking the Human Brain Can Save Civilization
Published By Mark Buchanan Sep 9, 2012 6:30 PM ET @bloomberg.com
Humans’ power to determine the future of planet Earth is increasing exponentially. The result could be disastrous unless we change the way we think.
Climate scientist Will Steffen of the Australian National University makes a powerful case about the uniqueness of our time: A few hundred years ago, more or less in sync with the Industrial Revolution, various indicators such as global population, water use, number of rivers dammed, global economic output, number of species extinctions and atmospheric carbon dioxide started following a steepening path upward.
The sudden explosion in human activity was a sharp break from the preceding tens of thousands of years, when things changed much more gradually. The shift is so pronounced that scientists now talk about a new geological era — the “Anthropocene,” in which all Earth processes come to be powerfully shaped by human activity. Of all the usable energy reaching the Earth from the sun, we humans already gather and exploit as much as 5 percent. Nearly half of the planet’s land surface has been altered by human action and practice.
This is all the result of the singular skill that sets us apart from all other species — our unparalleled capacity for innovation, especially through technology. New techniques for everything from farming to computation interact and combine to drive the creation of more innovations in an ever-accelerating spiral. Paradoxically, technological innovation has also created our biggest problems, including climate change, environmental destruction and the threat of nuclear annihilation.
Finite Planet
What comes next? Exponential growth on a finite planet simply cannot continue. If innovation is both the key to our success and the primary threat to our existence, what can we do? Can we innovate differently? More intelligently?
Some valuable thinking on the subject comes from Sander van der Leeuw, dean of the School of Sustainability at Arizona State University, who takes an optimistic view. We may indeed be able to use technology to find a path to a sustainable future, he suggests, if we use our technology in a fundamentally different way.
The gist of his argument: Humans suffer from a mismatch between our thinking about what we do and the truth of what we do. Our brains make sense of a multifaceted world by ignoring much of its complexity — a trait Van der Leeuw calls “low dimensional” thinking. In engineering a dam, assessing how agricultural runoff influences an estuary or figuring out how automobile emissions might alter the atmosphere, our conceptual models (or those of our scientists and engineers) at best consider only a few of the true pathways of cause and effect. As Van der Leeuw puts it, “every human action upon the environment modifies the latter in many more ways that its human actors perceive, simply because the dimensionality of the environment is much higher than can be captured by the human mind.”
This is a profound insight. It helps explain why our innovations, even as they help us in ways that we see clearly and understand, also end up affecting our environment in ways that we mostly fail to recognize. Effects build up in the environment — and this includes the social environment, as well as biological or physical — over the long term. We’re unaware, until eventually we have the famous “unintended consequences” so familiar from technological history. We may, for example, not yet know what lies behind the obesity epidemic in Western nations, but it is surely a consequence of one or more technologies — in food manufacturing and distribution, in human transportation, in entertainment and advertising.
Reckless Innovation
If Van der Leeuw’s analysis is right, then we should be worried about the future. We’re currently locked into a strategy of almost reckless innovation. If we seek further economic growth only through faster innovation, the unintended impact on our environment promises to grow even faster.
Van der Leeuw’s solution: Learn to innovate differently, by using technology to reduce the mismatch between our brains and reality. Computing and communications technology can improve our ability to handle large quantities of information. They make it possible, in principle, to help our brains build more accurate models of reality. Indeed, this is already happening in some areas, in large-scale models of climate, which include thousands or millions of atmospheric variables, or in new models of economies that try to include every last business or household.
In the end, Van der Leeuw’s perspective is both sobering and inspiring. We have an unprecedented opportunity as the first humans to be able to address our cognitive limitations consciously and directly, by using technology to increase our brain capacity and understand our interactions with the world in far more detail. All we require is the wisdom to make this our goal.
(Mark Buchanan, a theoretical physicist and the author of “The Social Atom: Why the Rich Get Richer, Cheaters Get Caught and Your Neighbor Usually Looks Like You,” is a Bloomberg View columnist. The opinions expressed are his own.)
How Better Interpersonal Relationships Lead to More Effective Team Members
Published: January 18, 2012 in Knowledge@Wharton
When it comes to teams, less is sometimes more. In a recent paper, Wharton management professor Jennifer Mueller found that while larger teams generally are more productive overall than smaller ones, members of the bigger groups were less fruitful individually than their counterparts on the smaller teams. The research, “Why Individuals in Larger Teams Perform Worse,” was published in the August issue of the journal Organizational Behavior and Human Decision Processes.
“There are costs to collaborating,” says Mueller. “In larger teams, one of those costs is that people may not have the time and energy to form relationships that really help their ability to be productive.” Mueller became interested in the issue of how team size impacted individual performance after reading through material collected from 26 corporate design teams as part of an ongoing research project led by Teresa Amabile, a professor at Harvard Business School. Through the research group, Mueller had access to journals and questionnaires provided by the 238 people on teams tasked with developing a host of products and services, including inventing a new type of dental floss, designing a new airline ticket purchase process and creating a cut-resistant fiber to be worn by factory workers. The content of the journals was eye-opening, Mueller says. “I started to recognize that employees in these larger design teams experienced incredible amounts of stress. People often said, ‘I don’t feel I can get the resources to do what they want me to do.’ One person referred to the experience as a ‘death march.'”
Mueller also began to see a pattern — the stress level seemed higher for members of larger teams. “On a smaller team, people knew what resources were available and felt they could ask questions when things went wrong. The situation was more controllable,” Mueller states. “But in these larger teams, people were lost. They didn’t know who to call for help because they didn’t know the other members well enough. Even if they did reach out, they didn’t feel the other members were as committed to helping or had the time to help. And they couldn’t tell their team leader because [it would look like] they had failed.”
The challenges of larger teams are well studied in academic literature. Mueller says that one meta-analysis showed that larger groups tend to perform better than small groups, but the group performance gains for every additional member are minimal because individuals in these larger groups perform worse than individuals in smaller groups. Previous work has focused on two culprits behind this: motivation and coordination loss. The first stems from the reality that people may not work as hard if their contribution is likely to be lost, or go unrewarded, due to the size of the team. Coordination loss refers to the difficulty getting all the disparate elements of a large team to work well together.
But Mueller suspected that there was a third force at work: relational loss. According to the paper, “Relational losses specifically involve perceptions about the extent to which teammates are likely to provide help, assistance and support in the face of struggle or difficulty.” Mueller’s theory was that this deterioration in connections between team members increased with team size, resulting in weaker performance on average by individual participants.
To assess the impact of relational loss, Mueller gathered questionnaires from the 238 team members from the Harvard study throughout their product or service development effort. The questionnaires included performance evaluations of each individual from both their peers and the team leader. The questionnaires also probed team members on their motivation, the team’s coordination and the degree to which they felt connected to other people in the group. By creating models around that data, Mueller was able to show that the stress caused by a lack of connection to other members of the group was a key driver behind the lower performance of individuals on the larger teams.
“There was some evidence of coordination loss, but I did not see evidence of motivation loss,” Mueller notes. “I saw compelling evidence for relational loss — it loomed larger than you might expect given how much emphasis is given to coordination.” Less-than-optimal relationships make people on a team feel isolated and unsupported, Mueller says, particularly when problems surface. That anxiety can have a direct impact on a team member’s performance. “Stress soaks up your cognitive resources and diminishes the extent to which you can hold information in your memory. That contributes to a decline in performance.”
Mueller’s findings offer important insights on how companies should be approaching team-based initiatives. Given the complexity of product development projects, it may be impossible to gather all of the needed expertise within a small group of people, necessitating the formation of a larger group. But Mueller says finding a way to enhance the connections between members of those large teams is critical to improving their individual effectiveness. “One thing teams could do is to have a person who has the role of troubleshooter — the one who steps in to help when stuff goes wrong.” The troubleshooter knows what skills and resources are available to the team, and can bring the right people together to address problems. “This role could help lubricate these relationships that don’t have the opportunity to form naturally,” Mueller notes. She adds that the “problem solver” position should not be filled by the team leader because team members may be reluctant to go to the boss to discuss problems.
What about trying to foster connections between members of a large team by simply creating opportunities for people to get to know each other better? If a team is likely to be in place for years, that sort of effort — including offsite team-building sessions — makes sense, Mueller says. But for teams that will only operate for a more limited period, those steps can simply take too long to bear fruit.
Essence of Leadership
The essence of leadership phenomenon is captured in the formula L = E3 (Where E stands for Envision, Energize and Execute). Leadership is a synergy among the three parameters. A leader must develop the qualities that allow all three parameters to have a value greater than zero. This simple mathematical formula encapsulates the essence of leadership and provides the framework in my quest to understand what leadership is, and what sets apart a few people to become religious, community, corporate, political or military leaders from the rest of the populous. These are the people that possess the necessary qualities and ingredients that allow them to reach the pinnacle of the pyramid to become leaders.
Team building one step at a time
Throughout the years I have managed several teams. I found the most effective way to build a team is to give the members small, feasible and time-bounded projects which force them to “live” together for a while.
Who defines the share value of an organization ?
Values tie in with credibility. The job of the leader is to survey the employees to find out their fundamental values and shape the strategy around those values. People have to know that someone is in charge, however they have to believe that the person in charge also shares their values.
The Sardis Press 