How Netflix plans to improve its Operational Visibility with Real-time Data Visualization?

This is the second post in our Data Visualization Spotlight series where we showcase how different organizations are using data visualization and analytics to solve their day to day problems.

In mid-March 2013, Netflix reported a global streaming subscriber list of 33 million. It increased to 36.3 million (29.2 million in U.S.) in April 2013. It had 40.4 million subscribers (31.2 million in U.S.) in September 2013.  By Q4 2013, it reported 33.1 million subscribers in U.S alone.

Netflix

With its subscriber list growing by leaps and bounds, Netflix faces the daunting task of supporting millions of connected devices spread across 40+ countries. At such a big a scale, it is impossible to manually monitor all that data. Imagine having to detect a system fault in an environment that is not only large and complex but also highly distributed!

To thwart such operational nightmares from occurring, the Netflix team is working on a greenfield project that focuses on building the next generation tools for operational visibility that can proactively detect and communicate system faults and identify areas of improvement.

Building systems to create greater visibility into an increasingly complicated and evolving world

In an article posted on the Netflix blog, Ranjit Mavinkurve, Justin Becker and Ben Christensen share their ideas on how they want to extend and improve the existing insight tools at Netflix.

Netflix Dashboard Annoted Circuit

Fig: An excerpt from the current Netflix dashboard along with explanations of what all the data represents. Source: Techblog.netflix

The tools that are currently in Netflix’s arsenal include dashboards that display the status of their systems in near real time, and alerting mechanisms that notify them of major problems. While these tools are very helpful, the team feels there is a huge scope of improvement in them.

“With our next generation of insight tools, we have the opportunity to create new and transformative ways to effectively deliver insights and extend our existing insight capabilities. We plan to build a new set of tools and systems for operational visibility that provide the insight features and capabilities that we need.”

So what will be new in the system?

Among other things, the team wants to focus on improving the data visualization capabilities of its tools. At Netflix, data visualization has always been of paramount importance. Many of Netflix’s major systems contain significant data visualization components and employees routinely look to existing data viz tools to tweak algorithms, garner new insights, and solve pressing business issues. (Related read: How Netflix uses data viz to garner customer insights?)

The existing insight tools at Netflix are system-oriented. They are built from the perspective of the system providing the metrics. This results in a proliferation of custom tools and views that require specialized knowledge to use and interpret. Also, some of the tools tend to focus more on system health and not as much on the customers’ streaming experience.

With the new set of tools, the team wants to tailor the insights and views to meet the needs of the tool’s consumers—internal staff members such as engineers who want to view the health of a particular part of the system or look at some aspect of the customers’ streaming experience.

“Our existing insight tools have dashboards with time-series graphs that are very useful and effective. With our new insight tools, we want to take our tools to a whole new level, with rich, dynamic data visualizations that visually communicate relevant, up-to-date details of the state of our environments for any operational facets of interest. For example, we want to surface interesting patterns, system events, and anomalies via visual cues within a dynamic timeline representation that is updated in near real-time.” 

Netflix WIP Dashboard  - 1

Fig: The mockup (with dummy data) for one of the views shows several components within the design: a top level navigation bar to switch between different views in the system, a breadcrumbs component highlighting the selected facets, a main view module (a map in this instance), a key metrics component, a timeline and an incident view, on the right side of the screen. The main view communicates data based on the selected facets. Source: Techblog.netflix

Netflix WIP Dashboard - 2

Fig: Another mockup (with dummy data) represents another view in the system and displays routes in their edge tier with request rates, error rates and other key metrics for each route updated in near real-time. Source: Techblog.netflix

As per the team, all views in the new system will be dynamic and will reflect the current operational state based on selected facets. A user can modify the facets and immediately see changes to the user interface.

Final thoughts

With the help of dynamic data visualization and real-time insights, Netflix aims to achieve an in-depth understanding of operational systems, make product and service improvements, and find and fix problems quickly so that it can continue to innovate rapidly and delight customers at every interaction. And a happy customer is what it takes to be successful in the long run and Netflix knows that for sure!

In the next post of the Data Visualization Spotlight series, read how P&G uses Data Visualization to uncover new opportunities for growth.

Reference:

Techblog.netflix

 

Leave a Comment