Tag Archive for 'high availability'

Continuous Availability versus High Availability

Wikipedia’s page on Continuous Availability is available here:


A quick perusal tells us that High Availability can be ‘accomplished by providing redundancy or quickly restarting failed components’. This is very different from ‘Continuously Available’ systems that enable continuous operation through planned and unplanned outages of one or more components.

As large global organizations move from using Hadoop for batch storage and retrieval to mission critical real-time applications where the cost of even one minute of downtime is unacceptable, mere high availability will not be enough.

Solutions such as HDFS NameNode High Availability (NameNode HA) that come with Apache Hadoop 2.0 and Hadoop distributions based on it are subject to downtimes of 5 to 15 minutes.  In addition, NameNode HA, is limited to a single data center, and only one NameNode can be active at a time, creating a performance as well as an availability bottleneck. Deployments that incorporate WANdisco Non-Stop Hadoop are not subject to any downtime, regardless of whether a single NameNode server or an entire data center goes offline. There is no need for maintenance windows with Non-Stop Hadoop, since you can simply bring down the NameNode servers one at a time, and perform your maintenance operations.  The remaining active NameNodes continue to support real-time client applications as well as batch jobs.

The business advantages of a Continuously Available, multi-data center aware systems are well known to IT decision makers. Here are some examples that illustrate how both real-time and batch applications can benefit and new use cases can be supported:

  • A Batch Big Data DAG is a chain of applications wherein the output of a preceding job is used as the input to a subsequent job. At companies such as Yahoo, these DAGs take six to eight hours to run, and they are run every day. Fifteen minutes of NameNode downtime may cause one of these jobs to fail. As a result of this single failure, the entire DAG may not run to completion, creating delays that can last many hours.
  • Global clickstream analysis applications that enable businesses to see and respond to customer behavior or detect potentially fraudulent activity in real-time.
  • A web site or service built to use HBase as a backing store will be down if the HDFS underlying HBase goes down when the NameNode fails. This is likely to result in lost revenue and erode customer goodwill.  Non-Stop Hadoop eliminates this risk.
  • Continuous Availability systems such as  WANdisco Non-Stop Hadoop are administered with  fewer staff. This is because failure of one out of five NameNodes is not an emergency event. It can be dealt with by staff during regular business hours. Significant cost savings in staffing can be achieved since Continuously Available systems do not require 24×7 sysadmin staff .  In addition, in a distributed multi-data center environment, Non-Stop Hadoop can be managed from one location.
  • There are no passive or standby servers or data centers that essentially sit idle until disaster strikes.  All servers are active and provide full read and write access to the same data at every location.

See a demo of Non-Stop Hadoop for Cloudera and Non-Stop Hadoop for Hortonworks in action and read what leading industry analysts like Gartner’s Merv Adrian have to say about the need for continuous Hadoop availability.



About Jagane Sundar

Answers to questions from the Webinar of Dec 11, 2012

 Download the webinar slides here.

Question 1: Are there any special considerations or support of Spring technologies for this (i.e. Spring-Data, Spring-Integration, Spring-Batch)?

Answer: We are continuously looking at technologies that make Hadoop easier to use and program. Spring-Data, Spring-Integration and Spring-Batch show promise. When sufficient momentum is gathered by these projects, we will work with the Spring community to include a tested version of these technologies in the AltoStor Appliance.

Question 2: What is the hadoop version underneath the appliance?

Answer: Hadoop 2. We intend to remain close to the latest version of Hadoop at any given point in time, modulo fixes and changes for bug fixes.

Question 3: Will the pricing model based on number of name nodes or size of the cluster?

Answer: Pricing decisions have not been made yet. We will announce pricing in the first quarter of 2013.

Question 4: Can you comment on how load balancing is resolved across active nodes? Is there a load balance router concept?

Answer: We do not require/depend on any specialized hardware such as load balancers or NFS filers. By “load balancing” we simply mean that application requests (read or write) can be directed to any NameNode based on its proximity to the client or available resources. Thus NameNodes can share the workload and provide higher overall cluster performance compared to active-standby architecture.

Question 5: How does Active-Active replication impact processing time relative to current Hadoop architectures?

Answer: Active-Active replication will result in load balancing of clients across many NameNodes, i.e. fewer clients will be serviced by each NameNode. Since NameNodes share the workload on a busy cluster, you should expect faster response time for clients. Generally, more active NameNodes can perform a proportionally larger amount of work.


About Jagane Sundar

Can’t live without Subversion – even for a minute?

Can't Live without SVN - even for a minute?

Enterprise Subversion Webinar by WANdisco

The popularity of Apache Subversion (SVN) as a version control system has grown significantly in recent times and is widely recognized as the sole leader in Standalone Software Configuration Management (SCM). With more than five million users, an increasing share of the market and hundreds of major global companies deploying Subversion for their development needs it’s clear that many can’t live without Subversion – even for a minute!

Subversion maintains current and historical versions of files such as source code, web pages, and documentation, making it a highly compatible successor to various traditional SCM solutions. Subversion is open source (and therefore free), which suggests some of the reasons for its popularity but it also performs and scales in some of the most aggressive SCM environments on the planet where some of the traditional SCM products could not.

WANdisco supports the Subversion open source project in a number of tangible ways and one of those is through providing free training webinars. It’s important that users of SVN are maximising their skill set for their own benefit – and that of course strengthens the Subversion community in the process.

The suite of Subversion training courses provided by WANdisco caters for developers and administrators from many different sectors but our latest addition to the free webinar schedule is aimed at major firms that demand the very best from their SVN implementation.

Following the success of our Enterprise Hardened Subversion webinar, ‘Can’t live without Subversion – even for a minute’ will provide excellent content for users who demand that Subversion is always on and always accessible.

A featured case study from one of WANdisco’s valued Subversion customers will help participants learn why companies have become dependent on Subversion for mission critical applications and why they can’t afford to be down, even for a single minute!

The case study will highlight a very typical Subversion deployment: a technology company with around 100 developers started using Subversion as an ad hoc project implementation and it grew to become the company standard.  However, what they thought was a secure automated backup system actually failed and brought the entire system down, resulting in lost productivity of 100 idle developers for an entire day and project delays for an entire week lost before full functionality was restored.  That’s a lot of minutes – and dollars!!!  Could this happen to you?!

WANdisco’s Subversion Clustering provides an excellent solution and the webinar will explain how a three node cluster results in no single point of failure and extremely high availability. We will also tell you how implementation services were used to accelerate adoption with costs easily justified – the outlay being a fraction of the cost of real downtime.

If that wasn’t enough we will take you through a more detailed account of WANdisco’s Subversion Clustering, including replication, intelligent load balancing, flexible deployment options, extension with Multisite and implementation services.

If you can’t live without Subversion then make sure you register for this free webinar – and don’t waste a minute.

Click here to register for this webinar now.