HDFS Blog

Why Data Driven Companies Rely on WANdisco Fusion

Hadoop is now clearly gaining momentum. We are seeing more and more customers attempting to deploy enterprise grade applications. Data protection, governance, performance and availability are top concerns. WANdisco Fusion’s level of resiliency is enabling customers to move out of the lab and into production much faster.

As companies start to scale these platforms and begin the journey to becoming data driven, they are completely focused on business value and return on investment. WANdisco’s ability to optimize resource utilization by eliminating the need for standby servers resonates well with our partners and customers. These companies are not Google or Facebook. They don’t have an endless supply of hardware and their core business isn’t delivering technology.

As these companies add data from more sources to Hadoop, they are implementing backup and disaster recovery plans and deploying multiple clusters for redundancy. One of our customers, a large bank, is beginning to utilize the cloud for DR.

I’ve met 11 new customers in the past eight days. Five of them have architected cloud into their data lake strategy and are evaluating the players. They are looking to run large data sets in the cloud for efficiency as well as backup and DR.

One of those customers, a leader in IT security, tells me they plan to move their entire infrastructure to the cloud within the next 12 months. They already have 200 nodes in production today, which they expect to double in a year.

Many of our partners are interested in how they can make it easy to onboard data from behind the firewall to the cloud while delivering the best performance. They recognize this is fundamental to a successful cloud strategy.

Companies are already embarking on migrations from one Hadoop platform to another. We’re working with customers on migration from MapR to HDP, CDH to HDP, CDH to Oracle BDA, and because we are HCFS compatible, GPFS to IOP. Some of these are petabyte scale.

For many of these companies, WANdisco Fusion’s ability to eliminate downtime, data loss and business disruption is a prerequisite to making that transition. Migration has never been undertaken lightly. I’ve spoken to partners who are unable to migrate their customers due to the required amount of downtime and risk involved.

One customer I met recently completed a large migration to HDP and just last week acquired a company that has a large cluster on Cloudera. We’re talking to them about how we can easily provide a single consistent view of the data. This will allow them to get immediate value from the data they have just acquired. If they choose to migrate completely, they are in control of the timing.

Customers measure their success by time to value. We’re working closely with our strategic partners to ensure our customers don’t have to worry about the nuts and bolts, irrespective of distributions, on-prem, cloud, or hybrid environment so customers can concentrate on the business outcome.

Please reach out to me if these use cases resonate and you would like to learn more.

Peter Scott
SVP Business Development

avatar

About Mackensie Gibson

The inspiration for WANdisco Fusion

Screen Shot 2015-04-21 at 10.08.22 PM

Roughly two years ago, we sat down to start work on a project that finally came to fruition this week.

At that meeting, we had set ourselves the challenge of redefining the storage landscape. We wanted to map out a world where there was complete shared storage, but where the landscape remained entirely heterogeneous.

Why? Because we’d witnessed the beginnings of a trend that has only grown more pronounced with the passage of time.

From the moment we started engaging with customers, we were struck by the extreme diversity of their storage environments. Regardless of whether we were dealing with a bank, a hospital or utility provider, different types of storage had been introduced across every organization for a variety of use cases.

In time, however, these same companies wanted to start integrating their different silos of data, whether to run real-time analytics or to gain a full 360 perspective of performance. Yet preserving diversity across data center was critical, given that each storage type has its own strengths.

They didn’t care about uniformity. They cared about performance and this meant being able to have the best of both worlds. Being able to deliver this became the Holy Grail – at least in the world of data centers.

This isn’t quite The Gordian Knot but it’s certainly a very difficult, complex problem and possibly one that could only be solved with our core, patented IP DConE.

Then we had a breakthrough.

Months later and I’m proud to formally release WANdisco Fusion (WD Fusion), the only product that enables WAN-scope active-active synchronization of different storage systems into one place.

What does this mean in practice? Well it means that you can use Hadoop distributions like Hortonworks, Cloudera or Pivotal for compute, Oracle BDA for fast compute, EMC Isilon for dense storage. You could even use a complete variety of Hadoop distros and versions. Whatever your set-up, with WD Fusion you can leverage new and existing storage assets immediately.

With it, Hadoop is transformed from being something that runs within a data center into an elastic platform that runs across multiple data centers throughout the world. WD Fusion allows you to update your storage infrastructure one data center at a time, without impacting your application ability or by having to copy vast swathes of data once the update is done.

When we were developing WD Fusion we agreed upon two things. First, we couldn’t produce anything that made changes to the underlying storage system – this had to behave like a client application. Second, anything we created had to enable a complete single global name-space across an entire storage infrastructure.

With WD Fusion, we allow businesses to bring together different storage systems by leveraging our existing intellectual property – the same Paxos-powered algorithm behind Non-Stop Hadoop, Subversion Multisite and Git Multisite – without making any changes to the platform you’re using.

Another way of putting it is we’ve managed to spread our secret sauce even further.

We have some of the best computer scientists in the world working at WANdisco, but I’m confident that this is the most revolutionary project any of us have ever worked on.

I’m delighted to be unveiling WD Fusion. It’s a testament to the talent and character of our firm, the result of looking at an impossible scenario and saying: “Challenge accepted.”

avatar

About David Richards

David is CEO, President and co-founder of WANdisco and has quickly established WANdisco as one of the world’s most promising technology companies. Since co-founding the company in Silicon Valley in 2005, David has led WANdisco on a course for rapid international expansion, opening offices in the UK, Japan and China. David spearheaded the acquisition of Altostor, which accelerated the development of WANdisco’s first products for the Big Data market. The majority of WANdisco’s core technology is now produced out of the company’s flourishing software development base in David’s hometown of Sheffield, England and in Belfast, Northern Ireland. David has become recognised as a champion of British technology and entrepreneurship. In 2012, he led WANdisco to a hugely successful listing on London Stock Exchange (WAND:LSE), raising over £24m to drive business growth. With over 15 years' executive experience in the software industry, David sits on a number of advisory and executive boards of Silicon Valley start-up ventures. A passionate advocate of entrepreneurship, he has established many successful start-up companies in Enterprise Software and is recognised as an industry leader in Enterprise Application Integration and its standards. David is a frequent commentator on a range of business and technology issues, appearing regularly on Bloomberg and CNBC. Profiles of David have appeared in a range of leading publications including the Financial Times, The Daily Telegraph and the Daily Mail. Specialties:IPO's, Startups, Entrepreneurship, CEO, Visionary, Investor, ceo, board member, advisor, venture capital, offshore development, financing, M&A

Name Node単一障害点回避 (QJMとNon-Stop Name Node)

ThinkIT Web記事で「NameNode障害によって発生するSingle Point of Failure問題を解決するソリューション」として弊社のNon-Stop Hadoopを紹介して頂きました。http://thinkit.co.jp/story/2014/11/11/5413

同様のソリューションとしてQJMもありますので、少し補足をしておきたいと思います。

QJMについてスライド1

2012年にCloudera社のTodd Lipcon氏が提案したものです。HadoopのアーキテクチャではNameNodeが単一障害点になっていたのを冗長化するものです。StandbyのNamaNodeを追加し、Journal Nodeで複数のジャーナルをとり、Zookeeperにより障害を検出し手動・自動でのフェールオーバーが可能となりました。右上図のような構成になります。

 

Non-Stop NameNodeについて

Paxos(パクソス)を拡張したActive-Activeの複製を行う仕組み(Distributed Coordination スライド1Engine: DConE)をNameNodeに組みこんでいます。これによりHSDF-6469に準拠したConsensusNodeとして複数のNameNodeが同一のメタデータを保持し同等に動作することになります。あるNameNodeが障害になっても多数決論理によりNameNodeのメタデータの更新を行うので、過半数のNameNodeが生きていれば継続稼働が可能です。右下図では5つNameNodeがあるので2つが落ちてもHadoopが止まることはありません。当該NamaNodeの障害が復旧すれば最新のメタデータまで自動的に復元されます。

QJMと比較した特長は以下の通りです。

  • 100%の稼働を、運用者に障害時・回復時の負担を掛けずに実現できます
  • 全てのNameNodeはActiveであり、またQJMで必要となるJournal、Zookeeperも不要です。リソースは100%活用されます
  • 複数NameNodeによる負荷分散が可能となり、性能向上が可能です。またシステムを止めないで拡張が可能です

更にNon-Stop HadoopはDataNodeのデータを、指定した範囲で自動的に複製する機能を提供しています。これにより以下のようなことも可能となります。

  • NameNodeがWANを跨がった別のデータセンタにあってもメタデータの一貫性は保障されます。容量の大きいDataNodeの複製は非同期に行います。遠隔地にあるデータセンタに複製が自動的に作られ、ディザスタリカバリも可能となります。
  • 異なる場所のデータセンタにその地域で発生したデータを格納し、別の場所から使用することも可能になります。例えばクレジットカード使用データは東京、NY、シンガポールのデータセンタに適宜格納し、不正検出のアプリは東京で動かすといった使い方が可能となります。

要は複数のHadoopクラスタを、仮想的に一つに見せることが可能という事です。これはクラスタが別のデータセンタに分散している場合も可能です。

NameNodeのメタデータの一貫性が保障されることで、述べてきたようなことが可能になっています。分散環境での一貫性の保障を行うのがPaxosを拡張した弊社の特許技術であるDistribution Coordination Engineです。これについては別途、紹介したいと思います。

avatar

About Kenji Ogawa (小川 研之)

WANdisco社で2013年11月より日本での事業を展開中。 以前は、NECで国産メインフレーム、Unix、ミドルウェアの開発に従事。その後、シリコンバレーのベンチャー企業開拓、パートナーマネージメント、インドでのオフショア開発に従事。

Big Data ETL Across Multiple Data Centers

Scientific applications, weather forecasting, click-stream analysis, web crawling, and social networking applications often have several distributed data sources, i.e., big data is collected in separate data center locations or even across the Internet.

In these cases, the most efficient architecture for running extract, transform, load (ETL) jobs over the entire data set becomes nontrivial.

Hadoop provides the Hadoop Distributed File System (HDFS) for storage and YARN (Yet Another Resource Negotiator) as the programming model in Hadoop 2.0. ETL jobs use the MapReduce programming model to run on the YARN framework.

Though these are adequate for a single data center, there is a clear need to enhance them for multi-data center environments. In these instances, it is important to provide active-active redundancy for YARN and HDFS across data centers. Here’s why:

1. Bringing compute to data

Hadoop’s architectural advantage lies in bringing compute to data. Providing active-active (global) YARN accomplishes that on top of global HDFS across data centers.

2. Minimizing traffic on a WAN link

There are three types of data analytics schemes:

a) High-throughput analytics where the output data of a MapReduce job is small compared to the input.

Examples include weblogs, word count, etc.

b) Zero-throughput analytics where the output data of a MapReduce job is equal to the input. A sort operation is a good example of a job of this type.

c) Balloon-throughput analytics where the output is much larger than the input.

Local YARN can crunch the data and use global HDFS to redistribute for high throughput analytics. Keep in mind that this might require another MapReduce job running on the output results, however, which can add traffic to the WAN link. Global YARN mitigates this even further by distributing the computational load.

Last but not least, fault tolerance is required at the server, rack, and data center levels. Passive redundancy solutions can cause days of downtime before resuming. Active-active redundant YARN and HDFS provide zero-downtime solutions for MapReduce jobs and data.

To summarize, it is imperative for mission-critical applications to have active-active redundancy for HDFS and YARN. Not only does this protect data and prevent downtime, but it also allows big data to be processed at an accelerated rate by taking advantage of the aggregated CPU, network and storage of all servers across datacenters.

– Gurumurthy Yeleswarapu, Director of Engineering, WANdisco

Application Specific Data? It’s So 2013

Looking back at the past 10 years of software the word ‘boring’ comes to mind.  The buzzwords were things like ‘web services’, ‘SOA’.  CIO’s Tape drives 70sloved the promise of these things but they could not deliver.  The idea of build once and reuse everywhere really was the ‘nirvana’.

Well it now seems like we can do all of that stuff.

As I’ve said before Big Data is not a great name because it implies that all we are talking about a big database with tons of data.  Actually that’s only part of the story. Hadoop is the new enterprise applications platform.  The key word there is platform.  If you could have a single general-purpose data store that could service ‘n’ applications then the whole of notion of database design is over.  Think about the new breed of apps on a cell phone, the social media platforms and web search engines.  Most of these do this today.  Storing data in a general purpose, non-specific data store and then used by a wide variety of applications.  The new phrase for this data store is a ‘data lake’ implying a large quantum of every growing and changing data stored without any specific structure

Talking to a variety of CIOs recently they are very excited by the prospect of both amalgamating data so it can be used and also bringing into play data that previously could not be used.  Unstructured data in a wide variety of formats like word documents and PDF files.  This also means the barriers to entry are low.  Many people believe that adopting Hadoop requires a massive re-skilling of the workforce.  It does but not in the way most people think.  Actually getting the data into Hadoop is the easy bit (‘data ingestion‘ is the new buzz-word).  It’s not like the old relational database days where you first had to model the data using data normalization techniques and then use ETL to make the data in usable format.  With a data lake you simply set up a server cluster and load the data. Creating a data model and using ETL is simply not required.

The real transformation and re-skilling is in application development.  Applications are moving to data – today in a client-server world it’s the other way around.  We have seen this type of reskilling before like moving from Cobol to object oriented programming.

In the same way that client-server technology disrupted  mainframe computer systems, big data will disrupt client-server.  We’re already seeing this in the market today.  It’s no surprise that the most successful companies in the world today (Google, Amazon, Facebook, etc.) are all actually big data companies.  This isn’t a ‘might be’ it’s already happened.

avatar

About David Richards

David is CEO, President and co-founder of WANdisco and has quickly established WANdisco as one of the world’s most promising technology companies. Since co-founding the company in Silicon Valley in 2005, David has led WANdisco on a course for rapid international expansion, opening offices in the UK, Japan and China. David spearheaded the acquisition of Altostor, which accelerated the development of WANdisco’s first products for the Big Data market. The majority of WANdisco’s core technology is now produced out of the company’s flourishing software development base in David’s hometown of Sheffield, England and in Belfast, Northern Ireland. David has become recognised as a champion of British technology and entrepreneurship. In 2012, he led WANdisco to a hugely successful listing on London Stock Exchange (WAND:LSE), raising over £24m to drive business growth. With over 15 years' executive experience in the software industry, David sits on a number of advisory and executive boards of Silicon Valley start-up ventures. A passionate advocate of entrepreneurship, he has established many successful start-up companies in Enterprise Software and is recognised as an industry leader in Enterprise Application Integration and its standards. David is a frequent commentator on a range of business and technology issues, appearing regularly on Bloomberg and CNBC. Profiles of David have appeared in a range of leading publications including the Financial Times, The Daily Telegraph and the Daily Mail. Specialties:IPO's, Startups, Entrepreneurship, CEO, Visionary, Investor, ceo, board member, advisor, venture capital, offshore development, financing, M&A

Continuous Availability versus High Availability

Wikipedia’s page on Continuous Availability is available here:

http://en.wikipedia.org/wiki/Continuous_availability

A quick perusal tells us that High Availability can be ‘accomplished by providing redundancy or quickly restarting failed components’. This is very different from ‘Continuously Available’ systems that enable continuous operation through planned and unplanned outages of one or more components.

As large global organizations move from using Hadoop for batch storage and retrieval to mission critical real-time applications where the cost of even one minute of downtime is unacceptable, mere high availability will not be enough.

Solutions such as HDFS NameNode High Availability (NameNode HA) that come with Apache Hadoop 2.0 and Hadoop distributions based on it are subject to downtimes of 5 to 15 minutes.  In addition, NameNode HA, is limited to a single data center, and only one NameNode can be active at a time, creating a performance as well as an availability bottleneck. Deployments that incorporate WANdisco Non-Stop Hadoop are not subject to any downtime, regardless of whether a single NameNode server or an entire data center goes offline. There is no need for maintenance windows with Non-Stop Hadoop, since you can simply bring down the NameNode servers one at a time, and perform your maintenance operations.  The remaining active NameNodes continue to support real-time client applications as well as batch jobs.

The business advantages of a Continuously Available, multi-data center aware systems are well known to IT decision makers. Here are some examples that illustrate how both real-time and batch applications can benefit and new use cases can be supported:

  • A Batch Big Data DAG is a chain of applications wherein the output of a preceding job is used as the input to a subsequent job. At companies such as Yahoo, these DAGs take six to eight hours to run, and they are run every day. Fifteen minutes of NameNode downtime may cause one of these jobs to fail. As a result of this single failure, the entire DAG may not run to completion, creating delays that can last many hours.
  • Global clickstream analysis applications that enable businesses to see and respond to customer behavior or detect potentially fraudulent activity in real-time.
  • A web site or service built to use HBase as a backing store will be down if the HDFS underlying HBase goes down when the NameNode fails. This is likely to result in lost revenue and erode customer goodwill.  Non-Stop Hadoop eliminates this risk.
  • Continuous Availability systems such as  WANdisco Non-Stop Hadoop are administered with  fewer staff. This is because failure of one out of five NameNodes is not an emergency event. It can be dealt with by staff during regular business hours. Significant cost savings in staffing can be achieved since Continuously Available systems do not require 24×7 sysadmin staff .  In addition, in a distributed multi-data center environment, Non-Stop Hadoop can be managed from one location.
  • There are no passive or standby servers or data centers that essentially sit idle until disaster strikes.  All servers are active and provide full read and write access to the same data at every location.

See a demo of Non-Stop Hadoop for Cloudera and Non-Stop Hadoop for Hortonworks in action and read what leading industry analysts like Gartner’s Merv Adrian have to say about the need for continuous Hadoop availability.

 

avatar

About Jagane Sundar

WANdisco Announces Free Online Hadoop Training Webinars

We’re excited to announce a series of free one-hour online Hadoop training webinars, starting with four sessions in March and April. Time will be allowed for audience Q&A at the end of each session.

Wednesday, March 13 at 10:00 AM Pacific, 1:00 PM Eastern

A Hadoop Overview” will cover Hadoop, from its history to its architecture as well as:

  • HDFS, MapReduce, and HBase
  • Public and private cloud deployment options
  • Highlights of common business use cases and more

March 27, 10:00 AM Pacific, 1:00 pm Eastern

Hadoop: A Deep Dive” covers Hadoop misconceptions (not all clusters include thousands of machines) and:

  • Real world Hadoop deployments
  • Review of major Hadoop ecosystem components including: Oozie, Flume, Nutch, Sqoop and others
  • In-depth look at HDFS and more

April 10, 10:00 AM Pacific, 1:00 pm Eastern

Hadoop: A MapReduce Tutorial” will cover MapReduce at a deep technical level and will highlight:

  • The history of MapReduce
  • Logical flow of MapReduce
  • Rules and types of MapReduce jobs
  • De-bugging and testing
  • How to write foolproof MapReduce jobs

April 24, 10:00 AM Pacific, 1:00 pm Eastern

Hadoop: HBase In-Depth” will provide a deep technical review of HBase and cover:

  • Its flexibility, scalability and components
  • Schema samples
  • Hardware requirements and more

Space is limited so click here to register right away!

Hadoop Console: Simplified Hadoop for the Enterprise

We are pleased to announce the latest release in our string of Big Data announcements: the WANdisco Hadoop Console (WHC.) WHC is a plug-and-play solution that makes it easy for enterprises to deploy, monitor and manage their Hadoop implementations, without the need for expert HBase or HDFS knowledge.

This innovative Big Data solution offers enterprise users:

  • An S3-enabled HDFS option for securely migrating from Amazon’s public cloud to a private in-house cloud
  • An intuitive UI that makes it easy to install, monitor and manage Hadoop clusters
  • Full support for Amazon S3 features (metadata tagging, data object versioning, snapshots, etc.)
  • The option to implement WHC in either a virtual or physical server environment.
  • Improved server efficiency
  • Full support for HBase

“WANdisco is addressing important issues with this product including the need to simplify Hadoop implementation and management as well as public to private cloud migration,” said John Webster, senior partner at storage research firm Evaluator Group. “Enterprises that may have been on the fence about bringing their cloud applications private can now do so in a way that addresses concerns about both data security and costs.”

More information about WHC is available from the WANdisco Hadoop Console product page. Interested parties can also download our Big Data whitepapers and datasheets, or request a free trial of WHC. Professional support for our Big Data solutions is also available.

This latest Big Data announcement follows the launch of our WANdisco Distro, the world’s first production-ready version of Apache Hadoop 2.

Running the SLive Test on WANdisco Distro

The SLive test is a stress test designed to simulate distributed operations and load on the NameNode by utilizing the MapReduce paradigm. It was designed by Konstantin Shvachko and introduced into the Apache Hadoop project in 2010 by him and others. It is now one of the many stress tests we ran here at WANdisco in testing our distribution, WANdisco Distro (WDD).

You can read the original paper about how this test works here:
https://issues.apache.org/jira/secure/attachment/12448004/SLiveTest.pdf
You can view the associated Apache JIRA for the introduction of this test here:
https://issues.apache.org/jira/browse/HDFS-708

This blog will provide a short tutorial on how you can run the SLive test on your own cluster of Hadoop 2 and YARN / MapReduce. Before we begin, please make sure you are logged in as the ‘hdfs’ user:

su – hdfs

The first order of business is to become familiar with the parameters supported by the stress test.

The percentage of operation distribution parameters:
-create <num> -delete <num> -rename <num> -read <num>  -append <num> -ls <num> -mkdir <num>

Stress test property parameters:
-blockSize <min,max> -readSize <min,max> -writeSize <min,max> -files <total>

The first set of parameters controls “how many of this kind of operation do you want?”. For example, if you want to simulate just a create and delete scenario, with no reads or writes, you would run the test with -create 50 -delete 50 (or any other percentages that add up to 100) and set the others in that first set to 0, or just don’t specify them and the test will automatically set them to 0.

The second set of parameters controls properties that extend throughout the entire test. “How many files do you want to make?,” “What is the biggest and smallest that you want each block in the file to be?” They can be ignored for the most part, except for “-blockSize”. Using the default block size, which is 64 megabytes, may cause your run of the SLive test to take longer. In order to make this a speedy tutorial, we will use small block sizes. Please note that block sizes must be in multiples of 512 bytes. We will use 4096 bytes in this tutorial.

There are other parameters available, but they are not necessary in order to provide a basic understanding and run of this stress test. You can refer to the document at the top of this entry if your curiosity of the other parameters is getting the best of you, or you can run:

hadoop org.apache.hadoop.fs.slive.SliveTest –help

The second step is to understand how to run the test. Although it is advised NOT to do this just yet, you can make the following call to instantly run the test with default parameters by executing the following command:

hadoop org.apache.hadoop.fs.slive.SliveTest

However, since we have no initial data within the cluster, you should notice that most, if not all, of the operations in the report are failures. Run the following to initialize the cluster with 10,000 files, all with a tiny 4096 byte block size, in order to achieve a quick run of the SLive test:

hadoop org.apache.hadoop.fs.slive.SliveTest -create 100 -delete 0 -rename 0 -read 0 -append 0 -ls 0 -mkdir 0 -blockSize 4096,4096 -files 10000

On a cluster with 1 NameNode and 3 DataNodes, running this command should take no longer  than about 3 – 4 minutes. If it is taking too long, you can try re-running with a lower “-files” parameter number and/or a smaller “-blockSize” parameter as well.

After you have initialized the cluster with data, you will need to delete the output directory that your previous SLive test run had created:

hadoop fs -rmr /test/slive/slive/output

You will need to do this after every time you have run an SLive test; otherwise your next run attempt will fail, telling you that the output directory for your requested run already exists.

You can now run the default test, which performs an equal distribution of creates, deletes, reads, and other operations across the cluster:

hadoop org.apache.hadoop.fs.slive.SliveTest

Or you can specify the parameters of your own choosing and customize your own load to stress test with! That is the purpose of the test, after all. Enjoy!

Here are the results obtained from our own in-house run of the SLive test for you to compare your own results with. I ran the following command after initialization:

hadoop org.apache.hadoop.fs.slive.SliveTest -blockSize 4096,4096 -files 10000

And I got the following results:

13/02/11 11:00:36 INFO slive.SliveTest: Reporting on job:
13/02/11 11:00:36 INFO slive.SliveTest: Writing report using contents of /test/slive/slive/output
13/02/11 11:00:36 INFO slive.SliveTest: Report results being placed to logging output and to file /home/hdfs/part-0000
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type AppendOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “bytes_written” = 4317184
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “failures” = 1
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “files_not_found” = 365
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 59813
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 1054
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “bytes_written” = 0.067 MB/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 23.741 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 17.622 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type CreateOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “bytes_written” = 1490944
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “failures” = 1056
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 19029
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 364
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “bytes_written” = 0.053 MB/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 74.623 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 19.129 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type DeleteOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “failures” = 365
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 4905
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 1055
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 289.501 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 215.087 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type ListOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “dir_entries” = 1167
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “files_not_found” = 1145
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 536
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 275
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “dir_entries” = 2177.239 directory entries/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 2649.254 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 513.06 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type MkdirOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 5631
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 252.175 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 252.175 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type ReadOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “bad_files” = 1
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “bytes_read” = 25437917184
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “chunks_unverified” = 0
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “chunks_verified” = 3188125200
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “files_not_found” = 342
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 268754
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 1077
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “bytes_read” = 90.265 MB/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 5.284 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 4.007 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type RenameOp
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “failures” = 1165
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 1130
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 1420
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “successes” = 255
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 1256.637 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “successes” = 225.664 successes/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Basic report for operation type SliveMapper
13/02/11 11:00:36 INFO slive.ReportWriter: ————-
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “milliseconds_taken” = 765862
13/02/11 11:00:36 INFO slive.ReportWriter: Measurement “op_count” = 9940
13/02/11 11:00:36 INFO slive.ReportWriter: Rate for measurement “op_count” = 12.979 operations/sec
13/02/11 11:00:36 INFO slive.ReportWriter: ————-

avatar

About Plamen Jeliazkov

Introduction to Hadoop 2, with a simple tool for generating Hadoop 2 config files

Introduction to Hadoop 2
Core Hadoop 2 consists of the distributed filesystem HDFS and the compute framework YARN.

HDFS is a distributed filesystem that can be used to store anywhere from a few gigabytes to many petabytes of data. It is distributed in the sense that it utilizes a number of slave servers, ranging from 3 to a few thousand, to store and serve files from.

YARN is the compute framework for Hadoop 2. It manages the distribution of compute jobs to the very same slave servers that store the HDFS data. This ensures that the compute jobs do not reach out over the network to access the data stored in HDFS.

Naturally, traditional software written to run on a single server will not work on Hadoop. New software needs to be developed using a special programming paradigm called Map Reduce. Hadoop’s native Map Reduce framework uses java, however Hadoop MR programs can be written in almost any language. Also, higher level languages such as Pig may be used to write scripts that compile into Hadoop MR jobs.

Hadoop 2 Server Daemons
HDFS:  HDFS consists of a master metadata server called the NameNode, and a number of slave servers called DataNodes.

The NameNode function is provided by a single java daemon – the NameNode. The NameNode daemon runs on just one machine – the master. DataNode functionality is provided by a java daemon called DataNode that runs on each slave server.

Since the NameNode function is provided by a single java daemon, it turns out to be Single Point of Failure (SPOF). Open source Hadoop 2 has a number of ways to keep a standby server waiting to take over the function of the NameNode daemon, should the single NameNode fail. All of these standby solutions take 5 to 15 minutes to failover. While this failover is underway, batch MR jobs on the cluster will fail. Further, an active HBase cluster with a high write load may not necessarily survive the NameNode failover. Our company WANdisco has a commercial product called the NonStop NameNode that solves this SPOF problem.

Configuration parameters for the HDFS deamons NameNode and DataNode are all stored in a file called the hdfs-site.xml. At the end of this blog entry, I have included a simple java program that generates all the config files necessary for running a Hadoop 2 cluster. This convenient program generates a hdfs-site.xml.

YARN:
The YARN framework has a single master daemon called the YARN Resource Manager that runs in a master node, and a YARN Node Manager on each of the slave nodes. Additionally, YARN has a single Proxy server and a single Mapreduce job history server. As indicated by the name, the function of the mapreduce job history server is to store and serve a history of the mapreduce jobs that were run on the cluster.

The configuration for all of these daemons is stored in yarn-site.xml.

Daemons that comprise core Hadoop (HDFS and YARN)

Daemon Name Number Description Web Port (if any) RPC Port (if any)
NameNode 1 HDFS Metadata Server, usually run on the master 50070 8020
DataNode 1 per slave HDFS Data Server, one per slave server 50075 50010 (Data transfer RPC),
50020 (Block metadata RPC)
ResourceManager 1 YARN ResourceManager, usually on a master server 8088 8030 (Scheduler RPC),
8031 (Resource Tracker RPC),
8032 (Resource Manager RPC),
8033 (Admin RPC)
NodeManager 1 per slave YARN NodeManager, one per slave 8042 8040 (Localizer),
8041 (NodeManager RPC)
ProxyServer 1 YARN Proxy Server 8034
JobHistory 1 Mapreduce Job History Server 10020 19888

A Simple Program for generating Hadoop 2 Config files
This is a simple program that generates core-site.xml, hdfs-site.xml, yarn-site.xml and capacity-scheduler.xml. You need to supply this program with the following information :

  1. nnHost: HDFS Name Node Server hostname
  2. nnMetadataDir: The directory on the Name Node server’s local filesystem where the NameNode metadata will be stored – nominally /var/hadoop/name. If you are using our WDD rpms, note that all documentation will refer to this directory
  3. dnDataDir: The directory on each DataNode or slave machine where HDFS data blocks are stored. This location must have be the biggest disk on the DataNode. Nominally /var/hadoop/data. If you are using our WDD rpms, note that this is the location that all of our documentation will refer to.
  4. yarnRmHost: YARN Resource Manager hostname

Download the following jar: makeconf
Here is an example run:

$ java -classpath makeconf.jar com.wandisco.hadoop.makeconf.MakeHadoopConf nnHost=hdfs.hadoop.wandisco nnMetadataDir=/var/hadoop/name dnDataDir=/var/hadoop/data yarnRmHost=yarn.hadoop.wandisco

Download the source code for this simple program here: makeconf-source

Incidentally, if you are looking for an easy way to get Hadoop 2, try our free Hadoop distro – the WANdisco Distro (WDD). It is a packaged, tested and certified version of the latest Hadoop 2.

 

avatar

About Jagane Sundar