Tag Archive for 'wdd'

WANdisco Releases New Version of Hadoop Distro

We’re proud to announce the release of WANdisco Distro (WDD) version 3.1.1.

WDD is a fully tested, production-ready version of Apache Hadoop 2 that’s free to download. WDD version 3.1.1 includes an enhanced, more intuitive user interface that simplifies Hadoop cluster deployment. WDD 3.1.1 supports SUSE Linux Enterprise Server 11 (Service Pack 2), in addition to RedHat and CentOS.

“The number of Hadoop deployments is growing quickly and the Big Data market is moving fast,” said Naji Almahmoud, senior director of global business development, SUSE, a WANdisco Non-Stop Alliance partner. “For decades, SUSE has delivered reliable Linux solutions that have been helping global organizations meet performance and scalability requirements. We’re pleased to work closely with WANdisco to support our mutual customers and bring Hadoop to the enterprise.”

All WDD components are tested and certified using the Apache BigTop framework, and we’ve worked closely with both the open source community and leading big data vendors to ensure seamless interoperability across the Hadoop ecosystem.

“The integration of Hadoop into the mainstream enterprise environment is increasing, and continual communication with our customers confirms their requirements – ease of deployment and management as well as support for market leading operating systems,” said David Richards, CEO of WANdisco. “With this release, we’re delivering on those requirements with a thoroughly tested and certified release of WDD.”

WDD 3.1.1 can be downloaded for free now. WANdisco also offers Professional Support for Apache Hadoop.

WANdisco’s March Roundup

Following the recent issuance of our “Distributed computing systems and system components thereof” patent, which cover the fundamentals of active-active replication over a Wide Area Network, we’re excited to announce the filing of three more patents. These patents involve methods, devices and systems that enhance security, reliability, flexibility and efficiency in the field of distributed computing and will have significant benefits for users of our Hadoop Big Data product line.

“Our team continues to break new ground in the field of distributed computing technology,” said David Richards, CEO for WANdisco. “We are proud to have some of the world’s most talented engineers in this field working for us and look forward to the eventual approval of these most recent patent applications. We are particularly excited about their application in our new Big Data product line.”

Our Big Data product line includes Non-Stop NameNode, WANdisco Hadoop Console and WANdisco Distro (WDD.)

This month, we also welcomed Bas Nijjer, who built CollabNet UK from startup to multimillion dollar recurring revenue, to the WANdisco team. Bas Nijjer has a proven track record of increasing customer wins, accelerating revenue and providing customer satisfaction, and he takes on the role of WANdisco Sales Director, EMEA.

“Bas is an excellent addition to our team, with great insight on developing and strengthening sales teams and customer relationships as well as enterprise software,” said David Richards. “His expertise and familiarity with EMEA and his results-oriented attitude will help strengthen the WANdisco team and increase sales and renewals. We are pleased to have him join us.”

If joining the WANdisco team interests you, visit our Careers page for all the latest employment opportunities.

We’ve also posted lots of new content at the WANdisco blog. Users of SmartSVN, our cross-platform graphical Subversion client, can find out how to get even more out of their installation with our ‘Performing a Reverse Merge in SmartSVN’ and ‘Backing Up Your SmartSVN Data’ tutorials. For users running the latest and greatest, 7.5.4 release of SmartSVN, we’ve put together a deep dive into the fixes and new functionality in this release with our ‘What’s New in SmartSVN 7.5.4?’ post. If you haven’t tried SmartSVN yet, you can claim your free trial of this release by visiting http://smartsvn.com/download

We also have a new post from James Creasy, WANdisco’s Senior Director of Product Management, where he takes a closer look at the “WAN” in “WANdisco:”

“We’ve all heard about the globalization of the world economy. Every globally relevant company is now highly dependent on highly available software, and that software needs to be equally global. However, most systems that these companies rely on were architected with a single machine in mind. These machines were accessed over a LAN (local area network) by mostly co-located teams.

All that changed, starting in the 1990’s with widespread adoption of outsourcing. The WAN computing revolution had begun in earnest.”

You can read “What’s in a name, WANdisco?” in full now.

Also at the blog we address the hot topic of ‘Is Subversion Ready for the Enterprise?’ And, if you need more information on the challenges and available solutions for deploying Subversion in an enterprise environment, be sure to sign up for our free-to-attend ‘Scaling Subversion for the Enterprise’ sessions. Taking place a few times a week, these webinars cover limitations and risks related to globally distributed SVN deployments, as well as free resources and live demos to help you overcome them. Take advantage of the opportunity to get answers to your business-specific questions and live demos of enterprise-class SVN products.

WANdisco Files Three New Patents with USPTO

We are pleased to announce the filing of three new patents with the United States Patent and Trademark Office (USPTO) related to distributed computing.

These three innovations involve methods, devices and systems that enhance security, reliability, flexibility and efficiency in the field of distributed computing. The patents are expected to have significant benefits for users of our new Hadoop Big Data product line.

Our team continues to break new ground in the field of distributed computing technology,” said David Richards, CEO for WANdisco. “We are proud to have some of the world’s most talented engineers in this field working for us and look forward to the eventual approval of these most recent patent applications. We are particularly excited about their application in our new Big Data product line.”

Our Big Data product line includes Non-Stop NameNode, which turns the NameNode into an active-active shared-nothing cluster, and the comprehensive wizard-driven management dashboard ‘WANdisco Hadoop Console.’ We also offer a free-to-download, fully-tested and production-ready version of Apache Hadoop 2. Visit the WANdisco Distro (WDD) to learn more.

This news comes after we announced the issuance of our “Distributed computing systems and system components thereof” patent, which covers the fundamentals of active-active replication over a Wide Area Network.

 

WANdisco’s February Roundup

This month, we launched a trio of innovative Hadoop products: the world’s first production-ready distro; a wizard-driven management dashboard; and the first and only 100% uptime solution for Apache Hadoop.

hadoop big data

We started this string of Big Data announcements with WANdisco Distro (WDD) a fully tested, free-to-download version of Apache Hadoop 2. WDD is based on the most recent Hadoop release, includes all the latest fixes and undergoes the same rigorous quality assurance process as our enterprise software solutions.

This release paved the way for our enterprise Hadoop solutions, and we announced the WANdisco Hadoop Console (WHC) shortly after. WHC is a plug-and-play solution that makes it easy for enterprises to deploy, monitor and manage their Hadoop implementations, without the need for expert HBase or HDFS knowledge.

The final product in this month’s Big Data announcements was WANdisco Non-Stop NameNode. Our patented technology makes WANdisco Non-Stop Namenode the first and only 100% uptime solution for Hadoop, and offers a string of benefits for enterprise users:

  • Automatic failover and recovery
  • Automatic continuous hot backup
  • Removes single point of failure
  • Eliminates downtime and data loss
  • Every NameNode server is active and supports simultaneous read and write requests
  • Full support for HBase

To support the needs of the Apache Hadoop community, we’ve also launched a dedicated Hadoop forum. At this forum, users can get advice on their Hadoop installation and connect with fellow users, including WANdisco’s core Apache Hadoop developers Dr. Konstantin V. Shvachko, Dr. Konstantin Boudnik, and Jagane Sundar.

subversion

For Apache Subversion users, we announced the next webinars in our free training series:

  • Subversion Administration – everything you need to administer a Subversion development environment
  • Introduction to SmartSVN – a short introduction to how Subversion works with the SmartSVN graphical client
  • Checkout Command – how to get the most out of the checkout command, and the meaning of the various error messages you may encounter
  • Commit Command – learn more about this command, including diff usage, working with unversioned files and changelists
  • Introduction to Git – everything a new user needs to get started with Git
  • Hook Scripts – how to use hook scripts to automate tasks such as email notifications, backups and access control
  • Advanced Hook Scripts – an advanced look at hook scripts, including using a config file with hook scripts and passing data to hook scripts

We’ve announced an ongoing series of free webinars, which demonstrate how you can overcome these challenges from an administrative, business and IT perspective, and get the most out of deploying Subversion in an enterprise environment. These ‘Scaling Subversion for the Enterprise’ webinars will be conducted by our expert Solution Architect three times a week (Tuesday, Wednesday and Thursday) at 10.00am PST/1.00pm EST, and will cover:

  • The latest technology that can help you overcome the limitations and risks associated with globally distributed deployments
  • Answers to your business-specific questions
  • How to solve critical issues
  • The free resources and offers that can help solve your business challenges

Hadoop Console: Simplified Hadoop for the Enterprise

We are pleased to announce the latest release in our string of Big Data announcements: the WANdisco Hadoop Console (WHC.) WHC is a plug-and-play solution that makes it easy for enterprises to deploy, monitor and manage their Hadoop implementations, without the need for expert HBase or HDFS knowledge.

This innovative Big Data solution offers enterprise users:

  • An S3-enabled HDFS option for securely migrating from Amazon’s public cloud to a private in-house cloud
  • An intuitive UI that makes it easy to install, monitor and manage Hadoop clusters
  • Full support for Amazon S3 features (metadata tagging, data object versioning, snapshots, etc.)
  • The option to implement WHC in either a virtual or physical server environment.
  • Improved server efficiency
  • Full support for HBase

“WANdisco is addressing important issues with this product including the need to simplify Hadoop implementation and management as well as public to private cloud migration,” said John Webster, senior partner at storage research firm Evaluator Group. “Enterprises that may have been on the fence about bringing their cloud applications private can now do so in a way that addresses concerns about both data security and costs.”

More information about WHC is available from the WANdisco Hadoop Console product page. Interested parties can also download our Big Data whitepapers and datasheets, or request a free trial of WHC. Professional support for our Big Data solutions is also available.

This latest Big Data announcement follows the launch of our WANdisco Distro, the world’s first production-ready version of Apache Hadoop 2.

WANdisco Joins Fusion-io Technology Alliance Program

WANdisco is excited to announce its partnership with Fusion-io. Following the launch of our first Big Data offering, the world’s first production-ready Apache Hadoop 2 distro, we’ve joined Fusion-io’s Technology Alliance Program. This program focuses on working with leaders in strategic market segments to deliver proven solutions, access to resources and expertise to enhance the value of technology offerings.

“With rapid growth in big data demands around the world, customers require proven solutions and expertise that deliver Hadoop availability with no downtime or data loss,” said Tyler Smith, Fusion-io’s Vice President of Alliances. “WANdisco is a valuable addition to our Technology Alliance Program as we work together to fulfill the market demand for innovative and proven big data solutions.”

As mentioned, this partnership news follows the launch of WANdisco Distro (WDD), a fully tested, production-ready version of Apache Hadoop, based on the most recent Hadoop release. WDD lays the foundation for WANdisco’s upcoming enterprise Hadoop solutions, including the WANdisco Hadoop Console, a comprehensive, wizard-driven management dashboard and the Non-Stop NameNode, which combines our patented replication technology with open source Hadoop to deliver optimum performance, scalability and availability on a 24-by-7 basis.

You can find out more about the Technology Alliance announcement by reading the press release, or visiting Fusion-io’s Technology Alliance Program webpage.

Introduction to Hadoop 2, with a simple tool for generating Hadoop 2 config files

Introduction to Hadoop 2
Core Hadoop 2 consists of the distributed filesystem HDFS and the compute framework YARN.

HDFS is a distributed filesystem that can be used to store anywhere from a few gigabytes to many petabytes of data. It is distributed in the sense that it utilizes a number of slave servers, ranging from 3 to a few thousand, to store and serve files from.

YARN is the compute framework for Hadoop 2. It manages the distribution of compute jobs to the very same slave servers that store the HDFS data. This ensures that the compute jobs do not reach out over the network to access the data stored in HDFS.

Naturally, traditional software written to run on a single server will not work on Hadoop. New software needs to be developed using a special programming paradigm called Map Reduce. Hadoop’s native Map Reduce framework uses java, however Hadoop MR programs can be written in almost any language. Also, higher level languages such as Pig may be used to write scripts that compile into Hadoop MR jobs.

Hadoop 2 Server Daemons
HDFS:  HDFS consists of a master metadata server called the NameNode, and a number of slave servers called DataNodes.

The NameNode function is provided by a single java daemon – the NameNode. The NameNode daemon runs on just one machine – the master. DataNode functionality is provided by a java daemon called DataNode that runs on each slave server.

Since the NameNode function is provided by a single java daemon, it turns out to be Single Point of Failure (SPOF). Open source Hadoop 2 has a number of ways to keep a standby server waiting to take over the function of the NameNode daemon, should the single NameNode fail. All of these standby solutions take 5 to 15 minutes to failover. While this failover is underway, batch MR jobs on the cluster will fail. Further, an active HBase cluster with a high write load may not necessarily survive the NameNode failover. Our company WANdisco has a commercial product called the NonStop NameNode that solves this SPOF problem.

Configuration parameters for the HDFS deamons NameNode and DataNode are all stored in a file called the hdfs-site.xml. At the end of this blog entry, I have included a simple java program that generates all the config files necessary for running a Hadoop 2 cluster. This convenient program generates a hdfs-site.xml.

YARN:
The YARN framework has a single master daemon called the YARN Resource Manager that runs in a master node, and a YARN Node Manager on each of the slave nodes. Additionally, YARN has a single Proxy server and a single Mapreduce job history server. As indicated by the name, the function of the mapreduce job history server is to store and serve a history of the mapreduce jobs that were run on the cluster.

The configuration for all of these daemons is stored in yarn-site.xml.

Daemons that comprise core Hadoop (HDFS and YARN)

Daemon Name Number Description Web Port (if any) RPC Port (if any)
NameNode 1 HDFS Metadata Server, usually run on the master 50070 8020
DataNode 1 per slave HDFS Data Server, one per slave server 50075 50010 (Data transfer RPC),
50020 (Block metadata RPC)
ResourceManager 1 YARN ResourceManager, usually on a master server 8088 8030 (Scheduler RPC),
8031 (Resource Tracker RPC),
8032 (Resource Manager RPC),
8033 (Admin RPC)
NodeManager 1 per slave YARN NodeManager, one per slave 8042 8040 (Localizer),
8041 (NodeManager RPC)
ProxyServer 1 YARN Proxy Server 8034
JobHistory 1 Mapreduce Job History Server 10020 19888

A Simple Program for generating Hadoop 2 Config files
This is a simple program that generates core-site.xml, hdfs-site.xml, yarn-site.xml and capacity-scheduler.xml. You need to supply this program with the following information :

  1. nnHost: HDFS Name Node Server hostname
  2. nnMetadataDir: The directory on the Name Node server’s local filesystem where the NameNode metadata will be stored – nominally /var/hadoop/name. If you are using our WDD rpms, note that all documentation will refer to this directory
  3. dnDataDir: The directory on each DataNode or slave machine where HDFS data blocks are stored. This location must have be the biggest disk on the DataNode. Nominally /var/hadoop/data. If you are using our WDD rpms, note that this is the location that all of our documentation will refer to.
  4. yarnRmHost: YARN Resource Manager hostname

Download the following jar: makeconf
Here is an example run:

$ java -classpath makeconf.jar com.wandisco.hadoop.makeconf.MakeHadoopConf nnHost=hdfs.hadoop.wandisco nnMetadataDir=/var/hadoop/name dnDataDir=/var/hadoop/data yarnRmHost=yarn.hadoop.wandisco

Download the source code for this simple program here: makeconf-source

Incidentally, if you are looking for an easy way to get Hadoop 2, try our free Hadoop distro – the WANdisco Distro (WDD). It is a packaged, tested and certified version of the latest Hadoop 2.

 

avatar

About Jagane Sundar

New Webinar Replay: The Future of Big Data for the Enterprise

You may have heard that we’ve just launched the world’s first production-ready Apache Hadoop 2 distro. This WANdisco Distro (WDD) is a fully tested, production-ready version of Apache Hadoop, based on the most recent Hadoop release. We’re particularly excited, as the release of WDD lays the foundation for our upcoming enterprise Hadoop solutions. If you want to find out more about WANdisco’s plans for big data, the replay of our ‘The Future of Big Data for the Enterprise’ webinar is now available.

This webinar is led by WANdisco’s Chief Architect of Big Data, Dr. Konstantin Shvachko, and Jagane Sundar, our Chief Technology Officer and Vice President of Engineering for Big Data. Jagane and Konstantin were part of the original Apache Hadoop team, and have unparalleled expertise in Big Data.

This 30 minute webinar replay covers:

  • The cross-industry growth of Hadoop in the enterprise.
  • The new “Active-Active Architecture” for Apache Hadoop that improves performance.
  • Solving the fundamental issues of Hadoop: usability, high availability, HDFS’s single-point of failure and disaster recovery.
  • How WANdisco’s active-active replication technology will alleviate these issues by adding high-availability, data replication and data security to Hadoop, taking a fundamentally different approach to Big Data.

You can watch the full ‘The Future of Big Data for the Enterprise’ replay, along with our other webinars, at our Webinar Replays page.

WANdisco Launches World’s First Production-Ready Apache Hadoop 2 Distro

hadoop

We’re excited to announce the launch of our WANdisco Distro (WDD) a fully tested, production-ready version of Apache Hadoop 2. WDD is based on the most recent Hadoop release, includes all the latest fixes and undergoes the same rigorous quality assurance process as our enterprise software solutions.

The team behind WDD is led by Dr. Konstantin Boudnik, who is one of the original Hadoop developers, has been an Apache Hadoop committer since 2009 and served as a Hadoop architect with Yahoo! This dedicated team of Apache Hadoop development, QA and support professionals is focused exclusively on delivering the highest quality version of the software.

We are also now offering enterprise-class professional support for organizations deploying Hadoop clusters that utilize WDD. Delivered by our team of open source experts, WANdisco’s professional support for Hadoop includes online service request and case tracking, customer discussion forums, online access to service packs and patches, indemnification coverage, Hadoop cluster health checks, consulting and training and more. You can find out more about the available support options at www.wandisco.com/support/hadoop

We’re particularly excited to make this announcement, as WDD lays the foundation for our enterprise Hadoop solutions that deliver 24-by-7 availability, scalability and performance globally, without any downtime or data loss.

“This is one of a number of key Big Data product announcements WANdisco will be making between now and the upcoming Strata 2013 Big Data conference in Santa Clara, CA, February 26-28. It’s a great time for enterprises requiring a hardened, non-stop Hadoop,” said David Richards, CEO of WANdisco. “Only our patented active-active technology removes the single point of failure inherent in Hadoop and works locally and globally. We are excited to have Dr. Konstantin Boudnik, one of the original developers of Hadoop, leading this rollout.”

You can learn more about WDD at the official press release, or by visiting the Download WANdisco Distro webpage.