Tag Archive for 'cloudera'

The inspiration for WANdisco Fusion

Screen Shot 2015-04-21 at 10.08.22 PM

Roughly two years ago, we sat down to start work on a project that finally came to fruition this week.

At that meeting, we had set ourselves the challenge of redefining the storage landscape. We wanted to map out a world where there was complete shared storage, but where the landscape remained entirely heterogeneous.

Why? Because we’d witnessed the beginnings of a trend that has only grown more pronounced with the passage of time.

From the moment we started engaging with customers, we were struck by the extreme diversity of their storage environments. Regardless of whether we were dealing with a bank, a hospital or utility provider, different types of storage had been introduced across every organization for a variety of use cases.

In time, however, these same companies wanted to start integrating their different silos of data, whether to run real-time analytics or to gain a full 360 perspective of performance. Yet preserving diversity across data center was critical, given that each storage type has its own strengths.

They didn’t care about uniformity. They cared about performance and this meant being able to have the best of both worlds. Being able to deliver this became the Holy Grail – at least in the world of data centers.

This isn’t quite The Gordian Knot but it’s certainly a very difficult, complex problem and possibly one that could only be solved with our core, patented IP DConE.

Then we had a breakthrough.

Months later and I’m proud to formally release WANdisco Fusion (WD Fusion), the only product that enables WAN-scope active-active synchronization of different storage systems into one place.

What does this mean in practice? Well it means that you can use Hadoop distributions like Hortonworks, Cloudera or Pivotal for compute, Oracle BDA for fast compute, EMC Isilon for dense storage. You could even use a complete variety of Hadoop distros and versions. Whatever your set-up, with WD Fusion you can leverage new and existing storage assets immediately.

With it, Hadoop is transformed from being something that runs within a data center into an elastic platform that runs across multiple data centers throughout the world. WD Fusion allows you to update your storage infrastructure one data center at a time, without impacting your application ability or by having to copy vast swathes of data once the update is done.

When we were developing WD Fusion we agreed upon two things. First, we couldn’t produce anything that made changes to the underlying storage system – this had to behave like a client application. Second, anything we created had to enable a complete single global name-space across an entire storage infrastructure.

With WD Fusion, we allow businesses to bring together different storage systems by leveraging our existing intellectual property – the same Paxos-powered algorithm behind Non-Stop Hadoop, Subversion Multisite and Git Multisite – without making any changes to the platform you’re using.

Another way of putting it is we’ve managed to spread our secret sauce even further.

We have some of the best computer scientists in the world working at WANdisco, but I’m confident that this is the most revolutionary project any of us have ever worked on.

I’m delighted to be unveiling WD Fusion. It’s a testament to the talent and character of our firm, the result of looking at an impossible scenario and saying: “Challenge accepted.”

avatar

About David Richards

David is CEO, President and co-founder of WANdisco and has quickly established WANdisco as one of the world’s most promising technology companies.

Since co-founding the company in Silicon Valley in 2005, David has led WANdisco on a course for rapid international expansion, opening offices in the UK, Japan and China. David spearheaded the acquisition of Altostor, which accelerated the development of WANdisco’s first products for the Big Data market. The majority of WANdisco’s core technology is now produced out of the company’s flourishing software development base in David’s hometown of Sheffield, England and in Belfast, Northern Ireland.

David has become recognised as a champion of British technology and entrepreneurship. In 2012, he led WANdisco to a hugely successful listing on London Stock Exchange (WAND:LSE), raising over £24m to drive business growth.

With over 15 years’ executive experience in the software industry, David sits on a number of advisory and executive boards of Silicon Valley start-up ventures. A passionate advocate of entrepreneurship, he has established many successful start-up companies in Enterprise Software and is recognised as an industry leader in Enterprise Application Integration and its standards.

David is a frequent commentator on a range of business and technology issues, appearing regularly on Bloomberg and CNBC. Profiles of David have appeared in a range of leading publications including the Financial Times, The Daily Telegraph and the Daily Mail.

Specialties:IPO’s, Startups, Entrepreneurship, CEO, Visionary, Investor, ceo, board member, advisor, venture capital, offshore development, financing, M&A

A View From Strata NY: Big Data is Getting Bigger

In general a trade show is a dangerous place to gauge sentiment.  Full of marketing & sales, backslapping & handshakes and marketecture rather than architecture the world is indeed viewed through rose-tinted-spectacles. Strata, the Hadoop Big Data conference in New York last week was very interesting albeit through my rose-tinted-spectacles.

Firstly, the sheer volume of people, over 3,500 is telling.  This show used to be a few hundred, primarily techies inventing the future.  The show is now bigger, much bigger.  A cursory glance at the exhibit hall revealed a mix of the biggest tech companies and hot start-ups.  The keynotes, to the disappointment of those original techies, were primarily press-driven product releases lacking real technical substance.  This is not such a bad thing though. It’s a sign that Hadoop is coming of age. It’s what happens when technology moves into the main stream.

Second, the agenda has changed quite dramatically.  Companies looking to deploy Hadoop are no longer trying to figure out how it might fit into their data centers. They are trying to figure out how to deploy it.  2014 will indeed be the end of trials and the beginning of full-scale enterprise roll-out.  The use-cases are all over the place.  Analysts yearn for clues and clusters to explain this “Are you seeing mainly telco’s or financial services?”  Analysts of course must try to enumerate in order to explain but the wave and shift is seismic and the only explanation is a fundamental shift in the very nature of enterprise applications.

My third theme is the discussion around why Hadoop is driving this move to rewrite enterprise applications.  As someone at the show told me, “the average age of enterprise application is 19 years”.  Hence,this is part of a classic business cycle.  Hadoop is a major technological shift that takes advantage of dramatic changes in the capabilities and economics of hardware.  Expensive spinning hard-disk, processing speeds, bandwidth, networks, etc. were limitations and hence assumptions that the last generation of enterprise applications had to deal with.  Commodity hardware and massive in-memory processing are the new assumptions that Hadoop takes advantage of.  In a few years we will not be talking about ‘Big Data’ we will simply use the term ‘Data’ because it will no longer be unusual for it to be so large in relative terms.

My fourth observation was that Hadoop 2 has changed the agenda for the type of use case.  In very rough terms Hadoop 1 was primarily about wall ststorage and batch processing.  Hadoop 2 is about yarn and run-time applications. In other words processing can now take place on top of Hadoop rather than storing in Hadoop but processing somewhere else.  This change is highly disruptive because it means that software vendors cannot rely on customers to use their products in conjunction with Hadoop.  Rather, they are talking about building on top of Hadoop.  To them Hadoop is a new type of operating system.  This disruption is very good news for the new brand of companies that are building pure applications built from the ground up and really bad news for those who believe that they can mildly integrate or even store data in 2 places. That’s not going to happen. Some of the traditional companies had a token presence at Strata that suggests they are still unsure of exactly what they are going to do – they are neither fully embracing or ignoring this new trend.

My final observation is about confusion.  There’s a lot of money at stake here so naturally everyone wants a piece of the action.  There’s a lot of flashing lights and noise from vendors, lavish claims and a lack of substance.  Forking core open source is nearly always a disaster. As open-source guru Karl Fogel says ‘forks happen due to irreconcilable disagreements, technical disagreements or interpersonal conflicts and is something developers should be afraid of and try to avoid it in any way’.  It creates natural barriers to use tertiary products and with an open source project moving as quickly as this, one has to stay super-close to the de facto open source project.

A forked version of core Hadoop is not Hadoop, it’s something else.  If customers go down a forked path it’s difficult to get back and they will lose competitive edge because they will be unable to use the community of products being built as part of the wider community.  Customers should think of Hadoop like an operating system or database.  If it’s merely embedded and heavily modified then this is not Hadoop.

So 2014 it is then.  As the Wall St Journal put it the Elephant in the Room to Weigh on Growth for Oracle, Teradata

Here’s a great video demo of the new @WANdisco continuous availability technology running on Hortonworks Hadoop 2.2 Distro

 

avatar

About David Richards

David is CEO, President and co-founder of WANdisco and has quickly established WANdisco as one of the world’s most promising technology companies.

Since co-founding the company in Silicon Valley in 2005, David has led WANdisco on a course for rapid international expansion, opening offices in the UK, Japan and China. David spearheaded the acquisition of Altostor, which accelerated the development of WANdisco’s first products for the Big Data market. The majority of WANdisco’s core technology is now produced out of the company’s flourishing software development base in David’s hometown of Sheffield, England and in Belfast, Northern Ireland.

David has become recognised as a champion of British technology and entrepreneurship. In 2012, he led WANdisco to a hugely successful listing on London Stock Exchange (WAND:LSE), raising over £24m to drive business growth.

With over 15 years’ executive experience in the software industry, David sits on a number of advisory and executive boards of Silicon Valley start-up ventures. A passionate advocate of entrepreneurship, he has established many successful start-up companies in Enterprise Software and is recognised as an industry leader in Enterprise Application Integration and its standards.

David is a frequent commentator on a range of business and technology issues, appearing regularly on Bloomberg and CNBC. Profiles of David have appeared in a range of leading publications including the Financial Times, The Daily Telegraph and the Daily Mail.

Specialties:IPO’s, Startups, Entrepreneurship, CEO, Visionary, Investor, ceo, board member, advisor, venture capital, offshore development, financing, M&A

WANdisco Teams up with Cloudera

We’re pleased to announce that WANdisco is now an authorized member of the Cloudera Connect Partner Program. This program focuses on accelerating the innovative use of Apache Hadoop for a range of business applications.

“We are pleased to welcome WANdisco into the Cloudera Connect network of valued service and solution providers for Apache Hadoop and look forward to working together to bring the power of Big Data to more enterprises,” said Tim Stevens, Vice President of Business and Corporate Development at Cloudera. “As a trusted partner, we will equip WANdisco with the tools and resources necessary to support, manage and innovate with Apache Hadoop-based solutions.”

As a member of Cloudera Connect, we are proud to add Cloudera’s extensive tools, use case insight and resources to the expertise of our core Hadoop committers.

You can learn more about this program at Cloudera’s website and by reading the official announcement in full.

At WANdisco, we’re working on our Hadoop-based products, including WANdisco Non-Stop NameNode, which will enable each NameNode server to support simultaneous read and write requests, alongside balancing workload across servers for optimum scalability and performance.

You can learn more about Non-Stop NameNode, and our other upcoming Hadoop-based offerings at our Hadoop Big Data Products page.

WANdisco’s January Roundup

Happy new year from WANdisco!

This month we have plenty of news related to our move into the exciting world of Apache Hadoop. Not only did another veteran Hadoop developer join our ever-expanding team of experts, but we announced a partnership with Cloudera, and WANdisco CEO David Richards and Vice President of Big Data Jagane Sundar met with Wikibon’s lead analyst for an in-depth discussion on active-active big data deployments.

WANdisco big data

You may have heard that AltoStor founders and core Apache Hadoop creators, Dr. Konstantin Shvachko and Jagane Sundar joined WANdisco last year. Now we’re excited to announce that another veteran Hadoop developer has joined our Big Data team. Dr Konstantin Boudnik is the founder of Apache BigTop and was a member of the original Hadoop development team. Dr. Boudnik will act as WANdisco’s Director of Big Data Distribution, leading WANdisco’s Big Data team in the rollout of certified Hadoop binaries and graphical user interface. Dr. Boudnik will ensure quality control and stability of the Hadoop open source code.

In building our Big Data team, we’ve been seeking Hadoop visionaries and authorities who demonstrate leadership and originality,” said David Richards, CEO of WANdisco. “Konstantin Boudnik clearly fits that description, and we’re honored that he’s chosen to join our team. He brings great professionalism and distribution expertise to WANdisco.”

Also on the Big Data-front, CEO David Richards, and Vice President of Big Data Jagane Sundar, spoke to Wikibon’s lead analyst about our upcoming solution for active-active big data deployments.

We can take our secret sauce, which is this patented active-active replication algorithm, and apply it to Hadoop to make it bullet-proof for enterprise deployments,” said David Richards. “We have something coming out called the Non-Stop NameNode … that will ensure that Hadoop stays up 100% of the time, guaranteed.”

Watch the ‘WANdisco Hardening Hadoop for the Enterprise’ video in full, or read Wikibon’s Lead Big Data Analyst Jeff Kelly’s post about the upcoming Non-Stop NameNode.

Capping off our Big Data announcements, WANdisco is now an authorized member of the Cloudera Connect Partner Program. This program focuses on accelerating the innovative use of Apache Hadoop for a range of business applications.

We are pleased to welcome WANdisco into the Cloudera Connect network of valued service and solution providers for Apache Hadoop and look forward to working together to bring the power of Big Data to more enterprises,” said Tim Stevens, Vice President of Business and Corporate Development at Cloudera. “As a trusted partner, we will equip WANdisco with the tools and resources necessary to support, manage and innovate with Apache Hadoop-based solutions.”

As a member of Cloudera Connect, we are proud to add Cloudera’s extensive tools, use case insight and resources to the expertise of our core Hadoop committers.

You can learn more about this program at Cloudera’s website and by reading the official announcement in full.

apache subversion logo

On the Subversion side of things, the SVN community announced their first release of 2013, with an update to the Subversion 1.6 series.

Apache Subversion 1.6.20 includes some useful fixes for 1.6.x users:

  • Vary: header added to GET responses
  • Fix fs_fs to cleanup after failed rep transmission.
  • A fix for an assert with SVNAutoVersioning in mod_dav_svn

Full details on Apache Subversion 1.6.20 can be found in the Changes file. As always, the latest, certified binaries can be downloaded for free from our website, along with the latest release of the Subversion 1.7 series.

How many developers can a single Apache Subversion server support? In his recent blog post, James Creasy discussed how DConE replication technology can support Subversion deployments of 20,000 or more developers.

“While impressive, DConE is not magic,” writes James. “What DConE delivers is a completely fault tolerant, mathematically ideal coordination engine for performing WAN connected replication.”

In another new DConeE post, James explains where DConE fits into the ‘software engineering vs. computer science’ debate, and warns “in the world of distributed computing, you’d better come armed with deep knowledge of the science.”

Finally, WANdisco China, a Wholly Foreign Owned Enterprise was announced this month, following WANdisco’s first deal in China with major telecommunications equipment company Huawei. From this new office we’ll be providing sales, training, consulting and 24/7 customer support for WANdisco software solutions sold in China, and are excited to be expanding our activities within this region.

We view China as an emerging and high growth market for WANdisco,” said David Richards. “It was a natural progression to establish our Chengdu office as a WFOE and ramp up staff there as so many companies have operations in the country. We are excited about this announcement and look forward to the growth opportunities this brings.”

To keep up with all the latest WANdisco news, be sure to follow us on Twitter.