Tag Archive for 'svnrdump'
The Apache Software Foundation have announced the general availability of 1.7; but how exactly will the new features in 1.7 affect the user? And why should you upgrade to version 1.7? Here are five good reasons, to make the leap to Subversion 1.7 today:
1) Completely Rewritten Working Copy Metadata System
In Subversion 1.7, the working copy metadata system has been completely overhauled. For the user, this means:
- An immediate performance boost. All the metadata for the whole working copy is now stored in a single datastore in the root of the working copy, so Subversion operations no longer have to walk the entire directory tree to gather all the necessary information about a working copy.
- Many small fixes (e.g WC-NG uses a generated index key to avoid case-sensitive file name issues) that together add up to a much more robust metadata storage system.
- Get ready for the future! The original WC library had grown so complex, that introducing new features was becoming a real effort. WC-NG paves the way for new features, beyond 1.7.
2) Improved HTTP Protocol
For 1.7, the Subversion team decided to drop DeltaV in favor of a new HTTP protocol variant, ‘HTTPv2.’ This has a number of benefits for the Subversion user:
- Removes the overheads associated with DeltaV (e.g additional port find requests in the Apache log file of a Subversion server.)
- Fewer client-server round trips per request means enhanced performance between Subversion clients and the server.
- Reduced server load (due to fewer requests to log and fewer accesses to the repositories.)
3) Get Control Over Your Cache
Version 1.6 introduced an in-memory caching system, but many users experienced problems with an increasing amount of memory being allocated to the cache. Subversion 1.7 gives you greater control over your cache:
- Functionality for controlling how much memory is used by the cache, to prevent the amount of memory allocated to the caching growing out of control.
- New caching structure and new code that has a better behaviour, regarding memory usage.
4) Configurable Data Compression Rate
Users with a high enough bandwidth can end up spending more time on the compression and decompression of data, than they gain by sending compressed data. Subversion 1.7 allows the user to configure the data compression rate; by sending uncompressed data on a high bandwidth, the server has a lower load and can respond faster.
5) New ‘svnrdump’ Remote Dumpfile Tool
Subversion 1.7 gives users the ability to replicate the functionality of svnadmin dump and svnadmin load on remote repositories. No administrator access to the source or target repository required!
In the first of our posts on Subversion 1.7, we looked at some of the major enhancements in version 1.7: a complete rewrite of the working copy metadata system, merge-tracking enhancements, and a new way of storing text-bases. In this post, we will look at even more exciting new functionality included as part of 1.7:
- HTTPv2 – a new HTTP protocol variant designed to enhance performance between Subversion clients and the server.
- A new in-memory caching system for FSFS repository backends
- Network compression – a protocol for avoiding CPU bottlenecks on the compression side.
- svnrdump – a new client tool that provides the same functionality as svnadmin dump and svnadmin load, but on remote repositories.
For version 1.7, the Apache Subversion team made the decision to drop DeltaV and implement a new HTTP protocol variant, ‘HTTPv2.’ This is the protocol the Subversion client uses to communicate with the Apache server – it is distinct from the protocol used to talk to svn serve. Less well-known than HTTP and DAV, the DeltaV protocol extended the WebDAV protocol with resource revision tracking capabilities, provided by the methods checkin, checkout, uncheckout, version-control, and report. Back in the days of Subversion 1.0, the Apache Subversion team made the decision to base Subversion on Apache + WebDAV/DeltaV, and to make Subversion a DeltaV sever, so that other DeltaV clients could interact with it. This approach offered several advantages, including the ability to pass through corporate firewalls and caching with intermediate proxies.
However, DeltaV never caught on, and today Subversion is one of the few significant DeltaV clients. Consequently, none of the benefits of DeltaV ever materialized for Subversion users, but they still had all the overheads associated with DeltaV. Traditionally, a lot of the port find requests in the Apache log file of a Subversion server have been associated with the DeltaV protocol. Dropping the DeltaV protocol will remove these extra port find requests, along with a few other requests, and make the protocol more efficient. Essentially, this protocol has shifted from HTTP “DeltaV on top of DAV on top of HTTP,” to HTTPv2 “DAV on top of HTTP,” so all of the DAV stuff will still work in 1.7.
The new HTTP protocol variant enhances performance between Subversion clients and the server, as it requires fewer client-server round trips per request. The sever itself has fewer requests to log and fewer accesses to the repositories, which reduces the server load. Users will notice the biggest difference when the client has to send requests in a series, where each request would require a round trip to the client and back. Particularly on a slow link with a high latency, these delays add up, resulting in a significant delay just from supporting the DeltaV protocol.
Such a fundamental change will always raise questions regarding backwards compatibility. Subversion has always made the effort to maintain backwards compatibility, and 1.7 is no exception. Subversion 1.7 features built-in support for both the HTTP protocol, and the HTTPv2 protocol, in both the client and the server. This gives the Subversion community a number of options:
- If you upgrade your server to 1.7 and don’t upgrade your clients, the old clients will continue to communicate using the HTTP protocol.
- If you upgrade just your client to 1.7, it will continue to use the HTTP protocol to talk to old severs, and use the HTTPv2 protocol to talk to the new servers.
- If you upgrade both the client and the server, you automatically get the HTTPv2 protocol.
- There is also a switch to stop advertising the HTTPv2 protocol, even if the server has been upgraded to 1.7, for instances where someone with a 1.7 server wishes to stop people from using the new protocol.
Subversion’s Serf-based repository access library has received all of the 1.7 protocol changes, and will become the default library for HTTP in Subversion 1.8.
Previous releases of Subversion have supported various caching mechanisms, but version 1.7 introduces a new in-memory caching system for FSFS repository backends. Version 1.6 introduced an in-memory caching system, but many users experienced problems with the amount of memory allocated to the cache increasing. Even worse, it was the memory associated with the cache infrastructure that increased, and not the cache objects, which meant the user experienced very little benefit from the increasing memory usage. Version 1.7 introduces a new caching structure and new code that has a much better behavior when it’s using memory, which solves the problem with cache growth. These changes are accompanied by Subversion 1.7 caching a slightly wider variety of Subversion objects and functionality for allowing users to control just how much memory is used by the cache. In particular, when dumping a repository, and consequently accessing the FSFS backend of the repository, the new caching mechanism will significantly speed up dumps and loads.
However, caching is not a perfect solution: as you increase the size of a cache, the rate at which performance increases, typically decreases, until increasing the size of the cache makes very little difference. This is usually because the CPU load created by the accompanying data compression has become a bottleneck, as compression takes a significant amount of CPU. In some cases, you can actually spend more time on the compression and decompression of data, than you gain by sending compressed data. Subversion 1.7 introduces a switch for controlling how much compression goes on. Users with a high enough bandwidth, can set Subversion to send uncompressed data, so the server has a lower load and can respond faster.
Subversion users will notice a new client tool in 1.7, ‘svnrdrump.’ Like svnadmin, svnrdump allows users to dump and load repositories, but whereas svnadmin must be running on the server with direct access to the repository, svnrdump can dump on the remote repository across a network, or load into a remote repository across a network using standard Subversion protocols. svnrdump does not require administrator access to the source or target repository, which is achieved by leveraging the replay API, first added for the svnsync one way replication system.
The new svnrdump tool can raise some security concerns, namely: if I let people dump my repository, am I letting them have access to data they shouldn’t have? svnrdump is a standard Subversion client, therefore it uses the standard Subversion protocols and is subjected to the same authentication procedures as any other Subversion client. As it doesn’t have any special privileges, it cannot bypass any of the protection on a repository and give users access to data they could not already access, for example by performing a checkout or an update.
In addition to these major changes, Subversion 1.7 introduces plenty of user-facing updates designed to make the developer’s life easier. These include:
- A new ‘svn patch’ subcommand that can apply patch files in unidiff format to a working copy.
- The ability to detect MIME types of binary files that are added to version control, by compiling with support for the libmagic library.
- Subversion on Windows now fully supports changing the case of file and directory names.
- Optimizations in the svn diff algorithm. Users should notice a performance boost when working with large files with many identical lines at the beginning and/or the end, and files that have many lines unique to one side of the comparison.
A complete list of what’s new is available at: http://subversion.apache.org/docs/release-notes/1.7.html
Despite all the excitement surrounding Subversion 1.7, we know that any migration can be a daunting prospect – to help users on the path to 1.7, we are currently shipping both Subversion 1.6 and 1.7 as part of our popular open ALM platform, uberSVN. Users have the option of switching between the two releases, through a unique ‘toggle’ feature in uberSVN’s configuration settings. And, if you need that extra support, WANdisco are offering three migration packs as part of our professional support services, designed for organisations moving from Subversion 1.6 to 1.7, and for those looking to make the leap from alternate solutions (ClearCase, PVCS, StarTeam, MKS, CVS) to Subversion 1.7.
Finally, like any successful open source project, Subversion relies on active participation and feedback from its user community. Contribute to the success of this major new release by providing your feedback at: http://www.svnforum.org/forums/56-Apache-Subversion-1.7-Support