Revisiting END-TO-END

Designing a system to support reliable file transfer from one machine to another.

  • File Transfer
  • HTTP
  • TCP
Written by Mark McKeown • 10 Sep 2019 • 5 min read • Last updated 2 months ago

"END-TO-END Arguments in System Design(opens new window) " was presented in 1981 by Saltzer, Reed and Clark [1]. Since then the thoughts around the principle have expanded but for this piece we will limit the discussion to principle as defined is in the paper.

The abstract of the paper is as follows:

This paper presents a design principle that helps guide placement of functions among the modules of a distributed computer system. The principle, called the end-to-end argument, suggests that functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level. Examples discussed in the paper include bit error recovery, security using encryption, duplicate message suppression, recovery from system crashes, and delivery acknowledgement. Low level mechanisms to support these functions are justified only as performance enhancements.

# Reliable File Transfer

The paper uses the case of reliable file transfer to make its point. Given the problem of designing a system to support reliable file transfer from one machine to another, what solution would we come up with? At the time of the paper UDP, TCP and HTTP had not been defined but we will use them here to illustrate options for file transfer and update the example provided in the original paper.

Safety constraints would require the content of the file is the same on both machines after the process is complete and nobody sees a partial version of the file on the remote node, liveness constraints would say that given some period of node availability and good network behavior the file would appear on the remote node.

Our solution might decide that HTTP 1.1 would fit the task, HTTP is an application-level protocol and sits at layer 7 of the OSI model, this is important when thinking of the end-to-end principle.

We use an HTTP PUT as it is defined to be idempotent and so we know it is safe to retry, we include an MD5 header with the file checksum in the request headers, we send headers and file and wait for a "200 OK" response from the server. If there is any issue we repeat the operation until we get the "200 OK" from the server.

Conventionally HTTP is used with TCP. The first question is do we need the HTTP response from the server? Could we use the final TCP ACK from the server to determine if the file has been sent? The remote node's OS sends the TCP ACK, this does not mean the OS has passed the data to the application and the system could crash before it does. Also, the application will not have checked the MD5 sum at this stage, or there might be a failure to write the data to disk, etc.

Early versions of HTTP did not require a content-length header when returning data an HTTP server could close the connection to indicate that all the data had been sent. This was hard to deal with for a client when it sees the connection close it can not determine if it has received all of the data or whether there was a network failure and it only received a partial response.

Is the MD5 sum required? Amazon lost the S3 service for 8 hours when a bit got flipped in a message ( new window) ). TCP only provides weak checksums at the packet level, if you want to be reliable then you need to include checksums at the application level.

HTTP requires a connection-oriented network protocol such as TCP. However, let's use UDP instead, the original end-to-end paper describes solving the file transfer problem using a protocol with similar properties. UDP does not guarantee packet delivery, the order that packets arrive or even that there will not be duplicate packets. If we used UDP we could send a stream of UDP packets and wait for a response from the server in a UDP packet. If any UDP packet is lost, arrives out of order or is duplicated the MD5 sum will fail and the server will send back some form of NACK and we will retry sending the file.

TCP provides low-level support for error handling (lost packets, packet reordering, packet duplication) but it is not actually required, however, they do provide a good performance benefit which justifies adding them, per the last sentence in the abstract. Using the simple UDP implementation described above we would have to re-transfer the whole file for a single packet drop which is sub-optimal.

There are systems that use UDP for file transfer and promise better performance than TCP. They necessarily have to provide similar features to those included in TCP (retransmits, data ordering, etc.).

TCP is often used as the canonical reference for an end-to-end design. TCP sits on top of the IP protocol but it does not rely on IP to provide any error handling functions. TCP also does not rely on any devices between the two endpoints for providing any functionality other than the simple packet forwarding as provided by routers at the IP level. For example, if a packet is lost TCP does not rely on a router to try and resend. It treats the network as dumb and all the intelligence is in the TCP stack at the endpoints (dealing with packet loss, deduplication, re-ordering etc).

In the 1970s, around the time this paper was written, there was a lot of different approaches to building networks such that intermediate nodes on the communication path would handle packet loss, re-ordering, duplication, etc. The majority winner, TCP, is a very nice example of following the "end-to-end argument".