fsync troubles...

or "ensuring data integrity in the face of IO errors is a really hard problem"

  • fsync
  • PostgreSQL
  • Databases
Written by Mark McKeown • 19 Jun 2018 • 2 min read • Last updated 2 months ago

There is an interesting discussion here "PostgreSQL's fsync() surprise(opens new window) ". The PostgreSQL developers got some nasty surprises on how Linux fsync works, which means they could lose data without any errors being reported. NetBSD and OpenBSD can also behave in the same way, but FreeBSD does not. It is a bit scary to think that even database developers have trouble dealing with safely writing data to disk.

Included in the article is a link to a post by Dave Chinner(opens new window) , the XFS maintainer who points out that the close() call can return errors from previous writes—how many people handle errors from close?

If the problem is scary, then the solution is even more scary AIO+DIO (Asynchronous IO and Direct IO)—here is an example of some AIO, http://www.xmailserver.org/eventfd-aio-test.c. It's interesting to see this AIO+DIA is currently evolving(opens new window) with new additions to the kernel like RWF_NOWAIT. Using this leads to no-portable code as one of the PostgreSQL developers points out.
There is a post on how the different OS behave(opens new window) , with some interesting phrases in the post "EIO lies" and "EIO tells the truth". In distributed systems when a component "lies" it's known as a Byzantine fault, after the famous Bzyantine Generals problem(opens new window) . A real world example of something that might be classified as  Byzantine fault(opens new window) is. In practice people do not use the term Byzantine fault for these types of events, they are just classified as bugs - perhaps talking about Byzantine faults only makes sense in the design phase.