sync (Unix)
syncis a standardsystem callin theUnixoperating system, which commits all data from thekernelfilesystembufferstonon-volatilestorage, i.e., data which has been scheduled for writing via low-levelI/Osystem calls. Higher-level I/O layers such asstdiomay maintain separate buffers of their own.
As a function inC,thesync()call is typically declared asvoid sync(void)in<unistd.h>.The system call is also available via acommand lineutility also calledsync,and similarly named functions in other languages such asPerlandNode.js(in the fs module).
The related system callfsync()commits just the buffered data relating to a specifiedfile descriptor.[1]fdatasync()is also available to write out just the changes made to the data in the file, and not necessarily the file's related metadata.[2]
Some Unix systems run a kind offlushorupdatedaemon,which calls thesyncfunction on a regular basis. On some systems, thecrondaemon does this, and onLinuxit was handled by thepdflushdaemon which was replaced by a new implementation and finally removed from the Linux kernel in 2012.[3]Buffers are also flushed when filesystems areunmountedor remountedread-only,[4]for example prior to system shutdown.
Some applications, such asLibreOffice,also call thesyncfunction to save recovery information in an interval.
Database use
[edit]In order to provide properdurability,databases need to use some form of sync in order to make sure the information written has made it tonon-volatile storagerather than just being stored in a memory-based write cache that would be lost if power failed.PostgreSQLfor example may use a variety of different sync calls, includingfsync()andfdatasync(),[5]in order for commits to be durable.[6]Unfortunately, for any single client writing a series of records, a rotating hard drive can only commit once per rotation, which makes for at best a few hundred such commits per second.[7]Turning off the fsync requirement can therefore greatly improve commit performance, but at the expense of potentially introducing database corruption after a crash.
Databases also employtransaction logfiles (typically much smaller than the main data files) that have information about recent changes, such that changes can be reliably redone in case of crash; then the main data files can be synced less often.
Error reporting and checking
[edit]To avoid any data loss return values offsync()should be checked because when performing I/O operations that are buffered by the library or the kernel, errors may not be reported at the time of using thewrite()system call or thefflush()call, since the data may not be written to non-volatile storage but only be written to the memorypage cache.Errors from writes are instead often reported during system calls tofsync(),msync()orclose().[8]Prior to 2018, Linux'sfsync()behavior under certain circumstances failed to report error status,[9][10]change behavior was proposed on 23 April 2018.[11]
Performance controversies
[edit]Hard disks may default to using their own volatile write cache to buffer writes, which greatly improves performance while introducing a potential for lost writes.[12]Tools such ashdparm-F will instruct the HDD controller to flush the on-drive write cache buffer. The performance impact of turning caching off is so large that even the normally conservativeFreeBSDcommunity rejected disabling write caching by default in FreeBSD 4.3.[13]
InSCSIand inSATAwithNative Command Queuing(but not in plain ATA, even with TCQ) the host can specify whether it wants to be notified of completion when the data hits the disk's platters or when it hits the disk's buffer (on-board cache). Assuming a correct hardware implementation, this feature allows the disk's on-board cache to be used while guaranteeing correct semantics for system calls likefsync.[14]This hardware feature is calledForce Unit Access(FUA) and it allows consistency with less overhead than flushing the entire cache as done for ATA (or SATA non-NCQ) disks.[15]Although Linux enabled NCQ around 2007, it did not enable SATA/NCQ FUA until 2012, citing lack of support in the early drives.[16][17]
Firefox 3.0, released in 2008, introducedfsyncsystem calls that were found to degrade its performance; the call was introduced in order to guarantee the integrity of the embeddedSQLitedatabase.[18]
Linux Foundationchief technical officerTheodore Ts'oclaims there is no need to "fear fsync", and that the real cause of Firefox 3 slowdown is the excessive use offsync.[19]He also concedes however (quotingMike Shaver) that
On some rather common Linux configurations, especially using theext3filesystem in the "data=ordered" mode, calling fsync doesn't just flush out the data for the file it's called on, but rather on all the buffered data for that filesystem.[20]
See also
[edit]References
[edit]- ^fsync specification
- ^fdatasync specification
- ^"R.I.P. Pdflush [LWN.net]".
- ^"mount - Does umount calls sync to complete any pending writes".Unix & Linux Stack Exchange.Retrieved2021-05-02.
- ^Vondra, Tomas (2 February 2019)."PostgreSQL vs. fsync".Osuosl Org.Archived fromthe original(mp4)on 10 February 2019.Retrieved10 February2019.
- ^PostgreSQL Reliability and the Write-Ahead Log
- ^Tuning PostgreSQL WAL SynchronizationArchived2009-11-25 at theWayback Machine
- ^"Ensuring data reaches disk [LWN.net]".
- ^"PostgreSQL's fsync() surprise [LWN.net]".
- ^"Improved block-layer error handling [LWN.net]".
- ^"Always report a writeback error once - Patchwork".Archived fromthe originalon 2018-05-04.Retrieved2018-05-03.
- ^Write-Cache Enabled?
- ^FreeBSD Handbook — Tuning Disks
- ^Marshall Kirk McKusick."Disks from the Perspective of a File System - ACM Queue".Queue.acm.org.Retrieved2014-01-11.
- ^Gregory Smith (2010).PostgreSQL 9.0: High Performance.Packt Publishing Ltd. p. 78.ISBN978-1-84951-031-8.
- ^"Enabling FUA for SATA drives (Was Re: [RFC][PATCH] libata: Enable SATA disk fua detection on default) (Linux SCSI)".
- ^"Linux-Kernel Archive: [PATCH RFC] libata: FUA updates".
- ^"Shaver » fsyncers and curveballs".Archived fromthe originalon 2012-12-09.Retrieved2009-10-15.
- ^"Don't fear the fsync!".
- ^"Delayed allocation and the zero-length file problem".