| ========================== |
| General Filesystem Caching |
| ========================== |
| |
| ======== |
| OVERVIEW |
| ======== |
| |
| This facility is a general purpose cache for network filesystems, though it |
| could be used for caching other things such as ISO9660 filesystems too. |
| |
| FS-Cache mediates between cache backends (such as CacheFS) and network |
| filesystems: |
| |
| +---------+ |
| | | +--------------+ |
| | NFS |--+ | | |
| | | | +-->| CacheFS | |
| +---------+ | +----------+ | | /dev/hda5 | |
| | | | | +--------------+ |
| +---------+ +-->| | | |
| | | | |--+ |
| | AFS |----->| FS-Cache | |
| | | | |--+ |
| +---------+ +-->| | | |
| | | | | +--------------+ |
| +---------+ | +----------+ | | | |
| | | | +-->| CacheFiles | |
| | ISOFS |--+ | /var/cache | |
| | | +--------------+ |
| +---------+ |
| |
| Or to look at it another way, FS-Cache is a module that provides a caching |
| facility to a network filesystem such that the cache is transparent to the |
| user: |
| |
| +---------+ |
| | | |
| | Server | |
| | | |
| +---------+ |
| | NETWORK |
| ~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| | |
| | +----------+ |
| V | | |
| +---------+ | | |
| | | | | |
| | NFS |----->| FS-Cache | |
| | | | |--+ |
| +---------+ | | | +--------------+ +--------------+ |
| | | | | | | | | |
| V +----------+ +-->| CacheFiles |-->| Ext3 | |
| +---------+ | /var/cache | | /dev/sda6 | |
| | | +--------------+ +--------------+ |
| | VFS | ^ ^ |
| | | | | |
| +---------+ +--------------+ | |
| | KERNEL SPACE | | |
| ~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~|~~~~ |
| | USER SPACE | | |
| V | | |
| +---------+ +--------------+ |
| | | | | |
| | Process | | cachefilesd | |
| | | | | |
| +---------+ +--------------+ |
| |
| |
| FS-Cache does not follow the idea of completely loading every netfs file |
| opened in its entirety into a cache before permitting it to be accessed and |
| then serving the pages out of that cache rather than the netfs inode because: |
| |
| (1) It must be practical to operate without a cache. |
| |
| (2) The size of any accessible file must not be limited to the size of the |
| cache. |
| |
| (3) The combined size of all opened files (this includes mapped libraries) |
| must not be limited to the size of the cache. |
| |
| (4) The user should not be forced to download an entire file just to do a |
| one-off access of a small portion of it (such as might be done with the |
| "file" program). |
| |
| It instead serves the cache out in PAGE_SIZE chunks as and when requested by |
| the netfs('s) using it. |
| |
| |
| FS-Cache provides the following facilities: |
| |
| (1) More than one cache can be used at once. Caches can be selected |
| explicitly by use of tags. |
| |
| (2) Caches can be added / removed at any time. |
| |
| (3) The netfs is provided with an interface that allows either party to |
| withdraw caching facilities from a file (required for (2)). |
| |
| (4) The interface to the netfs returns as few errors as possible, preferring |
| rather to let the netfs remain oblivious. |
| |
| (5) Cookies are used to represent indices, files and other objects to the |
| netfs. The simplest cookie is just a NULL pointer - indicating nothing |
| cached there. |
| |
| (6) The netfs is allowed to propose - dynamically - any index hierarchy it |
| desires, though it must be aware that the index search function is |
| recursive, stack space is limited, and indices can only be children of |
| indices. |
| |
| (7) Data I/O is done direct to and from the netfs's pages. The netfs |
| indicates that page A is at index B of the data-file represented by cookie |
| C, and that it should be read or written. The cache backend may or may |
| not start I/O on that page, but if it does, a netfs callback will be |
| invoked to indicate completion. The I/O may be either synchronous or |
| asynchronous. |
| |
| (8) Cookies can be "retired" upon release. At this point FS-Cache will mark |
| them as obsolete and the index hierarchy rooted at that point will get |
| recycled. |
| |
| (9) The netfs provides a "match" function for index searches. In addition to |
| saying whether a match was made or not, this can also specify that an |
| entry should be updated or deleted. |
| |
| (10) As much as possible is done asynchronously. |
| |
| |
| FS-Cache maintains a virtual indexing tree in which all indices, files, objects |
| and pages are kept. Bits of this tree may actually reside in one or more |
| caches. |
| |
| FSDEF |
| | |
| +------------------------------------+ |
| | | |
| NFS AFS |
| | | |
| +--------------------------+ +-----------+ |
| | | | | |
| homedir mirror afs.org redhat.com |
| | | | |
| +------------+ +---------------+ +----------+ |
| | | | | | | |
| 00001 00002 00007 00125 vol00001 vol00002 |
| | | | | | |
| +---+---+ +-----+ +---+ +------+------+ +-----+----+ |
| | | | | | | | | | | | | | |
| PG0 PG1 PG2 PG0 XATTR PG0 PG1 DIRENT DIRENT DIRENT R/W R/O Bak |
| | | |
| PG0 +-------+ |
| | | |
| 00001 00003 |
| | |
| +---+---+ |
| | | | |
| PG0 PG1 PG2 |
| |
| In the example above, you can see two netfs's being backed: NFS and AFS. These |
| have different index hierarchies: |
| |
| (*) The NFS primary index contains per-server indices. Each server index is |
| indexed by NFS file handles to get data file objects. Each data file |
| objects can have an array of pages, but may also have further child |
| objects, such as extended attributes and directory entries. Extended |
| attribute objects themselves have page-array contents. |
| |
| (*) The AFS primary index contains per-cell indices. Each cell index contains |
| per-logical-volume indices. Each of volume index contains up to three |
| indices for the read-write, read-only and backup mirrors of those volumes. |
| Each of these contains vnode data file objects, each of which contains an |
| array of pages. |
| |
| The very top index is the FS-Cache master index in which individual netfs's |
| have entries. |
| |
| Any index object may reside in more than one cache, provided it only has index |
| children. Any index with non-index object children will be assumed to only |
| reside in one cache. |
| |
| |
| The netfs API to FS-Cache can be found in: |
| |
| Documentation/filesystems/caching/netfs-api.txt |
| |
| The cache backend API to FS-Cache can be found in: |
| |
| Documentation/filesystems/caching/backend-api.txt |
| |
| A description of the internal representations and object state machine can be |
| found in: |
| |
| Documentation/filesystems/caching/object.txt |
| |
| |
| ======================= |
| STATISTICAL INFORMATION |
| ======================= |
| |
| If FS-Cache is compiled with the following options enabled: |
| |
| CONFIG_FSCACHE_STATS=y |
| CONFIG_FSCACHE_HISTOGRAM=y |
| |
| then it will gather certain statistics and display them through a number of |
| proc files. |
| |
| (*) /proc/fs/fscache/stats |
| |
| This shows counts of a number of events that can happen in FS-Cache: |
| |
| CLASS EVENT MEANING |
| ======= ======= ======================================================= |
| Cookies idx=N Number of index cookies allocated |
| dat=N Number of data storage cookies allocated |
| spc=N Number of special cookies allocated |
| Objects alc=N Number of objects allocated |
| nal=N Number of object allocation failures |
| avl=N Number of objects that reached the available state |
| ded=N Number of objects that reached the dead state |
| ChkAux non=N Number of objects that didn't have a coherency check |
| ok=N Number of objects that passed a coherency check |
| upd=N Number of objects that needed a coherency data update |
| obs=N Number of objects that were declared obsolete |
| Pages mrk=N Number of pages marked as being cached |
| unc=N Number of uncache page requests seen |
| Acquire n=N Number of acquire cookie requests seen |
| nul=N Number of acq reqs given a NULL parent |
| noc=N Number of acq reqs rejected due to no cache available |
| ok=N Number of acq reqs succeeded |
| nbf=N Number of acq reqs rejected due to error |
| oom=N Number of acq reqs failed on ENOMEM |
| Lookups n=N Number of lookup calls made on cache backends |
| neg=N Number of negative lookups made |
| pos=N Number of positive lookups made |
| crt=N Number of objects created by lookup |
| tmo=N Number of lookups timed out and requeued |
| Updates n=N Number of update cookie requests seen |
| nul=N Number of upd reqs given a NULL parent |
| run=N Number of upd reqs granted CPU time |
| Relinqs n=N Number of relinquish cookie requests seen |
| nul=N Number of rlq reqs given a NULL parent |
| wcr=N Number of rlq reqs waited on completion of creation |
| AttrChg n=N Number of attribute changed requests seen |
| ok=N Number of attr changed requests queued |
| nbf=N Number of attr changed rejected -ENOBUFS |
| oom=N Number of attr changed failed -ENOMEM |
| run=N Number of attr changed ops given CPU time |
| Allocs n=N Number of allocation requests seen |
| ok=N Number of successful alloc reqs |
| wt=N Number of alloc reqs that waited on lookup completion |
| nbf=N Number of alloc reqs rejected -ENOBUFS |
| int=N Number of alloc reqs aborted -ERESTARTSYS |
| ops=N Number of alloc reqs submitted |
| owt=N Number of alloc reqs waited for CPU time |
| abt=N Number of alloc reqs aborted due to object death |
| Retrvls n=N Number of retrieval (read) requests seen |
| ok=N Number of successful retr reqs |
| wt=N Number of retr reqs that waited on lookup completion |
| nod=N Number of retr reqs returned -ENODATA |
| nbf=N Number of retr reqs rejected -ENOBUFS |
| int=N Number of retr reqs aborted -ERESTARTSYS |
| oom=N Number of retr reqs failed -ENOMEM |
| ops=N Number of retr reqs submitted |
| owt=N Number of retr reqs waited for CPU time |
| abt=N Number of retr reqs aborted due to object death |
| Stores n=N Number of storage (write) requests seen |
| ok=N Number of successful store reqs |
| agn=N Number of store reqs on a page already pending storage |
| nbf=N Number of store reqs rejected -ENOBUFS |
| oom=N Number of store reqs failed -ENOMEM |
| ops=N Number of store reqs submitted |
| run=N Number of store reqs granted CPU time |
| pgs=N Number of pages given store req processing time |
| rxd=N Number of store reqs deleted from tracking tree |
| olm=N Number of store reqs over store limit |
| VmScan nos=N Number of release reqs against pages with no pending store |
| gon=N Number of release reqs against pages stored by time lock granted |
| bsy=N Number of release reqs ignored due to in-progress store |
| can=N Number of page stores cancelled due to release req |
| Ops pend=N Number of times async ops added to pending queues |
| run=N Number of times async ops given CPU time |
| enq=N Number of times async ops queued for processing |
| can=N Number of async ops cancelled |
| rej=N Number of async ops rejected due to object lookup/create failure |
| ini=N Number of async ops initialised |
| dfr=N Number of async ops queued for deferred release |
| rel=N Number of async ops released (should equal ini=N when idle) |
| gc=N Number of deferred-release async ops garbage collected |
| CacheOp alo=N Number of in-progress alloc_object() cache ops |
| luo=N Number of in-progress lookup_object() cache ops |
| luc=N Number of in-progress lookup_complete() cache ops |
| gro=N Number of in-progress grab_object() cache ops |
| upo=N Number of in-progress update_object() cache ops |
| dro=N Number of in-progress drop_object() cache ops |
| pto=N Number of in-progress put_object() cache ops |
| syn=N Number of in-progress sync_cache() cache ops |
| atc=N Number of in-progress attr_changed() cache ops |
| rap=N Number of in-progress read_or_alloc_page() cache ops |
| ras=N Number of in-progress read_or_alloc_pages() cache ops |
| alp=N Number of in-progress allocate_page() cache ops |
| als=N Number of in-progress allocate_pages() cache ops |
| wrp=N Number of in-progress write_page() cache ops |
| ucp=N Number of in-progress uncache_page() cache ops |
| dsp=N Number of in-progress dissociate_pages() cache ops |
| CacheEv nsp=N Number of object lookups/creations rejected due to lack of space |
| stl=N Number of stale objects deleted |
| rtr=N Number of objects retired when relinquished |
| cul=N Number of objects culled |
| |
| |
| (*) /proc/fs/fscache/histogram |
| |
| cat /proc/fs/fscache/histogram |
| JIFS SECS OBJ INST OP RUNS OBJ RUNS RETRV DLY RETRIEVLS |
| ===== ===== ========= ========= ========= ========= ========= |
| |
| This shows the breakdown of the number of times each amount of time |
| between 0 jiffies and HZ-1 jiffies a variety of tasks took to run. The |
| columns are as follows: |
| |
| COLUMN TIME MEASUREMENT |
| ======= ======================================================= |
| OBJ INST Length of time to instantiate an object |
| OP RUNS Length of time a call to process an operation took |
| OBJ RUNS Length of time a call to process an object event took |
| RETRV DLY Time between an requesting a read and lookup completing |
| RETRIEVLS Time between beginning and end of a retrieval |
| |
| Each row shows the number of events that took a particular range of times. |
| Each step is 1 jiffy in size. The JIFS column indicates the particular |
| jiffy range covered, and the SECS field the equivalent number of seconds. |
| |
| |
| =========== |
| OBJECT LIST |
| =========== |
| |
| If CONFIG_FSCACHE_OBJECT_LIST is enabled, the FS-Cache facility will maintain a |
| list of all the objects currently allocated and allow them to be viewed |
| through: |
| |
| /proc/fs/fscache/objects |
| |
| This will look something like: |
| |
| [root@andromeda ~]# head /proc/fs/fscache/objects |
| OBJECT PARENT STAT CHLDN OPS OOP IPR EX READS EM EV F S | NETFS_COOKIE_DEF TY FL NETFS_DATA OBJECT_KEY, AUX_DATA |
| ======== ======== ==== ===== === === === == ===== == == = = | ================ == == ================ ================ |
| 17e4b 2 ACTV 0 0 0 0 0 0 7b 4 0 0 | NFS.fh DT 0 ffff88001dd82820 010006017edcf8bbc93b43298fdfbe71e50b57b13a172c0117f38472, e567634700000000000000000000000063f2404a000000000000000000000000c9030000000000000000000063f2404a |
| 1693a 2 ACTV 0 0 0 0 0 0 7b 4 0 0 | NFS.fh DT 0 ffff88002db23380 010006017edcf8bbc93b43298fdfbe71e50b57b1e0162c01a2df0ea6, 420ebc4a000000000000000000000000420ebc4a0000000000000000000000000e1801000000000000000000420ebc4a |
| |
| where the first set of columns before the '|' describe the object: |
| |
| COLUMN DESCRIPTION |
| ======= =============================================================== |
| OBJECT Object debugging ID (appears as OBJ%x in some debug messages) |
| PARENT Debugging ID of parent object |
| STAT Object state |
| CHLDN Number of child objects of this object |
| OPS Number of outstanding operations on this object |
| OOP Number of outstanding child object management operations |
| IPR |
| EX Number of outstanding exclusive operations |
| READS Number of outstanding read operations |
| EM Object's event mask |
| EV Events raised on this object |
| F Object flags |
| S Object work item busy state mask (1:pending 2:running) |
| |
| and the second set of columns describe the object's cookie, if present: |
| |
| COLUMN DESCRIPTION |
| =============== ======================================================= |
| NETFS_COOKIE_DEF Name of netfs cookie definition |
| TY Cookie type (IX - index, DT - data, hex - special) |
| FL Cookie flags |
| NETFS_DATA Netfs private data stored in the cookie |
| OBJECT_KEY Object key } 1 column, with separating comma |
| AUX_DATA Object aux data } presence may be configured |
| |
| The data shown may be filtered by attaching the a key to an appropriate keyring |
| before viewing the file. Something like: |
| |
| keyctl add user fscache:objlist <restrictions> @s |
| |
| where <restrictions> are a selection of the following letters: |
| |
| K Show hexdump of object key (don't show if not given) |
| A Show hexdump of object aux data (don't show if not given) |
| |
| and the following paired letters: |
| |
| C Show objects that have a cookie |
| c Show objects that don't have a cookie |
| B Show objects that are busy |
| b Show objects that aren't busy |
| W Show objects that have pending writes |
| w Show objects that don't have pending writes |
| R Show objects that have outstanding reads |
| r Show objects that don't have outstanding reads |
| S Show objects that have work queued |
| s Show objects that don't have work queued |
| |
| If neither side of a letter pair is given, then both are implied. For example: |
| |
| keyctl add user fscache:objlist KB @s |
| |
| shows objects that are busy, and lists their object keys, but does not dump |
| their auxiliary data. It also implies "CcWwRrSs", but as 'B' is given, 'b' is |
| not implied. |
| |
| By default all objects and all fields will be shown. |
| |
| |
| ========= |
| DEBUGGING |
| ========= |
| |
| If CONFIG_FSCACHE_DEBUG is enabled, the FS-Cache facility can have runtime |
| debugging enabled by adjusting the value in: |
| |
| /sys/module/fscache/parameters/debug |
| |
| This is a bitmask of debugging streams to enable: |
| |
| BIT VALUE STREAM POINT |
| ======= ======= =============================== ======================= |
| 0 1 Cache management Function entry trace |
| 1 2 Function exit trace |
| 2 4 General |
| 3 8 Cookie management Function entry trace |
| 4 16 Function exit trace |
| 5 32 General |
| 6 64 Page handling Function entry trace |
| 7 128 Function exit trace |
| 8 256 General |
| 9 512 Operation management Function entry trace |
| 10 1024 Function exit trace |
| 11 2048 General |
| |
| The appropriate set of values should be OR'd together and the result written to |
| the control file. For example: |
| |
| echo $((1|8|64)) >/sys/module/fscache/parameters/debug |
| |
| will turn on all function entry debugging. |
