Subject: [nfsv4] How_much_to_cache

From: Prasanna Wakhare <[email-masked]>
Date: Fri, 14 Jan 2005 12:53:50 +0530

Hi all,
I have a simple question in mind. I want to introduce client side
caching in any network or cluster file system. The caches are
in-memory cache not secondary cache as in AFS but as sprite.
My question is all the cache is in kernel space and if large file
happens to mmap or read/write say 2 GB etc.
What would be consequence if there is great limit on my cache.
We know kernel space virtual address are from 3GB to 4GB. If i start
allocating cache block as soon as i'm getting file from
storage node and if my cache size is 5GB then this will hang the
system as kernel cant have that much addresses refer to. And in that
case we need either CPU to address 64 bit or something like that.
But even then there is limit on cacheing the file data.
So how much shall i keep my cache size to have best performance.
I hope i'm pretty clear in what i asked?
Thanks
Prasanna

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Michael E. Thomadakis" <[email-masked]>
Date: Fri, 14 Jan 2005 02:06:37 -0600

>I hope i'm pretty clear in what i asked?

This service is very similar to the 'buffer cache' service in the kernel 
where file system data are treated as VM pages. UNIX and increasingly 
Linux are using VM to buffer file pages.

In my oppinion there should be a method that a user can communicate 
performance 'hints' to the kernel as to the behavior of a file or the 
entire file system. You have the entire spectrum of control here: fixed 
to adaptive.  'Hints' are a good idea when a particular file or file 
system has a known / expected behavior or service requirements. However 
there should always evaluated in the context of the current resource 
availability. Assuming large file sizes, let me just give a few simple 
examples.

- A user scans a file from beginning to end and never rewrites it or 
modifies it. Thus, read-ahead could be of benefit. The question then is 
how much to read ahead, if the user does not know this value. One way to 
answer this is to monitor within a (sliding) window of time the request 
pattern of the process. The disclaimer is that recent past is not always 
a good indicator of the near future, but again, in many useful cases it 
is. Given the read once pattern, there is no reason to maintain the 
buffer cache pages for this file.

- A user writes always at the end of a file (appending to it) as is the 
case of log files or results from computation/reports, etc. Output 
buffering is of benefit and caching is not needed.

- A process that deviates from the two ideal ones above is one that 
re-reads or (worse) re-writes a portion of the file it read previously. 
Existing VM caching algorithms can be used here for the benefit of the 
system (LRU page replacement, etc).

- It would be interesting of a user could specify a heuristic or at 
least hint values (such as read ahead counts or 'release behind') at 
mount time for the entire file system.

In general I, don't think that one needs to cache entire files in the 
limited kernel memory space. This open  a pandora's box with all other 
system and user activities suffering degradation and then eventually 
leading to VM thrashing.  Notice though that the above discussion 
becomes immaterial for the small files, so one parameter of interest may 
be how much (in KBs) to cache for a file. A more interesting parameter 
is the total number of real memory pages to dedicate to file caching. 
You will be very thankful later if you can enforce a true upper bound on 
this.

Finally, since all heuristics can be defeated by certain request 
patterns, the meta-heuristic should be to guard against overusing scarce 
or bottleneck resources, which ever those may be at different times. In 
short, it is worthwhile pursuing decent heuristics but you should always 
monitor the system to avoid forming bottlenecks. 

-MT

---

From: Erblichs <[email-masked]>
Date: Fri, 14 Jan 2005 14:40:12 -0800

Ok,

	Maybe some of this will help.

	1) are you implimenting write-thru caching
	or write behind caching? This will determine
	the consistency of the cached data if a
	crash occurs?

	2) can you cache data temporally so a disconnect
	from a server will not cause service issues to
	the full set of data? This can also be applied
	to CD ROM src files, and then modified and not
	restored to the original location

	3) Do you accept holes (discontigous sections)
	within your file object?

	4) Lastly, what are the access latencies for
	   read operations?

	Ok, with all that said, with a WAN env the access
	latencies can approach at max 150ms. With GBs
	of memory allocated for file operations, a rule
	of thumb for me in the past was anything less than
	16MBs of file gets read. This for me is the largest
	working set size (MMU dependent).

	Most files (cached objects) will be signifianctly
	less than that, so if the total amount of memory
	consumed is less, the main question is aging. When
	do you consider the data no longer useful to even
	yourself (assuming that you did no modifications)
	and discard the cached contents?

	If the cache was a write-back, every 15-30 secs,
	cached data was marked for flushing back to the
	original src location.

	Mitchell Erblich
	----------------