During the 08-09 school year, we ran our clients and servers on Mac OS X 10.5.4.
Never had an issue with performance or reliability.
This year (09-10) we upgraded our clients and servers to Mac OS X 10.5.8 during the Christmas break, and since then we have been dealing with sporadic bouts of the AFP servers that were acting as home directory servers becoming totally unresponsive. When viewing the server's activity at the time of AFP becoming unresponsive, the servers would always have extremely high (even 100%) CPU utilization.
It wasn't every day, and sometimes we would go a week or more without any issues. During this time of course, we tried several methods trying to resolve the issue. Usually after each new fix we tried, the issue would go away for a few days, so we would think it was fixed.
But eventually it would always return.
We thought it might be fragmentation (as we weren't able to do our usual clean installations over the summer due to construction), so we did a clean installation. That resolved the problem for several weeks.
We thought it might be people running applications from the server, because they had created what they thought were "shortcuts" to the application by dragging an application to their desktop, when in essence, they were making a copy of the application to their desktop, which resided on the server. So we spent days deleting applications from user desktops.
We tried other suggestions from the forums: enabling Network Home Redirector, to redirect the cache files (we have never had to that before, but we thought lets try it). We tried disabling Spotlight indexing. We looked for usage patterns. Was it too many people streaming music? Was it a rouge network device? Corrupt .DS_Store files?
Here is a list of the top "fixes" for this scenario, but by and large, these items ultimately did not resolve the issue for the majority of people:
1. Setting AFP wan threshold
2. Turning off spotlight on server by marking AFP volume as private
3. Turned off Time Machine even though no backup volumes where defined
4. Set kern.maxfiles=200000 and kern.maxfilesperproc=5000 sysctl’s
5. Turned DS_Store off on both clients and servers
6. Turned off smb (windows samba) file server
7. Disabled auto-disconnect in AFP after idle time.
8. Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes.
9. Verified that Host Cache Flushing is disabled on external RAID array.
10. Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true. Set as preference for all groups.
11. Disabled kerberos for AFP authentication.
12. Changed the fibre topology to Point to Point for all 4 fiber connections to the Promise VTrak array.
13. Stop spotlight indexing by using the command: touch /Volumes/Sharename/.metadata_never_index
14. Renamed odpac.bundle in /System/Library/KerberosPlugins/KerberosAuthDataPlugins/ to odpac.bundle_DISABLED
15. volume with our home dirs has to keep at least 10% free for performance reasons
None of these things resolved the issue. Some people claimed upgrading to 10.6 would resolve the issue. Others said 10.6 didn't fix their issues.
So we broke down and ordered 32GB of RAM for each of our home directory servers (an upgrade from 8GB), and we upgraded the home directory server to Mac OS X Server 10.6.2.
Its only been two weeks, but I have been monitoring every day for those two weeks, and it looks like the issue has been resolved by the RAM upgrade and u0grade to 10.6 server. The graphs below represent what I have seen every day for the last 10 business days on all four of the upgraded servers:
With our typical load of 100-120 users:
We aren't even using 20% CPU - probably closer to 10% if you exclude morning login:
