MN K12 Tech Coordinator Tips & Tricks

Tags

    HistoryEdited Mar 1, 2010 10:27 AM by Joel Anderson…

    Slow AFP Response due to High CPU Utilization on Mac OS X Server

    During the 08-09 school year, we ran our clients and servers on Mac OS X 10.5.4.

    Never had an issue with performance or reliability.

    This year (09-10) we upgraded our clients and servers to Mac OS X 10.5.8 during the Christmas break, and since then we have been dealing with sporadic bouts of the AFP servers that were acting as home directory servers becoming totally unresponsive. When viewing the server's activity at the time of AFP becoming unresponsive, the servers would always have extremely high (even 100%) CPU utilization.

    It wasn't every day, and sometimes we would go a week or more without any issues. During this time of course, we tried several methods trying to resolve the issue. Usually after each new fix we tried, the issue would go away for a few days, so we would think it was fixed.

    But eventually it would always return.

    We thought it might be fragmentation (as we weren't able to do our usual clean installations over the summer due to construction), so we did a clean installation. That resolved the problem for several weeks.

    We thought it might be people running applications from the server, because they had created what they thought were "shortcuts" to the application by dragging an application to their desktop, when in essence, they were making a copy of the application to their desktop, which resided on the server. So we spent days deleting applications from user desktops.

    We tried other suggestions from the forums: enabling Network Home Redirector, to redirect the cache files (we have never had to that before, but we thought lets try it). We tried disabling Spotlight indexing. We looked for usage patterns. Was it too many people streaming music? Was it a rouge network device? Corrupt .DS_Store files?

    Here is a list of the top "fixes" for this scenario, but by and large, these items ultimately did not resolve the issue for the majority of people:
    1. Setting AFP wan threshold
    2. Turning off spotlight on server by marking AFP volume as private
    3. Turned off Time Machine even though no backup volumes where defined
    4. Set kern.maxfiles=200000 and kern.maxfilesperproc=5000 sysctl’s
    5. Turned DS_Store off on both clients and servers
    6. Turned off smb (windows samba) file server
    7. Disabled auto-disconnect in AFP after idle time.
    8. Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes.
    9. Verified that Host Cache Flushing is disabled on external RAID array.
    10. Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true. Set as preference for all groups.
    11. Disabled kerberos for AFP authentication.
    12. Changed the fibre topology to Point to Point for all 4 fiber connections to the Promise VTrak array.
    13. Stop spotlight indexing by using the command: touch /Volumes/Sharename/.metadata_never_index
    14. Renamed odpac.bundle in /System/Library/KerberosPlugins/KerberosAuthDataPlugins/ to odpac.bundle_DISABLED
    15. volume with our home dirs has to keep at least 10% free for performance reasons

    None of these things resolved the issue. Some people claimed upgrading to 10.6 would resolve the issue. Others said 10.6 didn't fix their issues.

    So we broke down and ordered 32GB of RAM for each of our home directory servers (an upgrade from 8GB), and we upgraded the home directory server to Mac OS X Server 10.6.2.

    Its only been two weeks, but I have been monitoring every day for those two weeks, and it looks like the issue has been resolved by the RAM upgrade and u0grade to 10.6 server. The graphs below represent what I have seen every day for the last 10 business days on all four of the upgraded servers:

    With our typical load of 100-120 users:
    100 Users Connected
    We aren't even using 20% CPU - probably closer to 10% if you exclude morning login:
    Server CPU Utilization

    Comments

    Tired of Apple (unauthenticated)
    Mar 3, 2010

    Have you tried increasing the maxthreads on AFP? The default is very low (40, I think). This can be done in Terminal:
    serveradmin settings afp:maxThreads=400 is what I used and it resolved this problem for me. See: http://support.apple.com/kb/TA24270 and http://forums.macosxhints.com/archive/index.php/t-70565.html
    Hope this helps

    Danny (unauthenticated)
    Mar 12, 2010

    What is you memory usage at peak times? Since you had implemented two changes to your system, which do you suppose was the fix?

    Dominik Hoffmann (unauthenticated)
    Apr 12, 2010

    Having my own dual-G4 Xserve at home I am not so fortunate to be able to just upgrade. I wish I could. In the meantime, I'm stuck on how to fix this. I would upgrade RAM, but this machine maxes out at only 2 GB.

    Danny (unauthenticated)
    Oct 4, 2010

    Just to update: I had also added additional memory first, but upgrading to 10.5 resolved this problem for me.

    /groups/tech/search/index.rss?sort=modifiedDate&sortDirection=reverse&tag=mainlist/groups/tech/search/?sort=modifiedDate&sortDirection=reverse&tag=mainCore ArticlesCustomTagSidebarCustomTagSidebar?sort=modifiedDate&sortDirection=reverse&tag=main0/groups/tech/sidebar/CustomTagSidebarmodifiedDate5CustomTagSidebarreversemainCore Articlescustom/groups/tech/search/index.rss?tag=hotlist/groups/tech/search/?tag=hotWhat’s HotHotListHot!?tag=hot60/groups/tech/sidebar/HotListAdded iOS 5 and iCloud Article LinkjoelJoel Anderson2011-10-27 05:51:48+00:002011-10-27 05:51:48updated47joelJoel Anderson2011-03-28 23:40:40+00:002011-03-28 23:40:40updated46Moved Document Management & iWork Document Formatting under FeaturesjoelJoel Anderson2011-03-28 23:40:32+00:002011-03-28 23:40:32updated45Moved Paperless Workflow under features, moved iPad 2 Mini review under General PointersjoelJoel Anderson2011-03-28 23:39:09+00:002011-03-28 23:39:09updated44Added breadcrumbsjoelJoel Anderson2011-03-28 15:48:01+00:002011-03-28 15:48:01updated43Changed Title of iPad Document Strategies linkjoelJoel Anderson2011-03-28 15:43:54+00:002011-03-28 15:43:54updated42joelJoel Anderson2011-03-27 19:12:09+00:002011-03-27 19:12:09updated41Separated Productivity Outline into its own pagejoelJoel Anderson2011-03-27 19:11:22+00:002011-03-27 19:11:22updated40joelJoel Anderson2011-03-21 11:51:32+00:002011-03-21 11:51:32updated39fix shopping and travel pages from blog pages to wiki pagesjoelJoel Anderson2011-03-21 11:49:38+00:002011-03-21 11:49:38updated38joelJoel Anderson2011-03-21 11:47:11+00:002011-03-21 11:47:11updated37Moved link to iPad 2 Mini Review towards top of pagejoelJoel Anderson2011-03-19 13:23:03+00:002011-03-19 13:23:03updated36Added New to iPad pagejoelJoel Anderson2011-03-19 12:02:34+00:002011-03-19 12:02:34updated35Added link to iPad 2 Mini ReviewjoelJoel Anderson2011-03-19 10:31:38+00:002011-03-19 10:31:38updated34joelJoel Anderson2011-03-19 10:29:17+00:002011-03-19 10:29:17updated33Added iPad App pages for Shopping and TraveljoelJoel Anderson2011-03-19 10:28:06+00:002011-03-19 10:28:06updated32Removed Cases links, as it is now linked from the new Accessories pagejoelJoel Anderson2011-03-19 09:59:48+00:002011-03-19 09:59:48updated31joelJoel Anderson2010-04-26 14:49:15+00:002010-04-26 14:49:15updated30Added new structure for iPad case articlesjoelJoel Anderson2010-04-26 13:29:44+00:002010-04-26 13:29:44updated29Formatting cleaningjoelJoel Anderson2010-04-26 02:30:51+00:002010-04-26 02:30:51updated28Added use of keynote presentationjoelJoel Anderson2010-04-25 21:14:19+00:002010-04-25 21:14:19updated27joelJoel Anderson2010-04-25 20:55:07+00:002010-04-25 20:55:07updated26Added a strengths/weakness list to article; Created place holders for printing and remote controljoelJoel Anderson2010-04-25 20:54:50+00:002010-04-25 20:54:50updated25Organized and Separated LinksjoelJoel Anderson2010-04-17 02:12:22+00:002010-04-17 02:12:22updated24joelJoel Anderson2010-04-12 19:17:12+00:002010-04-12 19:17:12updated23joelJoel Anderson2010-04-12 19:12:10+00:002010-04-12 19:12:10updated22joelJoel Anderson2010-04-12 19:10:37+00:002010-04-12 19:10:37updated21joelJoel Anderson2010-04-12 18:58:05+00:002010-04-12 18:58:05updated20joelJoel Anderson2010-04-12 18:46:24+00:002010-04-12 18:46:24updated19joelJoel Anderson2010-04-12 18:45:36+00:002010-04-12 18:45:36updated18joelJoel Anderson2010-04-12 18:23:47+00:002010-04-12 18:23:47updated17joelJoel Anderson2010-04-12 18:19:55+00:002010-04-12 18:19:55updated16joelJoel Anderson2010-04-12 18:17:32+00:002010-04-12 18:17:32updated15Added NoteTaking AppsjoelJoel Anderson2010-04-12 18:13:40+00:002010-04-12 18:13:40updated14Added iPad Accessories PagejoelJoel Anderson2010-04-07 13:57:19+00:002010-04-07 13:57:19updated13Added iPad Cases pagejoelJoel Anderson2010-04-07 13:50:29+00:002010-04-07 13:50:29updated12joelJoel Anderson2010-04-07 13:49:43+00:002010-04-07 13:49:43updated11joelJoel Anderson2010-04-06 00:00:36+00:002010-04-06 00:00:36updated10Added tag - hotjoelJoel Anderson2010-04-05 23:05:53+00:002010-04-05 23:05:53addTag9joelJoel Anderson2010-04-05 22:54:36+00:002010-04-05 22:54:36updated8joelJoel Anderson2010-04-05 22:46:37+00:002010-04-05 22:46:37updated7joelJoel Anderson2010-04-05 20:35:18+00:002010-04-05 20:35:18updated6joelJoel Anderson2010-04-05 20:33:42+00:002010-04-05 20:33:42updated5joelJoel Anderson2010-04-05 20:31:37+00:002010-04-05 20:31:37updated4joelJoel Anderson2010-04-05 20:27:51+00:002010-04-05 20:27:51updated3Added tag - ipadjoelJoel Anderson2010-04-05 20:26:48+00:002010-04-05 20:26:48addTag2First createdjoelJoel Anderson2010-04-05 20:15:04+00:002010-04-05 20:15:04created1wiki2011-10-27T05:51:48+00:00groups/tech/wiki/17816FalseiPad/groups/tech/wiki/17816/iPad.htmlJoel Anderson47 updatesiPad Personal Productivity Strategies > iPad Did you just get a new iPad, and want a know what you should get right away? Try my iPad Quick-St...Falsejoel2011-10-27T05:51:48+00:00joelJoel Anderson2010-05-01 02:22:25+00:002010-05-01 02:22:25updated11Changed the name from Personal Productivity to Personal Productivity StrategiesjoelJoel Anderson2010-05-01 02:14:34+00:002010-05-01 02:14:34updated10Added text descriptors to each categoryjoelJoel Anderson2010-05-01 02:05:53+00:002010-05-01 02:05:53updated9Added tag - paperlessjoelJoel Anderson2010-05-01 01:46:35+00:002010-05-01 01:46:35addTag8Added tag - mainjoelJoel Anderson2010-04-05 20:17:08+00:002010-04-05 20:17:08addTag7joelJoel Anderson2010-04-05 20:15:34+00:002010-04-05 20:15:34updated6Added tag - hotjoelJoel Anderson2010-04-05 20:15:11+00:002010-04-05 20:15:11addTag5Added tag - ipadjoelJoel Anderson2010-04-05 20:11:26+00:002010-04-05 20:11:26addTag4Added tag - gtdjoelJoel Anderson2010-04-05 20:11:21+00:002010-04-05 20:11:21addTag3Added tag - productivityjoelJoel Anderson2010-04-05 20:11:16+00:002010-04-05 20:11:16addTag2First createdjoelJoel Anderson2010-04-05 20:10:54+00:002010-04-05 20:10:54created1wiki2010-05-01T02:22:25+00:00groups/tech/wiki/56867FalsePersonal Productivity Strategies/groups/tech/wiki/56867/Personal_Productivity_Strategies.htmlJoel Anderson11 updatesPersonal Productivity Strategies This section of the site will focus on personal productivity, and will include topics such as Getting Things Done (GTD), OmniFoucs, paperless wor...Falsejoel2010-05-01T02:22:25+00:00joelJoel Anderson2010-05-01 02:22:25+00:002010-05-01 02:22:25updated11Changed the name from Personal Productivity to Personal Productivity StrategiesjoelJoel Anderson2010-05-01 02:14:34+00:002010-05-01 02:14:34updated10Added text descriptors to each categoryjoelJoel Anderson2010-05-01 02:05:53+00:002010-05-01 02:05:53updated9Added tag - paperlessjoelJoel Anderson2010-05-01 01:46:35+00:002010-05-01 01:46:35addTag8Added tag - mainjoelJoel Anderson2010-04-05 20:17:08+00:002010-04-05 20:17:08addTag7joelJoel Anderson2010-04-05 20:15:34+00:002010-04-05 20:15:34updated6Added tag - hotjoelJoel Anderson2010-04-05 20:15:11+00:002010-04-05 20:15:11addTag5Added tag - ipadjoelJoel Anderson2010-04-05 20:11:26+00:002010-04-05 20:11:26addTag4Added tag - gtdjoelJoel Anderson2010-04-05 20:11:21+00:002010-04-05 20:11:21addTag3Added tag - productivityjoelJoel Anderson2010-04-05 20:11:16+00:002010-04-05 20:11:16addTag2First createdjoelJoel Anderson2010-04-05 20:10:54+00:002010-04-05 20:10:54created1wiki2010-05-01T02:22:25+00:00groups/tech/wiki/56867FalsePersonal Productivity Strategies/groups/tech/wiki/56867/Personal_Productivity_Strategies.htmlJoel Anderson11 updatesPersonal Productivity Strategies This section of the site will focus on personal productivity, and will include topics such as Getting Things Done (GTD), OmniFoucs, paperless wor...Falsejoel2010-05-01T02:22:25+00:00joelJoel Anderson2010-05-01 02:22:25+00:002010-05-01 02:22:25updated11Changed the name from Personal Productivity to Personal Productivity StrategiesjoelJoel Anderson2010-05-01 02:14:34+00:002010-05-01 02:14:34updated10Added text descriptors to each categoryjoelJoel Anderson2010-05-01 02:05:53+00:002010-05-01 02:05:53updated9Added tag - paperlessjoelJoel Anderson2010-05-01 01:46:35+00:002010-05-01 01:46:35addTag8Added tag - mainjoelJoel Anderson2010-04-05 20:17:08+00:002010-04-05 20:17:08addTag7joelJoel Anderson2010-04-05 20:15:34+00:002010-04-05 20:15:34updated6Added tag - hotjoelJoel Anderson2010-04-05 20:15:11+00:002010-04-05 20:15:11addTag5Added tag - ipadjoelJoel Anderson2010-04-05 20:11:26+00:002010-04-05 20:11:26addTag4Added tag - gtdjoelJoel Anderson2010-04-05 20:11:21+00:002010-04-05 20:11:21addTag3Added tag - productivityjoelJoel Anderson2010-04-05 20:11:16+00:002010-04-05 20:11:16addTag2First createdjoelJoel Anderson2010-04-05 20:10:54+00:002010-04-05 20:10:54created1wiki2010-05-01T02:22:25+00:00groups/tech/wiki/56867FalsePersonal Productivity Strategies/groups/tech/wiki/56867/Personal_Productivity_Strategies.htmlJoel Anderson11 updatesPersonal Productivity Strategies This section of the site will focus on personal productivity, and will include topics such as Getting Things Done (GTD), OmniFoucs, paperless wor...Falsejoel2010-05-01T02:22:25+00:00joelJoel Anderson2010-05-01 02:22:25+00:002010-05-01 02:22:25updated11Changed the name from Personal Productivity to Personal Productivity StrategiesjoelJoel Anderson2010-05-01 02:14:34+00:002010-05-01 02:14:34updated10Added text descriptors to each categoryjoelJoel Anderson2010-05-01 02:05:53+00:002010-05-01 02:05:53updated9Added tag - paperlessjoelJoel Anderson2010-05-01 01:46:35+00:002010-05-01 01:46:35addTag8Added tag - mainjoelJoel Anderson2010-04-05 20:17:08+00:002010-04-05 20:17:08addTag7joelJoel Anderson2010-04-05 20:15:34+00:002010-04-05 20:15:34updated6Added tag - hotjoelJoel Anderson2010-04-05 20:15:11+00:002010-04-05 20:15:11addTag5Added tag - ipadjoelJoel Anderson2010-04-05 20:11:26+00:002010-04-05 20:11:26addTag4Added tag - gtdjoelJoel Anderson2010-04-05 20:11:21+00:002010-04-05 20:11:21addTag3Added tag - productivityjoelJoel Anderson2010-04-05 20:11:16+00:002010-04-05 20:11:16addTag2First createdjoelJoel Anderson2010-04-05 20:10:54+00:002010-04-05 20:10:54created1wiki2010-05-01T02:22:25+00:00groups/tech/wiki/56867FalsePersonal Productivity Strategies/groups/tech/wiki/56867/Personal_Productivity_Strategies.htmlJoel Anderson11 updatesPersonal Productivity Strategies This section of the site will focus on personal productivity, and will include topics such as Getting Things Done (GTD), OmniFoucs, paperless wor...Falsejoel2010-05-01T02:22:25+00:00hot/groups/tech/search/index.rss?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcomelist/groups/tech/search/?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcomeRecent ChangesRecentChangesListUpdates?sort=modifiedDate&kind=all&sortDirection=reverse&excludePages=wiki/welcome0/groups/tech/sidebar/RecentChangesListmodifiedDateallRecent ChangesRecentChangesListUpdateswiki/welcomeNo recent changes.reverse5searchlist/groups/tech/calendar/Upcoming EventsUpcomingEventsListEvents1Getting events…