Monday, April 16, 2012

Windows 7 Explorer RAM Usage

Want to see something really neat? Set up a media center with Windows 7, and have it record ~500 tv shows at 1-3GB a piece. Windows Media Center of course dumps all 1.2TB of video files into a single directory, which really shouldn't be a problem.

Now open that folder, and ask Windows Explorer to perform any sort of meta-data operation on all 1.2TB of files.  Just metadata.  For example, I asked Windows to sort these 500 files by date created.  RAM usage took off. Our system went from 1.5/4GB RAM usage to thrashing and completely catatonic in a matter of seconds, to the point where I couldn't even ask Windows to reboot, and had to yank the power cord.  I tried turning off indexing, compression, thumbnails, and anything else I could find in the control panel that at all sounded like it would be expensive on large files.  It still croaks, and I'm stumped.  The exact sequence is:
  • Open Windows Explorer and navigate to this magical huge folder
  • Switch to detailed view
  • Click one of the column headers to have Explorer sort by date or size, etc
  • RAM usage takes off. I saw it go as high at 6GB, but it took ~10 minutes of thrashing to get there. Task Manager nor Resource Monitor will show any processing taking this RAM; it just disappears.

How in the world can Windows 7 not be able to handle 500 files / 1.2TB directory with 4GB of RAM without even blinking?!?

We carefully cleaned out our recorded TV folder down to about half, and performing metadata requests now cause only a 1.5GB RAM usage jump, which hangs around for 5-15 minutes before just disappearing again.  I have no idea why Windows does this, and with >2TB file systems becoming the new norm, I sure hope there will be a solution to this.

Am I missing something? Has someone seen this before? I can't believe I wasn't able to find anything about this online.

8 comments:

  1. Same problem here man, I have a huge folder where I keep most of my media downloads and my system starts eating shit as soon as I open it. I discovered that I can reverse the memory spike by closing and re-opening Windows Explorer from the Task Manager (which I now keep on the taskbar for this very reason), but obviously that's not the root of the problem. I'm guessing it might be possible to prevent this from happening if you spread your media files across multiple directories, but that's just sidestepping the issue and sounds inconvenient as hell.

    ReplyDelete
  2. I remember Vista did that a LOT, haven't had that issue in Win7 though....

    ReplyDelete
  3. I`ve seen similar happen, but the actual problem turned out to be the virus checker installed. Disabling it dropped the RAM usage considerably.

    ReplyDelete
  4. I like the virus theory. I'm using MSE, so you'd THINK they'd have considered this. There is no way I'm running an unmaintained Windows box without virus. If Media Center will walk directories, maybe it's a weekly batch script to move the week's files into a YYYYWW sub-directory.

    ReplyDelete
  5. It's not due to a virus or antivirus software, Windows is just really terrible at working with lots of files. I have a folder with over 200 sub-directories and over 50 thousand files in it and every time I'm working with anything in those folders the system lags.

    It's really down to the number and size of files that Explorer is trying to work with. It also depends how big and if the Windows page file is on the same drive as all your files.
    Drive speed may also play into it.

    A short term fix would be to segregate everything into sub-directories and defrag the drive but a slightly better solution would be to get more RAM.

    ReplyDelete
    Replies
    1. We have weekly degrag running on this drive, so it's sitting at 0%, and with it being a DVR workload, it's not like it can get particularly fragmented in the first place (single write at a time, no files under 1GB, files are immutable). It's separated from the OS/Paging drive, and I watched it page out an additional 3GB, so even if I spent the money to max out the mobo to something ridiculous like 16GB, I'm not convinced that it wouldn't still try and use all of it; it might choke later, but a scaling issue is still a scaling issue.

      I'm just getting really frustrated, because I have Linux systems with larger file systems that handle acceptably well with only 256MB of RAM. These problems with Windows are stupid, and *shouldn't* be acceptable in this day and age.

      Delete
  6. Check this forum thread: http://social.technet.microsoft.com/Forums/en-US/w7itproui/thread/551eed27-6aaa-4ee2-8ab7-e843ae103c50/

    One of the MVPs suggests temporarily renaming the registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\PropertySystem\PropertyHandlers\.ts (where .ts would be whatever file extension your DVR ultimately saves to--.avi, .mkv, etc.) Let us know if that works for you.

    ReplyDelete
  7. Hi. I'm having the same problem with the memory spike while accessing the Recorded TV folder. It doesn't seem to release the memory until I reboot. Were you able to find any solution other than the registry edit.

    ReplyDelete