Swap file on Ramdisk?

[bsobel]

Originally posted by: opk
Dug,

You happen to be right about this. BUT, it's for extreme situations, not for the original question. This product sounds like a good idea IF your mobo memory is maxed out and you still need more. This obviously doesn't happen much as most mobos accept 1.5-2 GB of main memory. Most people (in excess of 99% i'd hazard) don't need that much.

So, assuming you're sitting there with this monster set of applications that you just have to run and your machine is maxxed out at 2GB of RAM. The applications really want, say, 3 GB of RAM. In that specific case, this product might be worth the money. It isn't the same as main system RAM, it isn't near as efficient, but it is many many times faster than paging out to hard disk.

Big servers and huge databases can need large amounts of RAM. That is behind the push (really more of a nudge) toward 64-bit computing. It's main benefit right now is it breaks the 4GB addressable RAM limit. But, again, we're talking ebay big here, nothing you, oog, or yam need worry their pretty little heads about.

This is a different topic than the original one yam broached. He wondered about excess main system memory. This product is explicitly for use when main system memory is at capacity.

Yea, what ^^ he ^^ said
Bill

 

[Caanon]

I didn't read whole thread just first few posts so if someone already said this, sorry.

The original poster said he wanted to reduce disk accesses. Here is something I've known and been doing since I had my pentium 233:

The main thing that causes a performance hit with swapfiles seems to be not the writing and reading from the file itself, but when it dynamically changes it size if your file needs are gradually increasing or decreasing for some reason it changes many times and can be especially annoying in games(this was more noticeable to me in 1996/97 =)

So anyway, I generally put swapfile size fixed at 1.5 or 2x the size of ram. A fixed swapfile also have the advantage that you can get all or most of the swapfile in one large chunk with no fragements.

So i've had:
48mb of ram: 96mb swapfile
128mb ram: 300mb swapfile
512mb ram: 768mb swapfile

 

[Windex]

just like the last thread i didnt bother to read it all, apologizein advance...

Im running windows xp pro with 768 MB ddr ram, and a SET swap file size of 1mb (this is for photoshop as it wont finish loading with out some swap space, before installing PS i had it set for none), i play NWN while running the toolset and the server without ever needing a swap file, runs just as fast (if not faster) then when i let widnows take care of things.

Occasionnally i get the windows is running out of memory, but it generally happens when i have somehting like outlook open while downloading tons of pr0n from the newsgroups. And occasionnally it pops up while all three NWN tools are loaded.

So who says you NEED a swap file?

 

[Nothinman]

So who says you NEED a swap file?

If you read your post again, you just did.

Occasionnally i get the windows is running out of memory, but it generally happens when i have somehting like outlook open while downloading tons of pr0n from the newsgroups. And occasionnally it pops up while all three NWN tools are loaded.

So using a stupidly small pagefile causes things to fail, thanks for satisfying our point.

 

[Windex]

my point is this, if it werent for ps i wouldnt be using ANY swap file and be running with no problems, in fact, if you re-read youll see i said i ran it this way for a good while before installing ps. Also if you re-read it only happens when i have the NWN client, toolset and ded. server running all at once, using 700+mb of physical memory, or hundreds of pictures open in one oe message.

UT2003, UT, half-life, nascar 2002, ms fs 2002, and other high end games run flawless, no annoying pop-up.

Hell all that NWN stuff will run fine as long as i close one of the three... something that should be done anyway.

On top of all that I dont think having a 1mb swap file even qualifies as having one. I belive its stupidly small as well, as i dont really need it at all. None of those messages would come up if it werent for PS having the need to view a swap file before it will agree to load, not my fault, but ill deal with the occasional "swap file too small" error for actually using the ram in my system.

 

[bsobel]

None of those messages would come up if it werent for PS having the need to view a swap file before it will agree to load, not my fault, but ill deal with the occasional "swap file too small" error for actually using the ram in my system.

Ok, read the darn thread. Some applications use API which are implemented in terms of the swap file. Yes, you usually don't need one with that much memory unless (now, get this it will be a fricken shock) you run so many applications that you use up your real memory and then need to swap. But you somehow think your running faster be having it disabled, your not, you weren't using it when it was set larger (except for photoshop) anyways.

Bill

 

[opk]

Point being, lots of programs bypass the Windows APIs and create their own memory management system. Photoshop is the one everyone knows about, i have no idea about NWN et al. Adobe knows Photoshop needs mucho RAM, more than most people have. It's prudent in that case to program for the ground up with some of your memory space out on disk. If it doesn't have to be, then bully. Most programs just rely on the Windows memory manager, and leave it at that.

The fact remains that under any NT system, it's best to leave it all alone, it works well without intervention. I don't see what the whole point of this is...

 

[Nothinman]

Point being, lots of programs bypass the Windows APIs and create their own memory management system.

I can't bypass the Win32 APIs in Windows any more than I can read/write kernel memory in Windows (assuming NT, Win9X doesn't qualify as a real OS). Certain applications try to outsmart the OS, but in the end the OS has the final say on where the memory is physicall locatated and how much of it gets swapped out when needed. Like I said, I don't even think Administrator can lock pages in memory by default, meaning absolutely everything is up for paging out when the time comes.

On a side note, Photoshop probably does this because it's roots are in Mac OS which before OS X had little or no real memory management, apps could do whatever they wanted to whatever memory addresses they wanted.

 

[bsobel]

On a side note, Photoshop probably does this because it's roots are in Mac OS which before OS X had little or no real memory management, apps could do whatever they wanted to whatever memory addresses they wanted.

Photoshop does this to preserve the performance of other applications. Otherwise it would wind up grabbing large chunks of memory and force everyone to page, by using the unnamed swap file backed memory mapped files, they have a consitant performance threshold without killing the rest of the system.
Bill

 

[Dug]

Original (me): No one should write a program that purposely uses up your physical ram and then go to the swapfile. This is because people operate many programs at the same time. This is also the reason you don't want to limit your swap file, but instead let Windows manage it, because you never know. A good example is the game NWN. It uses a great deal of your swapfile, up to 300MB at a time. Another is Photoshop.

Actually noone should write a program that cares where the memory it's using came from, it should just ask for memory and let the OS do it's job from there

If I was writing a program for the general public, I sure wouldn't want to do that.

No offense, but if you really believe that, you're an idiot. Modern OS design contradicts that 100%.

And then I hear from bsobel

Photoshop does this to preserve the performance of other applications. Otherwise it would wind up grabbing large chunks of memory and force everyone to page, by using the unnamed swap file backed memory mapped files, they have a consitant performance threshold without killing the rest of the system.

So am I the idiot for thinking the way I do? Or am I seeing a contradiction that doesn't exist.

 

[Nothinman]

So am I the idiot for thinking the way I do? Or am I seeing a contradiction that doesn't exist.

Not to be harsh, but yes. And Adobe is too. They should let the OS VM subsystem do it's job and let it maintain usability. If have so little memory that everything else gets paged to disk when you run Photoshop you need more memory.

 

[Dug]

They should let the OS VM subsystem do it's job and let it maintain usability

agreed in most cases,
But bsobel maintains that

they have a consitant performance threshold without killing the rest of the system.

I have talked with two instructors at school and two friends in programming that believe the same thing I do.
No one should write a program that purposely uses up your physical ram and then go to the swapfile. This is because people operate many programs at the same time. This is also the reason you don't want to limit your swap file.
They also added this is very rare for any program to gobble up all of your ram, and if you are using a program that does this that you probably aren't multi tasking with other programs anyway.


What are your qualifications to believe otherwise?
Do you write programs for Windows that are distributed to the general public?
BTW I'm not saying you are wrong. It just seems to go against conventional thinking.
I understand for performance that you want to use the amount of ram needed so your program doesn't slow down. But like I said, programmers write for a large customer base. Some with 64MB's ram and some with 1G of ram.

And its not just photoshop that does this. quite a few games and I'm sure CAD programs could swallow up any systems ram in no time by loading up everything it needs, but they don't (on purpose)

If every program that I had, used all the memory I have available, I would only be able to run one program at a time. Otherwise I would have heavy disk thrashing.

 

[Whitedog]

Not another SWAPFILE Thread! :|

Good God People.... just Get Over It already!

If you have a fast computer.... Lots of memory... running an OS like 2K or XP... Tweaking your Swapfile is pointless!

You'd probably "see" a more boast in performance if you "remove the shadow from your mouse" than you would from tweaking your swapfile. (if you get the sarcasm, you'd understand my point)

Get over it.

 

[Nothinman]

No one should write a program that purposely uses up your physical ram and then go to the swapfile.

Noone does (except proof of concept people writing DoS software), but if you open a 200M image in photoshop you should be aware that it'll need 200M+ overhead for the app memory available.

This is because people operate many programs at the same time. This is also the reason you don't want to limit your swap file.
They also added this is very rare for any program to gobble up all of your ram, and if you are using a program that does this that you probably aren't multi tasking with other programs anyway.

Yes, those are all true except for the pagefile limiting. I limit my pagefile size because I know I'll never need more than that, I do run the risk of getting into an OOM situation but I take that risk.

What are your qualifications to believe otherwise?

The fact that the OS VM is tuned for a reason, otherwise why not just let all the apps handle how much to swap by themselves? Why spend tons (and it does require tons) of time tweaking the OS' VM when people are just going to work around it?

But like I said, programmers write for a large customer base. Some with 64MB's ram and some with 1G of ram.

And that's why you should be as vague as possible when allocating memory, so that you don't penalize either group. There's no need to grab 200M of pagefile space immediately on startup on the 1G machine and the 64M machine is going to swap no matter what you do.

quite a few games and I'm sure CAD programs could swallow up any systems ram in no time by loading up everything it needs, but they don't (on purpose)

But all those machines _should_ have enough memory to handle the program and it's data, if you're trying to design a building on a 64M machine in AutoCad you're asking for trouble and deserve whatever you get.

If every program that I had, used all the memory I have available, I would only be able to run one program at a time. Otherwise I would have heavy disk thrashing.

I'm not saying that. I'm saying every program should just call malloc, GlobalAlloc, whatever and let the OS decide where to allocate it from, the app shouldn't care what swapping is going on that's up to the OS and user to resolve. If your program needs 2M memory then there's no need to allocate everything available, but if it needs 200M it also shouldn't look to hit the pagefile early either.

 

[bsobel]

I'm not saying that. I'm saying every program should just call malloc, GlobalAlloc, whatever and let the OS decide where to allocate it from, the app shouldn't care what swapping is going on that's up to the OS and user to resolve. If your program needs 2M memory then there's no need to allocate everything available, but if it needs 200M it also shouldn't look to hit the pagefile early either.

Nothinman, suprisingly I have to agree with Dug on this point (and this point only ). In *general* you are completely right, 99.9% of the applications written should not care how the OS resolves memory requests. However, there are certain types of applications that based on the work they need to do, and constraints of the current hardware/32bit OS world, must be more specific.

A couple of examples; Microsoft ISA server includes a web caching componet which can cache data in memory. In it's configuration you can specify the percentage of physical memory to reserve for caching. This is a reasonable way of provding the feature, but not destroying overall performance.

Photoshop, in may be editing a very large (100meg+) image file and need substantial room to perform additional operations on the image. Using unnamed memory mapped files (backed by swap) are perfect for this situation (and why they exist). Now, I haven't checked if Photoshop always does it this way, or only on low memory machines (not sure how 'cute' the developers where), but this implementation is reasonable given the common hardware they current Photoshop users are using. Things will change in a few years as multi gigabytes of memory are more standard.

Bill

 

[Nothinman]

A couple of examples; Microsoft ISA server includes a web caching componet which can cache data in memory. In it's configuration you can specify the percentage of physical memory to reserve for caching. This is a reasonable way of provding the feature, but not destroying overall performance.

That makes perfect sense for ISA server to do that. Continually resizing the web cache in memory would hurt performance, so grabbing X amount at start because you know eventually you'll use it makes sense because it's up to the administrator to make sure there's enough memory in the box.

The photoshop thing still seems fishy to me, but I'll let it go =)

 

[Oogle]

I keep reading this a few times, but I can't figure out what this means...

unnamed memory mapped files (backed by swap)

Is this a temporary file? Why is it unnamed? If it's unnamed, does that mean I can't find any trace of it in the file system? Why does a memory mapped file have to be backed by swap? Just the term alone (backed) makes the swapfile seem like an unnecessary redundancy.

Also, on another topic, I'm looking at my WinXP system here at work (512 RAM, 70 MB fixed swap), I noticed some strange stuff. I'm getting a "low on virtual memory" warning. I check my Task Manager. Physical memory says I have 300 MB RAM available. That doesn't sound like I have low memory. I check the PageFile usage % in the Performance Monitor. It says that my pagefile (\??\C:\pagefile.sys) is 100% used. What's happening in this scenario? Why should my pagefile be used at all if I have still have 300 MB RAM available? I thought the memory manager is only supposed to swap when I'm low on memory. What makes the memory manager think I'm low on memory?

 

[bsobel]

unnamed memory mapped files (backed by swap) Is this a temporary file? Why is it unnamed? If it's unnamed, does that mean I can't find any trace of it in the file system? Why does a memory mapped file have to be backed by swap? Just the term alone (backed) makes the swapfile <i>seem</i> like an unnecessary redundancy.

All I can recommend is you go read the API docs.

I'm getting a "low on virtual memory" warning. I check my Task Manager. Physical memory says I have 300 MB RAM available. That doesn't sound like I have low memory. I check the PageFile usage % in the Performance Monitor. It says that my pagefile (\??\C:\pagefile.sys) is 100% used. What's happening in this scenario? Why should my pagefile be used at all if I have still have 300 MB RAM available? I thought the memory manager is only supposed to swap when I'm low on memory. What makes the memory manager think I'm low on memory?

Re-read the thread, I mentioned how the first issue you asked can affect the second you asked 2 or 3 times.

Bill