best way to store & quickly access *lots* of tiny files?

[NTB]

I use an IDE at work that's based on IBM's Eclipse to create programs and UI screens for the school's data storage and management system (Datatel Colleague, if you're curious). I like the program, but the workspace is getting a little out-of hand:

Most, if not all, of the files that I import & work with are tiny, and they end up chewing up a lot of space: I just looked at the properties for my current workspace folder, and here's what I found:

68,300+ files
1,000+ directories
53MB of actual data
286MB used on-disk.

Any suggestions?

Nathan

 

[yh125d]

SSD is best at random small read/writes like that. If a small SSD is out of your budget just go for a single platter 7200RPM drive

 

[ilkhan]

ramdisk

 

[NTB]

Originally posted by: ilkhan
ramdisk

stuck with a 32-bit OS & 2GB RAM, at least for now I thought about getting one of Gigabyte's I-RAM boards, but at this point that would cost just as much, if not more, than a small SSD.

Nathan

 

[Idontcare]

Originally posted by: NTB

Originally posted by: ilkhan
ramdisk

stuck with a 32-bit OS & 2GB RAM, at least for now I thought about getting one of Gigabyte's I-RAM boards, but at this point that would cost just as much, if not more, than a small SSD.

Nathan

Try superspeed's ramdisk plus 10

RamDisk Plus 10 has a most unique feature. Our patent pending technology can access memory beyond the limitation imposed by a Windows 32-bit operating system! In other words, RamDisk Plus 10 can use "unmanaged" Windows' memory e.g. above 4GB. It can also use the stubbornly inaccessable memory between 3.2GB and 4GB.

There's a 30day free trial download there.

 

[ChaiBabbaChai]

why won't a high-end flash drive work?

 

[sxr7171]

The best RAMDisk IME is free: http://www.dataram.com/products-and-services/ramdisk/

Works great on Win 7 64-bit and performed better than the trials of paid RAMDisk software I've tried.

 

[Idontcare]

Originally posted by: sxr7171
The best RAMDisk IME is free: http://www.dataram.com/products-and-services/ramdisk/

Works great on Win 7 64-bit and performed better than the trials of paid RAMDisk software I've tried.

Only 4GB and smaller ramdisk for the free version, and doesn't access the installed system memory that the OS itself doesn't recognize.

I wouldn't be able to use dataram's free ramdisk to replace my 5GB created with ramdisk plus 10 software I use in WinXP pro 32bit. (8GB installed memory, windows sees 3.25GB, ramdisk plus 10 sees/accesses all 8GB even in winxp 32bit)

What does it cost to step up to the paid version for 64bit OS? I'm always on the lookout for ramdisks that are superior to superspeed's but I've yet to find one that doesn't come up short in the areas that matter to me.

 

[her209]

Get a second small drive and reformat it using a smaller block size, e.g. 512 bytes.

 

[NTB]

Originally posted by: ChaiBabbaChai
why won't a high-end flash drive work?

That was my first thought. Did you have any particular models in mind?

Nathan

 

[BaboonGuy]

http://www.zipzoomfly.com/jsp/...p?ProductCode=10009849

OCZ Vertex 30GB for $139. IIRC there were more of these small SSDs on newegg for cheaper a few weeks ago. Basically you just want anything with an Indilinx controller.

Prices will probably drop as production ramps up and supply meets demand.

 

[taltamir]

Originally posted by: yh125d
SSD is best at random small read/writes like that. If a small SSD is out of your budget just go for a single platter 7200RPM drive

QUALITY SSD that is... you want an intel or an indilinx based drive (vertex, falcon, ultradrive me, torqx, etc)

 

[DominionSeraph]

Originally posted by: NTB
stuck with a 32-bit OS & 2GB RAM, at least for now

Why does this prevent you from using a ramdrive when you only have 53MB of files?

And why exactly is 286MB a lot of space?

 

[Emulex]

alot of wasted space; i wonder if you put it into a compressed filesystem if you'd regain alot of that. cramfs or whatever microsoft uses.

 

[heymrdj]

The HD's you have will probably do fine with the files, what you need to do is format a drive for storage utilizing a much smaller cluster size, such as 512bytes. If your HD really is too slow to feed the program, which is doubtful, then get a good 20GB SSD or something with a small cluster size format, again like 512bytes.

 

[taltamir]

do NOT stick with 32bit... 64bit is much faster. up to a theoretical 4-5x faster. Now granted it varies by program... some games get 0% speed increase. 7z file compression is 23% faster. And some hash calculations are 300% faster.
64bit is NOT just about ram, it is a strawman...

Also 2GB of ram is terrible at this day and age... anyone should be using 4GB if they can.

 

[Nothinman]

do NOT stick with 32bit... 64bit is much faster. up to a theoretical 4-5x faster. Now granted it varies by program... some games get 0% speed increase. 7z file compression is 23% faster. And some hash calculations are 300% faster.

If you can get 64-bit binaries, which are pretty rare for Windows.

64bit is NOT just about ram, it is a strawman...

In Windows it's mostly about memory because the 32-bit clients are crippled and there's very few 64-bit builds of anything for Windows.

Also 2GB of ram is terrible at this day and age... anyone should be using 4GB if they can.

That's a retarded statement. Memory usage is determined by your usage patterns, 2G and under is still fine for a lot of people these days.

 

[taltamir]

If you can get 64-bit binaries, which are pretty rare for Windows.

Well duh. And they are quite common thank you very much.

In Windows it's mostly about memory because the 32-bit clients are crippled and there's very few 64-bit builds of anything for Windows.

There are plenty of things with 64bit builds, and what clients are crippled? its not about the whole conspiracy theory about MS not using that virtual trick to go beyond 2GB on 32bit is it?
1. individual programs are still limited to 2GB per program (which is already not enough).
2. There is no "crippling" in not including a feature.. .especially a difficult to do feature that serves no point, is obsolete, and causes BSODs if any of your drivers has not been made specifically to be compatible with it.

That's a retarded statement. Memory usage is determined by your usage patterns, 2G and under is still fine for a lot of people these days.

Who are you calling a retard?

2GB is not enough today, common programs in 2009 take so much ram that 2GB is just too little. Even if you use an ancient OS like windows XP, the amount of ram your games, office programs, browsers, etc take today is just much much more.
For example I routinely go over 2GB using web browsers (tabbed browsing, its awesome, it is a ram hog...)

Anyways, even the most mild usage patterns in 2009 using the latest versions of common programs for non intensive things requires more than 2GB (taking into account that the OS eats a big chunk of this... 700MB for winXP, 1.3GB for 64bit vista)

 

[Nothinman]

Well duh. And they are quite common thank you very much.

Quite common? The only ones that I'd say are common are the ones shipped with a 64-bit install. There are a few games, benchmarks and such that have 64-bit builds but just about everything else is 32-bit in the Windows world. And for good reason, most applications don't gain much, if anything, from a 64-bit build.

There are plenty of things with 64bit builds, and what clients are crippled? its not about the whole conspiracy theory about MS not using that virtual trick to go beyond 2GB on 32bit is it?
1. individual programs are still limited to 2GB per program (which is already not enough).
2. There is no "crippling" in not including a feature.. .especially a difficult to do feature that serves no point, is obsolete, and causes BSODs if any of your drivers has not been made specifically to be compatible with it.

I was talking about 32-bit Windows client OSes being crippled with regards to PAE. And it is indeed crippled because the feature is there, they had to add code to make it so that the system couldn't use that memory. Most people with a 64-bit Windows install are using it be able to use all of their memory. The rest that installed it because they think 64-bit is obviously faster than 32-bit don't understand the difference anyway.

Yes, each 32-bit process is limited to 2G of VM by default, but for the overwhelming majority of applications that is way more than enough. Sure there are exceptions, but they're the minority.

2GB is not enough today, common programs in 2009 take so much ram that 2GB is just too little. Even if you use an ancient OS like windows XP, the amount of ram your games, office programs, browsers, etc take today is just much much more.
For example I routinely go over 2GB using web browsers (tabbed browsing, its awesome, it is a ram hog...)

I have 2G here and in my laptop at work and it's enough.

Anyways, even the most mild usage patterns in 2009 using the latest versions of common programs for non intensive things requires more than 2GB (taking into account that the OS eats a big chunk of this... 700MB for winXP, 1.3GB for 64bit vista)

You're exaggerating the amount of memory the OS uses for itself by quite a bit there.

 

[NTB]

This is a *work* machine guys...I work at a university, so I have a lot more freedom with it than I did with the machine I used in my previous job (corporate), but it is still the school's machine Though I imagine that if I wanted to shell out for more RAM and / or an SSD, they'd let me use it. In the mean time though, as I said - it's Windows XP & 2GB of RAM.

Nathan

 

[Denithor]

USB flash drives are not fast at all - the fastest are still much slower than even slow HDD at transfer rates.

Just use one of those RAMdisk programs to create a 512MB drive and copy all the data files onto it. If you need the whole 2GB for system memory add another 1GB stick to the computer and use the whole stick for your RAMdisk.

Remember to back up when through working so you don't lose everything in case of power failure/reboot/etc.

 

[NTB]

Originally posted by: Denithor
USB flash drives are not fast at all - the fastest are still much slower than even slow HDD at transfer rates.

Just use one of those RAMdisk programs to create a 512MB drive and copy all the data files onto it. If you need the whole 2GB for system memory add another 1GB stick to the computer and use the whole stick for your RAMdisk.

Remember to back up when through working so you don't lose everything in case of power failure/reboot/etc.

But the access time is much lower, which should help - I'm dealing with *tons* of tiny files, as opposed to a few larger ones. For conditions like this, random read / write performance is more important than linear read / write, correct?

Nathan

 

[Denithor]

...which is exactly why I second the suggestion of a RAMdisk. The read/write times on a RAMdisk are one to two orders of magnitude faster than the fastest SSD drive you can find.

Meaning your small files will open faster than you can tell it to access them.

Just give that freebie a try and see how it works. Doesn't cost you anything but a little time.

 

[Nothinman]

You must consider that flash drives aren't persistent though, so you've got to store the files somewhere else if you use a RAM disk.

 

[tynopik]

what's your problem?

1. you desperately need another 100MB of disk space

if that's the only 'problem', then i would say it's time to push for a new computer

2. using the program is slow because of all the file accesses

move your project file to a ramdrive, it's small enough it shouldn't be a problem

3. loading the directory is painfully slow because windows sucks at dealing with 10,000+ files in a folder

split the project into more subfolders and bury them so accessing one subfolder doesn't cause all the files to be loaded at once

for instance
/projects/project1/(5000 files)
/projects/project2/(5000 files)

IS NO GOOD because going into the projects folder will cause it to read all files in the immediate subfolders

you have to do something like

/projects/project1/project1/(5000 files)
/projects/project2/project2/(5000 files)

making the directory another level deep 'hides' it from being loaded when entering the projects folder