Multiple file storage

[KingofFah]

I didn't know where to put this, and I guess this is the closest match.

Is there anyway to collect files into one file (before you start typing, I'm not talking about zip/rar/etc compression) so that all of the files can still be accessed by programs as if all the files were not within a single file (in a windows 2k/xp environment).

In case you have no idea what I'm talking about, here's a hypothetical situation:
You have a million files many of which are small in size
You want to store these files as a single file on a drive
You want programs (all programs) to be able to access all the files contained within the single file as if they were in their normal path.

Is there currently anything out there that can do this on windows 2k/xp?

An example of a solution would be to write a driver for a virtual hard drive that would mount a file which contains all the other files, allowing reading and writing to the files within the single large file, and hopefully being able to mount into a path on a real hard drive as if they were there.

 

[KingofFah]

Well, I googled some more, this time trying to find a way of mapping a local net share to a drive folder and discovered winbolic link (which solves the second part of the problem), and then discovered Pismo File Mount. This might allow for reading, but I'm not about writing yet. If not, then it would be the same as just making an ISO and doing the same thing with d tools and winbolic link. Still, if anyone has done something similar feel free to post.

 

[Fullmetal Chocobo]

Have you looked into TrueCrypt? It creates an encrypted file that when decrypted, creates it's own drive. All files can then be accessed normally. When you are done, unmount the encrypted volume, and it is back to a single file.

 

[Lorne]

To what logical perpose would you want to do this, Seems that corruption would be a real downside?

 

[Nothinman]

You have a million files many of which are small in size
You want to store these files as a single file on a drive
You want programs (all programs) to be able to access all the files contained within the single file as if they were in their normal path.

I can think of no good reason why one would actually want point 2 unless you're just looking for a way to over complicate things.

 

[Fullmetal Chocobo]

Also, you could burn the files to an ISO, then use a program like PowerISO to mount them as a drive.

 

[KingofFah]

Originally posted by: Nothinman

You have a million files many of which are small in size
You want to store these files as a single file on a drive
You want programs (all programs) to be able to access all the files contained within the single file as if they were in their normal path.

I can think of no good reason why one would actually want point 2 unless you're just looking for a way to over complicate things.

Remember, this is in windows. The purpose is to avoid MFT fragmentation without having to increase the size of the MFT for partitions storing many small files.

Originally posted by: Fullmetal Chocobo
Also, you could burn the files to an ISO, then use a program like PowerISO to mount them as a drive.

As I mentioned, I could just do this with dtools, but that's not the point. Unless PowerISO allows writing to a mounted ISO, then it won't help at all. Also, from what I've read about the iso format, it's not an efficient method of reading/writing.

TrueCrypt might be something if the decrypted data is not indexed in the MFT (instead processed by the applications I suppose).

Either way, pismo seems to get the job done well enough. Thanks.

 

[QuixoticOne]

TrueCrypt, or some file mounting utility.

Alternatively, run a virtual machine and have the files stored in a
VHD (microsoft virtual PC 2007 SP1 terminology) or the equivalent file system image for VMWARE server/player.
The virtual machine software can and generally does keep an entire hard disc "image" used by the virtual system
in a single filesystem file in the host filesystem. The guest OS you run sees it as being a regular hard disc drive.
Of course I/O to that virtual drive is pretty efficiently done in general.

You can usually set up the guest OS as being a network file-server so that its virtual hard disc and the files on it is
shared over the network so that the host OS or another PC on the LAN can mount the shared virtual drive.
Sometimes it isn't necessary to share the virtual drive over the network for the host to be able to access it
depending on the software you run. Sometimes you can directly (not over the network) share host files / paths to the
guest VM too.

Of course you can just plug in a USB flash disk or hard disc and then mount the filesystem on it and all of those files
will be stored on that single separable block device.

 

[Mr Fox]

Originally posted by: Fullmetal Chocobo
Also, you could burn the files to an ISO, then use a program like PowerISO to mount them as a drive.

This is the best way unless you require data protection, and then the encryption method is the way... as new ISO's are created the older ones are Archived, and eventually wiped.

This is the way that My Quality System Document Control System is set-up.

 

[Nothinman]

Remember, this is in windows. The purpose is to avoid MFT fragmentation without having to increase the size of the MFT for partitions storing many small files.

And of course you have proof that storing those small files in the MFT is causing problems, right?

 

[KingofFah]

Because utilities that defragment NTFS volumes cannot move MFT entries, and because excessive fragmentation of the MFT can impact performance, NTFS reserves space for the MFT in an effort to keep the MFT as contiguous as possible as it grows.

http://support.microsoft.com/kb/174619

But, because the MFT is constantly being used to access all other files on the disk, it also gradually becomes fragmented, resulting in longer disk access times and diminished performance.

http://technet.microsoft.com/e.../library/bb742585.aspx

Originally posted by: Nothinman
And of course you have proof that storing those small files in the MFT is causing problems, right?

No, and I don't even need to go find proof for it since I never made that claim. If you read what I said you would know that the purpose is to avoid MFT fragmentation without having to increase the size of the MFT. MFT fragmentation, according to MS -- the people who made it, is a bad thing. This is all I need to tell you, since I never claimed that many files in a correctly sized MFT would affect performance; however, if you want my opinion based on personal experience, yes, having 20000 files on a 5gb partition affects performance. That's personal experience, and I won't claim it to be fact. Please read more carefully; and if you don't have anything contributory to add to the thread (as is the case thus far in this thread), then don't bother posting.

 

[RebateMonger]

I'd certainly believe that having millions of files on a system can lead to system performance in some aspects of file maintenance, wouldn't that likely appy to ANY file system, including those used by whatever you use to compress the files into a single file?

 

[Lorne]

Ok dont get me wrong Im just looking for the logic to that side of the storyand I come from an older school of HD use and may have not seen the light yet..

I come a time when we used BAM (Block Alacation map) and have always thought MS's filesystems were always a waste of time redundancies but see how the filesystem is in actuallity a index itself and processor speed is faster reading the index then the HD head movment across the BAM (This is why drive indexing never works or makes thing slower on a file system drive, Nothing like looking into and index to tell you where to look into the MFT (which is an index) to get a file).

Now I still know that a fragmented MFT can be read 10 fold faster then a frag or non-frag sequental file on a HD.

Ok, So what I get is that you wish to make a single sequential file full of files (that could be fragmented inside and cant be defragged and one crossthatch could whipe it out) so that you can make only one entry into the MFT so that the MFT doesnt get fragmented?

 

[Nothinman]

If you read what I said you would know that the purpose is to avoid MFT fragmentation without having to increase the size of the MFT. MFT fragmentation, according to MS -- the people who made it, is a bad thing.

Sure it is, but you've got to weigh the options. Is your image filesystem more work and problematic than MFT fragmentation? IMO it probably is. And you'll still have to deal with whatever filesystem you use in that image so you're just shifting the problem around.

This is all I need to tell you, since I never claimed that many files in a correctly sized MFT would affect performance; however, if you want my opinion based on personal experience, yes, having 20000 files on a 5gb partition affects performance.

So what filesystem do you plan on using in that image? If you choose NTFS you'll still have an MFT to contend with, if you use FAT you'll have even more issues. And those are pretty much you're only well supported, read-write filesystems on Windows.

And just looking real quick I've got 743,581 inodes used on my ext3 / filesystem and I've never noticed an issue. An inode will be used for ever directory, named pipe, symlink, etc so that's not just a regular file count but they still count as files in ext3 terms. There are a lot of variables involved, you can't just say "20,000 files is bad!".

Please read more carefully; and if you don't have anything contributory to add to the thread (as is the case thus far in this thread), then don't bother posting.

If you can't take someone questioning your reasoning and looking for alternate solutions then maybe you shouldn't be asking questions in a public forum.

 

[KingofFah]

Well, your remark seemed to be snide in nature. Something like "Is there any proof that many small files in the MFT causes degraded performance?" would have been a little more appropriate.

From what I've read in the past, ext3, reiser, xfs, etc are not nearly as susceptible to degradation from large file counts. From personal experience, I'd agree with this, since I've stored around 20k files seemingly without performance degradation.

I would need to research into the method of mounting the images to be able to answer your question about the file system issues. However, these would be in zip files, cfs (an open source file system from the pismo software I mentioned), or the tempfs (another pismo fs that stores it in the memory). Since pismo mounts the images without extraction, I doubt these records will be entered into the NTFS MFT for the mount points. It would be more reasonable to assume it will be handled by the pismo program. A definite answer I do not have, though; as I said, I'd probably have to contact them to find out how the program functions in detail.

Edit
@Lorne: In my opinion it's easier to just defrag the small partition with a couple large files (something which I do weekly anyway) than to manage the MFT or defragment it.

Now I still know that a fragmented MFT can be read 10 fold faster then a frag or non-frag sequental file on a HD.

Maybe I have a poor understanding of this, but this doesn't make sense to me; the MFT is essentially a file at the beginning of the drive. Say it's 128mb, and right after it you have a "regular" file of the same size. How can the MFT be read 10 times faster than the second file which is relatively in the same, fast early sectors of the drive? Also, if MFT fragmentation occurs only after you've used up say 75 percent of the drive, then wouldn't the new MFT fragment be after all that data causing an increase in seek time (aside from now being in a slow area of the drive) just like with any other file?

a single sequential file full of files that could be fragmented inside

This is a good point. I should look into that for the CFS method and see what the case is. I know I've come across programs that defrag archives, but given that CFS is a file system made by a single company and not used by much, it seems unlikely I'd find one for it. The file system is open source, so I guess I could write a defragger for it; but then I would definitely have to admit it being a bigger hassle (I'm not saying I wouldn't do it, but I would have to admit it).

 

[Nothinman]

Well, your remark seemed to be snide in nature.

It was because your idea seems completely out there. No matter what you do you have to deal with a filesystem, whether it's being held by a partition or a file inside another partition.

However, these would be in zip files, cfs (an open source file system from the pismo software I mentioned), or the tempfs (another pismo fs that stores it in the memory). Since pismo mounts the images without extraction, I doubt these records will be entered into the NTFS MFT for the mount points.

If you don't use NTFS then you won't have to worry about the MFT but you still have to worry about whatever filesystem you choose. I'd wager that a zip file will be a lot less effecient, especially if you need write support. I can't really speak about CFS since I have no idea how it works but if tempfs works as it sounds then your files will probably be stored in memory and backed by the pagefile so you'll want to have at least 1.5x as much memory as you have files so they stay in memory. And at that point a ramdrive might be a better option.

 

[Lorne]

Isnt the MFT just a sequental lookup table that generated by the NTSF.sys during drive setup and is a pct of the drive size.

So when accessing a needed file the system looks to the MFT for the sequental then to the index in the sequental for the location of the file, Hmm

A sequental volume does get fragmented internaly, As you looks up a file from it and say edited and is resaved, If the newer version has added material and the new size extends into the next block size that new block is placed in the next available area which could be beyond the info/file you just saved, The standard defrag program wont touch this because it only see the main sequental and not the internal files with in.


Nothinman, Isnt this the same way the same way the PageFile works to in lamens terms?

You know I just thought (Its a miracle), I used to be into this kind of material many many years ago debating on this crap and was only half right then to, (Slaps forhead) Im a just go sit in my corner and shut up and listen because this thred does intrigue me.

 

[Nothinman]

Isnt the MFT just a sequental lookup table that generated by the NTSF.sys during drive setup and is a pct of the drive size.

Well it's more than just a lookup table and it's generated by format, but yea.

So when accessing a needed file the system looks to the MFT for the sequental then to the index in the sequental for the location of the file, Hmm

Kinda. If the file is small enough it's stored directly in the MFT so there's no need to go any further and that's what the OP is worried about. Lots of small files stored in the MFT causing it to be extended and become fragmented.

Nothinman, Isnt this the same way the same way the PageFile works to in lamens terms?

I think your description is fairly accurate for the pagefile but not the MFT. An MFT entry is a record with a predefined layout and different data in it, a pagefile entry is just a memory page or maybe a sequence of pages. There's not much logic needed for the pagefile since it's just storing random memory pages but the MFT is storing full files with ACLs, multiple locations on disk, time stamps, etc.

 

[KingofFah]

Zip files are extremely inefficient for reading and writing, even uncompressed -- this I know. I was just tossing out some examples. CFS seems more interesting but again I'd probably have to contact the makers to find out anything in detail.

If my idea seemed "completely out there", then your answer should be that what I am trying to accomplish, to your knowledge, is impossible. Why do you think I posted the thread in the first place? It was to see if anyone already knew a good method.

Yes, if I don't use NTFS I do not need to worry about the MFT, and that's the point: Finding a method that works. If various linux file systems are unaffected or less affected by lots of small files in small partitions, then wouldn't it be reasonable to say that I might find a file system for the mounted file that would also have this advantage over NTFS? One might consider finding such a file system the main purpose of the thread at this stage of discussion.