Using Data Structures in .NET

[SelArom]

Hi! I posted this in another community forum but never got a response. thought I'd try my hand at the programmers here at AT

A while back I developed a simple vb.net program for keeping track of rebates. I didn't want to use a database like access or msde because I don't want the overhead in file size and I especially don't want to have to have users install another piece of software in their system just to use my program...

so I implemented it using xml. I made a dataset and put in tables for the stores and the individual rebates. it worked really really well! the only problem is that because it's xml and a dataset, it has to load the entire file into memory when the program loads, and manipluate it in memory and write back to the file as a whole.

see, I like the idea of using a database because it only accesses the informaiton you need for that particular rebate. there is a performance hit because it has to retrieve each time you change rebates/stores, but you would always only be looking at one at a time, so there is no need to have them all in memory.

does any of this make sense? is one method better than the other in my particular case? is there nothing that can compromise? any suggestions about an alternative for storing and manipulating the data would be most welcome! thank you in advance for your insight and advice!

-SelArom
__________________

 

[replicator]

I'd definitely go with the database if this is business critical information that you can't afford to lose. The way you are storing it now is ok for an app that doesn't expect to hold a lot of records. With a small dataset it may not be a problem, but apps that are useful tend to collect more information that you would expect. XML doesn't scale very well. So you'll need to consider a few factors: importance of data integrity, how many concurrent users, how many records, how often records are added/updated, any integration required in the future.

In almost all cases, the database is the best idea. There really is no notice performance hit by using a database and I think this really shouldn't be your conern. DB's are so fast on modern hardware, i wouldn't worry about it at all.

There are some very small lightweight DB's that are available (firebird, SQLite, derby) that you can embed with your app.

 

[oog]

i think your choice of xml in this case was warranted, assuming that you don't have a huge number of rebates. you should make sure that your file writing logic is written so that if it crashes partway through you don't end up with a corrupt data file though.

databases are very important for non-trivial applications though.

 

[Snapster]

Installing additional software is understandable, but to say you are concerned about file size yet choose xml is like saying you don't like the colour red and buying a red ball. XML naturally has an overhead.

Anyway, you could try looking at the Office 2000 distributable interop assemblies that way you can use the likes of an access db without needing Office being installed on the target system.

 

[SelArom]

Originally posted by: replicator
I'd definitely go with the database if this is business critical information that you can't afford to lose. The way you are storing it now is ok for an app that doesn't expect to hold a lot of records. With a small dataset it may not be a problem, but apps that are useful tend to collect more information that you would expect. XML doesn't scale very well. So you'll need to consider a few factors: importance of data integrity, how many concurrent users, how many records, how often records are added/updated, any integration required in the future.

In almost all cases, the database is the best idea. There really is no notice performance hit by using a database and I think this really shouldn't be your conern. DB's are so fast on modern hardware, i wouldn't worry about it at all.

There are some very small lightweight DB's that are available (firebird, SQLite, derby) that you can embed with your app.

Wow an embedded database is EXACTLY the solution I was looking for!! since I program using .NET, I am totally excited to find out that firebird looks like it will totally fit my needs!! thanks for the info, I think this is going to be perfect!!

-SelArom

 

[Templeton]

How many records are we talking about here? I get the feeling that this is a standalone single user client app with maybe a couple hundred and certainly less then a couple thousand records. If this is the case, I don't see any reason to care about storing all data in memory, computers aren't at a shortage of ram, and how much extra is it really taking up? If the xml processing is really taking too long, consider using a flat file - you'll have to do your own parsing but won't have the overhead of building an xml tree and operations like adding a new record to the end of the file can be done in very fast time.

 

[kamper]

Originally posted by: Templeton
How many records are we talking about here? I get the feeling that this is a standalone single user client app with maybe a couple hundred and certainly less then a couple thousand records. If this is the case, I don't see any reason to care about storing all data in memory, computers aren't at a shortage of ram, and how much extra is it really taking up? If the xml processing is really taking too long, consider using a flat file - you'll have to do your own parsing but won't have the overhead of building an xml tree and operations like adding a new record to the end of the file can be done in very fast time.

A custom flat file sounds like a recipe for maintenance hell. Everybody understands xml and there will be tools capable of reading the structure for years to come, as opposed to a custom format which may become essentially unreadeable as soon as his code no longer runs on current platforms (like maybe it's not compatible with a future version of .NET for some reason). Of course, xml's not a silver bullet in this respect, but it is much better. As for the appending time, if you've got the whole thing in memory anyways, adding records at any point in the file should be fast.

I personally vote for the embedded db solution. It's got the most scalability built in, just in case you ever need it.

 

[SelArom]

Originally posted by: kamper

Originally posted by: Templeton
How many records are we talking about here? I get the feeling that this is a standalone single user client app with maybe a couple hundred and certainly less then a couple thousand records. If this is the case, I don't see any reason to care about storing all data in memory, computers aren't at a shortage of ram, and how much extra is it really taking up? If the xml processing is really taking too long, consider using a flat file - you'll have to do your own parsing but won't have the overhead of building an xml tree and operations like adding a new record to the end of the file can be done in very fast time.

A custom flat file sounds like a recipe for maintenance hell. Everybody understands xml and there will be tools capable of reading the structure for years to come, as opposed to a custom format which may become essentially unreadeable as soon as his code no longer runs on current platforms (like maybe it's not compatible with a future version of .NET for some reason). Of course, xml's not a silver bullet in this respect, but it is much better. As for the appending time, if you've got the whole thing in memory anyways, adding records at any point in the file should be fast.

I personally vote for the embedded db solution. It's got the most scalability built in, just in case you ever need it.

yeah, xml works fine for now but I do want scalability, and I do NOT want to put that msde bloat into my program. I think i've found the perfect solution! thanks again everyone

-SelArom