Calculating confidence level of RNG drop?

[ComputerGuy2006]

If I have a random chance (unknown percentage) of something happening what math formula do I use to determine my confidence of the data I have collected?

Example: Let's say you have a game that can drop a chest (you have no clue what the chances are). You try to get the chest 1000 times and get the chest 10 times (aka 1% of the time). Mathematically how confident can I say the drop rate is 1%? Similarly if you ran the test 100 times and lets say you got 2 drops.

 

[cbrunny]

a confidence interval?

http://www.dummies.com/education/sc...dence-interval-around-an-event-count-or-rate/

 

[brainhulk]

 

[mrjminer]

If you are using a true RNG, your confidence level should be 100% that it is 1% in all circumstances; non-compliance with 1% is simply randomness.

 

[cbrunny]

If you are using a true RNG, your confidence level should be 100% that it is 1% in all circumstances; non-compliance with 1% is simply randomness.

yeah that was my thought too. I think he is trying to estimate the actual drop rate though (that is, it is not known), in which case a confidence interval is a reasonable solution.

 

[Red Squirrel]

You need to collect tons and tons and tons of data, the more, the more accurate your result will be. I run a MMO game server (UO) and one of my friends got an account at one of the official shards and was doing just that. Keeping track of what kind of loot drops off certain mobs and what the percentage is. It's quite tedius to do but the more samples you have the more accurate it will be. Often there's info that's already available online too so use that as a base as someone else already may have done all the work.

If you are testing your own RNG one thing I've always done is have it spit out numbers into a text file, then tally up how many occurrences of each number. Assuming the drop formula is linear you just want each number to appear about the same number of times as the other. That will mean the RNG is random enough. If it keeps leaning towards certain values and does so consistently then perhaps there's a flaw in it. Depends how "accurate" you want the RNG to really be. It might still be acceptable.

Though there are different RNGs, some where it's exponential, so there is a higher chance of landing on say 1-10 than there is of landing on 11-20, and 90-100 being the rarest. A RNG like that can be used to determine how much of something drops, ex: gold. You then have a factor somewhere in the formula that is based off the actual mob itself. It can get complicated depending how far you want to go.

We actually want to redesign the loot as it kinda got out of hand how complicated it got and we will streamline it more.

 

[DaveSimmons]

For games with a test server people have opened 10,000 loot boxes to get an accurate drop rate.

But sorry, I'd have to use the All-knowing Google to find the formula for the interval given a uniform distribution, results of X and a sample size of N.

 

[dullard]

In statistics, you generally choose the confidence interval that you want to apply. Then you apply it and calculate the range that the data will be in. It isn't generally the reverse where you take numbers and estimate a confidence interval.

For example, if I want to be 100% confident, I need a large interval. Thus I can be 100% confident that your chest will be dropped between 0 times and 1000 times with 1000 attempts. Of course, that isn't a useful statistic, being 100% confident is just too strict. The lower and lower the confidence interval, the tighter and tighter the actual range will be.

I'm making up numbers here, but I could choose 90% confidence interval and the chest will appear between 5 and 15 times with 1000 attempts. Or I could be 80% confident and maybe then I could say the chest will appear between 6 and 14 times with 1000 attempts. Again, these are just example numbers.

If you want to know that you will get exactly 10 chests out of 1000 attempts, your confidence level is going to be virtually zero if it is truly a random occurrence.


For the close enough math, go here (there are better formulas but they are much more complex):
https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Normal_approximation_interval

Suppose you want to be 95% confident of the result. Then you will be wrong 5% of the time. Thus, alpha = 0.05. From that it is just easier to look up a table and get z = 1.959963 when alpha = 0.05 (source of that number is beyond the scope of my post). If you think that you will get the chest 1% of the time, then p = 0.01. Enter into the formula in the link above and you get that you are 95% confident that the chest will appear between 0.38% and 1.61% of the time (in other words, you are 95% confident that the chest will appear 4 to 16 times with 1000 attempts).

 

[mrjminer]

yeah that was my thought too. I think he is trying to estimate the actual drop rate though (that is, it is not known), in which case a confidence interval is a reasonable solution.

Ah, that makes sense now. I assumed he was trying to gage the RNG he included in a program. Silly me.

 

[ComputerGuy2006]

The reason why I asked this is because I collected numbers that contradicted the long standing beliefs of the drops in a new game I am playing. A group of us later also collected number which was close to matching mine (as opposed to the beleived drops) and two people got into an argument about if we needed more data to be sure. Which got me wondering how you can use a math formula to say something like 'this is within x% of accurate y% of the time', or the name of the math formula that would tell me with x roles and y hits the estimated accuracy is z%.

edit: If I use a normal approximation interval calculator with 1000 samples size with 10 hits I would know the number is between 0.544% and 1.831% with 95% confidence level... Or 99% confident it's between .453% and 2.1929% (tho I divided by 10 so not sure if this is right).

For the close enough math, go here (there are better formulas but they are much more complex):
https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Normal_approximation_interval

What are the more complex formulas?