Stop talking t-tests etc. Those tools are specifically for statistics properly modeled by a Normal distribution. But you are talking about the defect rate found in a sample of known finite size. This is Attribute statistics, modeled by a Binomial Distribution. In Statistical Process Control jargon, this is a p-Chart, where p is the Proportion of defects in the sample of size n. In your case, you have been told that the expected value of p is 0.0005 (based on prior studies of a large sample) and the sample in question had one defect in a sample of 3339, so n = 3339 and p = 0.00029949. Let's use the term "pbar" for the expected value of p.
For a p-Distribution, the formulae are that (for a Confidence level of 0.997, or 99.7%) the Lower Control Limit (LCL) is
LCL = (pbar) - 3 x SQRT{[(pbar) x (1 - (pbar)] / n},
and the Upper Control Limit (UCL) is
UCL = (pbar) + 3 x SQRT{[(pbar) x (1 - (pbar)] / n}
NOTE that there is NO "Standard Deviation" in this - the Standard Error for a binomial distribution is derived solely from pbar, the expected value of p, and the Confidence Interval uses this plus the sample size. In your case, plugging values in yields
LCL = -0.0006606, which is impossible so the real LCL is zero.
It also yields UCL = 0.0016606
If the value of p found for the current sample falls within this range, it is still considered part of the regular population being examined, but if it is outside those limits, it is considered statistically different. Well, your measured value of 1 defect in 3339 items was p = 0.00029949 and that is NOT outside the Limits, so your sample is NOT different from the whole population in question.
As a simpler view on this, consider that the expected value you were given is ½ defect per thousand, or 1 defect in 2,000 units, or 5 defects in 10,000 units. What you observed was 2.99 defects per 10,000 in your sample. Do you consider 3 significantly different from 5? Don't forget, in defect counting, there are no fractions in the actual counted number of defects. In a sample of n units, you could have 1 defect, or 2, or 3, or 4, etc. But you cannot have exactly 2.356 defects in a sample. The RATE of defects is 2.99 per 10,000, but we calculated that from an integer number, 1 defect, found in 3,339 units.