I'm assuming you were raised in the U.S. One reason it's hard to do is the algorithms that children are typically taught in this country for doing things like multiplication. For example, when multiplying 34 times 57, US children would multiply the 7 and 4 first, get 28, have to carry the two, multiply the 7 and 3, add the two. Then put a zero down to the right on the next line as a place holder, and multiply the 5 times the 4, carry the two, 5 times 3 and add the 2.
Other places would approach it like this: 30 times 50 = 1500. Easy to remember 1500. 4 times 50, add to the 1500. Now you're up to 1700. 7 times 30, add to 1700 is 1910. Just have to remember 1910. Lastly, 4 times 7, and add that to the 1910 = 1938.
In the first algorithm, it takes multiple steps to arrive at the first number and multiple steps to arrive at a 2nd number before you add them together. In the latter algorithm, you're keeping a running total while still doing one multiplication at a time.
Cue about middle school age. The brain has developed to the point where it can handle more abstract concepts (like variables.) Now we're teaching Algebra, and it takes a couple days of teaching before the kids get good at multiplying something like (2x+3) and (4x-5). Kids in other places think, "shit, this is what I've been doing for 4 years. This is nothing new." This, in part, explains some of the differences on standardized testing. Through, I think it's 4th or 5th grade, U.S. children compare quite favorably, then they fall way behind.
-----
Also, fucking calculators. If you had a little robot that carried absolutely everything for you, you never even had to lift up a glass to your mouth to drink, in a couple of years, you'd be asking, "why is it so hard to carry a bag of groceries." Without practice, you can't do arithmetic very quickly. The more often you use a calculator for simple arithmetic operations, the slower your brain gets at doing that mental arithmetic.
Further, if you don't have a calculator, you start learning tricks. 87 times 93? Why, that's just 8100 minus 9. Why? Because it's (90-3)(90+3); the product of conjugates which is 90²-9. Or, in a calculus class, faced with finding the area under a curve using the midpoint method (midpoint of the base of a rectangle determines the x-value at which you determine f(x) for the height of a rectangle). It's fairly common on those problems to have to find 6.5², 7.5², 8.5², etc.
6.5 squared: 6 squared (36) plus 6 = 42. Tack on a .25 to the end: 42.25
7.5 squared: 7 squared (49) plus 7 = 56. Tack on a .25 to the end: 56.25
8.5 squared: 8 squared (64) plus 8 = 72. Tack on a .25 to the end: 72.25
At which point you might see a pattern. "Hey, it goes up by 14, then 16, I'll bet the next one is 18 higher." Though the pattern might be a little harder to discern if f(x) is something like 2x²+x+5; but no more difficult to do in your head.
Why it works? Again with the algebra. (x + 0.5)(x + 0.5) = x² + x + .25
Note: if you did (10x + 5)(10x + 5), you get 100x²+100x+25. So, the pattern is the same without the decimal.
As your mind learns more of these tricks, you look absolutely amazing to people who can't do mental arithmetic. You might say, "but you're doing..." Doesn't matter - it's still mental arithmetic, it's just that with a fluency in Algebra, you have a lot more algorithms at your disposal. And, with practice (not relying on a calculator from 4th grade on), you get quicker and quicker at identifying which tricks work on which problems.
Fun problem: I'll get a class to give me digits of a 9 digit number. Once I have a lot of kids shouting out numbers, it's easy to be selective for the 2nd number to make it 142857143. I can multiply any 9 digit number times 142857143 faster than the students can type it into a calculator (only to find out that the display on the calculator isn't large enough. So, when they get a rounded number in scientific notation, we have to turn to a computer to verify my answer. But it's simple: I multiply by 142857143 indirectly. First I multiply the 9 digit number by 1000000001 (which just makes it an 18 digit number with the first 9 digits repeating. So, if the first 9 digits that the kids give me are 234897561, then the new number is 234897561234897561. Then, divide by 7. Long division - in your head. (omg, it's soooo hard, since you haven't done long division in your head in 20 years.)
7 goes into 23 3 times (write down the 3, mentally carry the 2; 7 goes into 23... )
So, in summary, why is mental math so hard for you? Because you're a lazy, stupid American.