A couple years ago, I made a New Year's Resolution to not eat sugar (i.e. added white refined sugar in stuff like candybars & donuts...fruit & honey & "natural" stuff were OK) for a whole year. I got about 2 months into it...without any changes. Weight was the same, emotionally felt the same (sugar doesn't make me hyper), didn't have teeth issues before (or after), etc. Note that I am a sugarholic, so I was giving up something I loved (sugary treats) just to see what it would do. Nada, no effect. I went back on sugar, also no effect. I should note here that I tend to eat 3 meals plus 2 or 3 snacks during the day (between meals), so I also don't like giving myself a chance to get hungry. Overall, this has been my health progression over the years:
1. Decided to get in shape, learned about dieting & bodybuilding, started eating healthy, homemade meals.
2. Switched to 6 meals a day. I found that bigger meals make me more tired, especially after lunch, and it's easier to stay full because everyone gets hungry for snacks between meals. I define a meal as anything I eat, including snacks...so that could be a handful of energy bites (basically a ball of homemade granola bars), or a protein smoothie, or half a sandwich, or carrots & hummus. That eventually turned into a nice mix of protein, carbs, and fats per "meal". Lost like 50 pounds doing this when combined with some light cardio.
3. Tried avoiding sugar for a couple months, no real effect positive or negative. This led me to dig deeper into nutrition & learning about IIFYM, which basically boils down to "it's not what you eat, it's primarily how many calories you eat in a day, and secondarily, how many grams of protein, carbs, and fat you eat in a day". I had a really hard time accepting that it really boiled down to calories in vs. calories out. The documentary "Fathead", a rebuttal to "SuperSize Me", got me considering that food quality, in terms of healthiness, had no relation to how fat or skinny you were, and that you could technically remove exercise from the equation as well if your only metric was weight. And in the show, he capped himself at 2,400 calories a day & got to an ideal weight by literally eating out every single meal...which was still plenty of food, but just not pigging out all the time (i.e. large soda instead of super-sized). More & more information started confirming this, like the Twinkie diet experiment:
http://www.cnn.com/2010/HEALTH/11/08/twinkie.diet.professor/
And this great article by Nicole Capurso on how eating donuts helped give her an 80kg snatch: (via IIFYM)
https://nicolecapurso.wordpress.com/2014/08/31/how-donuts-gave-me-abs-an-80kg-snatch/
Ultimately, I think the best combination if you want specific results is IIFYM with whole foods...which isn't to say that's the diet you should go on, because even IIFYM with goodies is still fine. I've done paleo before as well with good results, although if you look at the lady's pictures above, there's a clear difference between just eating clean & eating specifically for your body & your goals. Plus long-term, a diet consisting mostly of junk food will probably have negative health effects, so again...moderation.
Obviously, a plant-based (not saying vegan, but just integrating more fruits, veggies, nuts, leafy greens, etc.) diet is going to yield the best results because you're getting a lot of natural, healthy stuff into your body instead of processed chemicals. "Forks over Knives" really convinced me that a plant-based diet was the best way to go, especially after seeing those like 20 people get out of the hospital & that one lady start doing marathons & stuff after switching up her diet. But, based on my own experience, sugar, at least in moderation, isn't going to completely wreck your life or add to your waistline. As per the Twinkie diet guy, if you keep it at or under the max calories you need, your weight isn't going to be affected, nor are your various blood levels. From what I can tell, it seems that being overweight is pretty much the worst thing you can do for your body long-term because of the internal complications it leads to.
Regarding the physical & mental energy that you mentioned, I've found that's like 80% tied to sleep. When I go to bed early & get enough hours of sleep, and do that for a few days straight, I'm pretty much bulletproof to not working out, eating junk food, etc. because my body isn't already depleted. Sleep also has an enormous affect on my motivation levels, especially the mental friction I have from how hard it is to do everything from the dishes or laundry to my daily workout routine. I think & feel a very strong sense of "I don't wanna" when I'm wiped out from lack of sleep, which is a constant issue for me.
Ultimately for me, what it really boiled down to was going to bed early, eating smaller meals more frequently throughout the day, and doing at least a bit of cardio every day to keep my digestion going (at minimum, I do best with at least 30 minutes of cardio a day if I'm not doing anything else physically). Sugar hasn't really had an impact on me, surprisingly, because it's lauded as the great evil of our times. I don't think it's great for you, but neither is ingesting preservatives, smoking, drinking, doing drugs, not exercising, not getting enough sleep, etc., so it's all kind of a balance.