It seems maybe you're only considering this from an astronomical observation perspective?
Not entirely. I read that when our soul leaves, there's a cascading glow of light that can be detected throughout the intestines and some have surmised that it's the soul leaving the body. If we suppose that our soul is nothing more than light, then there is a possibility that we are all nothing but old light waves traveling through space and time and coming into existence every time cells are created.
This could mean:
1) Creation of cells involves a process that captures energy from space (left by traveling light waves).
2) If this lingering "light energy" is related to the passage of time, more lingering energy (fewer beings in existence) would result in slower passage of time, theory being that this unused energy acts as a deterrent to time.
3) More beings in existence would reduce this resistance to time hence the local time would speed up.
4) For an observer in a place with relatively little life, the place hustling and bustling with life would seem to exhibit accelerated time.
5) When scientists say millions and millions of years of evolution, it's coz there were fewer beings in existence so time was really, really slow. From our current perspective, that time seems on the order of millions of years.
6) If there is indeed no time inside a blackhole, it means that light entering a blackhole "expands" and "saturates" the virtually infinite space inside the blackhole. This light saturation prevents time from flowing at all.
7) If light rays could be made to retract over their exact path from source to destination with no further linear traversal, it could lead to time reversal. But doing so might require the intense pull of a blackhole. So in theory, do something or make something happen, create a small blackhole and then make it vanish into thin air before it is able to absorb the light rays permanently and you have a mini time machine at your disposal.
8) It follows that immortality can be achieved if a body gained the ability to create microscopic or even subatomic blackholes and make them vanish, over and over in an infinite cycle, preventing the local time inside the body from flowing normally.
9) Accelerated aging could be nothing more than light energy escaping from a body, reducing the resistance to time and causing time inside the body to accelerate and speed up decay of organic matter.
10) When we are young, it seems like we have all the time in the world. It's because we have more light "inside" us. The more light we lose with age, the more time seems to speed up for us until that very last glow in the intestines which signals complete loss of light energy, at least from the cells of our body (not the micro-organisms which still have some light in them so they get their chance at living the good life devouring the remains of the person/being).
11) We think of food as fuel that creates heat in our mitochondria and drives biochemical reactions. But this heat radiation causes an outward flow of the body's light energy, accelerating the local time and thus accelerating aging. This is why some indian gurus are able to live with just sunlight falling on their bodies. They have tuned their bodies to soak up as much light energy as possible which creates resistance to time flow inside their bodies and thus they are able to live longer without needing food because they are limiting the outward flow of energy and instead saturating their bodies with energy. In essence, they are making their bodies act as a sort of a quasi-blackhole where time becomes extremely slow.
There are definitely going to be chinks in this theory. Feel free to poke and prod and expand my knowledge and help me refine my thought process.
The past is whatever/whenever the interactions took place that caused those light waves/particles to make their journey from there to the observer.
That assumes an observer who is used to the forward direction of time. Theoretically, an observer that is able to conceive time backwards could "look" into the past. Similarly, an observer with "hyperthought" capability could peek into the future. Such hyperthought could be possible by creating microscopic blackholes in the brain that cause time to fly momentarily, affording the neurons of that person/being the capability to get glimpses of the future. In other words, a precognitive person or precog.