You do keep all your memories. In vivid detail, they are all there. It is just a matter of recalling them.
The best way to recall old memories is to associate sensory data. A sound, a smell, a texture can help activate an old memory. Lets say for example, you have a memory of when you were 6 years old sitting next to a river bed petting a turtle that you found in the road. You were about to set him in the river and that was when you imagined being a turtle. This could be a rich and detailed memory that could come rushing back to you simply by feeling the texture of a turtle shell. If there is a sound that goes with it, such as the same birds chirping, it would increase the likelihood of the memory coming back. As would the smell of the plants around the river, or the mud, or anything like that.
But there has to be an emotional component to it as well. You have to be feeling something, and you have to remember what that feeling was.
This is mostly just theorizing, but the way I believe the mind's memory works is that it compresses old memories by stripping out sensory data and replacing that data with something like a tag, ie a small code snippet that says "insert sensory data here". In order to recall the memory, you need to re-input the sensory and emotional data so that those tags get replaced by real data. Once you have all the data, the fine details are reconstructed into what feels like a whole unbroken and complete memory, and that is essentially what it is.