It depends on how you imagine time travel and causality. Is it a stable time loop? Or do you visit another version of reality with different outcomes? When you travel, are you unraveling the course of history to be redone? Or are you visiting an unyielding etching of the timespace continuum? If time is a set of dimensions, as all modern physics supports, then theoretically it wouls be possible to move through those dimensions in all directions. Special relativity confirms that movement affects how you move through time, but if you go backwards in time, you are still moving forward from your own reference point. That's the only way to retain your memories.