You are doing everything right. The thing is you cannot bring the book back to present with you. Not yet. Put the book back in place, and Death poker party should cease to come up.
By the way, while in the past, did you read the sign on the toilet door
