As we see in the flashback Calvin constantly plays chess with Hobbes, it's all he had to do in the hospital. We also see him play chess with Rosalyn when she first babysits him.
We start with him continuing to talk during his narration, then Rosalyn asks "So did you?" and he asks, confused, blinking "What?" and she asks if he stripped and swam naked to his house, and he says no, he just left that place, but then he blinks and looks around like he's confused again. He puts a hand on his head and winces, she asks what's wrong, he says nothing, sometimes he think he's just losing his mind (or maybe we save this for when he throws Hobbes) and Rosalyn says he shouldn't be worrying about losing his mind because he's already insane, to which he sarcastically replies something like "Thanks I forgot for a moment..."
At the beginning of this scene when he says no, he just walked home, Rosalyn waits and then says "It's your move, Calvin." and he realizes he's holding a chess piece and already knows where it's going to go, when he quickly puts it down she seems annoyed that he made a good move.
When she picks up the doll she says he really tore it up, he get into an argument? Calvin replies something like that.