And he actually left the final choice to Rory (so many lovely parallels with Amy's Choice) - Rory could have let her in.

That's why Amy has agency: she refused to let him-- and he listened to her.

What promise indeed? (And now I'm back to seriously contemplating that it's Amy. Because it makes so much sense.)

I can see him going because it's a fixed point, and it has to be kept fixed. But I'm presuming there's more to it than that...

There has to be. It's too flat otherwise. From how he acts (bowing his head, standing there), there's an emotional reason.