Seph wasn't near the castle but, he was in Radiant Garden.

He wandered about in Rainbow's End. It's emptiness disturbed him.

He'd already looked in the most likely places to find Jagannatha besides this world. That the Darkness seemed to have chosen the world many of the Refugees considered home...