Grease a 13x9-inch baking dish; set aside. Preheat oven to 350°F.
Arrange bread on a large, rimmed baking sheet. Bake until dry, about 14 minutes. Cool completely. Transfer bread cubes to the prepared baking dish.
In a large bowl, whisk together milk, eggs, granulated sugar, brown sugar, vanilla extract, cinnamon, salt, and nutmeg. Pour over the bread cubes; gently toss to combine.
Use a spatula to gently press down the bread so that it’s mostly covered and saturated. Cover and let stand for 30 minutes.
Remove the cover; bake until puffed and golden brown and a knife inserted in the center comes out clean, about 50-55 minutes. Let stand 30 minutes before serving. Dust with powdered sugar, if desired, and serve with bourbon sauce, vanilla ice cream, and/or whipped cream.