In a medium saucepan, combine the sugar, cornstarch, and salt. Gradually stir in the milk until smooth. Cook over medium heat, stirring constantly until mixture thickens and boils. Boil for 1 minute. Remove from heat.
Stir a small amount of the hot mixture into the egg yolks; return all to the pan, stirring constantly. Bring to a gentle boil and cook for 2 minutes longer.
Remove from heat and stir in butter and vanilla extract. Cool slightly.
Arrange banana slices over bottom of graham cracker crust; pour warm filling over top.