Place the potatoes in a large pot and cover with cold water. Bring to a boil over medium-high heat and cook until the potatoes are tender, about 15 minutes. Drain the potatoes, then rinse with cold water.
In a large bowl, whisk together the mayonnaise, sour cream, mustard, vinegar, sugar and celery seed. Season with salt and pepper.
Add the cooked potatoes to the bowl of dressing and stir gently to combine. Fold in the crumbled bacon.