In a medium saucepan off the heat, whisk together the milk, both sugars, and cornstarch until smooth. Whisk in the egg yolks, lemon zest, and salt. Set over medium heat and cook, whisking frequently at first and constantly toward the end, until thickened and the whisk leaves a very defined trail in the pudding, 9 to 12 minutes.

Remove the pan from the heat, add the lemon juice and butter, and stir until incorporated. Pour through a coarse sieve into a large serving bowl or four individual serving dishes. Let cool to room temperature. Refrigerate, loosely covered, until set and thoroughly chilled, at least 2 hours or up to two days. Serve chilled with whipped cream and candied violets, if you like.