In a medium saucepan over high heat, bring the potatoes to a boil in enough salted water to cover them by 1 inch. Reduce the heat to a brisk simmer and cook until tender but not falling apart, about 5 minutes. Drain. Set aside.

Pat the fish dry and season with 1/2 teaspoon salt and 1/4 teaspoon pepper. Heat 1-1/2 tablespoons of the olive oil in a 12-inch nonstick skillet over medium-high heat until shimmering hot. Arrange the fish skin side down in the pan so the fillets fit without touching. Cook undisturbed for 3 minutes. Flip the fillets and cook until the fish is cooked through, an additional 2 to 3 minutes. With a slotted spatula, transfer the fish to a serving platter or plates.

Add the remaining 1-1/2 tablespoons oil to the pan and heat until shimmering. Add the potatoes and rosemary and cook, flipping occasionally, until the potatoes are tender, 3 to 4 minutes. Add the olives, parsley, balsamic, and a pinch of salt and pepper and stir gently to heat. Arrange the potato mixture around the fish. Serve garnished with the lemon wedges.