Mix the olives, 1/3 cup of the oil, the lemon zest, lemon juice, garlic, and chile flakes in a small bowl with a fork; the vinaigrette doesn't need to emulsify. Season with salt to taste.

Pull out any bones in the fish with needlenose pliers or tweezers. Season both sides of the fish with salt and dredge very lightly in the flour. Heat 2 tablespoons oil in a large nonstick skillet over medium heat. When hot, add two of the fillets, skin side up; cook until light golden brown, 4 to 5 minutes. With a thin slotted metal spatula, turn the fish and cook until the second side is lightly browned and the fish is cooked through, about 3 minutes. Transfer the fillets, skin side down, to a plate, cover to keep warm. Repeat with the remaining fillets.

Stir the vinaigrette, toss enough of it with the arugula to coat lightly (1 to 2 tablespoons), and portion it among four plates. Lay the fish on top, spoon the remaining vinaigrette over the fillets, and serve with a wedge of lemon.