Preheat the oven to 325°Toast the almonds on a heavy rimmed baking sheet until golden brown inside, 25 to 30 minutes. Break one open to check for doneness. Let cool completely.
Raise the oven temperature to 350°Butter the bottom and sides of a 9-inch round cake pan, then line the bottom with parchment paper.
Put the whole almonds in a food processor with one-third of the cake flour and process until almost as fine as sand. In a bowl, whisk together the remaining cake flour, the baking powder, and salt, then whisk in the ground almonds until well blended.
In a stand mixer fitted with the whisk attachment, whip the eggs on high speed until well blended. Add the sugar gradually, then add the vanilla and orange zest. Continue whipping on high speed until the mixture triples in volume, about 3 minutes.
On low speed, add the flour mixture in three batches alternately with the olive oil in two batches, beginning and ending with the flour mixture and beating just until the batter is blended. Pour the batter into the prepared pan. Sprinkle the top evenly with the sliced almonds, if desired.
Bake until the cake is golden brown and just firm to the touch in the center, 40 to 45 minutes. Let cool in the pan on a rack for 10 minutes, then invert onto another rack, remove the parchment paper, and invert again. Finish cooling right side up on a rack. Transfer to a serving plate and cut into wedges to serve.