Position a rack in the center of the oven and preheat to 350 degrees F.
Brush a 9 inch spring form pan or an 8x8 baking dish with olive oil. Sprinkle evenly with flour and tap out the excess.
Whisk together the flour, baking powder, baking soda, and salt in a large bowl. Make a well in the center. Whisk together the buttermilk, oil, honey, and eggs in another bowl, pour into the well, and stir just until blended. Fold in the fruit and citrus zest, being careful not to overmix the batter. Pour the batter into the prepared pan and toss the almonds evenly over the surface.
Bake until the center springs back when pressed gently, about 30 to 35 minutes. Cool about 1/2 hour before serving. Ice cream or a dollop of freshly whipped cream, sweetened with a bit of honey, complements this nicely. Serves 8 - 10
*Alternately, use 1 1/2 cups chopped figs with zest of one orange.