Could I attach a wooden fence panel directly onto a brick wall?

I've a pretty crappy brick wall in my backyard covered with ever flaking pebble dash. To save time, cost and effort is it possible to attach some wooden fence panels to hide this? I was thinking either directly to the wall or by first attaching fence panels to the wall to create a gap between the wood and the brick. Any thoughts?