Well this is a tricky one and I also found myself in the same situation as you in the past flydrive.
It does not mention anything about premium baghage delivery on the Executive Club site, only that you can use the Club Check in if you are Silver and FIRST check in if you are Gold and that you can also use the Premium counters specifically at T5 if your bag is lost or damaged, not sure if that is such a massive benefit though.
Fact is sometimes they put a priority label on my bag when I travel in Traveller, sometimes they do not. But they always do when I fly in premium cabins, so sadly no consistency here. I was also told before that the label out of LHR makes not the slightest difference as the bags get sorted by the barcode on the actual bagtag and then prioritise them into the Premium Containers if you are travelling in a Premium Cabin. However on some routes, especially on European and Domestic flights there simply is no premium Container which probably was the case when your bag came off amongst the first even though it was not labelled. At some outstations, where the BA flights are handled by non BA staff they simply do not make the effort and prioritise bags, sadly.
So yes I agree it is very hit and miss and inconsistent but then again, so far, luckily, I never had to wait too long for my bags. Hipe my lengthy posting makes sense.