The interesting thing is that once you have a contactless card, it is active in your pocket/bag/wallet. If someone really wants to steal from it and has the equipment, they can steal from it just by being close enough with enough power to the reader.
Whether you use it or not is immaterial. I noticed that there are a few relatively good quality NFC blockers out there which you can put your card inside and block any reader but the one you do want to buy on.
Contactless is, essentially, only a problem if you actually lose the card. Where someone can go around making £30 purchases until the card is blocked. somewhat different to losing your card and pin, where the card can be compromised for much larger single purchases.
Essentially if the bank has given you a contactless card, then you might as well use it. If you really don't like it,
you can request a non contactless card.
I have a bigger issue that my BOS contactless will not work in France. However my Nationwide one will. I tend to use it for the vending machines in Paris, where I arrive without cash and want a drink before I get on the train and don't want to spend €1.3 more for the drink on the train.