Primary education is heading for becoming state childminding, as I've said many times.
The problem with physical education is the OFSTED mentality. "Things that get measured, get done".
So that if the focus is on English and Maths, the kids get crammed for English and Maths tests. Are the children measured and the school marked up our down on their physical fitness? No.
So as Jo posted on here a while ago, PE or games are among the first lessons to go when the school wants to get some extra cramming done for those subjects which are measured.
The problem is that education is a political football; every government wants to "leave their mark" on it; and many of them have, unfortunately.
Ofsted is a bullying organisation run by a bully; you can always rely on "Sir" Michael Wilshaw for a quote knocking teachers.
The pressure is coming from the top down remorselessly; from the government through Ofsted, who pressure Head teachers and governors, who pressure teachers. What this will lead to inevitably is pressure on children who should not even be be in formal learning.
Physical education has been in declne since the 80's, if not before.