Speculation is building within the Coalition that former cabinet minister Christian Porter will not recontest the next election.