Campaigners have expressed anger over proposals to move a Post Office away from a historic High Street in Lowestoft.