While the Surveys do show that men are more opposed than women. But a large percentage of women do in fact care.
"
But surveys show that those most concerned about transgender people using women’s restrooms aren’t women."
These surveys were just opinion polls.
A Reuters survey, meanwhile, found that
44 percent of women were okay with letting a transgender person use the women’s restroom, while
39 percent said they’d rather people use facilities that match their birth gender.
The 44 percent OK with it and the 39 percent not OK with it are probably within the margin of error anyway.
But don't make the assumption of some and say that:
"There are literally no women who have this fear". Quite a few actually do regardless if it is justified or not.
In my opinion they should make separate "Alphabet" Restrooms for these people so that nobody is imposed upon.