Some places already have gender neutral restrooms and haven't had too many issues. Some people might think something along the lines of what would prevent a man raping a woman in a public restroom if restrooms were gender neutral - the same thing that's preventing a potential rapist from walking into a women's public restroom. The little plastic sign on the door doesn't stop someone who is considering committing a violent act.
Next week, I will be attending the National Gay and Lesbian Task Force's Creating Change Conference in Detroit. The public restrooms will be designated as gender neutral during the conference. I haven't heard of any problems in the 3 conferences I've attended.