The combining form uni- does normally mean 'one, having or consisting of one': it comes from Latin unus 'one'. It forms words such as unicycle, a term for a cycle with just one wheel, and unicellular, meaning 'consisting of a single cell'. And in fact the 20-volume historical Oxford English Dictionary contains entries for the words unisexual, meaning 'of one sex or relating to one sex' and unisexuality, meaning 'the state of being unisexual'. Both these words date back to the early 19th century.
Unisex is a much newer word: it was coined in the 1960s and originally used in relatively informal contexts. Its formation seems to have been influenced by words such as union, united, and universal, from which it took the sense of something that was shared. So unisex can be understood as referring to one thing (such as a clothing style or hairstyle) that is shared by both sexes.