I’d make a couple points against that

1.We don’t “all agree” we’re born with specific sexual preferences. That’s not objective fact, it’s a hypothesis that can’t really be proven.

I’m Bi and don’t inherently know I was born this way. I have tons of personality and character traits that are impossible to assign to nature vs nurture, including sexual orientation.

2.Gender tropes change between cultures a ton, with various expectations and preferences. Differences in sex doesn’t just change by region like gender-tropes, humans are humans. Hence sexual preference based on sex and physical bodies is arguably more immutable