I am not staring at you, really
When I talk to anyone - a pretty girl I just met, a wrinkly old man, my seriously ugly friend Rust, my beloved mum - I look at their faces all the time. I give each one equal attention in terms of eye contact. Why?
One thing people with hearing loss have in common - no matter what they call themselves, HOH, HI, Deaf etc. - is the fact that we need to look at others’ faces during dialogue. Oftentimes, this may be miscontructed as ’staring’. Hearing people in conversation generally do not look at each other all the time when they talk; in fact, it’s possible to carry on a conversation without looking at the other party’s face at all.
So then, people with hearing loss (especially the Deaf) wonder why the hearing doesn’t look at one another when they talk, and in turn, the hearing wonder why people like me are staring at them during conversations. Alas, this sets the stage for misunderstandings and more.
The simplest and most fundamental answer to your question is that hearing people use prolonged eye contact to indicate sexual attraction. Hearing people have a very complicated set of well understood but unwritten rules concerning how eye contact is made and how long it is maintained. Even a slight deviation from these rules can lead to a socially uncomfortable situation. For this reason, hearing people must break eye contact regularly throughout a conversation in order to adhere to these rules. Among hearing people, prolonged eye contact is either interpreted as sexual attraction, or it is considered weird.
Also, prolonged eye contact among hearies can also be perceived as a sign of aggression or domination. A good example is during an interrogation.
