The rumors began swirling with the pair after she visited Charleston, but intensified once Conover joined Summer House, and Cavallari became part of the conversation.