"Big Brother Complex" is a problematic term due to usage of "complex" in some circles to refer to someone who idolizes (not necessarily romantically) the figure in question (their brother). All of these tropes can also apply to a big sister as well.
The trope you're looking for could be any one of these:
- Aloof Big Brother: A literal big brother who has no interest or time for the younger sibling
- Big Brother Bully: The big brother bullies or torments the younger sib.
- Big Brother Instinct: A male character has an instinctive desire to be a big brother to smaller, younger, weaker or female characters.
- Big Brother Attraction: For some reason a character wants to marry their sibling.
- Big Brother Worship: "My big brother is the best and I think he's perfect."
- Disappointing Older Sibling: An older sibling lets down their younger sibling.
- Sibling Incest: A romantic or sexual attraction between siblings.
- Sibling Rivalry: Siblings fall out with each other.
- Incest Subtext: The implication of same.
- Big Brother Mentor: a big brother figure who is also the mentor to the younger character.
- Father, I Want to Marry My Brother: A child too young to understand everything that marriage entails makes this announcement.
- Knight Templar Big Brother: A Big Brother Instinct carried to ridiculous extremes; the "big brother" will absolutely destroy anyone who has wronged or hurt their younger sib, no matter how minor the offense.
Other tropes that may be what you were thinking of:
- Big Brother Is Watching: An all-seeing governmental agency monitors everyone.
- Big Brother: The TV show.
If an internal link led you here, please correct it to point to the right page.