The Sentinel feat, found in the Player’s Handbook on page 114, allows an opportunity attack whenever you’re within 5 feet of a straight-line path and your opponent comes into that area. If your opponent has already passed through this area (or if there is no straight line between where you are standing and the target), then you can’t make an opportunity attack.
An opportunity attack does not use your reaction for which to make the Attack Roll bonus (+4), but does still allow you to add other damage bonuses normally associated with melee attacks (such as a Battlemaster’s Superiority Dice). You still get disadvantage on this attack if the target is out of reach or more than one size category larger than yourself, though. For example, you couldn’t make an opportunity attack against a Huge dragon (which is normally out of your reach), but could against a Large humanoid (which fits in one size category above you).
The Sentinel feat also doesn’t remove the bonus when it comes to moving away from an opponent. If this triggers, then the bonus still applies.
Only one opportunity attack can be made per turn – regardless of how many creatures end up in your threatened area – but more than one opponent can trigger this. For example, if two creatures move into a straight line that puts them within 5 feet of you, then you’ll get an opportunity attack against each of them even though they’re technically occupying the same space.
If this was not clear enough, here’s the exact text from the Player’s Handbook: “When a creature you can see moves out of your reach, you can make an opportunity attack against it. The creature must remain within 5 feet of you, unless it is flying. You make the attack immediately after the creature leaves your reach.”
This is a key point to remember for all martials who get their opponent in melee range, not just Battlemasters. If you’re grappling or being grappled then your opportunity attacks do not work against that target. However, if two creatures are grappled and one of them moves out of your reach (or 5 feet), then you can make an opportunity attack against them.
For Battlemasters, the Sentinel feat also works when you’re in melee with someone else, but it’s very situational when that happens. You would need to be surrounded by at least three enemies within your reach before this bonus applies. That means one of them could easily move out of range and take advantage of that fact. You can’t just move into reach of another opponent and have that count as triggering the opportunity attack, so keep your options open whenever possible.
The Battlemaster is all about mobility, so any time you find yourself in melee with more than one opponent then it’s likely a mistake on your part. If this happens to be the case, then you have to be very careful about your decision making or take advantage of any openings that may come your way.