Access Control on listening ports

I have a few servers that have ports open and listening for data, but

I've found some security vulnerabilities with leaving them open. The

ports are needed to talk between several known servers. Does anyone

know a way to make it so only the servers that need to talk to each

other can access the open port, any other server/machine that tries to

hit the port gets server refused or something of that nature?

I could have it done with a lot of firewall rules on my routers, but I

really want this taken care of on the servers rather then through the

routers.

Ideas oh smart admins?