FvM brings up a good point. Several manufactures of terminal/console servers offer chipsets which are designed specifically to provide the features you described.
I just took a quick look at a Lantronix development kit sitting on one of my shelves, which has a single 100Base-T Port, two RS-232 Serial Ports and the ability to be configured route packets with a specific port number to one or other RS-232 Serial Ports.
You may want to look for a "canned" module from Lantronix or Moxa as FvM suggested.
BigDog