The term mail server, according to the Wikipedia, is also used to mean a computer acting as a mail transfer agent (MTA) that is running the appropriate software. An MTA is a computer program or software agent that transfers electronic mail messages from one computer to another.
A more simple definition is given by SearchSOA.com. According to this site, a mail server (also known as a mail transfer agent, a mail transport agent, a mail router or an Internet mailer) is an application that receives incoming e-mail from local users (people within the same domain) and remote senders and forwards outgoing e-mail for delivery. A computer dedicated to running such applications is also called a mail server.
The mail server, in conjunction with other programs, works to create a messaging system that contains all the applications needed to keep e-mail messages moving. The messaging system uses SMTP (Simple Mail Transfer Protocol) or ESMTP (extended SMTP) for sending e-mail messages, and either IMAP (Internet Message Access Protocol) or POP3 (Post Office Protocol 3) for receiving messages.
List of some mail server programs:
Eudora Internet Mail Server
IBM Lotus Domino
Ipswitch IMail Server
Microsoft Exchange Server
Sun Java System
XMS Email Application Server