Is there any other easy to install server that could be run on the Apple TV that would allow faster file transfers into the box? SFTP speeds seem to peg out around 3-3.5MB/s, where itunes syncing is 9-10MB/s. Standard FTP or something would be nice, but what are the other options?

Background info:

100mbit wired connection, computer feeding it is gigabit that can dump out 40-45MB/s. Only bottleneck is the SCP/SFTP on the aTV right now.

enable ftp server ?

http://wiki.awkwardtv.org/wiki/Enable_FTP_Server

Tried that out, little bit better around 5MB/s. Any ideas on what iTunes is using to saturate the 100mbit connection?

According with InsanelyMac:
TCP port 3689 is used to communicate with iTunes while using the iTunes Library Sharing feature.
TCP port 3689 is used to communicate with iTunes while using the iTunes Library Sharing feature.
UDP port 5353 is used by Apple TV for automatically finding computers with iTunes on your network using Bonjour.
TCP port 80 and 443 are used for basic and secure communications with the iTunes Store via the Internet.

Have you tried AFP ?

i've used AFP and 11n and the transfer rate is arount 10-11mb/s

You should get better performance with ATVFiles plugin, which enables the AFP server on the ATV. Note that you will never actually get the full 100 mb/s, that is a theoretical limit - actual limits are more like the 9-10MB/s you are getting when syncing with iTunes. The overhead with scp is the encryption/decryption happening at both ends. Rsync is not an option until someone figures out how to enable it on ATV.