Yeah, I got it. There are just too many parameters you have to know.

I switched my ATV to 720p-50Hz. Next I activated in XBMC the vsync option. So the refreshrate in XBMC changed from 100Hz or 60Hz to 50Hz. The adjust refresh rate button in XBMC is not working or takes no effect.

So now my videos with 50Hz are looking fine.

But if there are very high motion scenes there are still droped frames. But I think this is realy a problem of the missing hardware acceleration.

So thanks for help very mutch!!!