All

Is it possible to book a USB HDD with 10.5 installed on it with appleTV ?

almost definitely not, the appletv only has 256 megs of ram and im pretty certain leopard will NEVER run with that meager amount of memory.

what about using a USB HDD to install OS X on. How can you make it book using appletv ? will the memory still cause an issue ? or will it boot ?

It is possible to get 10.4 to run externally, i don't think its worth it though the performance will be terrible. You need to fashion it the same way as a patchstick with a partition in the recovery format for the appletv and if done properly it will boot from start. i used to run a hacked 10.4.8 to do my patching pre patchstick era (i think 10.4.7 or 10.4.8 is the latest that will run) but i could only ever get it to boot once before having to do it all over again. Bottom line its a lot of effort and the end product isnt worth it, not even a little.