As stated it depends both on the species as well as the individual animals disposition. Some really do not like to be handled and display their displeasure in forms like struggling, biting, scratching, musking or just taking a good healthy dump on you!

Some will tolerate it and some (bearded dragons for instance) do seem to outright enjoy it. Probably more they enjoy the heat of your body!
As far as them always seeing you as a predator, a large percentage of herps are captive bred and have learned from day one of their lives that humans are the source of food, NOT a predator and as such do not fear us. That is one of the reason that captive bred animals make much better "pets".
Animals like iguanas if not handled regularly will become aggressive and unmanageable, while chameleons well actually stress, sicken, and die, if handled too much.
You have to let your knowledge of the species and observation of the particular animals tell you what is right for them. There is no blanket across the board rule that you can or cannot can handle every species.
That would be FAR too easy!
