I too have noticed the large guys on little pasos with all the comments in total agreement.
I think its worth adding that if a 'chunky' rider is atleast well balanced and the horse and rider are united so to speak it wouldn't be as bad as if they were a very poor rider. Or the horse didn't respect the rider.
If you watch any shows about China you'll see pretty large people on what appear to be ponies as I guess thats all they have out in the country.
A rather large man once wanted to ride my TB who is big and I flatly said no. (of course I did some tap dancing around the real reason!)
Maybe if the riders foot goes all the way down to the horses knee - thats a sign you need a larger horse.