What determines the size of the frictional force?
The nature of the the interacting objects, and their relative speed. In this case - how streamlined the pendulum is, and perhaps how dense the air (air friction would be zero in a vacuum).