In which direction does friction act?
In the opposite direction to the movement between the two objects - in this case - in the opposite direction to the pendulum's blue velocity arrow.
With sufficiently high levels of friction you'll the red friction force arrow grow as the pendulum swings through the bottom of its swing.