In order for the period to be independant of the size (or amplitude) of the swing, the force returning the pendulum to the middle position needs to be proportional to the displacement from the middle position.
The returning force, however, follows a sinusoidal not a linear law. For small swing amplitudes this isn't far off the linear ideal, but as the size of the swing increases, the returning force is significantly less than this - so the pendulum hangs around at the end its swing for a little longer, and the whole swing takes longer.
To observe this in the Posn:Force phase diagram; set friction to zero, reduce the pendulum's length and mass, and set it swinging with a swing going up to the horizontal.
You'll notice that the higher the pendulum's swing, the less the returning force achieves  straight line proportionality - and the longer the swing takes.