It turns out that converting mine to work without the pencil was not much trouble. I just swapped in square axles with Chinese coin style collars on them to give a round surface to pivot on at the necessary spots.
And just not adjusting for kerf on the round discs and holes should give it enough play to rotate freely.
ETA: the larger collars on the outside ends of the smaller pivot are just to keep it from walking side to side.