James,
Going by the picture in your Album, I think that your lathe pulleys are designed for a vee belt.
Have a look at the picture about 1/3 of the way down this page, to see what a groove properly shaped for round belt looks like.
As it is; your 6mm belt is probably riding on the top of the vee, and a 4mm belt may be better … BUT it might jam, because the vee is too narrow angle.
Now is probably a good time to take some measurements of the groove. 
MichaelG.