It is derived from the time of sailing ships when sailors would, occasionally but regularly, be allowed time to "make and mend" their uniforms, which were not then supplied by the Royal Navy.
Some sailors were, nevertheless, "on watch" to work (sail) the ship. The designated watch of sailors were still required to "turn to" if the ship's officers had to change the arrangement of the sails or rigging.
This afternoon off was often on Sunday. However, this time off was not a designated "holy day".