To “fly off the handle” literally refers to cheap axe heads flying off their handles when swung backward.