Check inside the spindle where the Bit is inserted to ensure there are no dirt or obstructions embedded inside that prevent the Bit from sitting correctly.
Depending on the Impact Driver model, pull the retaining collar forward and insert the Bit fully, making sure the hex base of the Bit touches the bottom of the spindle.
Release the collar and pull on the Bit to check it has been positioned properly.