Since the Xbox One has the ability to send both game audio and chat audio to the controller, you must make sure that the Xbox is not outputting any game audio to the controller.
When using the Stereo Headset Adapter, press and hold the button with the “chat” icon until you hear a beep. This will prevent game audio from coming through the controller.
When using the newer Xbox One controller with the 3.5mm jack built in, while the headset is turned on simply unplug the 3.5mm cable from the Xbox One controller and then connect it again. The controller will now see the LS30/LS31/LS40/LS41 as a chat only headset and only output chat audio.