I seriously doubt that you could develop serious hypoxia (low content of O2 in the blood/tissues) or hypercapnia (high concentration of CO2 in the blood/tissues) by sleeping deep in your bag and all other things were equal.
Even if you could you would have an overwhelming urge to escape the bag and reverse the condition. Though very high concentrations of CO2 will make you sleepy/drowsy, the initial physiological response is increased respiratory rate, shallowing of volume of air inspired and a hightened physiological state...think panic (in order to blow off CO2...get rid of it out of the blood stream). This state is even more dramiatic with hypoxia. So even if you were asleep you should theoretically wake up gasping for breath before you suffocate.
Small babies lack this response so the situation is much different with them. ALso, I think you might be at risk if you were already seriously hypothermic (clinically dangerously cold) as you would be already in a drowsy depressed state.
But generally speaking the small hole you describe and the breathability of your bag should keep you alive
Keeping your mouth right next to the opening should remove any risk and keep you dryer than breathing into your sleeping bag.
You can squeeze some extra warmth out of a higher temp rated bag too!
Having said that, the water resistant or water proof bivvy sacks might be a different story.
My 2c anyway.