Features are often engineered by combining a few of them, one of which could be say, gender. And because of the way it is engineered, the composite feature may in fact show correlation with outcomes even when it is not immediately obvious that it is the gender component which is driving this biasedness. Removing bias-inducing data should be part of data cleansing, the very first stage, because it becomes too difficult to do later on.