Suppose workers at company X strongly prefer to work with men. So, the algorithm learns men do well at company X, and women don’t. The algorithm will be predictive of performance, but not for the expected reasons. This is why naive algorithmic optimization is problematic.