Dataset Bias: Garbage In, Discrimination Out

In 2018, researchers tested commercial face-recognition systems and found error rates up to 34.7% for darker-skinned women. For light-skinned men: under 1%. The models weren't broken. Their training data was. That gap, between who your data represents and who your model serves, is dataset bias, and the consequences run far past a broken layout.

The "Works on My Machine" Problem

You know the drill. You build a beautiful UI, test it in Chrome on your MacBook, ship it, and immediately get bug reports from Firefox users, Windows users, and screen reader users. The problem wasn't your code. It was your testing coverage. You only validated against a narrow slice of your actual user base.

Dataset bias is the same trap. If your training data only represents certain demographics, your model will "work" for those groups and silently fail for everyone else.

Three Types of Bias

Selection Bias

Your dataset doesn't represent the full population. If you train a facial recognition model on photos that are 80% light-skinned faces, it will perform poorly on darker skin tones. The frontend version: testing only on desktop when 60% of your traffic is mobile.

Measurement Bias

Your data collection method itself introduces distortion. If you measure "employee productivity" using metrics that favor in-office workers, remote workers will be systematically rated lower. Not because they're less productive, but because your measuring tool is broken.

Historical Bias

Your data faithfully reflects an unfair world. If you train a hiring model on historical hiring decisions, and those decisions were biased against women in engineering, your model will learn to replicate that discrimination.

// Auditing a dataset for representation bias
interface DataPoint {
age: number;
gender: string;
ethnicity: string;
income: number;
label: number; // 0 = loan denied, 1 = loan approved
}

function auditRepresentation(data: DataPoint[]) {
const total = data.length;

// Count representation by group
const genderCounts: Record<string, number> = {};
const ethnicityCounts: Record<string, number> = {};

for (const point of data) {
  genderCounts[point.gender] = (genderCounts[point.gender] || 0) + 1;
  ethnicityCounts[point.ethnicity] = (ethnicityCounts[point.ethnicity] || 0) + 1;
}

// Compute proportions and flag underrepresentation
const report: string[] = [];

for (const [group, count] of Object.entries(genderCounts)) {
  const proportion = count / total;
  report.push(`Gender "${group}": ${(proportion * 100).toFixed(1)}% (${count}/${total})`);
  if (proportion < 0.1) {
    report.push(`  ⚠ WARNING: "${group}" is severely underrepresented`);
  }
}

for (const [group, count] of Object.entries(ethnicityCounts)) {
  const proportion = count / total;
  report.push(`Ethnicity "${group}": ${(proportion * 100).toFixed(1)}% (${count}/${total})`);
  if (proportion < 0.1) {
    report.push(`  ⚠ WARNING: "${group}" is severely underrepresented`);
  }
}

return report;
}

// Also check label distribution per group; disparate outcomes signal bias
function auditOutcomes(data: DataPoint[]) {
const groupOutcomes: Record<string, { approved: number; total: number }> = {};

for (const point of data) {
  const key = point.gender;
  if (!groupOutcomes[key]) groupOutcomes[key] = { approved: 0, total: 0 };
  groupOutcomes[key].total++;
  if (point.label === 1) groupOutcomes[key].approved++;
}

for (const [group, stats] of Object.entries(groupOutcomes)) {
  const rate = stats.approved / stats.total;
  console.log(`${group}: ${(rate * 100).toFixed(1)}% approval rate`);
}
}

Real-World Consequences

These aren't hypothetical problems. Amazon scrapped an AI recruiting tool that penalized resumes containing the word "women's" because it was trained on a decade of male-dominated hiring. Healthcare algorithms allocated less care to Black patients because they used healthcare spending (influenced by systemic inequality) as a proxy for health needs.

You would never close a bug report with "works for me." Hold your training data to the same standard: works for whom, exactly?

Challenge

Audit a dataset and identify representation gaps across demographic groups.