Fairness & Bias - Technical Essentials
Fairness & Bias Evaluations for LLMs [Unstructured Data]
Fairness & Bias Evaluations for Traditional Models [Structured Data]
Metrics
Based on the selected sensitive / protected attribute for the given dataset, the positive / favorable outcome distribution is compared with the rest of the groups in the dataset and the metrics are calculated.
a. Pretrain & Post-train Methods:
i. Statistical Parity Difference:
The Statistical parity difference metric calculates the difference in the ratio of favorable outcomes between privileged groups and un-privileged groups.
Interpretation
An SPD of 0 indicates perfect statistical parity (no difference in outcomes between groups).
A positive SPD means the privileged group has a higher rate of positive outcomes.
A negative SPD means the unprivileged group has a higher rate of positive outcomes.
The further the SPD is from zero, the greater the disparity.
ii. Disparate Impact Ratio:
The Disparate Impact Ratio metric calculates the ratio of favorable outcomes between privileged groups and un-privileged groups.
iii. Smooth Empirical Differential:
SED calculates the differential in the probability of favorable and unfavorable outcomes between intersecting groups divided by features. All intersecting groups are equal, so there are no unprivileged or privileged groups. The calculation produces a value between 0 and 1 that is the minimum ratio of Dirichlet smoothed probability for favorable and unfavorable outcomes between intersecting groups in the dataset.
iv. Four Fifths:
This function computes the four fifths rule (ratio of success rates) between group_unprivileged and group_privileged. The minimum of the ratio taken both ways is returned. A value of 1 is desired. Values below 1 are unfair. The range (0.8,1) is considered acceptable.
v. Cohen’s D:
This function computes the Cohen D statistic (normalized statistical parity) between group_unprivileged and group_privileged.A value of 0 is desired. Negative values are unfair towards group_unprivileged. Positive values are unfair towards group_privileged. Reference values: 0.2 is considered a small effect size, 0.5 is considered medium, 0.8 is considered large.
Model Analysis
Equalized Odds:
The greater of two metrics: true_positive_rate_difference and false_positive_rate_difference. The former is the difference between the largest and smallest of P[h(X) = 1|A = a, Y = 1], across all values 'a' of the sensitive feature(s). The latter is defined similarly, but for P[h(X) = 1|A = a, Y = 0]. The equalized odds difference of 0 means that all groups have the same true positive, true negative, false positive, and false negative rates.
Model Mitigation (Research)
Threshold optimizer:
Threshold optimizer is based on the paper Equality of Opportunity in Supervised Learning built to satisfy the specified fairness criteria exactly and with no remaining disparity. Threshold optimizer requires the sensitive features to be available at deployment time (i.e., for the predict method). For each sensitive feature value, Threshold optimizer creates separate thresholds and applies them to the predictions of the user-provided estimator. To decide on the thresholds, it generates all possible thresholds and selects the best combination in terms of the objective and the fairness constraints. The technique has too much randomness involved, and it's still in research phase.
Generic Audit
Objective
This functionality monitors AI-generated text outputs over time to detect bias trends. It is useful for auditing generative AI systems that produce unstructured content like summaries or text responses.
How it works
1.Upload a file containing AI-generated text.
2.In the Responsible AI toolkit, we:
a.Go to the structured data tab:
b.Click on ADD DATA & fill in the details
c.Click on Generate Report
d.Once the analysis is complete, click on download to download the report
3.The system analyzes each entry for bias in the statement and compiles a report with detailed metrics and visualizations
Data & Fields
Prompt:
Column containing the text to be analyzed. It is in the Question(Q) Answer(A) format where a question(Q) was asked and an answer(A) to that question was given by the AI system. It is for this statement that we will be analyzing for bias.
How to interpret results
There are two reports generated.
The first report is an excel report. It displays:
Privileged groups (which group from the statement has been favored by the AI response)
Unprivileged groups (which group from the statement has NOT been favored by the AI response)
Bias types (What is the type of bias present in the statement. E.g. Gender bias, Socio-economic bias, Racial bias etc.)
Bias level Indicator (The severity of the bias present in the statement. Can be Low/Medium/High)
The second report provides charts showing bias distribution across the dataset. This report shows charts on:
Bias types (e.g., age, gender, race) that occur most often
How frequent are the Privileged or Unprivileged groups occurring
Severity of bias based on scores
Frequency of different bias types
Average Bias Score by Bias Level –
This chart shows how strong the bias is for each category (High, Medium, Low). The threshold (70%) is the line we use to decide what counts as serious bias. Bias Density (68%) means the average bias score of all flagged items. In this report, all cases are in the Medium range, so there are no very high or very low bias issues, just moderate ones that need attention.
Result Screenshot
Decisive Audit
Objective
This functionality checks fairness in decision-making processes, such as loan approvals or hiring decisions. It calculates success rates for different groups to identify potential disparities.
How it works
1.We upload structured decision data
2.In the Responsible AI toolkit, we:
a.Go to the structured data tab:
b.Click on ADD DATA & fill in the details
c.Click on Generate Report
d.Once the analysis is complete, click on download to download the report
3.The system:
a.Takes the decision data (model predictions and actual outcomes) along with sensitive attributes like race or gender and groups the data by these attributes.
b.It then calculates the success rate (percentage of favorable outcomes) for each group.
c.Then it compares these success rates and shows them in charts. If one group has a much higher success rate than others, it may indicate bias.
Data & Fields
1.Prediction Column: The column in your dataset that contains the model’s predicted outcomes(e.g., labels_pred).
2.Favorable Outcome: The value that represents a positive result (e.g., 1 for approved).
3.Categorical Attributes: The sensitive attributes you want to analyze for fairness (e.g., race, sex).
4.File: The structured dataset that includes predictions and these attributes.
How to interpret results
The report shows the success rate (percentage of favorable outcomes) and population share for each group (e.g., race, sex, or combined race-sex). A higher success rate for one group compared to others may indicate potential bias. Look for large gaps between groups or patterns where smaller populations consistently have lower success rates—these are signals for deeper fairness review. The report displays the following information:
Grouped Success Rate: The total number of individuals (or data points) belonging to a particular group or category of an attribute. It allows comparison of fairness between different groups. If one group has a much higher or lower success rate than others, it indicates possible bias.
Success Rate: The percentage of individuals in the entire dataset who received a favorable outcome, regardless of group membership. It gives the overall success rate of the model, i.e.: how often it predicts a favorable outcome.
Population: The total number of individuals (or data points) belonging to a particular group or category of an attribute. Population tells us how big each group is in the dataset.
Result Screenshot
Extending Basic evaluation mechanisms to useful tools
Generative AI use case evaluation and monitoring tool
Generative AI generates unstructured data. With capabilities like RAG, its now being used for making decisions in binary classification tasks as well. Based on the wide nature of the use-case, we are working on two types of approaches.
Decisive use-case flow:
Decisive solutions or classification solutions where there is a decision . Decisive solutions or classification solutions where there is a decision is made like loan approvals etc., based on set of features and rules involved. In these cases, features importance scores are to be calculated for each decision and stored. Also, the distribution of success rates, population of sub-groups involved vs categories are also recorded. Using periodic audits or live dashboards, the success rate distribution is cross-checked to ensure fairness in the system. When there are indicators of bias like system is mostly favoring a single or few groups, the audit team would cross check the results with the explanations recorded and verify if they are false alarms or an actual bias in system which would trigger further investigation.
Generic use-case flow:
If the AI model is not involved in any decision making instead it generates some text or image or any content, like summarization tasks etc., we will be monitoring the outputs using our prompt classification tool. This would convert the unstructured out put to structured output which can be used for fairness analysis. This will contain the bias analysis, indicators, affected groups, type of bias and more details can also be added based on use case requirements. Now the bias distribution of the generated content can be monitored and audited to keep check of the generated content. As the bias analysis using the prompt template is the key, we additionally recommend to use chain-of-thoughts and chain-of-verification to ensure that the analysis done is as accurate as possible.
Traditional AI use case evaluation monitoring tool
We are working on a tool that combines all these metrics and combines the explainability features to get a complete analysis of the use case. Below are key highlights of this tool.
Many to Many analysis: All categorical columns and their sub-classes will be analyzed and their distributions based on representation and success rates are provided to user for acceptance.
Statistical Analysis of key features: Study the dataset based on the ground truth column statistically and understand the key columns which play a decisive role.
Explainability: Once the model is trained and predictions are made, use global and local Explainability to identify key features and their weightage that contribute to the outcome.
Review the metrics, key feature weights and data distribution for a comprehensive F&B analysis of the given use case.
Proposed flow of actions :
Prompt classification Tool
We end up with a huge number of prompts with help of open-source datasets, hackathons, synthetic generators etc. They form the essential block of Benchmarking, fine-tuning of Small Language Models and so on. We need to give structure to these prompts to know their properties / features to understand their overall distribution in the given dataset. The purpose of this tool is to create the respective features each prompt and classify them and thus converting it to a structured data. Now we can generate pivot tables and graphs to understand the over and underrepresented types of prompts which would help to enhance the Benchmarking process and also the fine-tuning process of Small Language Models.
Sample Graph - Bias Indicator Distribution:
Below graph provides insights to the dataset by showing the distribution of bias severity.
Sample Graph - Bias Type Distribution:
Below graph provides insights to the dataset by showing the distribution of bias types.
