Machine Learning for Software Defect Prediction

2.1 What is software defect prediction? SDP is a supervised-learning problem when historical examples have labels. For a module-level task, the model estimates P(defect | features) for each module. The result is a risk score or class label, not a guarantee. A high score indicates that a component resembles historically defect-prone examples under the learned data distribution. Prathamesh Kadam · Dattakala Group of Institutions, Faculty of Engineering · prathameshkadam3935@gmail.com · Guide: Dr. Taware G.G. Prathamesh Kadam¹ ¹ Dattakala Group of Institutions, Faculty of Engineering, Pune, Maharashtra, India Corresponding author: prathameshkadam3935@gmail.com Guide / Supervisor: Dr. Taware G.G. 2.2 Prediction granularity Granularity Input example Typical use File / module A source file or class with static metrics Allocate review and testing effort Change-level (JIT) A commit or code change with churn and author/context features Flag risky changes during code review Cross-project A component from a project with limited or no labels Support projects with little defect history Release / component history Aggregated historical quality and change information Plan regression testing and maintenance 2.3 Defining the target label Labels are commonly derived from bug-tracking systems, issue links, post-release reports, or repositories that connect bug-fixing commits to earlier changes. Label construction is difficult: a defect may be discovered long after introduction, bug-fix commits may contain unrelated edits, and unreported defects remain invisible. These uncertainties can create noisy labels and make different datasets incomparable.

This document is a focused narrative review and practical synthesis of established SDP research and public benchmark resources; it is not a new systematic literature search or a report of a newly run experiment. No accuracy figure in this paper should be interpreted as the result of an experiment conducted for this document. The workflow below describes a reproducible approach for a student or engineering team to implement and evaluate an SDP model. Stage Main activities Output 1. Data collection Collect repository history, static analysis metrics, issue tracker labels and release dates Versioned dataset 2. Label construction Link defects to modules or changes; document assumptions and time windows Target variable and label policy 3. Preprocessing Remove duplicates, handle missing values, encode categories, inspect imbalance Training-ready features 4. Feature engineering Compute size, complexity, churn, ownership and historical defect features Feature matrix 5. Training Fit a baseline and compare selected algorithms Trained candidate models 6. Validation Use temporal or project-separated splits; tune only within training data Unbiased evaluation estimate 7. Deployment Display ranked risks, explanations and confidence limitations Testing-prioritization support 8. Monitoring Track drift, false alarms, missed defects and feedback Retraining or retirement decision

5.1 Dataset

sources Researchers commonly use public repository and benchmark datasets, including NASA Metrics Data Program (MDP) datasets, PROMISE repository datasets, and defect data extracted from open-source systems such as Eclipse or Apache projects. JIT research often uses commit histories linked to bug reports. Dataset names alone do not guarantee comparability: versions, label rules, feature definitions, and preprocessing may differ. 5.2 Feature families Static code metrics: lines of code, cyclomatic complexity, coupling, cohesion, inheritance depth, and code duplication. Change metrics: lines added/deleted, files modified, change size, and dispersion across modules. Historical metrics: prior defects, previous changes, recent bug-fix activity, and file age. Process and ownership metrics: number of contributors, code ownership, review activity, and developer experience—used with care to avoid privacy and fairness issues. Code representations: tokens, abstract syntax trees, embeddings, or graph representations, where adequate data and compute are available. 5.3 Evaluation metrics Defect datasets are often imbalanced: most modules may be labeled non-defective. Accuracy can therefore appear high even when the model misses many defects. Precision measures the fraction of predicted positives that are actually positive; recall measures the fraction of actual positives found; and F1 combines precision and recall using their harmonic mean. ROC-AUC summarizes ranking over thresholds, while PR-AUC is often informative when positives are rare. Metric Interpretation When it helps Precision Of flagged items, how many are defective? Controls wasted testing or review effort Recall Of defective items, how many are flagged? Measures missed-defect risk F1-score Balance of precision and recall Single summary when both matter PR-AUC Precision-recall trade-off across thresholds Imbalanced classification ROC-AUC Ranking quality across thresholds General discrimination, but may look optimistic with rare positives Effort-aware recall Defects found within a testing/review budget Practical prioritization under limited effort Calibration Whether predicted probabilities match observed frequencies Risk communication and threshold selection 5.4 Validation and data leakage Random train/test splits can mix examples from the same project or release across both sets, allowing a model to exploit project-specific patterns. If the real use case is predicting the next release, use chronological training and testing. If the target is a new project, hold out entire projects or use cross-project evaluation. All preprocessing and feature selection must be fitted only on training data. Features computed after the prediction point—such as future bug-fix information—must be excluded. A credible study should report the split strategy, dataset version, label definition, class distribution, baseline models, hyperparameter-selection procedure, random seeds where relevant, and uncertainty across folds or projects. Compare models under the same splits and avoid presenting one favorable score as universal evidence.

6.1 Explainability

Developers are more likely to use predictions when they can understand why a file or change was flagged. Global feature importance can describe broad patterns; local explanations can summarize factors associated with an individual score. Explanations are approximate and may be unstable when features are correlated. They should support investigation rather than claim that a particular metric caused a defect. 6.2 Human-in-the-loop workflow A practical deployment displays a ranked list of components or changes, the risk score, a concise explanation, and links to relevant tests or code review. Developers and testers retain responsibility for decisions. A prediction should trigger focused review or additional testing—not automatically block a release or label a developer as poor-performing. 6.3 Operational risks Concept drift: coding practices, architecture, dependencies and team composition change over time, so model performance can decay. False positives: excessive alerts can create fatigue and reduce trust. False negatives: a low score can create false confidence; testing must not be removed solely because a model says risk is low. Fairness and privacy: author-related features can encode organizational bias or encourage inappropriate individual surveillance. Security: repository data and model endpoints should be protected; training data and feature pipelines should be auditable. 6.4 Monitoring After deployment, monitor precision and recall where labels become available, calibration, alert volume, review effort, and performance by project or component type. Record model and feature versions. Retrain only with a documented process and evaluate a candidate model against the existing model before replacement.

Start with a simple baseline such as logistic regression and a tree ensemble; add complexity only when evaluation shows a meaningful improvement. Define the prediction point and label policy before selecting features. Exclude future information and document uncertain bug-to-code links. Choose validation to match deployment: temporal splits for future releases and project-separated splits for new-project prediction. Report precision, recall, F1, PR-AUC and effort-aware results where applicable; select thresholds according to testing capacity and the cost of missed defects. Explain predictions in developer-centered terms and integrate them into existing code review and testing tools. Use risk scores to prioritize effort, not to replace testing, code review, static analysis, or engineering judgment. Monitor drift, alert fatigue and subgroup performance; define clear ownership for retraining and rollback.

SDP research faces persistent threats to validity. Public datasets may be old, small, project-specific, or built with inconsistent labels. Defect reports capture only discovered defects, and historical associations do not establish causation. Different studies may use different train/test splits and metrics, making headline scores difficult to compare. Repository features can also leak future information if their timestamps are not carefully controlled. Important research directions include better cross-project transfer, calibrated and explainable predictions, effort-aware evaluation, robust handling of label noise, and evaluation in real developer workflows. More studies should measure whether SDP actually improves defect discovery or reduces testing cost, not merely whether a model predicts benchmark labels accurately. 9. Conclusion Machine learning can help software teams prioritize testing, review, and maintenance by estimating defect risk from code metrics, change history, and repository data. Traditional models and tree ensembles provide practical starting points, while deep-learning and JIT approaches may be useful when data and deployment needs justify them. The credibility of an SDP system depends less on a single algorithm than on careful labels, leakage-free validation, suitable metrics, understandable outputs, and ongoing monitoring. Used as decision support—not as a substitute for engineering judgment—defect prediction can help allocate limited quality-assurance effort more effectively.