GitHub's Cheaper AI Model Detects Secrets in Developers' Code

Post Views: 1

GitHub has developed a credential classifier to enhance the detection of sensitive information in code and minimize false positives.

Overview of GitHub’s Credential Classifier

GitHub has developed a credential classifier to enhance the detection of sensitive information in code and minimize false positives. The updated system reduced median post-push scan execution time by 86% and model serving expenses by over 98% compared to its prior large language model (LLM) framework.

Secret Scanning Process

Push Protection

Push protection analyzes changes before acceptance, enabling developers to address issues proactively. The scanner evaluates potential credentials through structured patterns and contextual analysis.

Structured vs. Generic Secrets

Structured credentials typically exhibit identifiable prefixes, lengths, character sets, or checksums. Generic secrets may resemble build identifiers, asset hashes, or sample values, requiring contextual evaluation within files to determine their sensitivity.

“These scenarios represent challenging negatives: strings matching credential patterns but lacking usability. Expanding rule sets can mitigate some cases, but exceptions risk concealing actual secrets. A machine-learned classifier leverages signals like file paths, secret types, variable names, and surrounding code to improve accuracy,” developers noted.

Pipeline Details

The pipeline initiates with regular expressions to detect strings resembling passwords, access tokens, or other credentials. A refined ModernBERT model then assesses each candidate using file path data, credential type, variable names, and adjacent code or text. The classifier generates a score indicating the probability of the candidate being a genuine secret. Thresholds determine whether alerts are triggered during post-push scans or blocked during push protection. False positives are systematically excluded.

Model Training and Validation

The dual-stage approach combines pattern recognition with contextual evaluation, allowing ModernBERT to handle short code and text segments while maintaining throughput and response times necessary for GitHub’s scale. Model training and validation involved datasets derived from product feedback, repository examples identified via regex scanning, and synthetic examples addressing data-scarce or ambiguous scenarios. These datasets included real secrets and false positives, with human validation ensuring accurate labeling.

Evaluation and Results

Data grouping by repository, candidate value, or example ensured consistency before splitting for training. Offline evaluation guided classification threshold selection, followed by validation on production-like traffic prior to deployment. The objective was to sustain or enhance detection coverage while reducing false alerts and operational costs. The trained model was exported in ONNX format with 8-bit quantization to accelerate inference.

Evaluation results demonstrated improvements in post-push scanning precision and recall compared to the previous LLM pipeline. Offline testing revealed a 13% increase in precision and 19% higher recall, reflecting improved accuracy in identifying real secrets and detecting a greater proportion of actual credentials. A customer A/B experiment indicated an 11.5% rise in repository owner-weighted alert acceptance, accounting for variations in user behavior.

“Enhanced precision reduces disruptive alerts, while improved recall ensures more genuine secrets are identified,” the developers stated.

Future Plans and Integration

GitHub plans to refine the system using production feedback to address missed secrets and false positives, integrating these insights into future training data. The company is also exploring architectures that could unify secret detection and extraction into a single process, streamlining the workflow. The initiative aligns with broader efforts to strengthen security practices in software development, emphasizing the integration of AI-driven tools to mitigate risks associated with exposed credentials.