To assess a supplier's SDLC and QA, choose one recent change yourself and trace it from ticket to production, asking to see review, test, security scan, release and rollback evidence. Score what you see against the NIST SSDF, and treat uncontrolled releases or missing patch timelines as gates.
A secure development policy says what should happen to a code change. Following one real change to production shows what actually happens.
Reading a supplier's secure development policy is like reading a restaurant's food safety manual without visiting the kitchen. The manual can be flawless while the fridge is broken. This guide shows you how to inspect the kitchen: trace one real change to production, request evidence at each step and score it against a neutral standard.
Why development practice predicts your risk
How software is built decides how it fails. Weak review lets defects through, thin testing lets them reach production, and slow patching leaves known weaknesses open. Each of those becomes your outage or your incident.
Verizon's 2025 Data Breach Investigations Report found that exploitation of vulnerabilities as a way in grew by 34%. It now accounts for 20% of breaches, and third-party involvement doubled to 30%. A supplier's ability to find and fix flaws quickly is therefore part of your own exposure.
Most assessments still stop at a policy and a questionnaire. A SOC 2 report (an independent auditor's attestation on controls) helps, but it shows that a change was approved, not that it was tested well. It may not even cover the product you are buying.
Use the NIST SSDF as the yardstick
Without a neutral standard, every supplier defines good as whatever it already does. The NIST Secure Software Development Framework (SSDF), SP 800-218 version 1.1, is the most practical choice. It underpins the attestation form CISA released on 11 March 2024 for software producers serving the US federal government. Ask whether the supplier has completed it, since that is a fast signal.
| SSDF practice group | What to ask | Red flag |
|---|---|---|
| Prepare the Organization | Who owns secure development and training? | No owner, or training ended at onboarding |
| Protect the Software | How are code and releases protected from tampering? | Engineers can push straight to production |
| Produce Well-Secured Software | How is each change reviewed and tested? | Security testing only before major releases |
| Respond to Vulnerabilities | How fast are flaws fixed and disclosed? | No patch timelines or disclosure policy |
Trace one change from ticket to production
Request a 60-minute screen-share with an engineering lead, not a sales engineer. Choose the change yourself from the public release notes, ideally one from the last 90 days that touched authentication, payments or data export. A supplier's own choice shows its best day, whereas yours shows an ordinary one. Then ask to see the artefact, suitably redacted, at six checkpoints:
- Ticket: the original requirement, with security notes and acceptance criteria.
- Code review: who approved it, and whether the system enforced an independent reviewer.
- Automated tests: the test run attached to the change and the share of unreliable tests.
- Security checks: code, dependency and secret scanning results, and what happens on failure.
- Release: how it was deployed, and whether it went out gradually or to everyone.
- Aftercare: monitoring, the rollback path and any postmortem.
A mature supplier completes this in minutes, because the evidence is generated automatically. An immature one starts searching email threads. The speed and confidence of the answers are themselves evidence.
Then ask for three figures over six months: change fail rate, median recovery time from a failed deployment and defects found by customers. These are DORA delivery metrics or close relatives. Finally, request a software bill of materials (SBOM), a list of the components the product is built from, as defined in NTIA's minimum elements of 12 July 2021.
Score only what you saw
Score each area on evidence seen, not answers heard. A three-level scale keeps scoring consistent, and short notes let your risk committee audit the decision later.
| Area | Evidenced | Partly evidenced | Not evidenced |
|---|---|---|---|
| Code review | System-enforced, independent reviewer | Policy exists, manual enforcement | Optional or self-approved |
| Testing | Runs on every change, blocks release | Failures can be overridden freely | Mostly manual |
| Security testing | Scans on every change | Periodic or pre-release only | Annual penetration test only |
| Release control | Gradual rollout, tested rollback | Rollback possible but untested | Direct changes in production |
| Vulnerability response | Published timelines, dated examples | Timelines stated, no examples | No timelines or disclosure policy |
| Supply chain | SBOM available, build records kept | SBOM on request only | No component inventory |
Consider a hypothetical comparison of two HR platforms. Supplier A shows enforced review, a test run that blocked a faulty commit and a gradual rollout, but has no SBOM. Supplier B has a polished policy, yet its traced change was self-approved and deployed directly to production. B fails the release gate, and the team chooses A with SBOM delivery written into the contract.
Findings protect you only once they are contractual. Ask counsel to consider patch timelines, SBOM delivery and notice of major process changes in the agreement.
The bottom line
Conclusion
A food safety manual tells you what a kitchen intends, and a walk through the kitchen tells you what it does. One real change, chosen by you and traced to production, reveals more than any policy or questionnaire.
Scoring that evidence against the SSDF, with clear gates, turns an impression into a decision your risk committee can audit. This week, open the release notes of your most critical supplier, pick one recent change and request a 60-minute trace with an engineering lead.
Frequently asked questions
What is an SBOM and why should I ask for one?
A software bill of materials lists the components a product is built from, so known flaws in them can be tracked. NTIA's minimum elements cover data fields, automation support and practices for its use.
Does a SOC 2 report prove good development practice?
Not on its own. A SOC 2 Type II report shows how controls such as change approval operated over a period, but it rarely shows test depth and may not cover the product being bought.
What is the NIST SSDF?
The Secure Software Development Framework, NIST SP 800-218, groups secure development into four practice groups: Prepare the Organization, Protect the Software, Produce Well-Secured Software and Respond to Vulnerabilities.
What are the biggest red flags in a supplier's development process?
Self-approved code changes, direct changes in production, security testing only before major releases and no stated timelines for patching critical vulnerabilities.
How long does an SDLC assessment take?
A focused assessment needs a 60-minute screen-share tracing one change, plus a short follow-up for the SBOM, patch timelines and delivery metrics.
References
- Verizon, 2025 Data Breach Investigations Report (press release)
- Secureframe, SOC 2 Trust Services Criteria explained
- NIST, SP 800-218 Secure Software Development Framework (SSDF) Version 1.1 (February 2022)
- CISA, Secure Software Development Attestation Form (11 March 2024)
- DORA, Software delivery metrics: the four keys
- NTIA, NTIA releases minimum elements for a software bill of materials (12 July 2021)
Disclaimer: General information only, not legal, security, compliance or financial advice. Check the cited sources and seek professional advice. Names belong to their owners; no endorsement implied.