Back to Registry
code.python.tests_pass
HARD
Writes agent-generated Python code to a temp directory and runs pytest via the sandbox runner. Score is based on the fraction of tests that pass.
Scorecard
| Determinism | deterministic |
| Evidence Quality | hard-state |
| Intended Use | eval-and-train |
| Gating Required | No |
| Permissions | fs:write_tmp exec:pytest |
Test Fixtures
9 total
| Type | Count |
|---|---|
| Positive | 3 |
| Negative | 3 |
| Adversarial | 3 |
Metadata
| Version | 0.1.0 |
| Domain | code |
| Task Type | test_execution |
| Contributor | vr.dev |
| Source | code-quality |
Use in SDK
# CLI
vr verify --verifier code.python.tests_pass --ground-truth '{"order_id": "ORD-42"}'
# Python
from vrdev import verify
result = verify("code.python.tests_pass", ground_truth={"order_id": "ORD-42"})
# API
curl -X POST https://api.vr.dev/v1/verify \
-H "X-API-Key: vr_live_..." \
-d '{"verifier": "code.python.tests_pass", "ground_truth": {"order_id": "ORD-42"}}'