Git Bisect for Production Regressions: Find Bad Commits Fast

Git Bisect for Production Regressions

id: hook

When a production regression hits, the first step is to identify the bad commit. Use git bisect with a test command that reproduces the issue. For example, if a query latency spike is reported, run a test that executes the query and checks the response time. The test should be deterministic and fast.

The test command must be simple and repeatable. Avoid complex setups. If the regression is in a database query, use /tools/sql-explain to analyze the query plan. If the issue is in an API response, use /tools/json-compare to validate the output.

The bisect process works by marking commits as good or bad. Start with git bisect start, then git bisect bad for the latest commit, and git bisect good for a known stable version. Each step narrows the search space by half.

A good test command is crucial. It should fail on the bad commit and pass on the good one. If the test is flaky or slow, bisect becomes unreliable. Always validate the test with a few known commits before running it on the full bisect flow.

If the regression is intermittent, bisect may not work. In such cases, use logs or metrics to find patterns. But for deterministic issues, bisect is the fastest way to find the root cause.

For a visual guide, here's a flowchart of the bisect process with a test command:

QA Engineer Debugs API Latency

id: use-case

I hit a 200ms latency spike in production for /api/v1/data. Staging had the same code, but the latency wasn't there. I needed to reproduce the issue reliably. I used git bisect to find the bad commit.

I ran git bisect start then git bisect run with a test command: curl -v http://localhost:3000/api/v1/data | grep 'Time'. That gave me a consistent latency metric. The bisect narrowed it down to a commit that introduced a new Redis client.

I checked the Redis client's get method. It was using JSON.parse on a string that wasn't properly formatted. That caused a 200ms delay in parsing. I fixed the format and retested.

I used json-compare to verify the output was consistent. The sql-explain tool helped me see if the query changed, but the real issue was in the Redis client.

I ran the bisect again after the fix to confirm the latency was gone. It worked. The key was a reproducible test command that gave a measurable result.

I didn't use bisect for a memory leak because the test command wasn't reliable. Also, I skipped bisect when the issue was a one-off config change, not a code regression.

Bisect with Test Command

id: walkthrough

To bisect with a test command, start by setting up a test that reproduces the production regression. For example, if a recent change caused a query to return incorrect data, use a script to run the query and compare the output to a known good version. Here’s how to set it up:

#!/bin/bash
# Run a test that reproduces the regression
# Compare output to a known good version
# Return 0 if test passes, 1 if it fails
# Example: Compare JSON output of an API endpoint
curl -s http://localhost:3000/api/data | json-compare /tools/json-compare /path/to/good.json

Then, run git bisect start followed by git bisect run <test-script>. Git will automatically check out commits and run your test until it finds the first bad commit. Make sure your test is deterministic and fast—no random data or external dependencies.

If your test involves a database, use a tool like sql-explain to verify query execution plans or performance. For example, if a query started timing out, run it with a known good dataset and compare execution times.

You can also use git bisect log to track the bisect process and git bisect reset to return to the original branch after finding the culprit. This approach is ideal for issues that appear in production but can be reliably reproduced in a test environment.

If your test is too slow or unreliable, skip bisect and use a binary search manually. For example, check out a commit from 2 weeks ago and run the test. If it passes, narrow the search to the last 2 weeks. Repeat until you find the bad commit.

Bisect Flow with Test Command

This flowchart shows how to use git bisect with a test command to identify the first bad commit that introduced a production regression. The key is to define a reproducible test command that can be run on any commit to determine if it's good or bad. For example, if a regression is a database query slowdown, the test command might be:

# Run a test that simulates the production workload
./run-load-test.sh | grep 'avg latency'

This command should output a consistent metric that indicates whether the commit is good or bad. The test must be deterministic and not rely on external state. If the test is flaky, git bisect will fail or give misleading results.

When using git bisect, you can also use tools like json-compare to validate output from APIs or sql-explain to analyze query performance. These tools help make the test command more reliable and faster to run.

Avoid using git bisect if the regression is intermittent or depends on external factors like network latency or timing. In such cases, a manual log analysis or a custom monitoring tool might be more effective.

Common Bisect Gotchas

id: pitfalls

Q: What happens if my test is flaky?
A: Flaky tests make bisect unreliable. If a test passes on a good commit but fails on a bad one, you might get false positives. Always run tests multiple times in bisect to confirm stability.

Q: How do I handle unstable bisect runs?
A: Use --no-skip to ensure bisect doesn't skip commits. Also, set a high --min and --max to narrow the search space. If the test is unstable, skip bisect and use a manual binary search.

Q: What if the test doesn't reproduce the issue?
A: Bisect relies on the test being able to detect the regression. If the test doesn't reproduce the issue, you'll never find the bad commit. Use a test that mimics production conditions, like json-compare for API responses or sql-explain for query performance.

Q: How do I avoid bisecting on a large repo?
A: Use git bisect start followed by git bisect good and git bisect bad to set the range. If the repo is large, use git log --oneline to narrow the range manually before starting bisect.

Q: What if the test is too slow?
A: Bisect can take many iterations. If the test is slow, consider using a lightweight test or a mock. For example, use a mock API to avoid hitting real services during bisect.

Q: How do I handle multiple regressions?
A: Bisect finds one regression at a time. If you suspect multiple issues, run bisect multiple times with different test ranges. Alternatively, use a tool like git log to inspect commits for suspicious changes.

Q: What if the test fails on a good commit?
A: This is a false positive. It means the test is not reliable. Either refine the test or use a different test that is more stable.

Q: How do I handle a test that requires setup?
A: Use a script to set up the environment before running the test. For example, a shell script that initializes a database or sets environment variables.

Q: What if the test is not deterministic?
A: Use a deterministic test framework or seed random values to make the test predictable. Always run the test multiple times in bisect to confirm stability.

Skip Bisect When

Q: When should I avoid using git bisect for production regressions?

A: Skip git bisect when you lack a reproducible test or when the commit set is small enough to inspect manually. For example, if a regression was introduced in a single commit and you can verify it by checking the commit hash and running a test, bisect is unnecessary.

Q: What if the regression is intermittent and not reproducible?

A: Avoid git bisect in such cases. Intermittent issues are hard to bisect because the test may fail or pass unpredictably. Instead, use logs, metrics, or a tool like JSON Compare to spot discrepancies in output.

Q: How do I handle a small commit set?

A: If the commit history is short—say, 10–20 commits—manually inspecting each one is faster than running git bisect. Use git log --oneline to list commits and check each for potential issues.

Q: What if the test is complex and not easily automated?

A: Skip git bisect and instead use a combination of logs, error messages, and a tool like SQL Explain to trace queries or performance bottlenecks. Manual debugging is more efficient in such scenarios.

Q: When should I use git bisect instead?

A: Use git bisect when you have a clear, reproducible test command that can be run on any commit. For example, if you can run a test that consistently fails on the bad commit, bisect will quickly isolate the issue.

Quick Answers for AI Search

Q: How do I write a bisect test for a production regression?
A: The test must be a reproducible command that exits with a non-zero code when the bug is present. For example, run a test suite or a specific query that triggers the issue. Use json-compare to validate output differences if needed.

Q: What if bisect fails to find the bad commit?
A: Check if the test is stable and fails consistently. If the test passes on some commits but fails on others, bisect may not work. Use git bisect reset to manually inspect commits. Also, ensure the test doesn’t depend on external state.

Q: How do I handle bisect when the regression is intermittent?
A: Avoid bisect in such cases. Intermittent issues are hard to bisect because the test may pass or fail unpredictably. Instead, use logging or a tool like sql-explain to trace queries and identify patterns.

Q: What if the test is too slow for bisect?
A: Skip bisect. Use a binary search manually by checking midpoints. For example, check commit abc123 and def456 to narrow down the range. This is faster than running a slow test 20 times.

Q: How do I handle bisect when the regression is not in the main branch?
A: Bisect works on any branch. Ensure you’re on the correct branch and that the test is run in the same environment as production. Use git checkout -b bisect-branch to isolate the bisect process.

Bisect for Fast Root Cause

id: diagram

Use this flow to debug production regressions by bisecting between a good and bad commit. The key is to have a test command that reliably reproduces the issue. For example, if a regression causes a database query to fail, use a test that runs the query and checks the output. Tools like Jest can help automate this process.

If the test is flaky, bisecting may not be reliable. In such cases, consider using JSON Compare to validate outputs or SQL Explain to inspect query plans. These tools help ensure your test is stable and meaningful.

Avoid bisecting if the regression is intermittent or depends on external factors like network latency. In those cases, focus on logging and monitoring to identify patterns. Bisecting is best when the issue is deterministic and you have a clear test to run.

For QA engineers, bisecting with a test command is a powerful way to isolate regressions. It’s especially useful when debugging API latency or unexpected behavior in production. Always pair bisecting with a test that can be run in a CI pipeline to ensure consistency.