5.1 · Compare designs without hiding the tradeoffs
Chapter 5: Comparing and Testing Ideas
Watch the chapter or work through its reading pages at your own pace.
Chapter video not yet released. Registered participants will be notified when it becomes available.
Development edition. Each chapter has its own follow-along PDF, which may be updated when that chapter’s video is released.
Reading pages in this chapter
Chapter 5 · Page 1 of 2
Compare designs without hiding the tradeoffs
Learning objective: Use explicit criteria and feasibility conditions to select a testable candidate.
Separate mandatory conditions from desirable performance. A design that cannot meet a binding authorization requirement does not become acceptable because it scores well on speed. Screen mandatory conditions first. A condition with unresolved feasibility receives “unproven,” not an optimistic pass.
Then compare feasible candidates using a small set of defined criteria: outcome quality, elapsed time, staff effort, exception resilience and maintainability, for example. Avoid counting the same benefit several times under different labels. Explain whose interests the weights represent and agree them before looking at the scores.
Numbers structure a conversation; they do not create evidence. If using a 1–5 scale, define the anchors. For elapsed time, 1 could mean no credible reduction, 3 a plausible modest reduction with an identified mechanism, and 5 a strong mechanism supported by relevant test evidence. Early concepts will rarely justify 5.
Conduct a sensitivity check. If a small weight change reverses the preferred option, report that fragility. If two options are close but depend on different unknowns, choose the next experiment that best informs the decision rather than declaring a premature winner.
Record rejected options and the conditions under which they would become worth reconsidering. A distributed approval model might be deferred until authority rules are clarified; it is not necessarily invalid forever. A decision record should let a later team understand the reasoning without reconstructing the workshop conversation.
Practice
Use W7 to compare the three W6 options. Apply a mandatory-condition screen, score no more than five criteria with defined anchors, and change the two most influential weights. Record whether the preferred option changes and which uncertainty should be tested first.
Check your reasoning
What should happen when the highest-scoring design fails a mandatory condition?
Worked response
Reject it as currently formulated or retain it as conditional on an authorized change. A weighted score cannot compensate for a failed mandatory condition.
Capacity and the whole service · 2 minutes
Simplified fictional assumptions: 12 arrivals/day, capacity for 10 completions/day, constant rates, available work and no other inflows/outflows. Backlog grows by 2/day: from zero to 10 after five days. Faster routing alone cannot remove the mismatch.
If completions rise to 12/day, growth stops under those assumptions; an existing backlog does not disappear. Identify what would invalidate the calculation: changing volume, case complexity, absence, rework or abandoned cases. State what you would measure. Include client effort as well as staff effort when assessing a proposed change.
Read-along slides and narration script (audio not yet available)
5.01 · Comparing and Testing Ideas
Welcome to Colabmo.
I'm Diggy Breiling, founder of Colabmo. Welcome to First Principles Thinking for Business Process Redesign. This is chapter five, Comparing and Testing Ideas.
In this chapter, we'll compare alternatives and design a test of the assumption that matters most. BrightPath will help us examine capacity, user evidence and a revised recommendation.
5.02 · Mandatory conditions first
We begin with comparing alternatives.
A promising design still needs a fair comparison and an evidence plan. We will examine mandatory conditions, tradeoffs, and the result that would change our recommendation.
Separate mandatory conditions from desirable performance. A design that cannot meet a binding authorization requirement does not become acceptable because it scores well on speed. Screen mandatory conditions first. A condition with unresolved feasibility receives “unproven,” not an optimistic pass.
5.03 · A transparent comparison
Then compare feasible candidates using a small set of defined criteria: outcome quality, elapsed time, staff effort, exception resilience and maintainability, for example. Avoid counting the same benefit several times under different labels. Explain whose interests the weights represent and agree them before looking at the scores.
Numbers structure a conversation; they do not create evidence. If using a 1–5 scale, define the anchors. For elapsed time, 1 could mean no credible reduction, 3 a plausible modest reduction with an identified mechanism, and 5 a strong mechanism supported by relevant test evidence. Early concepts will rarely justify 5.
5.04 · Sensitivity and the next experiment
Conduct a sensitivity check. If a small weight change reverses the preferred option, report that fragility. If two options are close but depend on different unknowns, choose the next experiment that best informs the decision rather than declaring a premature winner.
Record rejected options and the conditions under which they would become worth reconsidering. A distributed approval model might be deferred until authority rules are clarified; it is not necessarily invalid forever. A decision record should let a later team understand the reasoning without reconstructing the workshop conversation.
5.05 · Demand, capacity and the growing queue
A service also has behavior over time. In this simplified fictional exercise, twelve requests arrive each day and the service can complete ten. With no other inflows or outflows, constant rates and sufficient available work, backlog grows by two per day. Starting at zero, it reaches ten after five days. Faster routing alone does not change that arithmetic.
Real work differs: requests vary, arrivals fluctuate, absence changes capacity, and rework can create more demand. Before redesigning, examine those conditions. Moving clarification to the client may reduce recorded adviser time while increasing client effort or abandonment. A faster local step can make the overall service worse.
Use this as a two-minute sensitivity exercise in W7. What if completions rise to twelve? Under the same simplified assumptions, backlog stops growing but an existing backlog does not disappear. What evidence would show that the proposed capacity is sustainable? Return to the whole outcome after examining the parts.
5.06 · W7: compare the options
Now apply the reasoning. Use W7 to compare the three W6 options. Apply a mandatory-condition screen, score no more than five criteria with defined anchors, and change the two most influential weights. Record whether the preferred option changes and which uncertainty should be tested first.
Pause the video while you work. Before resuming, record your answer to this question: What should happen when the highest-scoring design fails a mandatory condition?
When you return, compare your reasoning with this explanation. Reject it as currently formulated or retain it as conditional on an authorized change. A weighted score cannot compensate for a failed mandatory condition.
Chapter 5 · Page 2 of 2
Design a test that could prove the idea wrong
Learning objective: Write a falsifiable hypothesis and a practical evidence plan before testing.
Test the riskiest assumption, not merely the easiest feature to demonstrate. “People liked the prototype” tells you little if the design depends on reliable exception routing. Write a hypothesis connecting a mechanism to an expected outcome for a defined population.
For example: “For standard requests with complete inputs, allowing independent checks to start together will reduce elapsed time without increasing incorrect dispositions or total staff effort.” This contains several measurable claims. Define complete inputs, standard requests, elapsed time, incorrect disposition and effort before collecting results.
A test plan specifies eligible cases, exclusions, assignment or comparison method, observation period, sample rationale, measures, permissions, stop conditions and decision rules. Use a simulation or historical replay first when live consequences would be difficult to reverse. A replay tests some mechanics, but cannot establish real customer behavior or adoption.
If feasible, use a concurrent comparison with comparable cases. If using before-and-after data, record differences in volume, complexity, staffing and season. Do not attribute every change to the redesign. NIST’s experimental-design guidance explains that designs should be selected against objectives and factors; its catalogue includes randomization, blocking and confounding. Seek appropriate statistical help when a decision requires more than a small learning trial.
Set the decision rules before results arrive. A pilot may justify revise, stop, or expand the test. It rarely establishes universal effectiveness. Record adverse outcomes, missing observations and excluded cases, including whether exclusions could make the result look better.
Separate the test’s success threshold from the evidence strength needed for wider adoption. Ten successful cases may show that a mechanism is possible. They provide limited assurance about rare failure modes, unusual requests or sustained performance. State what remains unknown and how the next stage will address it.
Practice
Use W8 for one W7 uncertainty. Specify what result would make you abandon or change the design. Include one quality guardrail, one workload guardrail, an exception case, a comparison and an explicit decision owner.
Check your reasoning
A trial reduced elapsed time but required a specialist to rescue every case. Did the design succeed?
Worked response
Only against a narrow time measure. The rescue effort may violate the workload condition and hide an incomplete process. Record it, examine the causal explanation and revise before claiming a sustainable improvement.
BrightPath evidence update · 7 minutes
Record your initial option and decisive assumption in W8 before opening the card.
Open after recording your initial recommendation
Invented observations: 3 of 6 deliberately selected users could not express their need adequately through the written prompt. Two of those three could do so in a short conversation. No other results are supplied.
These observations describe those six users; they do not establish a population failure rate or a winning channel. Revise your premise and applicability boundary. Propose a discriminating next test. A better prompt, assisted route, conversation, different combination or further investigation may be defensible.
Measure first request to usable next step, adviser effort, client effort including clarification, unsuitable advice and recontact due to misunderstanding. Ask the client to explain the next step and when it would not apply. Include unresolved and abandoned cases. Set decision rules and sample rationale before the next test; no improvement claim is supplied.
Read-along slides and narration script (audio not yet available)
5.07 · A testable hypothesis
We now begin the section on Testing the assumption that matters.
Test the riskiest assumption, not merely the easiest feature to demonstrate. “People liked the prototype” tells you little if the design depends on reliable exception routing. Write a hypothesis connecting a mechanism to an expected outcome for a defined population.
For example: “For standard requests with complete inputs, allowing independent checks to start together will reduce elapsed time without increasing incorrect dispositions or total staff effort.” This contains several measurable claims. Define complete inputs, standard requests, elapsed time, incorrect disposition and effort before collecting results.
5.08 · A practical evidence plan
A test plan specifies eligible cases, exclusions, assignment or comparison method, observation period, sample rationale, measures, permissions, stop conditions and decision rules. Use a simulation or historical replay first when live consequences would be difficult to reverse. A replay tests some mechanics, but cannot establish real customer behavior or adoption.
If feasible, use a concurrent comparison with comparable cases. If using before-and-after data, record differences in volume, complexity, staffing and season. Do not attribute every change to the redesign. NIST’s experimental-design guidance explains that designs should be selected against objectives and factors; its catalogue includes randomization, blocking and confounding. Seek appropriate statistical help when a decision requires more than a small learning trial.
5.09 · What the result can support
Set the decision rules before results arrive. A pilot may justify revise, stop, or expand the test. It rarely establishes universal effectiveness. Record adverse outcomes, missing observations and excluded cases, including whether exclusions could make the result look better.
Separate the test’s success threshold from the evidence strength needed for wider adoption. Ten successful cases may show that a mechanism is possible. They provide limited assurance about rare failure modes, unusual requests or sustained performance. State what remains unknown and how the next stage will address it.
5.10 · BrightPath: new evidence changes the proposal
This is BrightPath’s new evidence section. First record your initial recommendation and its decisive assumption in W8. Pause before revealing the evidence card.
In our invented trial, three of six deliberately selected users could not express their need adequately through the written prompt. Two of those three could do so in a short conversation. These observations reveal problems in those six users. They do not estimate a population failure rate or prove that conversation is always best.
Allow seven minutes to revise the claim, applicability boundary and next test. A clearer prompt, an assisted route, a conversation, a different combination or further investigation can be defensible. A hybrid is not the required answer.
Measure first request to a usable next step, adviser effort, client effort including clarification, unsuitable advice and recontact caused by misunderstanding. Ask the client to explain the proposed next step and when it would not apply. Satisfaction or an email delivery receipt alone is insufficient. Keep unresolved and abandoned requests visible. Agree decision rules before the next test and justify the sample; do not invent an improvement result.
5.11 · W8: the assumption that matters
Now apply the reasoning. Use W8 for one W7 uncertainty. Specify what result would make you abandon or change the design. Include one quality guardrail, one workload guardrail, an exception case, a comparison and an explicit decision owner.
Pause the video while you work. Before resuming, record your answer to this question: A trial reduced elapsed time but required a specialist to rescue every case. Did the design succeed?
When you return, compare your reasoning with this explanation. Only against a narrow time measure. The rescue effort may violate the workload condition and hide an incomplete process. Record it, examine the causal explanation and revise before claiming a sustainable improvement.
You now have a comparison and a proposed test. Next, we will work on the facilitator’s role and on keeping the team’s scope clear.
5.12 · Next: Facilitation and Scope
You’ve completed chapter five, Comparing and Testing Ideas, in First Principles Thinking for Business Process Redesign.
Thank you for learning with Colabmo. Bring your expertise, keep your reasoning visible, and put what you’ve learned to work.
Scroll within the reading area. Use the page buttons above to review within this chapter. The course sidebar and Prev / Next buttons open another course page.
There are no comments for now.
to be the first to leave a comment.