[Bug]: Invalid suspension logs a WARN and returns a PENDING result that the service rejects

[yaythomas]

Expected Behavior

When the SDK suspends with no pending operation, it returns an output that the service rejects. The SDK should report this state as an SDK error. The error should name the operations that the suspended threads were waiting for.

The invocation should stay retryable. The race in #370 reached this state, and the execution still succeeded on a later invocation.

Actual Behavior

1. ExecutionManager calls preSuspendCheck() before it suspends. It does so in deregisterActiveThread and in finishCheckpointProcessing.
2. preSuspendCheck() looks for a pending operation: a STEP in PENDING, a WAIT or CALLBACK in STARTED, or a CHAINED_INVOKE in PENDING or STARTED.
3. If it finds none, it logs one WARN line: Invalid suspension. No operation is pending. The line does not name an operation or a thread.
4. The SDK then suspends and returns Status: PENDING with no error. The Lambda invocation succeeds, because the handler returns normally.
5. The service rejects the output with InvalidParameterValueException: Cannot return PENDING status with no pending operations.
6. If the cause is deterministic, every replay reaches the same state. The execution fails, and GetDurableExecution reports Status: FAILED with that error.

The customer sees Lambda invocations that succeed, one WARN line per invocation, and a failed execution. None of these names the operation or the SDK defect that caused the failure.

Steps to Reproduce

1. Run case 1 from [Bug]: get() on a SKIPPED parallel branch never returns, and the execution fails #752 with LocalDurableTestRunner. That handler runs a firstSuccessful() parallel with maxConcurrency(1), then calls get() on the skipped second branch.

2. Each invocation returns status PENDING with no error. No operation is pending.

3. Each invocation logs:

   WARN software.amazon.lambda.durable.execution.ExecutionManager - Invalid suspension. No operation is pending
   

4. LocalDurableTestRunner.runUntilComplete returns PENDING. It does not report that the service would reject the output.

SDK Version

2.2.1. Reproduced on main at ef88276 (2.2.2-SNAPSHOT).

Java Version

21

Is this a regression?

Unknown

Additional Context

Three SDK defects have reached this state. Each ended with the service error above.

• [Bug]: failed tests due to unexpected PENDING status  #370: a race between checkpoint completion and a waiting thread. Fixed in fix: prevent invalid PENDING suspension after checkpoint completion #659.
• [Bug]: Parallel allCompleted can combine a stale branch count with joined state #736: a race in allCompleted() branch registration. Fixed in Fix allCompleted parallel branch registration race #737.
• [Bug]: get() on a SKIPPED parallel branch never returns, and the execution fails #752: get() on a SKIPPED parallel branch. Open.

A fix needs to handle three things.

1. Keep the retry. A transient race can clear on a later invocation, as in [Bug]: failed tests due to unexpected PENDING status  #370. DurableExecutor already rethrows a retryable UnrecoverableDurableExecutionException. Its comment says this is to "let the backend retry the invocation". An error on that path could keep the retry and name the cause.
2. Allow the no-token path. [Feature]: Exit gracefully with PENDING when a checkpoint response has no CheckpointToken #706 proposes a PENDING return when a checkpoint response has no token. That path abandons in-flight operations, so the state can hold no pending operation. A stricter check must not reject that return.
3. Report it in local tests. LocalDurableTestRunner returns PENDING for this output. It could report the output as invalid, so that a local test shows the outcome the service produces.

A smaller change keeps the current behavior and logs at ERROR with the ID and name of the awaited operation.