Scope Kamek bl-patch LR-continuation detection to genuine skip-return… (#218)

* Scope Kamek bl-patch LR-continuation detection to genuine skip-return targets

Fix crash from Kamek skip-return hooks (Item Rain crash) (#182) added every
Kamek BranchLink patch target to lrContinuationCallTargets unconditionally,
with no filter analogous to the RetroWfcHookSetsLinkRegister check already
used for RetroWFC hooks. Since bl is the ordinary PowerPC call instruction,
this made the codegen treat effectively every patched call in the mod as a
potential skip-return hook, forcing conservative handling (full register
reload, disabled resident-call fast paths, local LR-continuation dispatch
tables) onto thousands of calls that just return normally.

For Retro Rewind this inflated total translated mod size by +42%
(1,414,327 -> 2,005,284 lines), concentrated in ~10 unrelated overlay
functions that happened to call a patched target, and was enough to make
one aggregate build shard pathologically slow to compile (hangs Linux CI).

Instead, only mark a bl target as LR-continuation-aware if a lightweight
discovery-only decode of its own body actually finds evidence of
skip-return behavior via DiscoverLrRelativeIndirectJumpOffsets. Falls back
to the conservative (old) behavior if a target can't be statically
analyzed, so no skip-return case is silently missed.

Verified against the real Retro Rewind mod: total mod size returns to
1,416,350 lines (+0.14% vs. pre-fix, down from +42%), all 6 genuinely new
continuation functions from the original fix are preserved, zero
functions lost, and all 609 existing translator tests still pass.

* Distinguish exhausted from truncated LR-relative offset search

CodeRabbit flagged that TargetExhibitsLrSkipReturn (added in ad2d4e7) treated
an empty DiscoverLrRelativeIndirectJumpOffsets result as a verified "this
target never skip-returns," but the analysis silently drops any path state
once more than MaxStatesPerInstruction (16) distinct states reach one
instruction - a bctr/return on a dropped state can never contribute its
offset, so an empty result could be an incomplete search rather than a real
negative. Treating every capped case as "skip-return possible" outright was
rejected as too broad a fallback given how conservative/expensive that path
already is.

Instead: raise MaxStatesPerInstruction 16 -> 512 (an arbitrary conservative
bound to begin with, not something correctness depended on) so genuinely
branchy functions have far more headroom to reach an exhaustive answer, and
give DiscoverLrRelativeIndirectJumpOffsets an optional onStateCapExceeded
callback that fires exactly when a state is dropped. TargetExhibitsLrSkipReturn
now only falls back to the conservative "treat as skip-return" answer when
the search both found nothing and the cap was actually hit during that run -
not whenever the cap merely exists - so a target is trusted as clean once the
search genuinely exhausts it.

Verified: all 609 translator tests pass, and a full translate-mod run against
the real Retro Rewind mod produces byte-for-byte identical output to the
prior fix (same 4,065 functions, 1,416,350 total lines) - confirming the
16-state cap was never actually the limiting factor in practice and this
change is a pure safety-net closure, not a behavior change for this mod.

* Add LR continuation regression tests

* Refine LR continuation hook analysis

---------

Co-authored-by: patchzyy <64382339+patchzyy@users.noreply.github.com>

translator/src/Translator.Cli/Program.cs

@@ -2282,9 +2282,37 @@ int EmitModCpp(
                 .Where(h => h.TargetAddress.HasValue && RetroWfcHookSetsLinkRegister(h))
                 .Select(h => (h.TargetAddress!.Value, h.ContinuationAddress)));
     }
+    var hookLrAnalysis = new Dictionary<uint, LrContinuationAnalysis>();
+    var hookDiscoveryCache = new Dictionary<uint, IReadOnlyList<PpcInstruction>>();
+    IReadOnlyList<PpcInstruction> DiscoverHookBody(uint target)
+    {
+        if (!hookDiscoveryCache.TryGetValue(target, out var instructions))
+        {
+            instructions = modTranslator.Discover(target,
+                new TranslationOptions(KnownFunctionEntryPoints: knownFunctionEntryPoints)).Instructions;
+            hookDiscoveryCache.Add(target, instructions);
+        }
+        return instructions;
+    }
+
+    LrContinuationAnalysis AnalyzeHook(uint target)
+    {
+        if (!hookLrAnalysis.TryGetValue(target, out var analysis))
+        {
+            analysis = LrContinuationAnalysis.Analyze(target, DiscoverHookBody);
+            hookLrAnalysis.Add(target, analysis);
+        }
+        return analysis;
+    }
+
     foreach (var patch in patchPlan.ExecutablePatches.Where(p => p.CommandId == KamekCommandId.BranchLink && p.Arguments.Count > 0))
     {
         var target = KamekAddress.Resolve(patch.Arguments[0], patchPlan.ModuleGuestBase);
+        if (!AnalyzeHook(target).MaySkipReturn)
+        {
+            continue;
+        }
+
         hookLrBases.Add((target, checked(patch.CommandAddress + 4u)));
     }
 
@@ -2387,50 +2415,42 @@ int EmitModCpp(
         }
     }
 
-    void RecordDiscoveredLrRelativeBaseContinuations(
-        FunctionTranslationResult result,
-        IReadOnlyList<uint> lrBases,
-        string reason)
+    void RecordHookContinuations(uint hookTarget, IReadOnlyList<uint> lrBases)
     {
-        if (lrBases.Count == 0)
+        var analysis = AnalyzeHook(hookTarget);
+        foreach (var lrBase in lrBases)
         {
-            return;
-        }
-
-        foreach (var offset in DiscoverLrRelativeIndirectJumpOffsets(result).Distinct())
-        {
-            foreach (var lrBase in lrBases)
+            var targets = analysis.Offsets.Select(offset => unchecked(lrBase + (uint)offset));
+            if (analysis.WasTruncated && baseFunctions.FindContaining(lrBase - 4u) is { } caller)
+            {
+                // Unknown offsets can resume at any aligned instruction in this caller.
+                targets = targets.Concat(Enumerable.Range(0, checked((int)((caller.End - caller.Start) / 4)))
+                    .Select(index => caller.Start + (uint)index * 4u));
+            }
+            foreach (var target in targets.Distinct())
             {
-                var target = unchecked(lrBase + (uint)offset);
                 var section = baseManifest.Sections.FirstOrDefault(s => target >= s.GuestStart && target < s.GuestEnd);
-                if (section is null || !section.Executable)
-                {
+                if (section is null || !section.Executable || (target & 3u) != 0)
                     continue;
-                }
-
                 var containing = baseFunctions.FindContaining(target);
-                if (containing is null || containing.Start == target)
-                {
+                if (containing is null || containing.Start == target || !queuedContinuationAddresses.Add(target))
                     continue;
-                }
-
-                if (!queuedContinuationAddresses.Add(target))
-                {
-                    continue;
-                }
 
                 discoveredContinuationQueue.Enqueue(new ContinuationEntry(
                     target,
                     containing.Start,
                     containing.End,
                     section.Name,
-                    result.EntryPoint,
+                    hookTarget,
                     KamekCommandId.Branch,
-                    $"{reason}; LR-relative jump offset {offset:+#;-#;0}"));
+                    $"LR-relative hook target 0x{hookTarget:X8}"));
             }
         }
     }
 
+    foreach (var (target, lrBases) in linkedHookLrBasesByTarget)
+        RecordHookContinuations(target, lrBases);
+
     ModTranslationWork CreateContinuationWork(ContinuationEntry continuation)
     {
         var name = $"rr_continue_{continuation.Address:X8}";
@@ -2596,16 +2616,7 @@ int EmitModCpp(
         }
 
         CommitWave(attempts, (result, work) =>
-        {
-            RecordDiscoveredBaseContinuations(result, $"base continuation discovered from module 0x{work.Address:X8}");
-            if (linkedHookLrBasesByTarget.TryGetValue(work.Address, out var lrBases))
-            {
-                RecordDiscoveredLrRelativeBaseContinuations(
-                    result,
-                    lrBases,
-                    $"base continuation discovered from LR-relative hook target 0x{work.Address:X8}");
-            }
-        });
+            RecordDiscoveredBaseContinuations(result, $"base continuation discovered from module 0x{work.Address:X8}"));
     }
 
     DrainDiscoveredContinuations();
@@ -2763,9 +2774,6 @@ IEnumerable<uint> DirectModuleTargets(FunctionTranslationResult result, uint mod
     }
 }
 
-IEnumerable<int> DiscoverLrRelativeIndirectJumpOffsets(FunctionTranslationResult result) =>
-    ContinuationPlanner.DiscoverLrRelativeIndirectJumpOffsets(result.Instructions);
-
 static bool RetroWfcHookSetsLinkRegister(RetroWfcExecutableHookPlan hook) =>
     hook.TypeName is "call" or "branchCtrLink" ||
     hook.Intent.Contains("Call", StringComparison.Ordinal);

translator/src/Translator.Core/Mods/ContinuationPlanner.cs

@@ -275,7 +275,13 @@ public static class ContinuationPlanner
         AddSigned
     }
 
-    public static IEnumerable<int> DiscoverLrRelativeIndirectJumpOffsets(IReadOnlyList<PpcInstruction> instructions)
+    // Dropped states make a negative result inconclusive.
+    private const int MaxStatesPerInstruction = 512;
+
+    public static IEnumerable<int> DiscoverLrRelativeIndirectJumpOffsets(
+        IReadOnlyList<PpcInstruction> instructions,
+        Action? onStateCapExceeded = null,
+        Action? onUnresolvedExit = null)
     {
         if (instructions.Count == 0)
         {
@@ -296,7 +302,6 @@ public static class ContinuationPlanner
 
         var seenOffsets = new HashSet<int>();
         var worklist = new Queue<(int Index, PathState State)>();
-        const int MaxStatesPerInstruction = 16;
 
         void Enqueue(int targetIndex, PathState stateToEnqueue)
         {
@@ -325,6 +330,7 @@ public static class ContinuationPlanner
 
             if (visited[idx].Count > MaxStatesPerInstruction)
             {
+                onStateCapExceeded?.Invoke();
                 continue;
             }
 
@@ -484,6 +490,9 @@ public static class ContinuationPlanner
                     yield return state.CtrOffset.Value;
                 }
 
+                if (!state.CtrOffset.HasValue && instruction.BranchTargets.Count == 0)
+                    onUnresolvedExit?.Invoke();
+
                 nextState = nextState.WithCtrOffset(null);
                 if (instruction.BranchTargets.Count == 0)
                 {
@@ -495,6 +504,8 @@ public static class ContinuationPlanner
                 (mnemonic.StartsWith("b", StringComparison.Ordinal) && mnemonic.EndsWith("lr", StringComparison.Ordinal)));
             if (isReturn)
             {
+                if (!state.LrReturnOffset.HasValue)
+                    onUnresolvedExit?.Invoke();
                 if (state.LrReturnOffset.HasValue && state.LrReturnOffset.Value != 0 && seenOffsets.Add(state.LrReturnOffset.Value))
                 {
                     yield return state.LrReturnOffset.Value;
@@ -524,7 +535,7 @@ public static class ContinuationPlanner
                     Enqueue(fallthrough.Value, nextState);
                 }
 
-                if (!isReturn)
+                if (!isReturn && !instruction.IsCall)
                 {
                     foreach (var target in instruction.BranchTargets)
                     {

translator/src/Translator.Core/Mods/LrContinuationAnalysis.cs

@@ -0,0 +1,83 @@
+using Translator.Core.Disassembly;
+
+namespace Translator.Core.Mods;
+
+// Opaque exits prevent classification; truncated exploration may also hide offsets.
+public sealed record LrContinuationAnalysis(IReadOnlyList<int> Offsets, bool IsComplete, bool WasTruncated)
+{
+    public bool MaySkipReturn => !IsComplete || Offsets.Count != 0;
+
+    public static LrContinuationAnalysis Analyze(
+        uint entryPoint,
+        Func<uint, IReadOnlyList<PpcInstruction>> discover)
+    {
+        const int maxFunctions = 256;
+        const int maxInstructions = 65536;
+        var instructions = new Dictionary<uint, PpcInstruction>();
+        var pending = new Queue<uint>();
+        var visited = new HashSet<uint>();
+        var complete = true;
+        var truncated = false;
+        pending.Enqueue(entryPoint);
+
+        while (pending.TryDequeue(out var entry))
+        {
+            if (instructions.ContainsKey(entry) || !visited.Add(entry))
+                continue;
+            if (visited.Count > maxFunctions)
+            {
+                complete = false;
+                truncated = true;
+                break;
+            }
+
+            IReadOnlyList<PpcInstruction> body;
+            try
+            {
+                body = discover(entry);
+            }
+            catch (Exception ex) when (ex is InvalidOperationException or ArgumentException
+                or IndexOutOfRangeException or NotSupportedException or OverflowException)
+            {
+                complete = false;
+                truncated = true;
+                continue;
+            }
+            if (!body.Any(instruction => instruction.Address == entry) ||
+                instructions.Count + body.Count > maxInstructions)
+            {
+                complete = false;
+                truncated = true;
+                continue;
+            }
+
+            foreach (var instruction in body)
+                instructions.TryAdd(instruction.Address, instruction);
+            foreach (var instruction in body)
+            {
+                if (!instruction.IsCall)
+                {
+                    foreach (var target in instruction.BranchTargets)
+                        if (!instructions.ContainsKey(target)) pending.Enqueue(target);
+                }
+                if ((!instruction.IsReturn && !instruction.IsUnconditionalBranch) ||
+                    instruction.IsConditionalBranch)
+                {
+                    if (!instructions.ContainsKey(instruction.EndAddress))
+                        pending.Enqueue(instruction.EndAddress);
+                }
+            }
+        }
+
+        if (!instructions.TryGetValue(entryPoint, out var first))
+            return new LrContinuationAnalysis([], false, true);
+
+        // Keep the original entry first, including when a tail target precedes it.
+        var ordered = new[] { first }.Concat(instructions.Values
+            .Where(instruction => instruction.Address != entryPoint)
+            .OrderBy(instruction => instruction.Address)).ToArray();
+        var offsets = ContinuationPlanner.DiscoverLrRelativeIndirectJumpOffsets(
+            ordered, () => { complete = false; truncated = true; }, () => complete = false).ToArray();
+        return new LrContinuationAnalysis(offsets, complete, truncated);
+    }
+}