[SPARK-1429] Spark shell fails to start after "sbt clean assemble-deps package" - #337

Mllib 16 bugfix

Bug fix: https://spark-project.atlassian.net/browse/MLLIB-16

Hi, I fixed the bug and added a test suite for `GradientDescent`. There are 2 checks in the test case. First, the final loss must be lower than the initial one. Second, the trend of loss sequence should be decreasing, i.e., at least 80% iterations have lower losses than their prior iterations.

Thanks!

Remove duplicate job define "cloud-provider-openstack-test"

### What changes were proposed in this pull request?

Step 4 of task 87 (`sql/varka/plans/PLAN_TASK_87.md`, sections 3.1 and 8): behind the `methodByteBudget` switch, the epilogue is emitted as one method per output group beside its loop method, `epilogueDense<g>` and `epilogueMasked<g>`, instead of one method over every output. Stacked on apache#337 (step 3a); the diff here is the last commit once that merges.

An emitted kernel has three kinds of method: a driver per side, a loop method per group of outputs, and the epilogue, which runs the loop body once over the partial lane group past `loopBound`. The loop was grouped from the start (task 10) because a hot method's C2 compile has a node budget; the epilogue was left as one method because it runs once per batch and that budget had nothing to bound in it. The JVM has a second limit that reads bytes rather than heat: `DontCompileHugeMethods` refuses any method over `HugeMethodLimit` (8000 bytes) at every tier, and a single epilogue carrying every output's tail crosses it at thirteen `make_date` outputs (apache#336's test), after which every ragged batch runs that method in the interpreter for the life of the JVM.

The change splits the epilogue by the loop's groups. `VarkaLoopEmitter` emits `epilogue<side><g>` next to `loop<side><g>` when the switch is set, `VarkaBodyEmitter`'s `EPILOGUE` arm takes a group like the `LOOP` arm does and plans and sets up only that group (step 3a's per-group prologue, now for both kinds of method), and the driver calls each epilogue after the loops, OR-ing the statuses as it does for the loops. Each epilogue keeps its own even-batch return rather than the driver testing once for all of them: on a scan whose batches divide evenly, the calls that return at once are what warm the method past C2's invocation threshold (plan 2.6.3).

From `dev/varka_emit.sh`, the `make_date` ladder, bytes with the `IntVector` count beside them:

| outputs | single `epilogueMasked` | `epilogueMasked<g>` under the switch | `runMasked` |
| ---: | :--- | :--- | :--- |
| 16 | 10314 (918) | 3257, 3269, 3844, 1236 (313, 313, 313, 93) | 824 -> 888 |
| 60 | 41338 (3338) | 3257 to 3972, twelve of them (313 each) | 2700 -> 2924 |

Every method of the ladder now fits `HugeMethodLimit` through sixty outputs at both widths, and each `epilogue<g>` carries exactly its `loop<g>`'s op count - one lane group of the same outputs. The sum over the epilogues is larger than the single method's by the shared civil-from-days prefix repeated per group (12 x 313 = 3756 against 3338; the difference is 11 x 38), which section 3.3 of the plan registered. The driver gains one call per epilogue, about twenty bytes each.

Also here: `VarkaCompilationWatch`'s class doc names the per-group epilogues, `VarkaEmittedBytesSuite`'s option-arm inventory lists `methodByteBudget=8000`, and section 9.3 of the plan records the ladder and what it leaves for step 5 (the regroup, which on this family has nothing to do).

### Why are the changes needed?

It is the property task 87 exists for: a kernel past the legacy crossing whose every method the JVM compiles. The switch stays off in this PR; step 6 measures both forms with `VarkaMethodSizeBenchmark` and flips the default.

### Does this PR introduce _any_ user-facing change?

No. Without the switch the emitted classes are byte-identical.

### How was this patch tested?

New `VarkaHugeMethodSuite` forks a JVM under `-Xbatch -XX:+PrintCompilation` (`VarkaHugeMethodProbe`, in the style of the width audit), runs the sixteen-output ladder hot on ragged batches, and reads the tiers HotSpot printed for the emitted class: under the switch every loop and epilogue method reaches tier 4; without it the single `epilogueDense` and `epilogueMasked` never appear at any tier while every loop method reaches tier 4. That is the crossing asserted from the JVM rather than inferred from a size.

A new test in `VarkaEmitterBudgetSuite` walks the ladder at 4, 8, 12, 13, 14, 16, 32 and 60 outputs at the JVM's width and at 128 bits: one epilogue per group, each with its loop method's `IntVector` count, the loop methods' counts unmoved by the switch, every method at most `HUGE_METHOD_LIMIT` bytes, and the sixteen-output kernel answering as the reference evaluator at lengths 1, 1024 and 1031 on both bodies. The step 3a test's driver assertion now allows the epilogue calls it gains.

The whole `codegen.varka` test package passes locally (279 tests, 27 cancelled as opt-in or hsdis-gated). `dev/scalastyle`, `dev/lint-java`, `dev/varka_quote_check.py` and the non-ASCII and 100-column scans pass.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5.1)

### What changes were proposed in this pull request?

Step 6a of task 87 (`sql/varka/plans/PLAN_TASK_87.md`, section 6): `VarkaMethodSizeBenchmark` gains its second arm and its committed files are regenerated with both forms in one table per rung, banded over ten runs. Stacked on apache#339 (step 5); the diff here is the last two commits once that merges.

The two forms are named explicitly so the labels survive the default changing: `single epilogue`, the legacy emission (`methodByteBudget` 0), and `epilogue per group`, the emission under the budget at `HugeMethodLimit` - a group's methods set up only their group (apache#337), the epilogue is one method per group (apache#338), and the class is measured and regrouped (apache#339). `VarkaEmitBudget` is public so a benchmark outside the emitter's package can name the limit. The results, in M rows/s at 512 bits on the masked arm (the arm that reaches the single epilogue's crossing at thirteen outputs):

| outputs | even batches | ragged batches |
| ---: | :--- | :--- |
| 4 | 441.9 -> 434.1 | 431.2 -> 442.8 |
| 12 | 134.0 -> 143.1 | 135.2 -> 142.3 |
| 13 | 120.9 -> 132.9 | 40.7 -> **131.7** |
| 16 | 94.4 -> 106.3 | 32.9 -> 104.9 |
| 60 | 24.1 -> 29.4 | 8.7 -> 28.9 |

The cliff is gone: on ragged batches the per-group form reads within a few percent of its own even-batch rate at every rung, where the single form had lost three quarters of its rate from thirteen outputs. And from twelve outputs the per-group form is faster on even batches too, by 7% at twelve and 22% at sixty, which no prediction had: each loop method now sets up only its group's segments and literals, and that outweighs the driver's extra epilogue calls. The 128-bit file reads the same way (13 outputs ragged: 24.9 -> 44.0). Section 9.5 of the plan scores the registered predictions: 3 held and understated the gain, 4 fell on the fast side, 5 is superseded by the design; the default's rule - nowhere worse beyond its band on the masked arm at any rung, length or width, and on the null-free arm up to twelve outputs - is met, and the flip is step 6b.

One lead for task 189: in this fork the single form's null-free rows at 128 bits from twelve outputs sit in the deoptimization cycle (0.3 to 0.7 M rows/s ragged) and the per-group form's do not (40 to 56). One fork proves nothing about a bimodal effect; the plan records it as a lead, not a property.

### Why are the changes needed?

The plan's rule for flipping the default is a measurement, and this is it: both forms in one file, at both widths, with the band that says which differences mean anything.

### Does this PR introduce _any_ user-facing change?

No. The default is unchanged in this PR.

### How was this patch tested?

The benchmark is the test: `dev/varka_bench_regen.sh catalyst VarkaMethodSizeBenchmark` (pinned, load 0.57 at start, canary clean, invariants ok at both widths) and `dev/varka_bench_repeat.sh catalyst VarkaMethodSizeBenchmark 10 --band`. `dev/varka_quote_check.py` passes for every number quoted in the plan. The suites do not change.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5.1)