support for Kinesis - #223

Mark partitioner, name, and generator field in RDD as @transient.

As part of the effort to reduce serialized task size.
(cherry picked from commit d6e5473)

Signed-off-by: Patrick Wendell <[email protected]>

Mark partitioner, name, and generator field in RDD as @transient.

As part of the effort to reduce serialized task size.
(cherry picked from commit d6e5473)

Signed-off-by: Patrick Wendell <[email protected]>

(cherry picked from commit 13f16d5)

* Made the HDFS keytab secret configurable.

* Updated dcoscli version

### What changes were proposed in this pull request?
The pr aims to upgrade `commons-codec` from `1.16.0` to `1.16.1`.

### Why are the changes needed?
1.The new version brings some bug fixed, eg:
- Fix possible IndexOutOfBoundException in PhoneticEngine.encode method #223. Fixes [CODEC-315](https://issues.apache.org/jira/browse/CODEC-315)
- Fix possible IndexOutOfBoundsException in PercentCodec.insertAlwaysEncodeChars() method #222. Fixes [CODEC-314](https://issues.apache.org/jira/browse/CODEC-314).

2.The full release notes:
    https://commons.apache.org/proper/commons-codec/changes-report.html#a1.16.1

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #45152 from panbingkun/SPARK-47083.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

The pr aims to upgrade `commons-codec` from `1.16.0` to `1.16.1`.

1.The new version brings some bug fixed, eg:
- Fix possible IndexOutOfBoundException in PhoneticEngine.encode method #223. Fixes [CODEC-315](https://issues.apache.org/jira/browse/CODEC-315)
- Fix possible IndexOutOfBoundsException in PercentCodec.insertAlwaysEncodeChars() method #222. Fixes [CODEC-314](https://issues.apache.org/jira/browse/CODEC-314).

2.The full release notes:
    https://commons.apache.org/proper/commons-codec/changes-report.html#a1.16.1

No.

Pass GA.

No.

Closes #45152 from panbingkun/SPARK-47083.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

…apers read against the engine, and Gandiva as prior art (apache#223)

### What changes were proposed in this pull request?

This pull request grows the milestone 6 scope catalogue, `sql/varka/plans/SCOPE_MILESTONE_6.md`, with a survey of eleven systems, HotSpot itself, and twenty-seven papers read against Varka's design on 16 September 2026, and records what each has that Varka should take, what Varka arrived at independently, and where Varka is right to differ. Every item names the files and pages it read so the record can be checked. Nothing in the source changes.

**Items 16 to 24, one system each.** PolarDB-X (per-node fallback via derived inputs, widen-on-overflow, selection narrowing, adaptive dense-or-selection batches), Apache Druid (statistics-driven bucket short-circuit, cost-ordered conjuncts), Velox (dictionary peeling with cross-batch memoisation, the clocks-per-row-eliminated conjunct metric, benchmark sets as differential tests), Trino (conjunct reordering allowed only for terms that cannot fail, which corrects Item 18 since task 84's range analysis is Varka's per-node proof of "cannot fail", and a CPU-flag gate on `compress`), DuckDB (the monotone-preimage rewrite that turns `year(d) = 2021` into a range on `d`, and statistics propagated through the calendar parts as a refinement of Item 15), DataFusion (an interval lattice run in both directions, whose top-down pass lets a conjunct retire a sibling's guard; the preimage declared per function as half-open intervals; selectivity from the same lattice), Arrow compute (a comparison proved true under a guarantee becomes the operand's validity bitmap; `PEXT` trusted on Intel only, so the compress gate has two instructions to consider, with HotSpot's own AVX2 lowering read from its match rules), ClickHouse's evaluator, scoped to what Item 10's calendar read left (lazy arguments of `if`, `and` and `or` evaluated only on undecided rows and only when the function says it can throw or is heavy, which answers when to narrow rather than blend; a null-ratio threshold that filters before an expensive kernel), and Comet, the one system with Varka's exact contract (a per-expression kill switch; the batch kernel compiled from Spark's own generated code to evaluate one inexact expression inside the columnar pipeline, which is Item 16's per-node fallback built and shipped; a test form asserting both engines throw or both agree; and the evidence that raising Spark's ANSI errors natively costs a subsystem with a published list of residual differences, which is why Varka declines the batch instead).

**Item 25, the papers.** Fifteen SIMD query-execution papers the owner downloaded, transcribed mechanically by the method `sql/varka/papers/README.md` describes and read against the record. The three structural choices Varka made without them are the ones they defend by measurement: boundary compaction (Lang 2020), fused loops for compute-bound work with the 68-against-162 instructions per tuple number (Kersten 2018), and explicit vector emission over the auto-vectoriser (Benson 2023). Nine amendments are applicable now, among them Lang's threshold rule for narrowing in place of "build both", re-measuring every selectivity policy and long multiply at 64-bit lanes before milestone 5 inherits an int32 number, a batch-size sweep derived from the touched set, a committed partitions ladder so the public write-up can state the concurrency regime, log-scale low-selectivity benchmark rungs, Photon's two testing forms, and a "would add a pivot" decline reason. Later items include the vertical probe and the fingerprint bucket as the two arms against the sorted `IN` chain, `BloomFilterMightContain` as a filter kernel on long lanes, and mask-to-bitmask as the first NEON measurement. Where two readings disagree, on whether narrowing should skip per lane group or per batch, the disagreement is recorded as the thing Item 16's task measures.

**Item 26, HotSpot read as a system.** JDK 25's instruction selectors and match rules, read as a table of Vector API operations against CPU features: a call ends as one instruction, a C2 sequence, or a silent Java fallback visible only under `PrintIntrinsics`. The table says which 64-bit operations are sequences or Java on the AVX2 runners milestone 5 will meet, that `fromLong` on AVX2 begins with `PDEP`, that scatter and long absolute value are AVX-512 only, that HotSpot itself defaults to AVX2 on Skylake server parts before Cascade Lake, and that NEON alone refuses masked access, compress, gather and scatter while SVE1 refuses `fromLong`. Four things to take: a no-fallback proof in CI from the JVM's own diagnostic output, the 64-bit operations to measure before publishing, two more selectors for Item 19's gate, and the refused list as the start of any ARM plan.

**Item 27, twelve more papers.** The papers Item 25 named as unread, read the same way: Ross 2002 and Menon 2017 on conjunction plans and the staging point; VOILA and Gubner 2017 on the design space, where FUJI's four components place Varka precisely and its Q6 result is Item 16's loss case; Kohn 2018 and Kersten 2021 on tiers, which map onto the JVM exactly and show that time-to-C2 is the only lever; Data Blocks, BLU and the vectorised Bloom filter on layouts, block statistics and the probe; Raasveldt 2018 and the two Gottschlag papers on benchmark credibility and the frequency licence. None prints a Creative Commons Attribution licence, so none is copied into the papers directory. Two claims about this repository were verified before being recorded: the surface benchmark's engine token also turns on the Arrow cache serializer while the README says the arms differ only by the engine flag, so a fifth arm is needed; and Spark's Bloom filter fixes the hash, index arithmetic and word layout, which corrects Item 25's Bloom note.

**Item 28, Gluten.** The other Spark accelerator, read on its Spark side since its kernels are Velox's and ClickHouse's. It arrived at the same surface as Comet and makes the strongest form of Item 18's choice by falling the whole plan back under ANSI. Three things are worth taking: a fallback policy that counts columnar-to-row transitions against a threshold and costs the reverted plan too, together with a fifty-expression threshold that leaves deep trees to whole-stage codegen, an accelerator's own statement of the regime a fused loop is built for; the partial project, a JVM island that converts only the columns the unsupported expressions read and composes the result back into the batch, which with Comet's codegen dispatch settles Item 16's derived-input question; and transitions planned as a shortest path over batch conventions with a rough cost table, which Item 11's second representation will need.

**Item 29, StarRocks.** A ClickHouse-descended vectorised engine that added an LLVM expression JIT late, the fifth engine here to leave lanes to the auto-vectoriser. Three things are worth taking: JIT admission by a benefit vote over the expression tree, compiled only when the score exceeds a ratio of the node count, which is the cheap form of the "would add a pivot" rule at Varka's row boundary; a popcount guard on 8-byte compress that copies survivors one by one below six set bits in thirty-two, with the source's own note that the instruction loses to scalar on down-clocking Intel parts, the row to add before milestone 5's 64-bit compaction is written; and `IN` as a dense bitset over the literal span with a range check, a third arm for date `IN` lists within a short span that needs no gather at all.

**Item 30, the JDK's own Vector API microbenchmarks.** The thirty hand-written cases the intrinsic authors keep, named against Varka's operations: a compress column filter forked at AVX2, the mask conversions per species, masked against partially masked logic, gathers and masked access past an array's end, segment access by arena kind and under profile pollution, and long-lane multiplies with thirty-two-bit operands. Three things are worth taking: a reference results file beside Item 26 that separates the engine's costs from the platform's; an emitter rule that a long-lane multiply with a widened-int or sub-thirty-two-bit operand is one instruction on every runner when the cast or mask stays adjacent, which C2 matches explicitly; and two cheap segment measurements. The cases are GPL with the Classpath exception and are run from the JDK tree, never copied.

**Four transcriptions** join `sql/varka/papers`, the ones whose printed terms are Creative Commons Attribution: Lang 2020, Ngom 2021, Benson 2023 and Schmidt 2025, each with the directory's header naming what it loses and which items read it. The other eleven are ACM copyright, BY-NC-ND or arXiv postings, so Item 25 keeps their notes and citations, as the directory's licensing rule says. The pre-commit non-ASCII check now exempts that directory the way it exempts benchmark results.

**`sql/varka/VISION.md` section 14** gains Gandiva as the closest architectural relative, paragraphs on Trino, DuckDB, DataFusion, ClickHouse and Comet, the third earlier JVM attempt (Shen's VEE, ICPP 2021, whose own decomposition shows plain X100-style vectorisation on Spark slower than whole-stage codegen on 21 of 22 TPC-H queries), and a paragraph on where the literature says the fused-loop argument holds and where it stops, at hash tables.

Candidate rows for milestone 5, to be filed together once apache#220 has merged so they land on the same table lines as task 123: the benchmark checksum per arm (Item 18), the compress and PEXT capability gate with an AVX2 measurement (Items 19, 22), the monotone-preimage rewrite (Item 20), the backward interval pass (Item 21), the per-expression kill switch (Item 24), and the amendments Item 25 lists.

### Why are the changes needed?

The milestone 6 catalogue is where ideas wait with their provenance until they become tasks. Each of these surveys was made once, against the record, so that no system or paper is read twice for the same question and so that a later task can cite a file and a page rather than a memory. Several findings change work already planned: Trino's rule corrects Item 18, DuckDB's and DataFusion's rewrites give the range analysis its next two steps, Comet settles Item 16's open design question, and the papers turn Item 16's "build both" into a rule with a measurement plan.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Documents only; the pre-commit checks, including the quote check, pass on every commit. The transcriptions were spot-checked against the PDFs (headings against the papers' section lists, and formulas with exponents in the ByteSlice paper) and no transcription was edited by hand. The fork CI runs are documents-only and are one more observation of task 106's docs gate.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5.1)

The milestone table and section 2 both grew on master while this branch was
open: task 120's row and section 2.55 outcome, task 122 and section 2.57 from
apache#220, and apache#223's scope work. Master's additions come first in both places -
section 2.57 before this branch's 2.59, row 122 before row 124 - and the gap at
2.58 and row 123 is the CI-gate task apache#224 named.