EventJournal: after #7461, a custom journal cannot decode EventLog payloads with one union codec #8476

What version of Effect is running?

effect 4.0.0-rc.117 (upgrading from 4.0.0-rc.112)

What steps can reproduce the bug?

A custom EventJournal receives write({ event, primaryKey, payload }) from EventLog and must decode payload itself, for example to validate the entry or to store it in its own format. On rc.112 the payload was MessagePack, which describes itself, so a journal could decode every event in a group with one codec built from the union of the payload schemas.

#7461 removed effect/unstable/encoding/Msgpack. The obvious port, Msgpack.schema(Union) → SchemaBinary.toCodec(Union), compiles and type-checks, but it rejects every payload that EventLog writes. SchemaBinary bytes follow the layout of the codec that wrote them, and EventLog writes each payload with its own event's payloadSchemaBinary, not with a codec for the union.

Minimal repro (both versions installed side by side):

{
  "type": "module",
  "dependencies": {
    "effect-old": "npm:[email protected]",
    "effect": "4.0.0-rc.117"
  }
}

import * as OldSchema from 'effect-old/Schema'
import * as OldMsgpack from 'effect-old/unstable/encoding/Msgpack'
import * as OldEventGroup from 'effect-old/unstable/eventlog/EventGroup'
import * as Schema from 'effect/Schema'
import * as SchemaBinary from 'effect/unstable/encoding/SchemaBinary'
import * as EventGroup from 'effect/unstable/eventlog/EventGroup'

const oldPayloads = [
  OldSchema.Struct({ name: OldSchema.Literal('user/create'), id: OldSchema.String }),
  OldSchema.Struct({ name: OldSchema.Literal('user/rename'), id: OldSchema.String, to: OldSchema.String })
] as const
const payloads = [
  Schema.Struct({ name: Schema.Literal('user/create'), id: Schema.String }),
  Schema.Struct({ name: Schema.Literal('user/rename'), id: Schema.String, to: Schema.String })
] as const

const OldGroup = OldEventGroup.empty
  .add({ tag: 'user/create', primaryKey: (p) => p.id, payload: oldPayloads[0] })
  .add({ tag: 'user/rename', primaryKey: (p) => p.id, payload: oldPayloads[1] })
const Group = EventGroup.empty
  .add({ tag: 'user/create', primaryKey: (p) => p.id, payload: payloads[0] })
  .add({ tag: 'user/rename', primaryKey: (p) => p.id, payload: payloads[1] })

const input = { name: 'user/rename', id: 'u1', to: 'Ada' } as const

// rc.112: EventLog writes event.payloadMsgPack bytes; one union codec decodes them.
const oldBytes = OldSchema.encodeSync(OldGroup.events['user/rename'].payloadMsgPack)(input)
console.log('rc.112 union decode:', OldSchema.decodeUnknownSync(OldMsgpack.schema(OldSchema.Union(oldPayloads)))(oldBytes))

// rc.117: EventLog writes event.payloadSchemaBinary bytes; the direct port of the same journal code fails.
const bytes = Schema.encodeSync(Group.events['user/rename'].payloadSchemaBinary)(input)
Schema.decodeUnknownSync(SchemaBinary.toCodec(Schema.Union(payloads)))(bytes)

Run it with bun repro.ts.

What is the expected behavior?

Either a journal can decode an EventLog payload without knowing which event wrote it, as on rc.112, or the API makes the per-event contract hard to miss.

What do you see instead?

rc.112 union decode: { name: "user/rename", id: "u1", to: "Ada" }
SchemaError(Missing key)

The union codec fails for every member of the union, not only for later ones. So every EventLog write through such a journal fails at runtime, and nothing in the types warns about it. The fix is to look up group.events[tag].payloadSchemaBinary for each entry, but nothing in the EventJournal docs or the #7461 changeset points to it.

Additional information

Possible fixes, from smallest to largest:

1. Document on EventJournal.write and Entry.payload that the bytes are encoded with the writing event's payloadSchemaBinary, so a reader must decode them with that same codec.
2. Add a migration note to the Remove MessagePack support #7461 changeset: SchemaBinary.toCodec(schema) is not a drop-in replacement for Msgpack.schema(schema) when the reader's schema differs from the writer's, for example a union reading a member's bytes.
3. Export a helper that decodes an entry for a group, for example EventGroup.decodePayload(group, entry), so custom journals do not have to write their own lookup by tag.