Repository navigation
Proposal: support opt-in streaming for multipart file uploads #3407
Description
Summary
I would like to propose an opt-in streaming mode for Swagger 2.0
multipart/form-data file parameters in generated go-swagger servers.
The current generated binding flow parses the complete multipart form before
invoking the application handler. As a result, the handler cannot consume or
forward file bytes while the client is still uploading them.
The existing behavior is useful and should remain the default. The proposal is
to add an explicitly enabled streaming alternative.
Current behavior
For a parameter such as:
- name: file
in: formData
required: true
type: filego-swagger v0.35.3 generates a BindRequest implementation that calls:
runtime.BindForm(
r,
runtime.BindFormMaxParseMemory(...),
runtime.BindFormMaxBody(...),
runtime.BindFormFile("file", true, o.bindFile),
)The generated file validation also seeks to the end of the parsed
multipart.File to determine its size.
Therefore, even though the generated handler parameter is an io.ReadCloser,
the application receives it only after the request body has already been
parsed.
Reproducer
Repository:
https://github.com/fpawel/go-swagger-multipart-streaming-repro
Exact tested revision:
fpawel/go-swagger-multipart-streaming-repro@0923884
The repository contains:
- a committed server snapshot generated by go-swagger v0.35.3;
- a multipart client that pauses after sending its first 64 KiB;
- a deterministic in-process comparison;
- a restricted streaming prototype based on
multipart.Reader; - captured client and server logs.
Reproduction with the generated server
Start the generated server:
make run-currentIn another terminal:
make client-currentRelevant output:
Client:
19:24:23.954257 client wrote 65536 bytes
19:24:23.954261 client paused for 3s
19:24:26.955331 client resumed
19:24:26.963224 multipart body closed
Generated server:
19:24:26.967635 application handler started
19:24:26.968255 application handler finished: 8388608 bytes
The application handler starts only after the client has resumed, sent the
remaining bytes, and closed the complete multipart body.
Streaming proof of concept
Start the prototype:
make run-streamingIn another terminal:
make client-streamingRelevant output:
Client:
19:24:46.859657 client wrote 65536 bytes
19:24:46.859665 client paused for 3s
19:24:49.860783 client resumed
19:24:49.871595 multipart body closed
Streaming prototype:
19:24:46.860641 application handler received "payload.bin"
19:24:49.872391 application handler finished: 8388608 bytes
Here the application receives the file part while the client is still paused,
approximately three seconds before the remaining bytes are uploaded.
The deterministic comparison also reports:
current:
application started while client was paused: false
streaming-poc:
application started while client was paused: true
Use case
An application may want to forward an incoming upload directly to object
storage or another downstream service:
HTTP request body
-> multipart part reader
-> application handler
-> object storage writer
With the current binder, this cannot be a true end-to-end streaming pipeline
because the generated handler is invoked only after multipart parsing has
completed.
Expected opt-in behavior
For an explicitly opted-in operation or file parameter, the generated handler
would receive a reader after the matching multipart part headers have been
parsed, while the part body is still arriving from the client.
The existing runtime.BindForm behavior should remain unchanged for all
operations that do not opt in.
One possible direction, only as a discussion starter, could be a vendor
extension:
- name: file
in: formData
required: true
type: file
x-go-streaming: trueThe exact extension name and whether it belongs at operation or parameter level
would need maintainer feedback.
Prototype scope
The prototype is intentionally restricted and is not presented as a drop-in
replacement for runtime.BindForm.
It currently assumes:
- exactly one streamed file part;
- the streamed file is the first multipart part;
- no trailing form fields;
- explicit request-body and file-size limits;
- the application consumes the reader before returning;
- existing generated APIs remain unchanged without opt-in.
Design questions
- Should streaming be enabled at operation level or file-parameter level?
- Should the runtime expose a separate streaming multipart binder?
- Who owns and closes the multipart part reader?
- How should generated
minLengthandmaxLengthvalidation work when the
size is only known while the handler reads the stream? - How should read-time limit violations be mapped to generated responses?
- Should form fields preceding the streamed file be supported initially?
- Should trailing form fields or multiple streamed files be explicitly
rejected in the first version? - Would maintainers prefer this proposal to be split between
go-swagger/go-swaggerandgo-openapi/runtime?
Environment
Go: go1.26.4 darwin/arm64
go-swagger: v0.35.3
runtime: github.com/go-openapi/runtime v0.32.6
OS: macOS 26.5.2, arm64
I would be interested in implementing this upstream once the desired public API
and initial scope are agreed upon.
Activity
fredbi commentedon Jul 22, 2026
Interesting idea.
Today we do not stream because we rely on stdlib net/http#Request.ParseMultipartForm, which consumes the whole body.
That part is handed over to go-openapi/runtime. So if you want to build a better (streaming) multipart handler, it should be there.
The codegen wiring should be limited to apply the proper options and take the right decision (e.g. if there is a maxLength validation
present, we must Seek). I've peeked a few time inside the standard library implementation: it tries to solve a lot of quirks with
multipart that are not always easy to resolve.
It could be a nice achievement indeed.
About your questions:
size is only known while the handler reads the stream? validations should disqualify streaming
rejected in the first version? no - there is no "first version"
go-swagger/go-swagger and go-openapi/runtime? essentially runtime, go-swagger would enable the right knobs
fredbi commentedon Jul 22, 2026
the spec is for both client and server, but I understand that your "x-go-streaming" only applies to the server
(our client already streams).
fpawel commentedon Jul 22, 2026
@fredbi Thanks, that clarifies the direction.
Yes, I intended x-go-streaming as a server-side code generation hint only. The generated client behavior should remain unchanged, since it already streams multipart uploads. The extension name was only a placeholder; something more explicit such as x-go-server-streaming may be less ambiguous, but I would prefer to define the name only after the runtime API and ownership semantics are clear.
My current understanding is:
I’ll inspect the existing client-side streaming/error handling and the current BindForm flow, then prepare a runtime-oriented design proposal before starting implementation.
fredbi commentedon Jul 22, 2026
Good.
Just fine.
Ideally yes. But as you mentioned, it may be more complicated than that to control how the readers / writers are closed / drained.
In the client we already have such an unfortunate legacy with unwaited for goroutines that can't be surfaced through the existing interface. So this is a real design problem. So indeed:
fredbi commentedon Jul 22, 2026
Another way could be to bind an almost transparent reader (it would just parse the multipart stuff and pass on the payload) and expose it to the handler, which would be free to do whatever they like with it. In that case, we'd just bind and leave up to the handler to consume it.
fpawel commentedon Jul 22, 2026
That alternative sounds promising.
I’ll explore a lazy, almost transparent multipart reader owned by the handler. The runtime would parse only the multipart framing and headers necessary to expose the file payload, without spawning background goroutines or consuming the complete request body beforehand.
The handler would then be responsible for consuming and closing the reader. The main design points I’ll investigate are:
I’ll prepare a small runtime-level prototype and compare this approach with extending the current BindForm behavior before proposing a concrete API.