WebStream API support

This has been sort of worked on in #140 but then was continued in another branch #140 (comment).

Is the WebStream API, that is published in FF & Chrome, like getReader() etc. too experimental for node-fetch?

Not sure it would be useful in a nodejs enviorment where everything is written with node streams.
the WebStream api isn't small and would add quite alot to node-fetch dependency.

There is a grate package called node-web-streams for conversion between those two

const webStreams = require('node-web-streams');
const readableStream = webStreams.toWebReadableStream(nodeReadable);

if we where to implement ReadableStream in node-fetch I'm guessing most ppl would just to the opposite and try to convert it back to a node stream

It would be a breaking change since Response.prototype.body is now a node stream and some are already using it.

I've recently been implementing the cloudflare-workers API as a local server for development, and support for WebStreams is something we need. Would it be possible to support passing in a set Stream constructors for node-fetch to use?

That way if people want the web-streams API, they can opt-in to it.

@ThisIsMissEm

@ThisIsMissEm There are nontrivial differences in the stream.Readable API vs. the web ReadableStream API, and using the constructors of the two classes the same way unfortunately doesn't work. See some examples for Web Streams vs. Node.js Streams.

There have some exciting developments over on the Node.js side (nodejs/node#22352), and if Web Streams eventually get in Node core we can certainly add support for it.

(Does this answer your question or am I missing something you are trying to say?)

Yeah, we realised this was the case, so we've forked node-fetch to use ReadableStream API. You can see the start of our attempt here: master...titel-media:whatwg-readablestreams

Whilst my usage of node-fetch is for reimplementing the API of CloudFlare Workers, I think it’s worth noting that several modules that depend on node-fetch claim that they expose the whatwg-fetch spec, which isn’t true, as that’d mean they expose whatwg streams; They’ve possibly missed the note about difference in stream APIs

@TimothyGu

@TimothyGu We actually have pretty much full tests passing — I removed a bunch of stuff and can do a PR, but in order for my team to move forward we'll be publishing a scoped fork. It's up to you if you want to take my changes into node-fetch, I'm happy to have a fork for now.

Found out something that is going to make this cross browser/node chunk reading much simpler! Async iterator!!!

Node implemented Symbol.asyncIterator into there stream specification (v10) and whatwg are planing on adding them too, until then here is a polyfill for the browser:

if (!ReadableStream.prototype[Symbol.asyncIterator]) {
  ReadableStream.prototype[Symbol.asyncIterator] = async function* () {
    const reader = this.getReader()
    while (1) {
      const r = await reader.read()
      if (r.done) return value
      yield r.value
    }
  }
}

now what you can do in both the browser and Node is:

const res = await fetch('https://httpbin.org/bytes/500000')
for await (const chunk of res.body) {
  console.log(chunk)
}

Both yields a Uint8Array array (well, a buffer in node, but they are inherited by Uint8array) so you can treat them the same

Hopefully this will make the WebStream api for node-fetch more irrelevant
(still think the node-web-streams is to much as a dependency)

this is something i would very much like to see in the readme 😛

@jimmywarting

@jimmywarting Unfortunately, async iterators don't provide a full replacement for streams with regards to things like backpressure. This is obviously a step in the right direction but there is still value in exposing Web streams.

For context, a PR that adds async iterators to web streams is whatwg/streams#950.

@jimmywarting

@jimmywarting wow! That makes it SO much nicer than using the reader / while loop interface.

@TimothyGu

@TimothyGu wouldn't backpressure with async iterator just be handled via something like:

const res = await fetch('https://httpbin.org/bytes/500000')
for await (const chunk of res.body) {
  await doSomework(chunk)
}

So because we're delaying in each iteration backpressure applies?

Hm, I guess I was more referring to the highWatermark-based backpressure. That does indeed work, yes.

@TimothyGu

Okay, so, an update from my end: We're publishing a fork with ReadableStreams as @titelmedia/node-fetch, such that we can use it in our Cloudflare Worker development server. It's up to you @TimothyGu if you choose to adopt the changes I've made.