Repository navigation
Parallelize reads - #1607
Parallelize reads#1607
Conversation
|
@lossyrob Grisha has opted here not to explicitly use He was seeing similar issues to what we had before, where Tasks weren't appearing to complete and using |
|
Without dedicated pool it will end up using
I think that works out alright for spark tasks |
| } | ||
|
|
||
| tileSeq.flatten | ||
| nondeterminism.njoin(maxOpen = 8, maxQueued = 8) { range map read }.runFoldMap(identity).unsafePerformSync |
There was a problem hiding this comment.
Notice 8 is magic number here but 32 is for Cassandra and File. Did it benchmark to be best that way?
There was a problem hiding this comment.
It would be best if we could configure this somehow, and not have magic numbers impossibly un-magicked
There was a problem hiding this comment.
How do you feel about using typesafe config for this? This is probably a system wide property rather than per-action property.
There was a problem hiding this comment.
just experimented how tests would run faster on a local machine; though they are still magical. probably it makes sense to make them configurable, as speed of them depends on certain machine(s) configuration.
|
@echeipesh in terms of scalaz, it would use def njoin[A](maxOpen: Int, maxQueued: Int)(source: Process[Task, Process[Task, A]])(implicit S: Strategy): Process[Task, A] |
|
This continues to be an informative PR. |
Uh oh!
There was an error while loading. Please reload this page.