@ApacheSpark varint nice addition indeed, howvr why nested [] of structs w/ explodes and other complex nesting schema usages are still not properly handled from column prunning perspective and consequently have worse performance...specifying schema on read helps but its not a sol
Schema-on-Read Without Regrets: Lessons Learned from Apache Spark Variant Shredding in Production x.com/i/broadcasts/1…
