Repository navigation
Improve the memory usage of SnappyFramedInput/OutputStreams #81
Description
@rxin reported the memory usage of SnappyFramedFormat was unexpectedly high compared to SnappyOutput/InputStreams using jmap -histo:live dump command:
https://gist.github.com/rxin/83d94bdf882d5f7f8bed
Activity
bokken commentedon Jul 19, 2014
What was the actual test scenario?
bokken commentedon Jul 19, 2014
Also, was there a report of a performance problem or memory leak? Or just a report that larger byte[] objects are created?
rxin commentedon Jul 19, 2014
I just created 1000 SnappyFramedOutputStream streams and the memory usage was much higher than SnappyOutputStream (see the jmap dump).
bokken commentedon Jul 19, 2014
What type of source (for input stream) and destination (for output stream)
did you use?
Is it possible that this simply shows the use of 1 more buffer and buffers
which are larger?
On Jul 18, 2014 10:41 PM, "Reynold Xin" [email protected] wrote:
rxin commentedon Jul 19, 2014
@bokken I'm not sure what you were suggesting. My test was very simple actually. I opened 1k FileOutputStream and 1k SnappyFramedOutputStream. I wrote 1 character to it.
bokken commentedon Jul 22, 2014
My point is that each time you create the Framed implementation streams they allocate an ~64KB byte[]. The SnappyOutputStream also allocates ~64KB (2 ~32KB byte[]) on instantiation, but this did not appear to be a case you posted results for. The big difference is on the input stream side where the framed implementation automatically allocates the ~64KB array based on the documented max fragment size in the format. The SnappyInputStream defers allocating byte[] until individual fragments are read. In practice, where streams of data are almost certainly going to contain more than 1 character of data and fragments are likely going to be around the full size, there is little difference between the 2 implementations. The biggest difference is the default size of fragment used (32KB vs 64 KB).
[-]Improve the memory usage of Snappy streams[/-][+]Improve the memory usage of SnappyFramedInput/OutputStreams[/+]xerial commentedon May 13, 2015
This is no longer a problem. Now I close the ticket.