Various Jpeg optimizations by antonfirsov · Pull Request #768 · SixLabors/ImageSharp

antonfirsov · 2018-11-04T03:00:21Z

Prerequisites

I have written a descriptive pull-request title
I have verified that there are no overlapping pull-requests open
I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
I have provided test coverage for my change (where applicable)

Summary

This PR contains various optimizations for JpegDecoder, resulting in a ~10% speedup. We are decoding common 420/YCbCr baseline images only 2.4-2.5 slower than System.Drawing now.
I had a hard time squeezing the last drops, we probably can't do much more here before .NET Core 3.0.

Optimizations

Replace JpegComponent.GetBlockReference(x,y) with .GetRowSpan(y)
Introduce Image.CreateUninitialized<TPixel>(...) because decoders do not need pre-initialized image buffers. Introduced in Jpeg, can be used in other decoders as well.
Optimized Block8x8 -> Block8x8F conversion (Block8x8F.LoadFrom(...)) using Vector.Widen(...) and Vector.ConvertToSingle(...) (.NET Core 2.1 / .NET 4.7.2 only)
Optimized Block8x8F.CopyTo(...) for 2x2 scale cases
Optimized ExifReader.ToEnum<T>(...)

Chore

Refactor and group profiling tests
Add JpegSnoop reports for all images under TestImages\Input\Jpg
TestImages.Jpeg.BenchmarkSuite
Add/refactor benchmarks
Execute benchmarks add results

Benchmark results

Only .NET Core 2.1 for now, using the benchmark DecodeJpeg_ImageSpecific (renamed from DecodeJpeg).

Before

         Method |                                   TestImage |       Mean |      Error |    StdDev | Scaled | ScaledSD |     Gen 0 |    Gen 1 |    Gen 2 |   Allocated |
--------------- |-------------------------------------------- |-----------:|-----------:|----------:|-------:|---------:|----------:|---------:|---------:|------------:|
 System.Drawing |                       Jpg/baseline/Lake.jpg |   6.460 ms |   1.432 ms | 0.0809 ms |   1.00 |     0.00 |   62.5000 |        - |        - |   205.83 KB |
     ImageSharp |                       Jpg/baseline/Lake.jpg |  20.764 ms |   2.747 ms | 0.1552 ms |   3.21 |     0.04 |         - |        - |        - |    19.95 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing |                Jpg/baseline/jpeg420exif.jpg |  17.156 ms |   5.838 ms | 0.3298 ms |   1.00 |     0.00 |  218.7500 |        - |        - |   757.04 KB |
     ImageSharp |                Jpg/baseline/jpeg420exif.jpg |  46.414 ms |   8.867 ms | 0.5010 ms |   2.71 |     0.05 |         - |        - |        - |    21.92 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing | Jpg/issues/Issue518-Bad-RST-Progressive.jpg | 428.494 ms | 124.232 ms | 7.0193 ms |   1.00 |     0.00 | 2375.0000 |        - |        - |  7403.76 KB |
     ImageSharp | Jpg/issues/Issue518-Bad-RST-Progressive.jpg | 432.931 ms |  44.287 ms | 2.5023 ms |   1.01 |     0.01 |  125.0000 | 125.0000 | 125.0000 | 35187.05 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing |       Jpg/issues/issue750-exif-tranform.jpg |  96.404 ms |  45.786 ms | 2.5870 ms |   1.00 |     0.00 | 1750.0000 |        - |        - |  5492.63 KB |
     ImageSharp |       Jpg/issues/issue750-exif-tranform.jpg | 267.698 ms |  37.973 ms | 2.1456 ms |   2.78 |     0.06 |  750.0000 | 750.0000 | 750.0000 | 58833.47 KB |

After

         Method |                                   TestImage |       Mean |      Error |    StdDev | Scaled | ScaledSD |     Gen 0 |    Gen 1 |    Gen 2 |   Allocated |
--------------- |-------------------------------------------- |-----------:|-----------:|----------:|-------:|---------:|----------:|---------:|---------:|------------:|
 System.Drawing |                       Jpg/baseline/Lake.jpg |   6.117 ms |  0.3923 ms | 0.0222 ms |   1.00 |     0.00 |   62.5000 |        - |        - |   205.83 KB |
     ImageSharp |                       Jpg/baseline/Lake.jpg |  18.126 ms |  0.6023 ms | 0.0340 ms |   2.96 |     0.01 |         - |        - |        - |    19.97 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing |                Jpg/baseline/jpeg420exif.jpg |  17.063 ms |  2.6096 ms | 0.1474 ms |   1.00 |     0.00 |  218.7500 |        - |        - |   757.04 KB |
     ImageSharp |                Jpg/baseline/jpeg420exif.jpg |  41.366 ms |  1.0115 ms | 0.0572 ms |   2.42 |     0.02 |         - |        - |        - |    21.94 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing | Jpg/issues/Issue518-Bad-RST-Progressive.jpg | 428.282 ms | 94.9163 ms | 5.3629 ms |   1.00 |     0.00 | 2375.0000 |        - |        - |  7403.76 KB |
     ImageSharp | Jpg/issues/Issue518-Bad-RST-Progressive.jpg | 386.698 ms | 33.0065 ms | 1.8649 ms |   0.90 |     0.01 |  125.0000 | 125.0000 | 125.0000 | 35186.97 KB |
                |                                             |            |            |           |        |          |           |          |          |             |
 System.Drawing |       Jpg/issues/issue750-exif-tranform.jpg |  95.192 ms |  3.1762 ms | 0.1795 ms |   1.00 |     0.00 | 1750.0000 |        - |        - |  5492.63 KB |
     ImageSharp |       Jpg/issues/issue750-exif-tranform.jpg | 230.158 ms | 48.8128 ms | 2.7580 ms |   2.42 |     0.02 |  312.5000 | 312.5000 | 312.5000 | 58834.66 KB |

…marks

…Suite as a static class

codecov · 2018-11-04T13:49:10Z

Codecov Report

❗ No coverage uploaded for pull request base (master@a2e09f1). Click here to learn what that means.
The diff coverage is 77.52%.

@@            Coverage Diff            @@
##             master     #768   +/-   ##
=========================================
  Coverage          ?   88.49%           
=========================================
  Files             ?      997           
  Lines             ?    42504           
  Branches          ?     3150           
=========================================
  Hits              ?    37613           
  Misses            ?     4204           
  Partials          ?      687

Impacted Files	Coverage Δ
...arp/Formats/Jpeg/Components/Block8x8F.Generated.cs	`100% <ø> (ø)`
tests/ImageSharp.Tests/TestImages.cs	`100% <ø> (ø)`
...eSharp.Tests/ProfilingBenchmarks/JpegBenchmarks.cs	`0% <0%> (ø)`
...ts/ProfilingBenchmarks/LoadResizeSaveBenchmarks.cs	`0% <0%> (ø)`
...rc/ImageSharp/MetaData/Profiles/Exif/ExifReader.cs	`73.55% <100%> (ø)`
...sts/Formats/Jpg/Block8x8FTests.CopyToBufferArea.cs	`100% <100%> (ø)`
src/ImageSharp/Image.Decode.cs	`73.91% <100%> (ø)`
...g/Components/Decoder/JpegComponentPostProcessor.cs	`100% <100%> (ø)`
...ts/Formats/Jpg/Utils/LibJpegTools.ComponentData.cs	`73.41% <100%> (ø)`
.../ImageSharp.Tests/Formats/Jpg/Utils/JpegFixture.cs	`85.52% <100%> (ø)`
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2e09f1...9a12287. Read the comment docs.

JimBobSquarePants

Impressive! The benchmark results make for interesting stuff.

How long were you working on this, looks like it must have taken a while?

JimBobSquarePants · 2018-11-04T15:43:15Z

@@ -506,8 +515,7 @@ private void DecodeBlockProgressiveDC(

        private void DecodeBlockProgressiveAC(
            JpegComponent component,


component is unused now.

thanks! will fix.

JimBobSquarePants · 2018-11-04T15:44:10Z

        private Image<TPixel> PostProcessIntoImage<TPixel>()
            where TPixel : struct, IPixel<TPixel>
        {
+            var image = Image.CreateUninitialized<TPixel>(


I wonder where else we can apply this?

Everywhere, where we can make sure that the operation is about to fill the whole image.
So all decoders. Copying processors: not sure, they are a bit more complicated because of the swapping logic.

We might not be able to use it in gif as individual frames sometimes only contain data in some pixels but both png and bmp should definitely benefit.

JimBobSquarePants · 2018-11-04T15:45:34Z

+            [MethodImpl(InliningOptions.ShortMethod)]
+            public static bool IsDefined(int value)
+            {
+                return Array.BinarySearch(Values, 0, Values.Length, value) > 0;


JimBobSquarePants · 2018-11-04T16:05:24Z

+        //  'Decode Jpeg - System.Drawing' |                Jpg/baseline/jpeg420exif.jpg |  17.063 ms |  2.6096 ms | 0.1474 ms |   1.00 |     0.00 |  218.7500 |        - |        - |   757.04 KB |
+        //      'Decode Jpeg - ImageSharp' |                Jpg/baseline/jpeg420exif.jpg |  41.366 ms |  1.0115 ms | 0.0572 ms |   2.42 |     0.02 |         - |        - |        - |    21.94 KB |
+        //                                 |                                             |            |            |           |        |          |           |          |          |             |
+        //  'Decode Jpeg - System.Drawing' | Jpg/issues/Issue518-Bad-RST-Progressive.jpg | 428.282 ms | 94.9163 ms | 5.3629 ms |   1.00 |     0.00 | 2375.0000 |        - |        - |  7403.76 KB |


This makes me think that your IDCT/Colorspace code is very fast and we're still missing an optimization opportunity somewhere in the Huffman decoding process.

Looking through the MozJpeg/LibJpegTurbo code there's decode_mcu_fast and decode_mcu_slow in jdhuff.c which seems to back this up.

My thought is that S.D is slow for progressive images for some reason.

For progressive images with many scans (like Jpg/issues/Issue518-Bad-RST-Progressive.jpg) huffman decoding is definitely a bottleneck:

For 420 baseline images (like pg/issues/issue750-exif-tranform.jpg) it's about 1:1 now:

My thought is that S.D is slow for progressive images for some reason.

That's what I mean. They're not able to use an optimized path. Looking at jdphuff.c there's no fast/slow branching so they have to do what we do. I think for us to match them, we should look at the jdhuff.c file and see what we can make of it and how it compares to our baseline decoder.

antonfirsov · 2018-11-04T19:27:02Z

Yeah I've been working on this a lot, almost all evenings during the last week. My next move is to apply similar basic refactors to ResizeProcessor before dismounting the whole thing.

…tion Various Jpeg optimizations

antonfirsov added 20 commits October 31, 2018 00:37

add some benchmark results

28b990b

results for DecodeJpeg

068b2d3

drop ComplexIntegrationTests, add ProfilingBenchmarks

099e38a

use InliningOptions.ShortMethod

bd4b785

InliningOptions.ShortMethod

b5e419d

AVX2 optimized Block8x8 -> Block8x8F conversion

7f113ab

Moved IJpegComponent.GetBlockReference(...) to an extension method

2acf80e

do not use GetBlockReference() on hot path

aa35af7

benchmarks for current state

265f279

reimplement Block8x8F.CopyTo2x2

941e51c

Optimization: do not initialize pixel buffer in JpegDecoder

bc79b18

improved Block8x8F.Copy2x2Scale

c1e4ccb

optimize ExifReader.ToEnum<TEnum>(...)

c34e4ff

Even better Copy2x2. Tests: Group together & refactor profiling bench…

b326945

…marks

minor optimization in JpegColorConverter.FromGrayscale

c791579

add JpegSnoop reports for all current input Jpeg

fa092d1

improved benchmarks + more benchmark results

8c69038

Drop all GetBlockDataReference() usages

5ea62ad

simplify benchmark suite

86489b8

drop old benchmark results

2ff5b9c

antonfirsov requested review from JimBobSquarePants, dlemstra and tocsoft November 4, 2018 03:14

antonfirsov added 2 commits November 4, 2018 14:24

fix Block8x8FTests.Copy1x1Scale, reorganize TestImages.Jpeg.Benchmark…

5c65cca

…Suite as a static class

oops

fd1ce98

JimBobSquarePants approved these changes Nov 4, 2018

View reviewed changes

antonfirsov mentioned this pull request Nov 4, 2018

Epic: ResizeProcessor performance improvements (Memory & CPU) #733

Closed

8 tasks

drop unused parameter

99a6416

Fix EnumHelper<T>.IsDefined

9a12287

antonfirsov merged commit 7e25ecd into master Nov 5, 2018

qdraw mentioned this pull request Apr 14, 2019

JpegEncoder producing incorrect output on linux-arm (since Beta006) #871

Closed

5 tasks done

JimBobSquarePants deleted the af/jpeg-microoptimization branch September 3, 2019 11:13

antonfirsov added a commit to antonfirsov/ImageSharp that referenced this pull request Nov 11, 2019

Merge pull request SixLabors#768 from SixLabors/af/jpeg-microoptimiza…

82bedd3

…tion Various Jpeg optimizations

Navigation Menu

Repository navigation

Various Jpeg optimizations - #768

Various Jpeg optimizations#768
antonfirsov merged 24 commits into
masterfrom
af/jpeg-microoptimization

antonfirsov commented Nov 4, 2018 •

edited

Loading

Uh oh!

codecov Bot commented Nov 4, 2018 •

edited

Loading

Uh oh!

JimBobSquarePants left a comment

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

antonfirsov Nov 4, 2018

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

antonfirsov Nov 4, 2018

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

antonfirsov Nov 4, 2018 •

edited

Loading

Uh oh!

JimBobSquarePants Nov 4, 2018

Uh oh!

antonfirsov commented Nov 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -506,8 +515,7 @@ private void DecodeBlockProgressiveDC(

		private void DecodeBlockProgressiveAC(
		JpegComponent component,

Conversation

antonfirsov commented Nov 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Prerequisites

Summary

Optimizations

Chore

Benchmark results

Before

After

Uh oh!

codecov Bot commented Nov 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JimBobSquarePants left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antonfirsov Nov 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antonfirsov commented Nov 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

antonfirsov commented Nov 4, 2018 •

edited

Loading

codecov Bot commented Nov 4, 2018 •

edited

Loading

antonfirsov Nov 4, 2018 •

edited

Loading