Skip to content

Jpeg HuffmanTable improvements - #1926

Merged
JimBobSquarePants merged 12 commits into
SixLabors:masterfrom
br3aker:dp/huff-table-improvements
Jan 9, 2022
Merged

JimBobSquarePants merged 12 commits into
SixLabors:masterfrom
br3aker:dp/huff-table-improvements

Conversation

@br3aker

@br3aker br3aker commented Jan 6, 2022 •

Copy link
Copy Markdown
Contributor

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

This PR removes redundant Configure() calls for HuffmanTable and removes stack pressure from stackalloc calls.

TODO

  • Remove Configure() method
  • Remove stackalloc calls
  • Create out-of-bounds huffman table test image

@br3aker br3aker changed the title Removed Configure() method Jpeg HuffmanTable improvements Jan 6, 2022
Comment thread src/ImageSharp/Formats/Jpeg/Components/Decoder/HuffmanTable.cs
@br3aker

br3aker commented Jan 6, 2022 •

Copy link
Copy Markdown
Contributor Author

Gotcha

Message: 
    Assert.Throws() Failure
    Expected: typeof(SixLabors.ImageSharp.InvalidImageContentException)
    Actual:   typeof(System.IndexOutOfRangeException): Index was outside the bounds of the array.
    ---- System.IndexOutOfRangeException : Index was outside the bounds of the array.

  Stack Trace: 
    HuffmanTable.ctor(ReadOnlySpan`1 codeLengths, ReadOnlySpan`1 values) line 58
    JpegDecoderCore.ProcessDefineHuffmanTablesMarker(BufferedReadStream stream, Int32 remaining) line 1098
    JpegDecoderCore.ParseStream(BufferedReadStream stream, HuffmanScanDecoder scanDecoder, CancellationToken cancellationToken) line 358
    ...

One of the invalid test images was going out of bounds during new HuffmanTable ctor. The only reason it wasn't spotted before is that exposed code in Configure() was called after checks for invalid image which led to these crashes.

@JimBobSquarePants

Copy link
Copy Markdown
Member

Ah nice, well done finding that. So now I guess we have to find a way to sanitise cheaply and then try to move the table building to the constructor.

@br3aker

br3aker commented Jan 7, 2022

Copy link
Copy Markdown
Contributor Author

It's actually a bug, there's a possibility for malformed image go out of bounds without any exceptions in current master. I found a way to check that without a noticable slowdown. I'd also create an example image for a test suite.

Comment thread src/ImageSharp/Formats/Jpeg/Components/Decoder/HuffmanTable.cs Outdated
@codecov

codecov Bot commented Jan 7, 2022 •

Copy link
Copy Markdown

Codecov Report

Merging #1926 (544c557) into master (ce34e84) will decrease coverage by 0%.
The diff coverage is 100%.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #1926   +/-   ##
======================================
- Coverage      88%     88%   -1%     
======================================
  Files         966     966           
  Lines       51350   51320   -30     
  Branches     6401    6397    -4     
======================================
- Hits        45218   45188   -30     
  Misses       5074    5074           
  Partials     1058    1058           
Flag Coverage Δ
unittests 88% <100%> (-1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...mats/Jpeg/Components/Decoder/HuffmanScanDecoder.cs 93% <100%> (-1%) ⬇️
...rp/Formats/Jpeg/Components/Decoder/HuffmanTable.cs 100% <100%> (ø)
src/ImageSharp/Formats/Jpeg/JpegDecoderCore.cs 89% <100%> (-1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce34e84...544c557. Read the comment docs.

@br3aker
br3aker marked this pull request as ready for review January 9, 2022 01:42
@br3aker br3aker mentioned this pull request Jan 9, 2022
4 tasks done
@br3aker

br3aker commented Jan 9, 2022

Copy link
Copy Markdown
Contributor Author

High hopes it would pass all the tests, it is ready to be reviewed:

  • Rewritten part of the huffman table building routine, it now requires 256 bytes less memory workspace
  • No stackalloc calls
  • Rewritten DHT marker parsing code, now it fetches a big single buffer for binary parsing and table building routine
  • Added sanity checks for huffman table input, now it should be impossible to break decoder with malformed DHT marker

There's no need for a new test image, image from #839 has malformed DHT marker which goes out-of-bounds during huffman table for unsafe pointers (which is unspottable without wrapping them with Span). I've left a note just in case but this PR should nullify this error.

this is my favourite
On top of a critical security breach fix and extra checks we are not actually slower:

MASTER
// Decoding
// Elapsed: 4643ms across 400 iterations
// Average: 11,6075ms

PR
// Decoding
// Elapsed: 4634ms across 400 iterations
// Average: 11,585ms

@JimBobSquarePants JimBobSquarePants left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faster... Fantastic 🤣

This is very, very good as always. Thanks! 👍

@JimBobSquarePants
JimBobSquarePants merged commit 576a69f into SixLabors:master Jan 9, 2022
@br3aker
br3aker deleted the dp/huff-table-improvements branch January 9, 2022 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants