Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Color conversion with ICC profiles #1567

Draft
wants to merge 99 commits into
base: main
Choose a base branch
from

Conversation

JimBobSquarePants
Copy link
Member

@JimBobSquarePants JimBobSquarePants commented Feb 27, 2021

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

Note: This is a replacement for the original PR #273 from @JBildstein that was automatically closed by our Git LFS history rewrite. Individual commits have unfortunately been lost in the process. Help is very much needed to complete the work.

As the title says, this adds methods for converting colors with an ICC profile.

Architecturally, the idea is that the profile is checked once for available and appropriate conversion methods and a then a delegate is stored that only takes the color values to convert and returns the calculated values. The possible performance penalty for using a delegate is far smaller than searching through the profile for every conversion. I'm open for other suggestions though.

There are classes to convert from the profile connection space (=PCS, can be XYZ or Lab) to the data space (RGB, CMYK, etc.) and vice versa. There are also classes to convert from PCS to PCS and Data to Data but they are only used for special profiles and are not important for us now but I still added them for completeness sake.

A challenge here is writing tests for this because of the complexity of the calculations and the big amount of different possible conversion paths. This is a rough list of the paths that exist:

  • "A to B" and "B to A" tags
    • IccLut8TagDataEntry
      • Input IccLut[], Clut, Output IccLut[]
      • Matrix(3x3), Input IccLut[], IccClut, Output IccLut[]
    • IccLut16TagDataEntry
      • Input IccLut[], IccClut, Output IccLut[]
      • Matrix(3x3), Input IccLut[], IccClut, Output IccLut[]
    • IccLutAToBTagDataEntry/IccLutBToATagDataEntry (Curve types can either be IccCurveTagDataEntry or IccParametricCurveTagDataEntry (which has several curve subtypes))
      • CurveA[], Clut, CurveM[], Matrix(3x1), Matrix(3x3), CurveB[]
      • CurveA[], Clut, CurveB[]
      • CurveM[], Matrix(3x1), Matrix(3x3), CurveB[]
      • CurveB[]
  • "D to B" tags
    • IccMultiProcessElementsTagDataEntry that contains an array of any of those types in any order:
      • IccCurveSetProcessElement
        • IccOneDimensionalCurve[] where each curve can have several curve subtypes
      • IccMatrixProcessElement
        • Matrix(Nr. of input Channels by Nr. of output Channels), Matrix(Nr. of output channels by 1)
      • IccClutProcessElement
        • IccClut
  • Color Trc
    • Matrix(3x3), one curve for R, G and B each (Curve types can either be IccCurveTagDataEntry or IccParametricCurveTagDataEntry (which has several curve subtypes))
  • Gray Trc
    • Curve (Curve type can either be IccCurveTagDataEntry or IccParametricCurveTagDataEntry (which has several curve subtypes))

The three main approaches in that list are

  • A to B/B to A: using a combination of lookup tables, matrices and curves
  • D to B: using a chain of multi process elements (curves, matrices or lookup)
  • Trc: using curves (and matrices for color but not for gray)

The most used approaches are Color Trc for RGB profiles and LutAToB/LutBToA for CMYK profiles.

Todo list:

  • Integrate with the rest of the project
  • Write tests that cover all conversion paths
  • Review architecture
  • Improve speed and accuracy of the calculations

Help and suggestions are very welcome.

@brianpopow
Copy link
Collaborator

I wonder why the test MatrixCalculator_WithMatrix_ReturnsResult only fails with netcoreapp2.1 and not with the other frameworks.

@JimBobSquarePants
Copy link
Member Author

@brianpopow It'll be an accuracy issue most likely. (I hope it's not a JIT issue). It should be possible to inspect the result and see.

@codecov
Copy link

codecov bot commented Jul 13, 2021

Codecov Report

❗ No coverage uploaded for pull request base (main@ca20c92). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 5834c39 differs from pull request most recent head f60d4b8. Consider uploading reports for the commit f60d4b8 to get more accurate results

@@          Coverage Diff           @@
##             main   #1567   +/-   ##
======================================
  Coverage        ?     87%           
======================================
  Files           ?    1023           
  Lines           ?   55212           
  Branches        ?    7052           
======================================
  Hits            ?   48227           
  Misses          ?    5768           
  Partials        ?    1217           
Flag Coverage Δ
unittests 87% <0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@brianpopow
Copy link
Collaborator

@brianpopow It'll be an accuracy issue most likely. (I hope it's not a JIT issue). It should be possible to inspect the result and see.

The issue only happens with a Release build. I think i found the reason, but it seems very weird. Vector3.Zero does not have the expected value (0, 0, 0).

This can be seen with the testoutput:

[xUnit.net 00:00:07.19]     MatrixCalculator_WithMatrix_ReturnsResult(matrix2D: { {M11:1 M12:0 M13:0 M14:0} {M21:0 M22:1 M23:0 M24:0} {M31:0 M32:0 M33:1 M34:0} {M41:0 M42:0 M43:0 M44:1} }, matrix1D: <-0,0007887525. 4,590794E-41. 1>, input: <0,5. 0,5. 0,5. 0>, expected: <0,5. 0,5. 0,5. 0>) [FAIL]

matrix1D is supposed to be Vector3.Zero

@JimBobSquarePants
Copy link
Member Author

Vector3.Zero does not have the expected value (0, 0, 0).

@brianpopow Woah! That's bonkers!

@brianpopow
Copy link
Collaborator

Vector3.Zero does not have the expected value (0, 0, 0).

@brianpopow Woah! That's bonkers!

I have reported this issue: dotnet/runtime#55623

They confirmed the issue, but they say its unlikely to be fixed because netcore2.1 is out of support in august.
So long story short: be careful with default values or Vector.Zero in testdata.

@brianpopow
Copy link
Collaborator

@JimBobSquarePants It would be really nice, if we could bring this PR forward. This would be a good addition to ImageSharp. I thought, I may ask you, if you know what the PR needs (besides tests) to be finished?

What would be the right way to apply an embedded color profile? My first attempt was:

var converter = new IccPcsToDataConverter(profile);
for (int y = 0; y < image.Height; y++)
{
    for (int x = 0; x < image.Width; x++)
    {
        var inputVec = image[x, y].ToVector4();
        Vector4 converted = converter.Calculate(inputVec);
        image[x, y] = new RgbaVector(converted.X, converted.Y, converted.Z);
    }
}

Here is an example image with adobe rgb color profile:

Momiji-AdobeRGB-yes

This does not seems to work, the colors seem to be wrong. Here are more example images

@JimBobSquarePants
Copy link
Member Author

@brianpopow Honestly.....

I don't know. I was hoping the OP would come back to finish things off. I've just kept things updated over the years and hadn't gotten involved at all in the implementation as yet.

Judging from the old comments in the previous PR I believe the code is based somewhat based on the following

https://github.com/InternationalColorConsortium/DemoIccMAX/tree/master/IccProfLib

As for accuracy. That result looks like it's just spitting out the sRGB values again.

I do agree that it would be an awesome addition to the library and would save a ton of headaches. I hoped we'd get it in V3 but that's a mighty big ask.

@brianpopow
Copy link
Collaborator

I think we definitely need a reference implementation to compare the results against. I tried BABL which gnome is using, but i could not get it to work on windows. I will take a look at DemoIccMAX

@waacton
Copy link
Collaborator

waacton commented Dec 12, 2024

@JimBobSquarePants a couple more changes since your review. What I plan to look at next:

  • ICC conversion that bypasses the perceptual adjustment (likely restoring your original code)
  • Writing known-value tests for conversions that Unicolour doesn't support yet
  • Looking into the Span variation of the convert function - I'll expect to need some help here

@@ -39,13 +39,16 @@ public ColorTrcCalculator(
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public Vector4 Calculate(Vector4 value)
{
// uses the descaled XYZ as DemoMaxICC IccCmm.cpp : CIccXformMatrixTRC::Apply()
Vector4 xyz = new(CieXyz.FromScaledVector4(value).ToVector3(), 1);
Copy link
Collaborator

@waacton waacton Dec 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amusingly the matrix is applied to the actual XYZ not the scaled XYZ. Is there a better way to descale this? (Apologies for my rookie vector wrangling)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmm… this means we are scaling then descaling each pixel. Do all other calculators use scaled values? If so can we avoid the initial scaling or update this to match?

Copy link
Collaborator

@waacton waacton Dec 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only calculator I've encountered so far that requires the non-scaled values. Won't be sure until we find profiles with the other types of curves but it's likely to be just this one, for the matrix multiplication to apply correctly.

We might be able to do something similar to the Is16BitLutEntry check and avoid scaling XYZ if we know we're using ColorTrcCalculator, or add a IVector4Calculator.NeedsScaled property for calculators themselves to define? It would add more complexity to the target PCS calculation but save on a couple of potential scaling roundtrips.

Also, I'm aware that in general I'm flopping between e.g. Vector and CieXyz quite a bit, is that problematic?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to avoid switching between the types where possible as that causes per-pixel overhead. Perhaps we should be trying to provide this information up-front? It's tricky...

Copy link
Member Author

@JimBobSquarePants JimBobSquarePants Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by this comment?

// when data to PCS, output from calculator is descaled XYZ
// but expected return value is scaled XYZ
// see DemoMaxICC IccCmm.cpp : CIccXformMatrixTRC::Apply()

How does the calculator know how to descale the output from the scaled input?

Also doesn't the Gray TRC caclulator require the same?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow your question but I'm still getting my head around it myself. Does it help if I say...

  • Matrix TRC can only be used with XYZ PCS
  • Simple curve lookups use the scaled XYZ
  • In this case, because the output of the curve needs to be multiplied with a matrix, we need to use the actual descaled XYZ it represents
  • Downstream operations tend to assume the PCS will be in scaled form

I've experimented with a different approach that might make things clearer, moving the responsibility of knowing about this from ColorTrcCalculator to ColorProfileConverter. The downside is you can't "just use" the calculator, but that's true of ICC transforms generally anyway once you start looking closely.

Imagine ColorTrcCalculator doesn't have these changes, and instead ColorProfileConverter has these extra steps:

// output of Matrix TRC calculator is descaled XYZ, needs to be re-scaled to be used as PCS
Vector4 sourcePcs = sourceParams.Converter.Calculate(source.ToScaledVector4());
sourcePcs = sourceParams.IsMatrixTrc ? new CieXyz(sourcePcs.AsVector3()).ToScaledVector4() : sourcePcs;
Vector4 targetPcs = ... // do conversion from source PCS to target PCS, potentially with perceptual adjustments

// input to Matrix TRC calculator is descaled XYZ, need to descale PCS before use
targetPcs = targetParams.IsMatrixTrc ? new Vector4(CieXyz.FromScaledVector4(targetPcs).ToVector3(), 1) : targetPcs;
return TTo.FromScaledVector4(targetParams.Converter.Calculate(targetPcs));

I can't tell if it feels better or worse. Depends what mood I'm in when I look!

As for Gray TRC, I can't see any evidence that it does the same thing, just multiplies the scaled monochrome by scaled D50.


💡 If that matrix multiplication in LutEntryCalculator does indeed need fixing, then presumably the matrix multiplication in ColorTrcCalculator does too.

@waacton
Copy link
Collaborator

waacton commented Dec 16, 2024

With the latest commit, I've been able to run each random-value-input test case of CanConvertCmykIccProfiles one million times with no failures 🙌

Summary of where this is at:

  • Good testing in place for converting between profiles with the following data spaces
    • CMYK ⇒ CMYK
    • CMYK ⇒ RGB
    • RGB ⇒ CMYK
    • RGB ⇒ RGB
  • No testing for other data spaces. If we encounter them I can attempt to add them; I briefly attempted to add a test for a profile with GRAY data space but I don't think ImageSharp has a suitable 1-channel structure
  • Only profiles with perceptual or relative colorimetric intent have been tested. I haven't found a profile yet that specifically targets saturation or absolute colorimetric
    • I expect that saturation intent would work OK
    • Absolute colorimetric intent would just be wrong as it requires its own form of PCS adjustment. Adding another layer of PCS adjustment is a bit intimidating right now, especially if profiles that target this don't exist in the wild
    • We can probably modify existing profiles to target these other intents if test data is desired
    • I'd be tempted to limit initial support to only perceptual or relative to avoid the extra complexity
  • Need to figure out how to handle the conversion function that works with Span arguments

Comment on lines 104 to 124
/* This is a hack to trick Unicolour to work in the same way as ImageSharp.
* ImageSharp bypasses PCS adjustment for v2 perceptual intent if source and target both need it
* as they both share the same understanding of what the PCS is (see ColorProfileConverterExtensionsIcc.GetTargetPcsWithPerceptualV2Adjustment)
* Unicolour does not support a direct profile-to-profile conversion so will always perform PCS adjustment for v2 perceptual intent.
* However, PCS adjustment clips negative XYZ values, causing those particular values in Unicolour and ImageSharp to diverge.
* It's unclear to me if there's a fundamental correct answer here.
*
* There are 2 obvious ways to keep Unicolour and ImageSharp values aligned:
* 1. Make ImageSharp always perform PCS adjustment, clipping negative XYZ values during the process - but creates a lot more calculations
* 2. Make Unicolour stop performing PCS adjustment, allowing negative XYZ values during conversion
*
* Option 2 is implemented by modifying the profiles so they claim to be v4 profiles
* since v4 perceptual profiles do not apply PCS adjustment.
*/
bool isSourcePerceptualV2 = sourceConfig.Icc.Intent == Intent.Perceptual && sourceConfig.Icc.Profile!.Header.ProfileVersion.Major == 2;
bool isTargetPerceptualV2 = targetConfig.Icc.Intent == Intent.Perceptual && targetConfig.Icc.Profile!.Header.ProfileVersion.Major == 2;
if (isSourcePerceptualV2 && isTargetPerceptualV2)
{
sourceConfig = GetUnicolourConfigAsV4Header(sourceConfig);
targetConfig = GetUnicolourConfigAsV4Header(targetConfig);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the last fly in the ointment for me at the moment and hopefully the comment makes sense. My goal was to be able to confirm that any source data values follow the same calculations as Unicolour, but in the case where both profiles need PCS adjustment I need to trick Unicolour into treating the profiles in a different way.

If this is too unpleasant, a simpler alternative is to detect if the source PCS ends up as XYZ with negative values, in which case generate a new vector of random numbers and restart the test, and just trust that negative XYZ PCS is converted correctly.

@waacton
Copy link
Collaborator

waacton commented Feb 9, 2025

I added support for TRC conversions in Unicolour at the start of the year which has unearthed a couple of extra discrepancies, probably related to PCS adjustment again. Once I've figured that out I'd feel confident saying that ICC conversion is working (to the best of our knowledge, for specific rendering intents).

@waacton
Copy link
Collaborator

waacton commented Feb 13, 2025

@JimBobSquarePants the ICC conversion tests are now working as far as I'm concerned, I don't think the current build failures are related.

Testing summary

  • Profile-to-profile conversions covering permutations of
    • v2 & v4 profiles
    • CMYK & RGB data spaces
    • Lut8, Lut16, LutAB, TRC transformations
  • Covers boundary cases, values that trigger specific perceptual adjustment branching, and random values
  • I've run all profile pairings using random values one million times each locally, all passed

Known issues

  • No testing for BToD / DToB tags (no reason to think it will be incorrect but not yet implemented in Unicolour, and I don't have any profiles with those tags)
  • No testing for GRAY data colour space (I don't think ImageSharp has a suitable 1D data structure?)
  • Profile with absolute intent will calculate incorrect values (needs its own form of PCS adjustment that will increase complexity a lot, and I don't have any profiles that reference it)
  • No testing for profile with saturation intent (no reason to think it will be incorrect, but I don't have any profiles that reference it)
  • Only working and tested with ConvertUsingIccProfile(TFrom source) - the ConvertUsingIccProfile(ReadOnlySpan<TFrom> source, Span<TTo> destination) variant needs to be updated and tested (beyond my current understanding)

Other than that last point, I don't think it would be unreasonable to release a first implementation with those known limitations, with some detection and error reporting.

@JimBobSquarePants
Copy link
Member Author

@waacton You’re a wizard!! I’ll have a look at your updates then look at updating the bulk code to match.

@waacton
Copy link
Collaborator

waacton commented Feb 14, 2025

I lie awake at night wondering if I've misinterpreted something with the DemoIccMAX reference implementation or the ICC spec (there's too much for a mere mortal like me to unpack), and if everything I've done here and in my own library is wrong 🫠

Quick note for the DToB / BToD tags, I eventually want to use this probe v2 profile as a smoke test for them. These multi-process elements are so flexible that testing every possible path through them would be a nightmare, and I suspect there's very little real-world data available. The probe profile wouldn't test complex tags, but the results should be predictable and might even be a nice visual confirmation for ImageSharp.

@saucecontrol
Copy link
Contributor

DirectX includes an scRGB profile with DToB/BToD tags, which they use to increase the precision of the RGB<->XYZ matrices. It makes for a good test because it should match a linear sRGB profile very closely.

d2d_scRGB.zip

I can also dig up some greyscale profiles if that would be helpful.

@waacton
Copy link
Collaborator

waacton commented Feb 15, 2025

... if everything I've done here and in my own library is wrong 🫠

I'll put the brakes on my celebrations for now; I updated my test-data-generation code with the latest code from the ICC reference implementation, and now I have 40,000+ failing tests 🙃

I don't have the energy to debug what's changed but since ImageSharp is a port of the reference implementation from an earlier time then presumably it's either A) something I've introduced to ImageSharp (e.g. the PCS adjustment routine is now wrong) or B) the reference implementation has changed, and both Unicolour and ImageSharp underlying implementations are wrong.

Either way, I've raised an issue to try to find out more: InternationalColorConsortium/DemoIccMAX#113

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants