Manage / remove colors from material picker

So I only use a full colors, including a few custom ones I've added to the Materials picker. But I find it a bit annoying to see "Deep orange" next to "Orange" and really just want to see the colors that I actually use on this picker.

So, it'd be