If I recall correctly, this color picker button area was added when the "Tungsten" theme was added - and did not support color tinting in the Track View. So there needed to be an area that displayed the color.
This is still a neat feature. For those of us that don't use tinted panels, having the color visible is still good to have - obviously useful. I'm not a fan of the gradient myself, but the visible region should not be removed without an option to display it.
I agree about the right-click context menu option - at this point my brain is basically hard-wired to go to the context menu to look for actions such as color selection.