Could or couldn't? - Let's put an end to the controversy.

[The Sleeper]

But, how is it derogatory to say, "I could care less".

It's assumed that you still don't care a lot. I don't think anyone would take that statement and put a positive connotation on it. However, you still leave the space open for other things that you do care less about.