Please note that this was actually written as a rant piece, not a scholarly essay.
People will tell you that the name spelled 〈Celt〉 should always be pronounced as if it were 〈Kelt〉. Some will actually get very loud and upset if you argue. If you ask why, they’ll typically give one of two rationalizations: 1) In Irish, the letter 〈c〉 is always pronounced like 〈k〉. 2) It was originally a Latin name, and in Classical Latin too, 〈c〉 was always pronounced with a /k/ sound.
Well, it’s true that Irish and Classical Latin use the letter 〈c〉 for only the one sound. The development of other sounds for this letter came later. But names often behave very much like other words, and have to be treated accordingly. In particular, they tend (for the most part) to evolve in the same ways as other words.
We’ve reached the point where the dictionary actually gives both pronunciations. But according to both the Oxford English and American Heritage Dictionaries, the name Celt began as a Greek tribe name in plural form, Keltoi, then was borrowed into Latin as Celtae, which became the French Celtes. The singular form in French, Celte, is what became our English word Celt.
In other words, the name wasn’t originally Latin, and it was never Irish (unless it has been borrowed into Irish in modern times; I was unable to find out if this is the case). Until a couple of centuries ago, it wouldn’t have made sense for the Irish (or the Welsh, or the Bretons) to have a single word for all the Celtic-speaking peoples, since they didn’t consider themselves related to each other. It would be like English having a single native word to cover all the peoples that traditionally used Germanic languages, including ancient Goths, Burgundians, and Franks as well as modern Germans, Dutch, Austrians, English, German-speaking Swiss, Flemish, and Alsatians. (Incidentally, English still has no such word: “Germans” refers to the people of Germany.)
In fact, in Greek and Latin the name probably didn’t even include the Irish — it meant the Celts of Gaul! How it came to refer to all the languages in this group, along with their speakers, is best left to another writer. What matters is that the Irish pronunciation is irrelevant to the English pronunciation.
As for the relevance of Latin: Do we say that the value of a penny is one kent? Do we eat kereal for breakfast and then use our kellular phones to say we’ll be late? Do we kelebrate graduations with keremonies of pomp and kirkumstank? Even if the word hadn’t come to us through French, enough Latin words have that we generally pronounce Latin 〈ce〉, 〈ci〉 the same way that the French do, even in words and names taken directly from Latin, like ciborium and Caesar. (Okay, I did cheat with “kirkumstank”.)
Those who spell the name as 〈Kelt〉 obviously can’t be expected to pronounce it as /selt/. This spelling is an accepted variant and can be justified as an adoption and adaptation of the Greek Keltoi. But it’s significant that this is the variant form, and 〈Celt〉 is the more normal spelling.