Now he has his toughest assignment of all. Yesterday, he accepted the challenge at Northwestern, a school that has never been invited to the NCAA tournament.
O'Neill, 40, decided to leave Tennessee where he spent the last three years and was in the midst of restructuring that program with two of the nation's top recruiting classes.
"I took a chance. Columbus did, too," O'Neill said after a news conference announcing him as the replacement for the fired Ricky Byrdsong.
"I can't talk about what's happened here in the past. We will be doing everything in our power to win. We've got to find kids who want to be part of doing something that hasn't been done before."
Sound familiar at Northwestern? Football coach Gary Barnett took over a program that hadn't won in 24 years and has led the Wildcats to two straight Big Ten titles and bowl appearances.
O'Neill had a brief phone conversation with Barnett.
"He said it was a great place to work. I didn't go, 'Gary do you think we can get it done in basketball?' That would be like Gary calling me and saying, 'Kevin do you think we should punt?'"
Before taking the Tennessee job, O'Neill compiled an 86-62 record in five years at Marquette, resurrecting a once powerful program and getting the school into the NIT once and to the NCAAs twice.
Tennessee finished the past season 11-16 and was 36-47 in the three years since athletic director Doug Dickey hired him from Marquette. The Vols were 5-22 the year before he took over.
O'Neill admitted there had been a disagreement with Dickey over the eligibility of Isiah Victor, who was cleared to play by the NCAA but then redshirted.
"We had a disagreement and we moved on. Did that affect my decision? Not in any way. I made my decision on where I was going to, not where I was leaving," O'Neill said.
Terms of O'Neill's contract were not released. Athletic director Rick Taylor said the number of years in O'Neill's contract was "more than two, less than 14."
O'Neill had been under contract at Tennessee through 2001. The fifth year of his contract has been renewed each spring. He was paid $375,000 a year, plus income from summer camps, after getting a $30,000 raise last year.
His compensation package could exceed $500,000 annually at Northwestern.
Taylor said O'Neill was his choice because of his tenacious recruiting and the way his Marquette teams always played while Taylor was athletic director at Cincinnati.
Northwestern went 7-22 this season and won only five Big Ten games in Byrdsong's last three years.