Born in London, he was educated at Highgate School, where a fellow pupil was John Tavener. He then read music at Clare College, Cambridge, where he was a member of the choir. He then became director of music at Clare College from 1975 to 1979 and led the choir to international prominence.

In 1974, Rutter visited the United States at the invitation of choral musician Melvin (Mel) Olson and conducted the premiere of his cantata "Gloria" in Omaha, Nebraska, in the Witherspoon Hall of Joslyn Art Museum. The composition, commissioned by Olson's Voices of Mel Olson chorale, has become a much-performed favorite over the years. (In the same concert, the Young People's Choir of the Midlands performed Rutter's "Eight Childhood Lyrics" in its premiere US performance.)

In 1981 Rutter founded his own choir, the Cambridge Singers, which he conducts and with which he has made many recordings of sacred choral repertoire (including his own works), particularly under his own label Collegium Records. He still lives near Cambridge, but frequently conducts other choirs and orchestras around the world.

In 1980 he was made an honorary Fellow of Westminster Choir College, Princeton, and in 1988 a Fellow of the Guild of Church Musicians. In 1996 the Archbishop of Canterbury conferred a Lambeth Doctorate of Music upon him in recognition of his contribution to church music. In 2008, he was made an honorary Bencher of the Middle Temple whilst playing a significant role in the 2008 Temple Festival.

From 1985 to 1992 he suffered severely from myalgic encephalomyelitis (ME; or Chronic fatigue syndrome), which restricted his output; after 1985 he stopped writing music to commission as he was unable to guarantee meeting deadlines.

He also works as an arranger and editor, most notably (in his youth) of the extraordinarily successful Carols for Choirs anthology series in collaboration with Sir David Willcocks.

He was inducted as a National Patron of Delta Omicron, an international professional music fraternity in 1985.