The bank, which counts the Queen and numerous pop and sports stars among its clients, received the fine from the Financial Services Authority (FSA) after a probe into its management of high-risk money laundering situations.
Tracey McDermott, acting director of enforcement and financial crime at the FSA, said Coutts' failings were "significant, widespread and unacceptable".
The FSA said Coutts had started improving its anti-money laundering systems and had agreed to settle at an early stage with the regulator over the fine.
Had it not done so, it would have been fined £12.5m.
Coutts, which accepted the FSA's findings, said the fine related to activity undertaken between December 2007 and November (Stuttgart: A0Z24E - news) 2010 and that no money laundering took place during that time.
Rory Tapner, chief executive of the wealth division of RBS, said they were disappointed with the results of the probe.
He added: "Since the FSA first raised its concerns, we have implemented a number of improvements to prevent any recurrence of these failings. Regulatory reforms continue apace.
"We remain committed to ensuring that our systems and controls are robust and counter the risk of financial crime in all the markets in which we operate."
The latest fine follows a £6.3m sanction in November for failings related to the sale of a fund product during the run-up to the 2008 financial crisis.
It adds to millions of pounds worth of other sanctions against RBS, which was rescued from collapse by the Government during the 2008 financial crisis, over the last 18 months.
Coutts, which was founded by philanthropist Thomas Coutts in the 18th century, was acquired by the RBS Group in 2000.