The deal, which was due to be announced by Health Secretary Andrew Lansley this week will involve better labelling of units in alcoholic drinks, warnings in pubs on the dangers of drinking too much and the banning of advertising of alcohol.
As well as alcohol there are separate "deals" on three other issues: food, physical activity and health at work.
The health organisations said they had lost confidence with the alcohol deal because it was not clear what would happen if alcohol industry did not meet the commitments. They also said the pledges were not measurable, lacked scope and there was no evidence they would even work, according to the BBC.
And they complained that the Government had still not done enough to tackle the issue of pricing of alcohol. The Government has proposed banning the sale of alcohol below cost, preventing retailers from selling it at less than the cost of VAT plus duty, but a recent Guardian investigation found that this would affect just one supermarket deal in thousands.
Don Shenker, chief executive of Alcohol Concern, said: "It's all carrot and no stick for the drinks industry and supermarkets.
"By allowing the drinks industry to propose such half-hearted pledges on alcohol with no teeth, this government has clearly shown that, when it comes to public health, its first priority is to side with big business and protect private profit."
Professor Vivienne Nathanson, of the British Medical Association, another of the groups which have pulled out of signing up to the deal, added: "The Government has talked the talk, but when it comes to taking tough action that will achieve results, it falls short."
Health Secretary Andrew Lansley told the BBC the Government was making progress and taking tough action where necessary.
"We have made clear from the start that the responsibility deal is just one strand of the government's public health policy. It explicitly excludes cost and price competition to avoid conflicts of interest, he said".
The full list of organisations which are refusing to sign up is: Alcohol Concern, the British Association for the Study of the Liver, the British Liver Trust, the British Medical Association, the Institute of Alcohol Studies and the Royal College of Physicians.
By Neil Gerrard
E-mail your comments to Neil Gerrard here.
If you have something to say on this story or anything else join the debate at Table Talk - Caterer's new networking forum. Go to www.caterersearch.com/tabletalk
Looking for a new job? Find your next job here with Caterersearch.com jobs