Whilst not trying to minimise the care with which plutonium must be handled, it isn't actually much different from other heavy metals in terms of chemical toxicity. Although it is radioactive, it is an alpha emitter, and so can be very easily shielded, plastic bag for instance. Where it is a problem is if it gets into the lungs by inhalation, or into the bloodstream, when the alpha radiation can damage internal organs. This doesn't mean we can treat it lightly, as it degrades slowly into other elements which are beta and gamma emitters. Even so it's not the killer element some would have us believe. To quote wikipedia "There were about 25 workers from Los Alamos National Laboratory who inhaled a considerable amount of plutonium dust during 1940s; according to the hot-particle theory, each of them has a 99.5% chance of being dead from lung cancer by now, but there has not been a single lung cancer among them."
The oft quoted Three Mile Island 'disaster' had very little radiological impact: "The average radiation dose to people living within ten miles of the plant was eight millirem, and no more than 100 millirem to any single individual. Eight millirem is about equal to a chest X-ray, and 100 millirem is about a third of the average background level of radiation received by US residents in a year." A financial disaster yes, but hardly a public health disaster.
To make bomb grade plutonium you have to design a special reactor, a commercial PWR would be pretty useless. How to do it is fairly well known, our not building more commercial stations won't stop rogue states.
Yes Nuclear Power has to be handled carefully, but remember Bhopal, where some estimates suggest 8000 deaths within 2 weeks, and a further 8000 have died since. Hydro isn't without its risks. In 1975 the Banqiao dam in China burst, resulting in 26,000 dead from flooding, 145,000 dead from subsequent famine and epidemics, 11 million homeless.