Most Science Studies Appear to Be Tainted

[Admiral Valdemar]

True, I tend to be somewhat on the harsh side of things. Although the reason I'm targeting med students is that med students are the ones who turn into doctors, who do conduct a fair amount of the medical research out there. The bad practices of medical school do carry over to some degree into the wider world of medical technology.

When you actually let the scientists take over, quality tends to improve. Although you'll forgive me if the idea of Big Pharma taking over the research and standards from medical doctors disquiets me as much as it comforts me.

For the many stages before human trials, Big Pharma most assuredly runs the game with medical honchos not getting a look in. Any CRO will conduct trials on everything from animal effects to "What happens if this compound gets into the water table?". By the time humans are getting involved, the compounds are more or less safe and just being looked into for any side-effects that may cause issues for certain people.

In any case, we've come a long way since the issues of lab practice in the late '70s/early '80s. The UK, US, Japanese and OECD standards are very high, and just a single critical breach in protocol is enough to lose you your licence. No licence, no business. Simple as. And having recently had the joy of the GLPMA doing their biennial inspection, if you so much as have a scruffy desk, they will consider that poor practice. FSM help you if you happen to screw up on animal welfare or stats.

We do manage to avoid or mitigate most and the worst of disasters, and I'd be the last to argue that our medical institution is a bad thing, just very far from perfect. I also seem to have gotten somewhat off topic, since my criticisms seemed to have wandered into blasting largely at MD (and related) guided research, and academia general.

In general, the MDs don't do the research, so while they may not be the best in science, they're typically not doing it anyway. Which is just as well, since they're meant to be properly applying what the scientists create for their patients, not doing it all themselves. I'd be concerned if med. students were our driving force for R&D.

[Fingolfin_Noldor]

As a psych major I'll jump in to explain.

It's because the vast majority of statistics in psychology are complete BS. The only way you learn to "interpret" statistical psych data is by taking courses like "Statistics in Psychology". Wherein you get an insufficnet set of data to fully plot a graph, and are then given a data sheet of numbers to select from a group to both fill in the missing data as well as fit a normal curve.

Which is why I don't bother reading psych statistics so much as read the experiment and draw my own conclusions.

That explains much. When a linear correlation |r| <= 0.5 is stated as a linear correlation I am utterly baffled.

Then again, in Physics, there are similar problems. I was once told a story of a Physicist postdoc or something giving a talk and splashed a graph that looked utterly horrible and apparently a professor queried on why the speaker can make claims based on that data. The person had no answer and the professor left the talk promptly.

[Eris]

For the many stages before human trials, Big Pharma most assuredly runs the game with medical honchos not getting a look in. Any CRO will conduct trials on everything from animal effects to "What happens if this compound gets into the water table?". By the time humans are getting involved, the compounds are more or less safe and just being looked into for any side-effects that may cause issues for certain people.

...

In general, the MDs don't do the research, so while they may not be the best in science, they're typically not doing it anyway. Which is just as well, since they're meant to be properly applying what the scientists create for their patients, not doing it all themselves. I'd be concerned if med. students were our driving force for R&D.

Hmm, while I was aware that a great deal of research is conducted before a drug enters even phase I clinical trials, and that a lot of biochemists, pharmacists, et alia were involved, I wasn't aware quite to quite the extent that was true. That's remarkably comforting. Sort of.

Now that is funny.

I never took a sociology class there but between them and the psych students I got hit up enough for questionaires/studies. Talk about a bunch of crap. It's not all that surprising that they wouldn't give credit to someone who really knows something about statistics.

Some of the other colleges and departments at the UofA aren't much better about giving tenure.

I have the best stories about the U of A, good and bad. My father's worked here since he got his PhD when I was fourish, and I'm now attending, and on occasion working as well, so I've gotten everything from the perspective of staff, faculty, and student.

The soc department is one of my favourite targets, since frankly I hold them a grudge. A list of people they haven't given tenure include:

Pat MacCorquodale, now Dean of the Honours College.
Juan Garcia, now Vice Provost for Academic Affairs.
Michael Sobel, now full tenured professor at Columbia.

And those people are just off the top of my head. Otherwise, though, our departments are sort of hit or miss, depending on the programme - not surprising considering we're a well-funded state school. It's one of those places that runs from degree mill (East Asian Studies, Women's Studies...) to one of the best in the country (Forensic Anthropology, Chemistry...).

[Tsyroc]

Funny you mention "Women's Studies" because I took quite a few classes that could be counted as such. The ones which came from the Humanities department were actually scary in how stupidly lame they were for upper division classes. (The same thing could be said about how lame some of the Women's Studies PhDs were that some of the professors had).

The Women's Studies style classes that were over in the History department were much better and at times pretty difficult. Trying to choke down reading most of the Malleus Maleficarum in a day or two was pretty bad.

I had one class that I took as a History class that was based as an East Asian Studies class. It was nominally a history of China but had a lot more to do with culture, art and philosophy than just pure history. It was interesting but it was a real conglomeration of things. It was also the only class I had at the UofA that had a main professor who lectured in one of the large lecture halls, but then on Friday we were all split up into small rooms with Graduate Assistants teaching the classes in separate class rooms. That's also where we took our tests and delivered our papers.

[Broomstick]

One factor I haven't seen mentioned yet is (at least in the US) the "publish or perish" rule in both academia and corporate science. A scientist's output must be measured and evaluated by his/her employers, and without exception that benchmark is how much and how often published. Quantity is valued over quality.

I have to disagree here. They are also judged by how often the work is cited by other researchers, which is a measure of quality. A researcher not only has to publish, he has to publish in good journals and the articles have to have enough impact that they get cited

True to an extent, but having dealt with quite a few 30-50 page CV's from various researchers over the years, as well as gathering articles at libraries and on-line services for our own researchers, it's quite clear that a lot of articles that are published, even by top-rate people, are seldom cited.

So I will ammend my statement to
1) You must publish on a regular basis, and
2) You must be cited on a regular basis.

But since everyone who publishes an article in such journals seems to cite multiple other articles number one takes care of much of the problem of number 2.

The result is that there is considerable pressure to put something out, sometimes before it is really refined into quality work.

[Alyrium Denryle]

That one in particular is the University of Arizona department of sociology

This pleases me in a strange way, I must confess. I am a biology major at ASU, which means I get the double digs in, on the rival school AND the social sciences. Our standards are better (I had a long conversation with my Herpetology professor who happens to be on the personell committee over on the west campus about this not to long ago) So my conscience is clear

I can sympathise. It took me until analytic chemistry until I met another student who wasn't in a chemistry course for pre-med, or something similar. The practice of cramming has never made much sense to me, but after attending some of the study sessions with classmates, it now terrifies me like few other things do.

Tell me about it. My chem professors ask who is pre-med in the class and everyone but me raises their hands. It is fucking disgusting. I cant help but cringe. Just the thought of cramming for an exam terrifies me. Partially because I WANT to understand and retain the information. I want to be able to apply the information to ask questions that are informative. To do this requires that You go over the course material in sequence. Review chapter 1. Next night, review chapter 1 and 2, then 1,2 and 3 etc. Until you know the material backwards, forwards, inside out, upside down and sideways, through the speen... All that so I can take the happy GRE and get into a good grad program... then be actually usefull while there. (this is partially because what I want to do requires a fairly broad knowledge base in animal behavior, genetics, evolution and developmental biology, as well as herpetology)

Tangenitally, do you know weird it is to be in an organic chemistry lab nominally for chem E, chem, and biochem majors and have all of your labmates be physiology, psychology, or similar majors? Pretty damn weird. The end result of all of which being that I think I may have picked up a bit of an elitism streak with regards to people like pre-med. Don't tell anyone: I don't think they've noticed yet.

Yes. And dont worry, I am the same way. I know I am smarter than each and every one of them. Even if they get a better grade than I do, I will remember the information in a month while they will not...

For the many stages before human trials, Big Pharma most assuredly runs the game with medical honchos not getting a look in. Any CRO will conduct trials on everything from animal effects to "What happens if this compound gets into the water table?". By the time humans are getting involved, the compounds are more or less safe and just being looked into for any side-effects that may cause issues for certain people.

Honestly I have philosophical issues with the approach. I dont like the idea of knowledge being proprietary. The problem with the initial trials being done "behind closed doors" is that one cannot verify or replicate it. Correct me if I am wrong, it is not often that I search for medical trials in Web Of Science or other archives (I prefer reading the behavioral bio literature) but they dont actually publish a lot of this stuff in any way that is replicable. My issues are more because it is antithetical to the way science should be practiced (I am an idealist, I admit it) than because the results are bad. But still....

[Zixinus]

In any case, we've come a long way since the issues of lab practice in the late '70s/early '80s. The UK, US, Japanese and OECD standards are very high, and just a single critical breach in protocol is enough to lose you your licence. No licence, no business. Simple as. And having recently had the joy of the GLPMA doing their biennial inspection, if you so much as have a scruffy desk, they will consider that poor practice. FSM help you if you happen to screw up on animal welfare or stats.

That does seem a wee bit overkill to me, or are you exaggerating?

[Darth Wong]

In any case, we've come a long way since the issues of lab practice in the late '70s/early '80s. The UK, US, Japanese and OECD standards are very high, and just a single critical breach in protocol is enough to lose you your licence. No licence, no business. Simple as. And having recently had the joy of the GLPMA doing their biennial inspection, if you so much as have a scruffy desk, they will consider that poor practice. FSM help you if you happen to screw up on animal welfare or stats.

That does seem a wee bit overkill to me, or are you exaggerating?

He's not exaggerating. Labs have to meet very stringent rules about cleanliness, documentation of activities, etc. Unfortunately, people tend to assume that this means any resulting studies must also be sound, and that's not the case. Those regulations are a lot like ISO9001 manufacturing regulations; they don't actually ensure a proper outcome at all, they just ensure that everything is clean and well-documented.

[Admiral Valdemar]

Well, actually, a lot of the rules apply to how you conduct your study with regards to validity of the data, although naturally bad conditions for the workplace are looked at as well, a key concern for when using live animals.

For instance, there are SOPs on how many data points are required to be sampled and how many are allowed to pass or not. For instance, if you have a table of respiration rates with Student's t-test results annotated by the stats. dept., you need to check a certain percentage, chosen at random, for suitability. If you have between 1 and around 90, it's 3 data points checked, one wrong and it's a reject. Above 100, it's 13 with 3 incorrect for a reject, and so on. The software we built for data verification and sell on to others also will point out any outliers that skew the results. There's a reason we mainly use the likes of rodents, because you can use literally hundreds for one study that gives you a lot of data points to eliminate any outliers or errors, and it's even easier with remote telemetry systems recording data 24/7 electronically. These SOPs cover everything from how reports are written and filed, to how to calibrate machinery. If I'm on inspection at formulation chemistry, I expect all aliquots to be labelled properly, all tare readings to be documented for the electronic mass and everything is checked again by the study director (because it's his head that rolls if the study is flawed).

Generally speaking, there is a written procedure for just about everything you can do and I have access to the training records of everyone for the company. If they're not currently assessed to satisfaction on the procedures they're working on, then that gets red flagged. Even if that means a whole study is jeopardised, which you really do not want to tell higher management. We have studies that have run for years using thousands of rats and costing millions of pounds. The client can't afford, and nor can we, to botch up the science because at the end of the day, if you get caught, it costs you a damn sight more in the long run, on top of the billions and years of R&D you spent trying to get one useful drug from tens of millions of contenders. Thankfully, we're very successful and have never failed any inspection from the GLPMA, FDA or any other body, because if we did, it would bankrupt the business overnight. The closest we've come was just this past inspection, I believe. There were some doubts raised over a French lab we used for outsourcing some research to. It was a non-GLP compliance one, which we documented clearly, but apparently the GLPMA guys wanted it to be struck off our list of labs to turn to in future because the French lab, not us, had failed to get a GLP certification after taking on several studies they should have been certified for (they normally didn't work to that standard, but had a change of heart and so require that standard now).

We even have a Home Office inspector permanently on site. She has an access all areas security pass and can come and go anywhere on site with no notice. That neither she, nor her predecessors, have ever found a fault with the facility's work in the 50+ years of operating is a good sign. We did, for one thing, pioneer the first heart transplants here and genetically engineered organisms.

Another example I like to mention is with respect to animal welfare. We had a study a few years ago with horses that required a several month study costing half a million to test some new compound for veterinary use. It was almost ours, before we found that the study required the horses be euthanised after the study. There were 24 horses. The company said no, on advice from the ERP, or Ethical Review Panel that is an open, specialised and layman populated panel that judges the merits of a study against the requirements from an animal perspective. In the end, we found that we didn't need to kill the horses, something the client was relieved about too, and we re-homed them after the minor invasive biopsy for the required data.

Given the UK has passed more legislation on this matter than any other nation on the planet and has the most stringent inspections, I can at least attest to the validity of the studies on this island. If we were to lose business, it'd go to the likes of China or India, where they simply don't give a shit about test animals or you and me.