There is utility in both speed and in a slow step. The truth is that we need both. A continuum of methods and ideas. After all, we are searching in the dark and our only real advantage is parallel Monte Carlo search. The problem is that this is incredibly difficult to accurately evaluate and we are arrogantly attempting to evaluate exceptionally noisy systems with extreme precision without even having a basic model of that system. We could relate this to businesses too, but the timetables for them are typically shorter and so the chaff can more clearly be gained (often post-explained through pure intent instead of it's combination with luck). Then again, we are trying to make academia a business, trying to apply their noisy successes to a completely different model.

Ugh.

There's plenty of honest researchers in the field.

You don't have to be corrupt to build a nice career and, now that I think of it, only people who are equally corrupt seem to embrace/justify/tolerate these practices.

This is multiplied several-fold when you are doing this populist intellectual dinner talk circuit in one of several countries whose elite are anxious to establish (or regain) their own scientific prestige on the world academic stage and will throw unbelievable amounts of money at the most brittle and flimsy proposals so long as there is an R1 institution attached.

What policies are those? I thought 'publish or perish' was an expected outcome given the extreme supply/demand situation in the academic job market.

Could it be there's more than one problem? :)

https://en.wikipedia.org/wiki/Grievance_studies_affair

I don't know that anyone else does, but I like to draw a distinction between experimental scientific inquiry and rigorously observed phenomenology. Properly speaking I say a great deal of what's called science is really a species of phenomenology. For example cosmology is pure phenomenology. We can't actually run experiments on what happens when you smash two black holes together. However we can rigorously observe the phenomenon and abduce a descriptive model that can potentially even make predictions. But the word "science" has become a totem, so there's a lot of cargo culting[1] and punishing blasphemers.

However, I don't really think that whether a field is science, phenomenology, or some blend of both has much effect on the rate of fraud. Rather, I hypothesize the dominant factor in the incidence of fraud is the perceived magnitude of the potential benefit to be obtained. The Alzheimer's fraud is a good example of that. There are hundreds of billions and maybe even trillions of dollars in play in that space. Drug companies have already poured billions down the drain chasing dead ends so anyone who can sell them hope can potentially get a substantial payday. An amusing corollary of this is that given the relatively small stakes social science fraudsters are playing for, we can conclude that they are small-minded in the sense that they aren't thinking nearly as big as the drug company fraudsters.

[1] https://calteches.library.caltech.edu/51/2/CargoCult.htm

The best experimenters I know in terms of monitoring and controlling external variables are chemists, but sometimes they just use the wrong statistical test.

Biologists seem to generally be bad at both, and physicists seem to generally be good at both. I honestly think this is about the culture of the fields rather than knowledge - they all start as good scientists and just unlearn what they don't "need."

Bayesian statistics is great, you can get any posterior you want by choosing good priors.

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” - Upton Sinclair

Except possibly even more high stakes than that, since academics' very reputations are on the line.

Of course the best irony is the current fad of rejecting "objective truth" (on whatever grounds are useful - it's racist, etc) and yet still claiming you are doing a science. Not the search for truth but rather the search for your truth.

https://www.cambridge.org/core/journals/modern-intellectual-...

Or we could use fancy Latin suffixes:

sociology, psychology, economicology, biology, chemiology, pharmacology, mathology (https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg).

I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.

Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.

Some results do replicate. Big Five personality traits, general classification of mental illness, mainstream IQ results, and human perception and performance. All psychology is not false and there are interesting things to learn from the field.

But the severity and frequency of fraud and non replication is worst in psychology. So much so that half of what you learned in Psych 101 ten years ago does not replicate.

The big challenge for the field is that humans know a lot of psychology. We can track our own thoughts and we are constantly interacting with people and trying to understand their psychology. A biologist who studies ants intensely for 5 years is maybe only one of 100 people have ever watched them that carefully. They'll find lots of new stuff.

Psychology doesn't have powerful techniques than that for determining new truths about humans. It's still mostly give a questionnaire or put people in weird situations.

So there are a lot of researchers hunting around for original ideas that are undiscoverable with our current tech. Some of them are bound to give into the temptation to just make up an interesting result with manufactured data. With p hacking they might not even be committing fraud, they are desperate for a positive result, when the get one they stop looking and publish.

So I would still guess in 2023 that about 80% of "new" discoveries published in journals won't replicate. That rises to 95% with journal publications that are published as university pr and reported in the media.

For coverage of what to remember to unlearn Rolf Degen is great: https://twitter.com/DegenRolf

Popular examples that do not replicate (mostly from Rolf):

Repressed childhood memories Social media harms Search bubbles and echo chambers Women prefer masculine men when fertile Priming (you see a fight in the hallway and then don't cooperate later) Power posing (puff out your chest to feel more confident) Watching eyes make people more honest (in Kahneman and Malcolm Gladwell)

Notice that a lot of these are interesting, and you want them to be true. That's not enough to make it so.

As a bonus, an interesting interview with Daniel Kahneman: https://www.edge.org/adversarial-collaboration-daniel-kahnem...

Adding to complication in the psych field is the widely observed phenomenon that it draws students with psych problems. Like those who would be willing to fabricate data, for example.

But hey, at least it isn't sociology.

There are resilient exceptions in academia in general, but soft science fields have led the way in the decay of academic standards.

Some of it is used in policy. So when some convicted murderer is out on furlough and kills again, am I wrong to wonder if that was some of the top-notch applied psychology at work?

Some are purely on the popular science side of failure. The 10k of practice was basically thrown out, but the original research still seems good. They never claimed "if you do 10k of work, you will be good."

The "marshmallow test," though, seems completely tossed? Maybe there was something there?

Anchoring and other items? Not sure how well those have survived. :(

https://en.wikipedia.org/wiki/Replication_crisis looks to be a good article, though I haven't finished reading it.

- Stanford Prison Experiment; Not an experiment, abuse was scripted, experimenter constantly intervened, reactions of participants were faked, and there was not even a scientific hypothesis they were testing. Definitely read the paper[0] debunking it.

- Milgram Experiment: (the one where people were ordered to shock actors) No good evidence for it. Researchers did not follow script, implausible levels of agreement between different experiments. Killer line is, “only half of the people who undertook the experiment fully believed it was real and of those, 66% disobeyed the experimenter.”

- Robber's Cave: (the one where two groups of kids immediately formed tribal hatred between one another) The conflict was orchestrated by experimenters and the experiment was actually repeated because the first time the kids absolutely refused to turn on one another. More information at [1].

- At best, weak evidence for implicit bias testing and stereotype threat.

- Weak evidence of "facial feedback" (smiling causes a good mood and frowning causes a bad mood)

- Good evidence against "ego depletion" (the idea that willpower is limited in a muscle-like fashion)

- Mixed evidence for Dunning-Kruger effect

- Questionable evidence for "hungry judge" effect (the idea that judicial sentences are massively more merciful in the morning and after a lunch recess due to "ego depletion" - this is also thoroughly debunked here [2])

- The 10,000 hours of practice leading to expertise idea has been disowned by its proponents

- No good evidence that tailoring teaching to students’ preferred learning styles has any effect on objective measures of attainment.

- No good evidence that brains contain one mind per hemisphere. i.e. the left-brain, right-brain split that people talk about, especially after the link between the hemispheres is severed.

- No good evidence for left/right hemisphere dominance correlating with personality differences.

Per https://danluu.com/dunning-kruger/:

- Most people talking about Dunning-Kruger have no idea what it actually means. The actual purported bias is much weaker than people claim - basically, that everyone either overestimates their ability, but that estimation is still positively correlated with actual ability, or estimated ability has basically no correlation with actual ability and everyone is just guessing.

- Increasing your wealth does in fact make you happier at a predictable rate. There is no "plateau" of wealth or income - what appears to be a plateau is misleading displays of data. In effect, increasing income by a proportional rate will increase reported happiness by a fixed rate. For example, say you make $10, and your happiness is 50. Then, your income increases to $20 and your happiness increases to 60. Then, doubling your happiness is necessary to increase your happiness by 10. It's a logarithmic function; plotted on a standard axis, it looks like a plateau, but plotted on a logarithmic scale and it's a constantly increasing line.

- Hedonic Adaptation (aka the hedonic treadmill) is a myth. Bad life events (divorce, disability, death of a loved one) all have negative long-term effects on happiness. Vice versa for positive events.

[0]: https://www.gwern.net/docs/psychology/2019-letexier.pdf

[1]: https://www.theguardian.com/science/2018/apr/16/a-real-life-...

[2]: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...

"Oh no! Anyways."

When most people are stuck in a rat race to get rich and "become successful", there's not much energy or reason to care about some drama, unless it's entertaining or political, perhaps. If my employer follows the newest workplace fads based on some studies, I might care. Otherwise, why should I spend mental energy (at the very least) trying to get the messes of scientific research cleaned up?

People are morons, generation after generation, what else do you expect?

Sorry if I'm missing your point?

I didn't say Humanities were worthless. I just said there are no provable results there. "Is X ethical?" - no one can prove it but we will discuss it as long as someone will pay us to. Since there are no firm truths, anyone can get a degree (or more) without actually gaining the skills you refer to. In fact actually gaining the skills you allude to makes someone LESS likely to get funded or famous. It is extremists filled with certainty who rule these subjects...

People who don't need guidelines gain nothing from them. People who need them won't sign up to them. Meanwhile I have to fill in a form and have a 2h meeting before I can shine a laser on an (already dead, commercially available) butterfly's wing?!

Like many things, research is often spoiled by human nature. Researchers want to prove their hypothesis so desperately that they’re willing to do whatever it takes to make that hypothesis true. Perverse incentives are everywhere in the research world.

Lots of people working in policy don't have any real ethical or political engagement with the problems they are supposed to be solving. They lack vision and purpose. They choose to believe that irrelevant and trivial interventions can make huge inroads into difficult social problems, because they are like the man looking for his wallet under the lamppost. If making progress on really difficult problems requires deep thought, courageous leadership and persistent hard work, then they aren't going to be able to solve them anyway.

The difference of "allegedly" is big to a publisher (missing it can break their own company), and to a journalist.

Because of this, I think it's also a big difference to the savvy reader. When an outlet dispenses with the "allegedly" or phrasing as someone else's claim/assertion, that feels to me like a rare occasion on which they really want to impress that it's definitive. And that they know what they're doing, and that experienced readers can tell. Never crying wolf. (A little like how you'd say "literally" once in a decade or lifetime, with emphasis, for impact, when it was really true.)

Also, financial success is on-topic here, as this forum originated for Silicon Valley bros seeking wealth. In past lives, we were "greed is good" Wall Street bros. I'm sure some percentage of our earlier counterparts started out really loving spreadsheets, before the schemes.

Similar to why so many people want to get Google, Meta, Microsoft, Apple, etc on their resume. While there are a ton of amazing small companies they just do not have the same brand recongition.

I think the economic term for this is "signalling." Having associating with these brand name schools gives you instant credibility: https://en.wikipedia.org/wiki/Signalling_(economics)

Because by and large people go to university to have a better chance of making more money

Most aspirational people who are out to make mega bucks are trying to put distance between themselves and the time they didn't have that, and to setup the structures in their life so if they have kids, they can give that easy going life to their heirs.

Because when a person has a lot of money, it vastly increases the scope of available activities that they enjoy and give them purpose.

It's not odd. If your metric was "be as useful as possible to others" then making lots of money is an excellent proxy metric.

Often for business school profs, their work will be judged less on academic acceptance (peer-reviewed publication, etc) and more on its 'real-world impact'. You are seen as a good academic of marketing if you can market yourself into Ted talks, pop science publications, and ultimately, influence on 'decision-makers' (the sort of politicians and thinktanks who believe that eg healthcare can be solved by painting hospitals orange).

So there are strong incentives to falsify results but they are quite different from those which bear on actual scientists.

Those are not weasel words. I said that in my original message. I anticipated your argument and addressed it, before you made it.