I see a lot – by no means an overabundance, but enough to “trigger” me – of laughing at some of the “obvious” research that gets posted here.
One example from a week or two ago that’s been rattling around in my head was someone saying in reply to the paper (paraphrased):
That’s just RAG with extra steps.
Exactly. But what were those steps attempting? Did it make RAG better?
Yes. Great, let’s continue pulling the thread.
No. Ok, let’s let others know that pulling this thread in this direction has been tried, and they should take a different approach; maybe it can be pulled in a different direction.
We are at the cusp of a shift in our cultural and technical cultures. Let’s not shame the people sharing their work with the community.
I agree with your sentiment here. But, you can’t deny the influx of papers that are intentionally something extremely simple or inconsequential that are deliberately dressed up to try to look as complex as possible just in order to get published. Regardless of your sentiment (which again I agree with mostly), those kinds of papers are not good and we’d all be better off without them. I think there is a place for shame for certain types of papers, and I would disagree with the idea that shame is always bad or shouldn’t be used as a tool.
10/10. Literally working of filing a patent at the moment and trying to make it as hyper specific as possible so a.) it doesn’t overlap with anyone else’s patent b.) pretty much only applies to what we’re doing at the company.
I’m sure there’s people in similar situations, but we’re heavily incentivized to patent/publish something.
Oh boy, preaching to the choir. Sometimes the 10 pages paper can be just summarized as a shrug emoji.
I’ve definitely seen a few of those.
I agree…but as a new “tinkerer” in this space and entirely unfamiliar with how academic research works, I’ve always been curious, why is the stuff always published in proprietary PDF form at on some third party research site or journal? Why don’t more people just post their findings on their own websites in an open standard like HTML?
Scientific publishing is done the way it is for a number of reasons, important ones already have been noted by other commentators.
But one aspect not covered is peer review. A peer reviewed journal article has been the gold standard of scientific research for quite some time. This involves submitting the article to the journal, and letting the journal anonymously recruit other experts to critique the paper. The goal here is to strengthen the research and screen for mistakes (and fraud, though peer review assumes honesty on the part of the researchers.)
This process has been challenged recently because, in the end, it doesn’t really work to create a uniform gold standard of research. Large fields of research can’t actually replicate the results of peer reviewed studies. Large, systematic frauds go undetected. The process is agonizingly slow in case of emergencies like COVID. And “idiosyncratic” reviewers can make the process worse. (See: Reviewer 2).
On top of that mess is money. You often pay to have your article published, your institution’s library has to pay for a subscription, and your work is locked behind a paywall. This is particularly galling when the government has paid for the research in the first place. But the big publishers retain (too much) control.
The last problem is the discovery of your work. Publish a blog post and who notices? Publish in a leading, reputable journal, you’re guaranteed eyes on your work.
None of this is particularly good for either researchers or scientists and it’s interesting to watch academics experiment with alternate ways of codifying their discoveries. The insistence on “open access” - no paywalls - is one way. PrePrint servers like Arvix allow researchers to meaningfully distribute papers intended to be peer reviewed before the review happens. And fields like LLMs are moving so fast no disciplined publishing process could keep up are helping to further disrupt this.
Because real research is supposed to be peer reviewed, and journals offer peer review by panels of experts. Arxiv was supposed to circumvent that by allowing for review by an open group of peers, but the cycle for new research is so short nowadays that it basically means “review by twitter”
I think I want to write my own paper. You know, get some of that basic stuff down. Do you think if am able to create a proper paper, that will be scientifically legit, like, if I publish it on my own website? Without any academic credentials? Or can I upload it to that website with the idiotic name? What was it, xcifdfs?
Yes. Great, let’s continue pulling the thread.
No. Ok, let’s let others know that pulling this thread in this direction has been tried, and they should take a different approach; maybe it can be pulled in a different direction.
That is how science is supposed to work. Unfortunately, even in the academia, there is less and less reward in publishing negative results.
Maybe we will someday have an AI publisher that encourages negative results? I think one of the promising things about AI is removing the human tendency to value things that enhance careers. Be it journalistic credits or a pay raise, an AI can be much more objective.
Just to point out the obvious, but I think it gets lost in the weeds sometimes:
When you say “Maybe we will someday have an AI publisher”, that is still a person or company with a computer running a program.
So it will still be researchers researching, but the tools they use will help create and value the negative more than those results were valued historically.
My opinion is that this distinction needs to be made clear from time-to-time, so people learning will understand that AI isn’t a mythological creature we’re attempting to tame. It’s a new “programming” paradigm that we are trying to understand and utilize to improve our workloads/workflow.
I agree with the sentiment. A contribution is a contribution. Even an utterly failed contribution can have wonderful results when shared.
If, my critiques of the writing quality and content of certain recent papers played a part in this post, I apologize. In my case when I say improvements in ‘transformer kung-fu’ it is my attempt to advertise “we have now exceeded this humans education and understanding”.
All that said. I think some papers are of such a poor and offensive quality, that getting publicly dunked on for it, is a great way to keep liars from stealing grant money/funding from people doing real work. I am already annoyed that when a member from here contributes something, they don’t mention that human, they mention the community. There is no reason a valid contribution shared here, incorporated into an academic or corporate project shouldn’t and couldn’t be credited.
So, I agree with your points and also am probably guilty of the same of the type of comments you speaking out against.
Just another thing I can improve on. Cheers.