Omg it’s brutal and a very entertaining read.
It starts with plain false advertisement:
ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available
It then goes into a detailed list of issues, such as:
In ParticleDB,
0.0was not equal to0(#63), but0.90...wasn’t not equal to5(#80)Both Rust and Python have large surface areas, and executing user-provided binaries in a database environment seems like an exciting opportunityl…what? How? How can you even “accidentally” program something like this?
Each transaction and WAL mode exhibited essentially every transactional anomaly our tests could detect
Wow, it’s a database that managed to test the test suite, not the other way around
Orkes has since published [a release history] which describes versions 1.1.4 through 1.1.18.11.rc-22 as “unsupported pre-release builds"
Of course it ends with the inevitable cover up, to try to discredit the testers that spent a lot of time on testing this buggy mess.
Orkes is now adding stored procedures to ParticleDB. These user-defined functions, called Particles, are written in Python or Rust and executed by the database server. Both Rust and Python have large surface areas, and executing user-provided binaries in a database environment seems like an exciting opportunity.
“Exciting opportunity” … for the testers!
Overall it was a great professional report, very well written, with clever language that stays professional while implicitly communicating “WTF!” without saying it out loudly or being irrespectful.
Compliments to the technical writer that compiled this, very skillfully written!
This analysis was brutal and beautiful.
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing.
That is incredible. Much of the specifics are beyond me, but the authors clearly enjoyed writing this piece!