I am currently trying to build small convolutional regression models with very tight constraints regarding model size (max. a few thousand parameters).
Are there any rules of thumb/gold standards/best practices to consider here? E.g. should I prefer depth of the model over width, do skip connections add anything in these small scales, are there any special training hacks that might boost performance, etc?
Any hints or pointers, where to look are greatly appreciated.
TL;DR The more constraints on the model, the more time should spend analyzing your data and formulating your problem.
I’ll agree with the top comment. I’ve also had to deal with a problem at work where we were trying to perform product name classification for our e-commerce product. The problem was that we couldn’t afford to have anything too large or increase infrastructure costs (i.e., if possible we didn’t want to use any more GPU computing resources than we already were).
It turns out that extensive EDA was what saved us. We were able to come up with a string-matching algorithm sophisticated enough that it achieved high precision with practically no latency concerns. Might not be as flexible as something like BERT but it got the job done.