I want to do an instance segmentation of objects in images.
Usually I would stick to something like an Mask R CNN and let it run. However additionally to the image itself and the pre-labeled images, I have additional features that might be interesting for the segmentation.
Example: I want to segment certain products in images from a factory and I have additional information about the products than run at a specific time (like product family, color, brand, etc.).
How do I add these additional features in an instance segmentation?
additional information about the products than run at a specific time (like product family, color, brand, etc.).
Seems easier to just search for images based on the additional information and then train on those images. You would have to first predict the brand(which again requires you to extract visual features from product images of that brand) and maybe do ocr(slow) and use that to filter region proposal.
You could just add an extra channel with the extra information. E.g. you want to use a pretrained model. You take (and freeze) everything but the last few layers. Let’s say the frozen model reduces the image to 224x224x100. Just add a few more channels (with constant value for all pixels), e.g. 10 channels for color, 20 for product type, etc. This will work if the you have training data. If not, maybe something like what u/saintshing said might be the way to go.