I want to do an instance segmentation of objects in images.
Usually I would stick to something like an Mask R CNN and let it run. However additionally to the image itself and the pre-labeled images, I have additional features that might be interesting for the segmentation.
Example: I want to segment certain products in images from a factory and I have additional information about the products than run at a specific time (like product family, color, brand, etc.).
How do I add these additional features in an instance segmentation?
You could just add an extra channel with the extra information. E.g. you want to use a pretrained model. You take (and freeze) everything but the last few layers. Let’s say the frozen model reduces the image to 224x224x100. Just add a few more channels (with constant value for all pixels), e.g. 10 channels for color, 20 for product type, etc. This will work if the you have training data. If not, maybe something like what u/saintshing said might be the way to go.