Patch an image with lots of holes

Given the binary attached, I'm look for a way to patch those white part without using contourf function.

My routine is: bwboudaries => get edge coordinates => Loop through each detected edge, patch(x,y, [0.5 0.5 0.5]). However, such method fills the holes as well. The reason I want edge coordinates is that I can choose a segment to patch. But contourf function doesn't give me such flexibility.