Very interesting concept. The only thing I can think of is using Texture2D.GetData() to see the pixel data of your damage mask, iterate to find a sufficiently dark pixel, then use some basic maze-solving to follow it in a path through adjacent dark pixels. A stack-based algorithm (probably the simplest maze-solving ones) may work for this, but you're going to have to find something that's efficient enough for your desires. Basically, find the longest string of dark pixels you can.
Storing the start and end as a Vector2, you could calculate a line between them and use the angle and the distance to see if it bisects the rectangle area you define as your ship (not necessarily the whole texture, you'd need to manually define it I guess). To find out if it fully bisects the rectangle... I'll try to whip something up here, no guarantees:
float hypotenuse = damageLine.Length(); //this allows us to calculate trig stuff
float boxHeight = shipBoundingBox.Height;
//assuming the height of the box is its smallest dimension (i.e. long ships only, no fat
//ones). it's possible to adapt this if your ship types vary; just use the smallest
//dimension.
if (hypotenuse < boxHeight)
return; //if the length is not long enough it could not possibly bisect it
float opposite = damageLine.X; //the x distance between start and end
float angle = (float)Math.Abs(Math.Asin(opposite/hypotenuse)); //this is the angle of the
//line from y-axis
//now we can use trig to calculate the largest angle, based on the hypotenuse and the
//vertical height of the bounding box, which would bisect the bounding box.
float largestAngle = (float)Math.Abs(Math.Acos(boxHeight / hypotenuse));
if (angle < largestAngle)
doBisect(); //damage has bisected the rectangle
else
return; //damage has not bisected the rectangle
Note that this would only check to see if you cut the ship in half width-wise (the way you've shown it in your example picture). You could easily use the dot product of the slice vector with Vector2.Right to see if the vector is actually length-wise on your ship, and switch a few values above for the calculations. Or, maybe there's a general formula you could adapt from that.
As for building new masks... perhaps extend the line you found so that the two points are at the edges of the texture, then use a RenderTarget2D to draw this new line on top of the existing damage texture. Using the equation of the line you could then fill in pixels above or below it for two successive draw calls, saving the RenderTarget2D as a different texture for both, then pass those on.
I'm sorry this is mostly ideas and less hard code, but I've never really thought about something like this before, and everything here would take some testing and tweaking to make it work. Hope it gives you something to go off of, though. The methods here would also only make straight cuts and not more complex ones, but I think that may be sufficient. If you use the damage texture with RenderTargets like I mentioned, it should retain the jagged edge.
