Currently you are trying to put a jQuery object - literally an object, not a string representation of it - into a string. This won't work.
You are already using jQuery, which is great for constructing elements and creating event handlers on those elements, without reverting to setting inline strings of onclick="". Create the button element separately, setup the click event handler, then append it to the modal:
// get parent element
var linha = $(this).parent().parent();
// create a button
var button = $('<button type="button" id="btnOK" class="btn btn-md" />');
// add click event handler to button
button.click(function() { RemoveLinha(linha); });
// append button and text to modal
$("#modal-footerMsg").append(button, "OK");
Or if you want to be concise but messy:
var linha = $(this).parent().parent();
$("#modal-footerMsg").append(
$('<button type="button" id="btnOK" class="btn btn-md" />')
.click(function() { RemoveLinha(linha); }),
"OK"
);
onclick='RemoveLinha(linha);'>? it's just a string...