When you set the origin you have to take into account the position of the card and its previous origin.
card.setOrigin(event.mouseButton.x - (card.getPosition().x - card.getOrigin().x),
event.mouseButton.y - (card.getPosition().y - card.getOrigin().y));
I don't really know how to explain, I will try with a drawing.
Your card is the red rectangle: its position is the dark green arrow and its origin is the light green arrow. The position of the click is the blue arrow.
So, when you subtract the light green arrow to the dark green arrow you get the purple arrow. So the purple arrow is card.position - card.origin.
Then you subtract that purple arrow to the blue arrow, and you get the pink arrow mouse.position - (card.position - card.origin). And the pink arrow in the new origin of the card, it is at the position where the player clicked with coordinates that are local to the card. i.e. on a card that is at position (0,0) and origin (0,0).

I hope that makes sense.