How to delete a pointer inside a struct object when the struct object goes out of scope?

I have a struct object that contains a pointer to an object.

struct Track
{
    KalmanFilter* kalmanFilter; //Kalman filter for this track
    ....
};

I instantiate the struct and pointer objects by

Track track;
track.kalmanFilter = new KalmanFilter(contours[contour].centroid);
....

I then check each frame if the track is still valid by

//Temporary vector to hold alive tracks
std::vector<Track> tempTracks;

for(...)
{
    //If track is valid save into temp tracks
    if(tracks[track].deadTime <= deadTime) tempTracks.push_back(tracks[track]);
}

//Save all temp tracks back into tracks - this removes dead tracks that were in tracks
tracks = tempTracks;

After a while the micro-controller i'm using runs out of memory, this is the only pointer in my code so i'm pretty sure the KalmanFilter pointer is not being deleted.

So far I have tried a destructor in the Track struct which causes my micro-controller to hard fault immediately.

//Destructor frees up Kalman filter memory when the Track object goes out of scope
~Track()
{
    if(kalmanFilter) delete kalmanFilter;
} 

and I have tried using a unique_ptr but I cannot get the code to compile, all the examples I have found instantiate the object being pointed to at the same time as instantiating the unique_ptr. In my case I don't think I can do this.

What is the correct way to handle a situation like this?

Thanks

EDIT

So based on the comments there seems to be two main points.

Allocate the memory in the constructor

I have rewritten my struct to be

struct Track
{
    Track(const Point& initialPosition) : kalmanFilter(initialPosition) {}
    ~Track() {}
    KalmanFilter kalmanFilter; //Kalman filter for this track
};

and I instantiate it by

Track track(contours[contour].centroid);

Is this correct?

Find a better way to delete the tracks

What is the best way to remove an object from a vector without messing up the indexing during the loop? Using an STL iterator?

I implemented the solution from @M.M that seems to work

tracks.erase(std::remove_if(tracks.begin(), tracks.end(), [&](Track const &t){return t.deadTime > deadTime;}));