Disable validation for specific inputs in ASP .Net Core

I am facing an issue when validating a form on ASP.

I have 2 models Person and Address. Each of them have their specific forms to add entries using EF Core.

In Person form I need hidden inputs to keep track of potential Address related data (It is perfectly fine there is no Address). To that matter I use hidden input fields but because the Address model have validation annotations, the generated markup reflects it which puts ModelState.IsValid to false upon validation unless, of course, I am editing a person with valid Address data.

I also tried to set Person.Address to null in the specific situation where it should be just before actually checking its value and it does not work. Probably it is already too late for ModelState to update by then.

if (Person.Address is not null && Person.Address.Id == 0)
    Person.Address = null;

if (ModelState.IsValid)
    // Further code to save data in DB
else
    // Code to redisplay the page

I would like to know if there is a way to tell ASP not to validate/ignore specific input fields but still allow me to keep a piece of data?

Thank you very much.

Edit: As per demand in the comments I add the classes with annotation

The Person

public class Person
{
  [Key]
  public int Id { get; set; }

  [Required]
  public string FirstName { get; set; }

  [Required]
  public string LastName { get; set; }

  [Range(0, 150)]
  public int Age { get; set; }

  public Address Address { get; set; }
}

The address class

public class Address
{
  [Key]
  public int Id { get; set; }

  [Required, MaxLength(50)]
  public string Street { get; set; }

  [Required, MaxLength(50)]
  public string City { get; set; }

  [Required, MaxLength(50)]
  public string Country { get; set; }
}