Typically you would generate a visibility graph and perform pathfinding on that. Refer to this excellent page on map representations by amitpamitp.
Given an open space with polygonal obstacles, the shortest path between any two points is either:
- directly from start to finish, or
- touches the boundaries of at least one of these obstacles
So if you generate a visibility graph, made up of the edges of all these obstacles and connected together based on visibility, any shortest path can be found using that graph. It looks something like this:
See how the shortest path touches some edges and corners, and the shortest path is completely within the visibility graph. This is a result that can be proven but it's pretty intuitive anyway.
