If you treat rooms as zones, joined by portals, you only only have to do visibility scans within each room and those parts of adjoining rooms that are visible through the pre-determined portals.
Your line-of-sight is likely not accounting for the direction the civilian is facing? In such a case, if you divide rooms that contain obstacles like pillars or goes around corners into separate convex zones and assuming that all civilians in the same zone are all visible to each other. You may take that further by having overlapping concave zones and allowing civilians to be in more than one zone at once as as to find as many as possible in the same zone as the other civilian so as to avoid all other visibility checks.
If your terrain is uneven and you have a large number of agents then you may be interested in this O(n) sweepthis O(n) sweep (where N is the granularity of a grid you've divided terrain into): 