Not quite how I read it.
If you have permission of an owner to fly over his house and the persons in that house know you are undertaking that flight or have even asked you to photograph their property, they are under your control. It is then up to you as the Pilot to manage that environment and if that means the occupants stay in the building until you're done that is one way it can be managed or if they wanted to watch, they stand beside the Pilot or behind him where he can control them if necessary. But the important thing is, they are under the Pilots control.
However unless you get it in writing from an adjoining owner that they consent to flying in proximity to their property, within 50m that is where the problem lies. The way the CAA restrictions are worded e.g. buildings under your control, from a legal view adjoining buildings will never be under your control as you are not the legal owner of that building, so how can you have control. Well, in my opinion, as long as you notify the adjacent owners of your intentions and have obtained their consent, advised them they should stay indoors for the duration of operations you have managed the environment as best you can. You just continue to manage and observe the area for anyone who may enter the area then make a decision to terminate of move or ask them politely to stay outside your area of operations.
Flying over people or within 50m in flight or 30m on take-off and landing is the restriction for people not under your control. You always have inquisitive people who may wander into the environment, but it is the Pilot in charge of the UAV to put in place procedures to manage that. This as far as I am aware can be achieved by using additional persons or observers with two way radios to look out for anyone entering the arena of operations and if it was someone undertaking paid work they could I suppose allow for providing cordoned areas to further manage an area.
Obviously if you as the Pilot of the UAV were getting paid for this you would also have to have a license from the CAA.