Walking past a soggy piece of A4 paper, crumpled among cigarette butts and dog urine at the bottom of a lamp-post, is the closest most people come to any interaction with the British planning system. The nosy might stop to uncurl the wind-battered notice, jot down the reference number and look up the application online, where, once they have navigated pages of documents, they might be lucky enough to download a file that shows what is being proposed – if they can decode architectural drawings, that is.

For those who aren’t prepared to go to such lengths, the first warning that a 40-storey tower is being built at the end of their street often comes in the shape of the tower itself.

The opacity of the planning system has long fostered a destructively combative culture on both sides of the building site fence. Developers go to increasing lengths to obscure the reality of proposals, organising token “consultation” sessions at times when few apart from the retired can attend, and concealing their calculations in pages of viability assessments. Opposition groups, meanwhile, have become ever more militant in their attacks on the developers, staging marches, protests and public meetings with the crazed fervour of a lynch mob.

Communities understandably feel cut out and disenfranchised from the shaping of their cities as the march of luxury flats continues, and developers feel they’re being held over a barrel, forever cast as villains while being forced to contribute to local authority coffers over and above what they deem fair. And with resources stretched to breaking point, councils are resorting to similar measures, as if to say “we’re not guilty, we’re being forced to do it”.

In Hackney, east London, a bold hoarding erected around an estate regeneration project declares: “28 new council homes for social renting, 39 for shared ownership and eight for private sale to help pay for them all in the absence of government funding”.

It is an unprecedented statement for a local authority to make, joined a little further south by another council-sponsored billboard warning of “A dark future for Shoreditch”. With big black and red letters giving it the look of a horror movie poster, it outlines exactly why the borough is opposed to controversial plans to develop the Bishopsgate Goods Yard, which could be the site for a cluster of 12 buildings of up to 47 storeys, where one-bed flats are likely to go for more than £1m.

“We’ve had the decision taken away from us,” proclaims the poster, in reference to the mayor of London, Boris Johnson, calling in the project to decide for himself, “but it’s not too late for you to have your say.”

Perhaps people could have had their say a little earlier, rather than the developer wasting millions on professional fees and the council now spending taxpayers’ money on a campaign to stop the project so far down the line – if they had been allowed to have input before the designs were already fixed.

These issues are the nub of a new report produced by the mayor’s Design Advisory Group, an independent body of experts from across the architecture, planning and property sectors, on tackling London’s growth up to 2030 – by which time an extra 1.5 million people will need to have been accommodated in the capital. Unlike the pages of generic platitudes that usually emerge from such groups, Growing London is a breath of fresh air, addressing longstanding flaws in the development system and weaknesses of the London Plan with commonsense recommendations that any new mayor would do well to take up.

The report is ambitiously broad in scope ranging from rethinking density and policy for tall buildings to how suburbia could be intensified, and the need for better understanding of the total lifetime maintenance costs of new buildings. There are good recommendations on breaking up large parcels of publicly owned land into smaller development sites, to give smaller house-builders a look-in, and on encouraging local authorities to share expertise in leading development projects themselves. There is important advice about measuring the qualitative impact of planning policies, given the strange new urban forms often spawned in a developer’s vain effort to tick all the boxes.

The most important thrust of the report, which should be embraced by the next administration, is about finally cracking open the byzantine development process in London, laying it out under the spotlight for all to see. For one thing, it advises that the feeble A4 planning notice should be rethought – perhaps a picture of the proposal might help? – and that it could be linked to a virtual 3D model, so you could see what was being proposed in its wider context. Astonishingly, most planning authorities still don’t have their own in-house virtual models, or request the submission of 3D models with big planning applications, meaning the cumulative impact of neighbouring developments can go critically ignored.

Building on this, the report stresses that public engagement in development needs to become less reactive and move to a much earlier stage in the process: consultation should inform the principles of any development before they are fixed, rather than simply present a finished design to be voted on. It also demands greater transparency and accountability over section 106 agreements, which mitigate impacts of schemes, and the viability appraisals that underpin them, to help reconcile the mismatch between communities’ demands and the reality of what developments can deliver.

If the potential wider benefits of a scheme are spelled out, at least people can judge whether it is in their interests, rather than resorting to a nimby knee-jerk no.

If London’s population is to grow by as much as 100,000 people every year – adding the equivalent of a town the size of Bath on an annual basis – then Londoners have to come to terms with the realities of what that kind of growth means. Opening up the planning process to full scrutiny, with choices presented in a way that people can actually understand and feel part of, is the crucial first step.