A heritage listed house on Wangaratta's Bruck Court will be demolished and seven two-storey townhouses will be built around Bruck House, a building which will be maintained.

The development will provide key worker housing for Bruck Textiles employees into the future.

Demolition of the caretaker's quarters, which were built in the 1950s, was cause for concern among local history buffs, and the development of the new units in the precinct also upset some residents in the area.

The caretaker's quarters and Bruck House are in a heritage overlay and they were designed by leading Australian modernist architect of that period, Robin Boyd.

At the December council meeting this week, director - sustainability & culture Stephen Swart recommended that the planning application be approved despite 20 objections to the permit.

Chris Spencer is the next door neighbour to the proposed development and she asked councillors whether they would like double storey town houses built along their fence lines.

"It's right in the middle of a heritage site designed by Robin Boyd," Ms Spencer said.

"This is Wangaratta's chance to hang onto something precious and unique to Wangaratta.

"Don't knock it down, don't allow this to happen."

The Bart Group, which owns Bruck Textiles, currently has another planning permit for a 33-lot subdivision of industrial land adjacent to Bruck Court.

A question from Wangaratta resident Anne Shaw asked why couldn't some of this land be rezoned residential to meet the company's worker accommodation needs instead of using land on Bruck Court.

However, Mayor Irene Grant told the gallery that each application must be assessed on its own merits.

Another local resident Claire Russell pointed out the heritage overlay clause on built form and heritage.

"Council seeks to encourage the restoration and reconstruction of heritage places in the heritage overlay...encourage built form that is proportionate in scale and respects places that are in heritage overlay," she read.

"Why would there be a recommendation to demolish that house and build two apartment blocks?"

Mr Swart explained that the heritage overlay is part of a comprehensive planning scheme and needs to be assessed in part with other matters.

"Our officers are asked to look at all of that information together and make a recommendation on balance of all of that information," he said.

"If I have made the impression that a heritage overlay is only a minor thing, that is certainly not the case.

"The officer's report attempts to generate a significant discussion about the heritage of this area."

In moving the motion to grant the permit, Cr Grant said she appreciated the significance of the caretaker's quarters and the relevance of Wangaratta's industrial history.

"Then, as now, worker accommodation was important to development and this was an example of what it provided at the time," she said.

"However, it must be said that the cottage is not in pristine condition and while it has some connection to the architect Robin Boyd, it was not the main focus of his work.

"That is Bruck House itself and that's being maintained."

Cr Harvey Benton also made the point that objectors want the buildings to be retained, but he asked who is going to maintain them, as the owners don't want to.

"At the end of the day I think we have to move forward and be practical," he said.

"If we don't demolish it, it's just going to fall down unless somebody in the community pulls their cheque book out."

Cr Allison Winters and Cr Dave Fuller did not support the granting of the permit.

"I personally wouldn't like five double storey townhouses overlooking my backyard and invading my privacy," Cr Winters said.

"Concerns around heritage significance, residential amenity and neighbourhood character are reasonable and I have concern there may be a demolition by neglect in this case."