Sandwich shop Appetite and newsagents Nisa Metro say the work on the Lumiere skyscraper is stamping out passing trade and profits are plummeting.
Vibrations from the diggers have even sent bottles of wine and shelves crashing to the ground, according to Nisa Metro boss Jazz Singh who says he is planning to take legal action.
As previously reported in the YEP, popular bar Breeze has recently closed down after two years – blaming its failure on the construction work.
The council's highways department has diverted pedestrians off Whitehall Road and Wellington Street and onto the opposite side of the road.
This has seen a severe drop in passing trade according to some of the shops, including Appetite.
Manager Steven Rush said: "We have definitely had a downturn in profits. People don't want to stand outside and be bombarded with noise and dust."
Mr Singh added: "Since they have been working on here, our sales have dropped drastically. We are losing between £300 and £400 a day.
"The drilling is getting worse and we had bottles of wine falling off our shelves and they all collapsed. When it's drilling like that it's not just a bit of movement, the whole place is shaking."
The twin towers – the tallest of which will be 54 storeys high – are expected to take four years to complete.
But not every business in the area says trade has been hit.
Two new restaurants say they were prepared and that the sealed off pavements have not put customers off.
However bosses at both Primo and the Lazy Lounge admit that the effect of the work is difficult to tell as they have both only opened within the last two months.
Richard Dean, joint managing director at developer KW Linfoot Plc, said: "Whilst we acknowledge the concerns raised by some of the local businesses, especially those situated on the ground floor of West Point, they were aware that there would be major construction work taking place on the site before agreeing to move into their respective properties.
"That said we will of course endeavour to keep the disruption to a minimum within the parameters possible whilst building a project of this scale.
"We believe that Lumiere will be a fantastic addition to the city once it is complete and the surrounding businesses will benefit greatly from the captive audience that a building of Lumiere's status will create."