Scandals regarding the treatment of workers have subsided for the time being, so the focus is once again on the performance of Sports Direct’s business. Oh dear.

Selfridges of Sport today announced a 72.5% drop in profits for the year ending April 29, due to founder and boss Mike Ashley’s eccentric habit of buying tracks other retailers – he has a soft spot for department stores – have returned to bite his investors.

Sports Direct is the largest shareholder in Debenhams, with almost 30% of the shares, and is also an investor in House of Fraser.

These are two department store chains that Selfridges bosses have to look at and think “phew, thank goodness we’re not them.”

The first was a dog. The collapse in its share price was not the only reason for Sports Direct’s fall in profits, and last year’s numbers were flattered by the sale of the Dunlop brand. But it played a big role.

House of Fraser, meanwhile, is undergoing a painful restructuring that will result in the loss of thousands of jobs. As a company, it also goes “whew”.

Unliked Sports Direct Chairman Keith Hellawell has sought to champion the company’s portfolio of retail holdings as “innovative strategic partnerships” that “will help differentiate our offering.”

Right now, they’re doing it the wrong way.

The company is a sports retailer with a special situations retail fund. This last part pulls the first down.

I have laughed at Mr. Ashley’s ‘Selfridges of Sport’ concept in the past because it is a bit silly and suggests that Sports Direct suffers from an inferiority complex. Why not just talk about how you elevate Sports Direct and be proud of the brand you’ve created, rather than caring about someone else’s?

The company’s move upmarket seems to be working. Lee Wild, head of equity strategy at Interactive Investor, said the concept was working “better than expected” and other analysts have said similar things.

The company’s retail operations in the UK and Europe had a difficult year, recording revenue declines of 2 and 0.1 percent respectively. But underlying pre-tax profits are clearly ahead – up by more than a third – and the group continues to grow at a rapid pace in the rest of the world.

It may hurt some to admit it, but Mr. Ashley is a very good retailer, and the heart of Sports Direct is a very good company that keeps up with an extremely tough retail climate.

The problem is, Mr. Ashley is also exceptionally good at cutting his nose despite his face, and the retail investments that “differentiate” Sports Direct do the same for his business.

“A sharp drop in annual profits at Sports Direct suggests the company should stick to its core competencies,” broker AJ Bell said. I expect Mr Ashley and his team to hear variations on this theme regularly, because it makes a lot of sense. Imagine what the business could be like if it did.

I wouldn’t count on listening to Mr. Ashley, however.