Sky News, citing the agency, reports that although the government had discussed the issue, it had resolved the challenges posed by Covid would be too great.
The games will be held from 23 July to 8 August and the Paralympic games from 25 August to 5 September. Both had been due to take place in 2020.
The International Olympic Committee is yet to formally confirm the decision.
A spokesperson for InsideJapan Tours, which had nearly 300 people due to travel to the rescheduled games, said that while the news game as a big blow, the operator understood why it was necessary.
"We are as disappointed as our customers will be with the Japanese government’s decision not to open the games to international visiting fans this summer," they said.
"Of course, we understand why this is necessary, but nonetheless it’s a big blow to everyone who booked the trip of a lifetime, to thousands of hotels, guides and restaurants, and to Japan, which has invested huge sums into the event.
"As a specialist Japan tour operator, we had hoped the Olympics would be a global celebration marking the reopening of the country for inbound tourists. Sadly, this isn’t to be, but the safety of all visitors and of the population of Japan is paramount.
"We fully support the decision and hope that by taking this measure now, it will enable Japan to safely open up its borders to international visitors later in the year."