Kawabuchi said he accepted Mori’s request in an emotional meeting on Thursday during which both men cried.
“Mr. Mori was straightforward saying ‘I want you to take over now this happened’,” Kawabuchi, 84, told reporters late on Thursday.
“I thought how hard it must be for him and I couldn’t stop crying,” said Kawabuchi.
The 83-year-old Mori, a former Japanese prime minister, sparked a global outcry with sexist comments that women talk too much, which he made during an Olympic committee meeting.
Mori has apologised for his comments but has so far refused to resign, despite growing calls for him to step down.
His resignation less than six months before the Summer Olympics are scheduled to begin would raise new doubts over the viability of holding the postponed Games this year.
Games officials are already struggling with how to hold a safe Olympics, with tens of thousands of athletes and possibly spectators, during the coronavirus pandemic.
Mori’s controversial comments were out of question, said Kawabuchi, former Japan Football Association president, however, he said wants Mori to play a consulting role in the Games to help make the event a success.
Kawabuchi represented Japan in football at the 1964 Tokyo Olympics and helped Japan co-host the 2002 FIFA World Cup with South Korea.
Later on Friday, the Tokyo Olympic organising committee, which has not officially commented on Mori’s resignation, plans to hold a meeting of its council and executive board, followed by a press conference.
Having initially said it considered the matter closed with Mori’s initial apology, the International Olympic Committee branded his remarks “completely inappropriate.” Reporting by Ju-min Park; Editing by Michael Perry