Rōshi was also an Iwagakure ninja, the Four Tails' Jinchuriki. He often wore purple clothes and old-fashioned armor like Hashirama's. Because he didn't get along with Ōnoki—and was even more stubborn than Ōnoki—he left the village.
He hadn't defected; he had simply left. It was like how Jiraiya and Tsunade had left the village to get away from what annoyed them and find somewhere secluded to train.
His stubborn personality was also why, despite getting along quite well with the Four Tails for over forty years, he had never been willing to call it by name.
Though Rōshi could not be entirely blamed for that. Naruto probably would not have wanted to say it either.
According to the Four Tails himself, "I am the Water Curtain Cave Handsome Monkey King, granted the Dharma name Sun by the Sage of Six Paths, Sun Wukong—the Great Sage Equal to Heaven!"
Just based on the length of that name, Minato might have had some blood relation to the Sage of Six Paths.
Naruto found it deeply uncomfortable that the guy was using Sun Wukong's name, while Rōshi might simply have failed to remember it.
With his stubborn personality, another way of putting it was that he was an old tsundere. After failing to remember it once, he had been too embarrassed to ask a second time, then thought, I'm not calling him that because I don't want to, not for any other reason.
And until the day he died, he had never become a perfect Jinchuriki. Tsundere tendencies really could ruin a life.
Now that Han, the Five-Tails Jinchuriki, had gone missing and was suspected of having been taken by Akatsuki, Ōnoki naturally needed to confirm Rōshi's safety as soon as possible. Bad blood or not, as Tsuchikage, he could not be too