Note on Manji's clothes

The symbol on the back of Manji's kimono is not a swastika. It is a sauvastika. Manji's name is the literal Japanese word for sauvastika. It is suppose to be a symbol of balanced opposites in the Buddhist religion. This means  symbol has a much different meaning for the Japanese culture.

This article and this article go into the difference a bit better than I can.

Or here is a Wikipedia article that goes through the long history of the symbol.

Or this person has one of my favorite explanations:

• But you're into Nazi comics.

No. The character on the back of Manji's kimono (robe) is not a swastika. It is a sauvstika. Adolf Hitler, in addition to being a mass-murdering f*&khead and a vegetarian painter, also had a slight tendency to steal other culture's well-known symbols and stick them onto Naziism. In the original Hindu religion, a swastika symbolised day and openness, while a sauvstika stood for night and secrets. The sauvstika is just a mirror image of the swastika. These symbols were widely known in Southeast and East Asia because of the popularity of Buddhism, and the fact that Buddhist monks had a tendency to travel and pick up bits of other religions, mostly Hinduism. Manji is actually the Japanese word for sauvstika. On a side note, Nazis did not yet exist during the time period in which Blade of the Immortal is set.