Well, i can't speak for all extention subjects, but tbh, i don't think it will be too big of an impact on your studies. Granted, i only do three propper subjects so my view may be a little skewed, but if you love a subject that much, it won't really be much of a burden on you. I can say that evey moment i spend doing maths is a moment i enjoy. If this is how you feel about jap, go for it.
Especially, becuse the stuff you learn in jap e.g. the new vocab, the more advanced grammar and such will directly help your 3/4 jap this is one more reason that doing uni jap is a good. idea. For most uni subjects, like uni chem, biol, maths, the stuff you learn in the uni corse, while it extends you and exposes you to new concepts, it dosen't directly help you with hte corresponding 3/4, however this is not the same for languages.
Not sure if this hold true for languages, but i have a 2~ hour lecture each week, and probably spend one to two hours a week doing uni maths hw.
Bottom line, because it directly helps your jap and you enjoy it, go for it. Just think of it as extra jap homework:)