I can't believe it took me 91 levels to realize this ><

That really changes everything lol

You can, but keep in mind that wearing armor too heavy will really screw you up in combat. Reduction in casting time for example.

I didn't notice this on my first char and couldn't figure out why my spells were casting so slow, then i realized while i could wear other armor, it's better not to usually.

Yes, it lowers all haste skills. Check your offensive skills tab. The heavier it is, the bigger the penalty.