0.1% of alpha particles were deflected back by nucleus, why did Rutherford say it very massive? and not small.?

He did not say "occupies most space in the atom" as you mention. In fact, he concluded precisely the opposite .... that there was a central +ve charge with less than 1/3000th of the diameter of the atom, i.e. the core is very very small part of the atom and since almost all the mass is stored there, it's very dense.