Ion implantation at low energy in silicon is known to generate collision cascades that produce heavily damaged zones called Amorphous Pocket (AP). Understanding how such APs accumulate to form a continuous amorphous layer is still an open problem. In this contribution, nanocalorimetry is used to measure the amount of heat released during a ramped annealing. This yields information about the annealing of defects in term of activation energies to track their evolution over a wide range of fluences (10¹¹-10¹⁶ ion/cm²). The presence of the amorphous phase in the material at high fluence is established by observing a recrystallization peak in the calorimetric measurements. Our results, at energies between 10 and 80 keV, show that rather simple defects with low activation energies build up to create more stable damage and eventually form an amorphous layer. Moreover, we show that 80 keV self-implantation yields a progressively more stable amorphous layer with increasing fluence. A supralinear behaviour in damage accumulation is observed between 2.5 and 5 x10¹⁴ ion/cm². This suggests that damaged zones interconnect and eventually merge to form a continuous amorphous layer. One the contrary, similar experiment carried out with 10 keV Si exhibits an amorphous layer of constant stability with increasing fluence. The different behaviour suggests that denser cascades form more stable structures yielding directly to a continous amophized layer. These results are interpreted in terms of a model based on simple defect accumulation with activation energy depending on the number of first neighbour defects. It will be shown that while such model qualitatively grasps many features of the defect annealing, it underestimates the complexity of the build up of defects and their annealing.