Abstract In this paper, we address one of the most typical problems of person detection: scenarios with the presence of groups of persons. In this kind of scenarios, traditional person detectors have difficulties as they have to deal with several simultaneous occlusions. In order to try to solve this problem, we propose the use of two different hierarchies. The first one consists of a hierarchy of persons, i.e., the use of the detection of different persons belonging to a group in order to refine the individual’s detections. The second one consists of a hierarchy of parts, i.e., the use of different combinations of body parts in order to refine the final detections. Experimental results over several video sequences show that the proposed hierarchies significantly improve the results with respect to different approaches from the state of the art.