Crushing of blasted ore is an essential phase in extraction of valuable minerals in mining industry. It is typically performed in multiple stages with each stage producing finer fragmentation. Performance and throughput of the first stage of crushing is highly dependent on the size distribution of the blasted ore. In the crushing plant, a metal grate prevents oversized boulders from getting into the crusher jaws, and a human-controlled hydraulic manipulator equipped with a rock hammer is required to break oversized boulders and ensure continuous material flow. This secondary breaking task is event-based in the sense that ore trucks deliver boulders at irregular intervals, thus requiring constant human supervision to ensure continuous material flow and prevent blockages. To automatize such breaking tasks, an intelligent robotic control system along with a visual perception system (VPS) is essential. In this manuscript, we propose an autonomous breaker system that includes a VPS capable of detecting multiple irregularly shaped rocks, a robotic control system featuring a decision-making mechanism for determining the breaking order when dealing with multiple rocks, and a comprehensive manipulator control system. We present a proof of concept for an autonomous robotic boulder breaking system, which consists of a stereo-camera-based VPS and an industrial rock-breaking manipulator robotized with our retrofitted system design. The experiments in this study were conducted in a real-world setup, and the results were evaluated based on the success rates of breaking. The experiments yielded an average success rate of 34% and a break pace of 3.3 attempts per minute.