It is situated in a street north of Odisha High Court. There was jungle & cremation ground at the present site of the temple during Marhatta rule. A sannyasin found a Shiva Linga in that jungle and started worshipping it. She built a kali idol using copper and soil taken from the bed of the Ganges.
Lord Shiva's idol is to be found at the feet of the present Kali idol and the orginal Shiva Linga in front of it. Kalipuja is performed on the Kartik Amavasya (new-moon) througout the state.