Kiviloo was founded as an Episcopal castle in 1413. The few ruins of the Episcopal castle (destroyed in the Livonian War in 1558) can be seen even at present. The current eclectic manor house (in private possession) was built during the ownership period of the von Stackelbergs in about 1906–1910 (the earlier wooden structure was burnt down in 1905).