The vassal castle of von Lodes, established in the 14th century, later belonged to the Saare-Lääne bishopric and was reconstructed as a major fortress with an internal yard and two corner towers.

Later on, Koluvere belonged to the von Loewens, Count Orlov and von Buxhoevdens, and the medieval fortress was reconstructed to resemble a palace. In 1783, the palace was bought by the Russian Empress Catherine II, to exile the Württenberg Princess Augusta Carolina (who died a year later). The interior of the palace has not been preserved as a result of many fires (1840, 1905, 1960), while the medieval exterior was restored to an extent in the 1960s. A care home was located there in the 20th century. The palace is in private possession since 2006.

By the historical distribution the manor was in Kullamaa Parish in Läänemaa County. By the present administrative distribution it belongs to the Kullamaa Municipality in Läänemaa County.