In 1310 the Count of Schaumburg enfeoffed the von Rottorp family with the court in Hülsede, which later became the ground floor of the subsequent manor house.
The castle was built as a four-winged building between 1529-48 by (Clawes) Claus von Rottorp and further extended up to 1600.
In 1584 the male line of the von Rottorps died out. [citation needed] The son-in-law, Ernst von Reden, was not able to establish his claim against the counts of Schaumburg. As a result, the Count of Schaumburg enfeoffed the Bishop of Minden, Hermann von Mengersen, (seneschal (Drost) in Rodenberg and Sachsenhagen) who married Ilse v. d. Born) with the Hülsede estate.
In 1908 the fief became the freehold of the Mengersens. In 1970 Monika von Bronsart, née von Mengersen, became the owner. In 1979 restoration of the castle was begun.