Already in the 11th century, the castle was built at this site and belonged to the Counts von Peilestein (known locally as Pilštajn), including Hemma of Gurk (Slovene: Sv. Ema Krška). Count Tattenbach rebuilt it around 1550 in a Renaissance style and added a defensive ditch to guard against Turkish incursions.
Thereafter the castle changed owners several times until 1663, when Janez Sakhmardi secured the agreement of Emperor Leopold I, the Bishop of Ljubljana and Count von Buchaim to found a Pauline Eremite monastery on the site. In the same year six brothers arrived from Croatia and settled in the castle, turning it into a monastery. In 1665–1675 a Baroque church with altar of the Assumption of Mary was added. The monk Ivan Ranger painted the entire presbytery in 1740, and a side chapel of St Francis Xavier was built in 1760–1766 and decorated by Anton Lerchinger.
The monastery was dissolved by the order of Joseph II in 1782. In 1805 it was bought by Attems, who pulled down the northwest and northeast wing, for they were unable to maintain such a great complex.
At the end of World War II the Attems were murdered and the complex nationalised. Later in 1974 it was badly damaged in an earthquake. In the years following it has been thoroughly renovated. Since 1990 Olimje Monastery has been occupied by monks of the Minorite order, who also manage the parish.
The three-storey complex consists of a southeast and southwest wing with oval towers in the corners. The façades were renovated after the earthquake in Renaissance style. The Baroque church with a bell tower is situated at the west side, while a herbal and vegetable garden faces south. Other structures or features belonging to the estate are a mill, a linked double hay-rack (toplar), and a pond.
In recent years conference facilities have been installed in the Renaissance cloister of the monastery, offering an area of 300 square metres with a capacity of 250 seats. This conference centre features interpretation booths for simultaneous translation and contemporary audio-visual equipment. The facilities have air conditioning, heating and thermostatic control. There is a large foyer which, together with the conference room, may be used as a 400 square metre exhibition area.
In the lobby on the ground floor there is a permanent exhibition of herbaria made by Marija Gaber.
A special attraction of Olimje Monastery is the pharmacy on the ground floor of the southern tower, which was established by the monks and was one of the first in Europe. The annals of the order contain several references to the Pauline monks' use of medical herbs and engagement in healing. The pharmacy was decorated with frescoes by the painter Anton Lerchinger in 1780, containing a fresco honouring Paracelsus. Frescoes represent biblical events with an emphasis on fruits and plants that have been used for healing purposes. The original pharmacy equipment is not preserved, only a few vessels are stored at the Maribor Regional Museum. Homemade herbal infusions and other remedies, produced in the traditional manner, are offered at the pharmacy shop.
The main altar of the Assumption of Mary, which is one of the largest gold baroque altars in Slovenia, was made in 1680. First it was black like the two present side altars and a pulpit, and later gilded. The organ was constructed by the Czech master Janez Frančišek Janaček in 1765.