The Imperial College is the highest academy set up in the Yuan, Ming and Qing dynasties to train talents for the feudal authorities. The Imperial College was first built in the 10^th year (1306) of the Dade reign of the Yuan Dynasty (1271-1368). According to the traditional rule that temples lie in the left while college lies in the right, the Imperial College is next to the Confucian Temple.

The principal building in the Imperial College is Piyong, which was exclusively built for emperors to give lectures. From Emperor Kangxi of the Qing Dynasty (1644-1911) on, every emperor would lecture here after his enthronement. Piyong was constructed according to the rules of the Zhou Dynasty (11^th century 　 256BC): facing the south; a square plane; 5 zhang (1 zhang =3.33 meters) and 3 chi (3 chi=1 meter) wide and long respectively; double-eaved pyramidal roof on four corners; yellow-colored glaze covering the roof with golden pearls on it. The wall has four doors in four sides and is surrounded by a cloister and pond. The pond is enclosed by white marble banisters with a stone bridge across it connecting the four doors of Piyong. They form a framework of Piyong by the water like that in the Zhou Dynasty.

North to Piyong, there are seven principal rooms called Yilun Tang for keeping books. 32 wing-rooms along each side are the places for lecturing and teaching. They share a collective name of Six Tang: Shuaixing Tang, Chengxin Tang and Chongzhi Tang in the east; Xiudao Tang, Zhengyi Tang and Guangye Tang in the west. Jing Yiting yard, which is behind them, is the place for the supreme instructor of the Imperial College to work.

In the passageway between the Imperial College and the Confucian temple, there are 189 pieces of carved stones of the Thirteen Sutra. There are up to 630,000 characters on the stones. A Gongsheng (student of the Imperial College)named Jiang Heng from Jintan City of Jiangsu Province spent 12 years writing them. In the 59^th year (1794) of the Qianlong reign of the Qing Dynasty, the steles were set up here. At that time, there were nearly one thousand students (called Jiansheng then) studying in the Imperial College with foreign students from Russia, Jiaozhi (now Burma), Gaoli (now Korea), etc. Students from the ethnic minorities like Mongolian, Hui, Tibetan and Manchu studied here as well. Thus, the Imperial College was not only a gathering place for the talents but also an important site for the cultural exchange between China and other countries.

The Imperial College has now been transformed into the Capital Library with a collection of 2.46 million books, together with many rare ancient books and their hand-copies, microgrooves of famous music and dramas all over the world, and so on. It has been an important databank for various studies.