Although an appointed commission chose Spencer Township to be the county seat, the citizens rejected that choice and elected Peterson to be the site. The voters also authorized $6,000 for construction of public buildings. Although the contractor's bill for the courthouse totaled $7,500, the county only paid him $6,900.
Clay County grew rapidly and in 1871, the county moved to Spencer. A courthouse was built for $1,333 but became too small so a new brick courthouse was constructed for $4,435. The old courthouse in Spencer was then used for a barn, a cement block factory, and finally remade into two houses.
Members of the public did not want a plain, unpretentious courthouse, so in 1900 the present courthouse, a beautiful red stone structure was built. This courthouse, with its pillars and a bronze dome cost $60,000.