The community of Clem was named for Clem Danneman. Clem lived in Scott's Canyon, three or four miles southeast of the townsite. His home was a stage stop between Arlington and Condon. The first Clem post office was at his home. The town of Clem was started when the railroad came through in 1905. Two competing towns were platted - Clem was laid out on the east side of the main road between Condon and Arlington; Welshons was laid out on the west side. Eventually, the towns were combined and called Clem.

There was a school house, three stores and two saloons. A church was never built, but sometimes Sunday school was held in one of the saloons Oregon in a small building across from the hotel. School was held in one of the saloons until the school building was finished. There was a grange hall and two livery stables, about a dozen houses, a garage and a blacksmith shop. One of the stores was a "blind pig." they even had a small golf course. On the railroad right-of-way stood the train depot, three grain warehouses and the Arlington lumber yard. Later, three grain elevators were built. Clem was the hub of several roads - the Cunningham Market Road from Mikkalo and Rock Creek to the north; the main road from Arlington to Condon; and a road coming from Rock Creek Canyon on the east to Hay Creek Canyon on the south and west.