Atomic memory instructions are indivisible operations performed with memory like swap or add. They are usually done outside of the CPU itself perhaps at the memory controller to guarantee the operation will be atomic. The memory controller needs to be able to process commands given to it by the CPU beyond LOAD or STORE. Normally for a CPU that only performs loads and stores the two are distinguished by a write enable signal. To support atomics more signals are needed. For my own CPU, I have it output a five-bit command code which includes all the atomic operations in addition to load and store, also output is the data associated with the command. The memory controller then takes care of the atomic part and sends back results if needed.
Probably the best place to put atomic memory operations is in the mem unit of the CPU.