Generally the stakes nearest to a particular temple are in that temple's district. However, there are exceptions. For example, there may be areas near church headquarters where some stakes are not in the nearest temple district, due to the large number of stakes in that area.
Members of the temple presidency generally speak at stake conferences in their temple's district and temple workers are ordained for the temple in their district. However, any recommend holder can attend any temple any time it is open. There may be times when it is difficult to attend a temple in one's district, or when one is out of town. Temple districts exist for reasons of order and convenience and a recommend gives you the privilege of attending any temple in the world.