Many people seem to get confused about where they are on this climb. Happened to us the first time we did it, and the second time we ran into a party that had also gotten off route at about the same place. See rec.climbing for several trip reports in this vein.
The Swain topo is confusing with regard to the first belay. It shows a tree on a ledge, close to the end of the first pitch, and indicates a belay above that ("on a sloping terrace"). Instead of the tree, you will see a three bolt belay / rap station on that ledge (the second set of bolts you encounter on the first pitch). Belay at the bolts, and climb a short (60 ft), 3d class section over the sloping terrace, up and to the right, to the base of a short black wall. You have now reached the "official" first belay as shown on the topo. Belay either at the base of the short black wall, or on the ledge above it. From there, both the topo and the description are accurate.
Also, watch out for the rope getting stuck on rappel. Lucky for us, we only spent about 10 minutes getting it free (on the second rap), and did not have to climb back up. Again, see rec.climbing for some other stories.
All in all, a great route.