You could consider the phantom an extension of the move wood phase. So then you would need to place the phantom before using a tower section to remove a follower. I might be persuaded to go with that - but I don't know if there is anything written to support it. The closest I can come up with is can you use a festival to retrieve your phantom and then place him on the tile?
CAR v7.3, footnote 410 states that "If the Festival is used to remove a follower, the Phantom can still be placed on that turn." This doesn't specifically state that the Phantom can be removed and then re-played, but the implication is that the Festival action happens (follower removed), then the Phantom can be placed. It seems that the Phantom could be removed and replaced, as the Phantom acts as a normal follower, so should be removable in Part 1, then would be available to place in Part 2 of the move wood phase.
The Tower rules state that a player can take a follower prisoner "whenever a player places a tower piece," so I would treat this as an analogously immediate event -- the tower action occurs when the piece is played, not at the end of the move wood phase.
One thing though is you have to place the phantom on an incomplete unoccupied feature on the tile, don't you? That would exclude you from placing a tower section as the first part of your wood move and putting the phantom on top of a tower section as the second part of your wood move. The same way a phantom cannot go through a magic portal or go to the wheel as the second part of a wood move. It has to be placed on a feature.
The phantom actually
can go through a magic portal or be placed on a tower segment or the Wheel of Fortune, you just can't use the magic portal if the first follower already did. (CAR v7.3, footnotes 406, 411, 412).