For example, a black regular fisherman is deployed on a river. Black draws a tile with chivalry icon and places it at this river. Now he can take back his black follower back and place a big follower there.
The builder, mayor, barn, pig and wagon cannot be placed on the river, so I think there is no other follower you can place there. But if you use the Fishing Boats, you could exchange the follower to a fishing boat, which is a big follower and "wagon" for the rivers.
You can change the followers too if there are other followers of other players and if you don't have the majority.