Should I swap or not and should I be concerned that my spouse views on it are so different then mine?

Your concerns are legitimate and if you don't want to participate and did it anyway, it would most likely destroy your marriage. Stick to your guns.