Swami refuses to take orders