I too am surrounded by non-believers, mostly. My partner is vehemently against it (he's a hardcore rationalist, the most pure left-brain rational person i know) but tolerates it (particularly when I connected it to his cultural background, Jewish Kabbalah, though at first he was offended)
I've come from a rationalist background myself (identified as atheist and nihilist from my early teens, continued doing so even as i actually started to learn what they meant, until i had an epiphany near the end of college, getting me into closed system thinking, myers-briggs, qualitative type systems, myth and folklore, all to diving into astrology about a year ago) and have some recollection of why I was against it -- and other reasons I've learned when bringing up astrology in mixed company
- many astrology-supporters and pop media explicitly say or implicitly suggest that planets have a causal effect on the goings-on of humans, which is really, really unlikely to be true. most academically-inclined astrologers i've found rebuke this causal explanation, but it persists in the collective ether that astrology is so opposed to the dominant rational view of the universe and its known forces
- astrology is connected to new ageism, which generally has a bad rap in culture and society as a leftover of the 60s, opposed to a budding system of study in its own right
- astrology seems like a scheme for good tone/behavior readers to manipulate clients with astrological babble to that confirms their existing assumptions about themselves, for their money. i've heard people refer to sun sign astrology as their basis for this claim, as it often is written in such general ways that they find it useless
- on top of the poor public image of astrology, science and academia is pretty swayed towards reductionism, materialism, atomism... in essence, only that which can be put into numbers or equations. the social science either tow towards the hard sciences and emphasize behavioralism, biochemistry and/or statistics (leading to manuals of vaguely-worded pathologies and a centerless amalgamation of psychology phenomena), though some like anthropology emphasize participant-observation, the inevitable interference of the measuring process, and faith that not all qualitative analysis is without import
it would also help if astrology had a clear center of professionalism and a clearer standard of research. and if its metaphysical principals could be better articulated. most rationalists i know can't stand believing something that can't be rectified with their total understanding of reality.