Step carefully around this man.
His Leo Sun is in wide conjunction to your Moon. That means his ambient presence lulls you into your passive, domestic personality. This synastric contact means you are easily led by him. Between a husband and wife, it tends to harmony with the man (or the Sun person) being in the lead. But, in your circumstance being his boss its not an influence you should readily accept without some consideration.
Neptune aspects indicate seduction and/or deception. His Neptune is in strong conjunction to your Sun, dissolving the boundaries of your ego and its standards in subtle ways...which honestly isn't very hard to do anyways with a Sag Sun to begin with. When other people have Neptune conjunctions with planets in our charts they have a talent for unconsciously leading us into things we probably would regret later. It's a fabulous contact if the person seducing is extremely evolved or advanced spiritually...not so much in the average horoscope, where its influence can be degrading. Its an aspect of vulnerability.
Your Mars is Trine to his Venus. And all Venuses and Mars positions are in compatible Water/Earth signs. This is a potential inherent in the physical presence of each other. Many men and women share the trine aspect, so its often a dumb note, or manifests as a simple liking for the other, but when other aspects combine it can lead to a sexual affair. It probably doesn't help you to say that it would probably be good sex too, him being Scorpio Moon.
Don't be fooled by how well you get along at work. He's a Leo Sun, a Fire Sign at work, which blends very nicely with your double Sun/Moon in Fire Signs (Sagittary and Leo). Its a very satisfying energy interaction for you, so long as you both keep your interactions in public. But, he's a Scorpio Moon (Water)...which means he'll be VERY different in a domestic setting. And if you share those private times, there will be moments where you feel completely shut off from him, as Water is hostile to Fire.
Astrology is not going to tell you whether to have the affair or not. It will only map out interactions already there, and show you the potentials, both good and bad.
Best of luck, but for a Sag, I'm sure you'll be fine.