Urban beekeeping in New Mexico's largest city.
I keep thinking about your question, and finally I have had enough time to digest your neighbors concern and come up with a thoughtful none emotional response, but I have to say (Does your neighbor know everyone is alergic to bee stings, the question is how allergic?)
Some of my thoughts, It is possible that a species of ground nesting bee that closely resembles a honey bee (Plasterer Bee or Sweat Bees) could be their problem. Other possibilities are that your bees or some other nearby beekeepers bees are looking for water in their garden or from a drip line, leaky faucet or hose. Note they like drip lines and dripping faucets with minerilized buildup. if this is the case you could remedy the situation by providing a preferred fresh water source (well water is usually ideal) with minerals. The direction of your bees flight path may be contributing to their fear and excerbating the problem, this can be easily corrected by re-orienting the hive opening to a new direction. Perhaps during certain times the growing season the only thing in bloom may be nearby flowering garden plants, perhaps feeding might help if this if this is the case.
Sometimes a gift jar of honey, alot of education and friendly discusion will make the difference, let them know you are passionate about your bees and our ecosystem, share your knowledge about bees with them.
Best wishes to you and your bees.
P.S. Sometimes complaints are pre-emptive or retailatory. Examples of a pre-emptive behavior might play out the following way; Some flower and landscape gardners use systemic pesticides and herbicides extensively and often incorrectly and they get annoyed when beekeepers live nearby, knowing the beek may complain, clue; If their addicted to pesticide you may note that their garden is weed and insect free, and they don't have a paid gardner.
Retailatory behaviors could be the result of past disagreements or neighborhood disputes.
**Sadly for the bees some people just hate bees and their is not much you can do. If this is the case and you continue to pursue beekeeping make sure you homeowners liability policy does not exclude beekeeping activites.