When I was a kid (middle of the last century), boric acid solution (5%, I think) was commonly used for eye-wash. Later turned out that enough of it got down the throat (via the nasolacrymal duct) that some kids got poisoned by it. We still use boric acid powder (with some kind of free-flowing agent) as an ant poison - it isn't very effective, btw.
Borax, though, is really not very scary stuff, and people have been using it for a long while. I think reasonable care will be adequate - don't eat it, don't eat your sandwich without washing your hands after handling it, and mainly, stay upwind of the fire.
Chances are that burning off the sulfur in some of what passes for coal these days is going to be more dangerous to your lungs.
Fluorite fluxes, on the other hand, can be seriously dangerous.
WRT brazing disease - I did that to myself once, back around 1975... I was building a sculpture, and liked the color effect of burnt brazing -- so I coated the whole piece with braze, and burnt it as I went along. Next day I had a fever, aching muscles, cough, and nausea. Thought it was 'flu. I didn't find out otherwise until I got back to work and talked with a guy who used to be a welder, who said it used to be called "Bell-founder's Ague" because the people who cast bells used to get it a lot. Not a lot of fun, I'll tell you.
I've also read that cadmium fumes can kill you. Not because cadmium is so toxic (it is), but because the oxide will cause a kind of chemical pneumonia. Check the MSDS on any silver-solder before using it. Many of them contain - or at least USED to contain - significant amounts of cadmium.
PeteH