Yellowish stains are typically from sweat and body grime (have tried taking a show doc? :biggrin
However, orange~ish discolorations are chemical reactions. I've seen it many times. Here are the few cases were I've seen it.
1) Do you swim or have a pool? Do you put your hat on after you swim? If so, your hair will still be holding chlorine and other pool chemicals. They will slowly turn your hat orange. Especially around the seams.
2) Do you use any face products? Chemicals in certain face products will also discolor your hat. Lotions, facial treatments (especially benzoil peroxide!) and other creams will turn your hat orange, especially around the forehead area and anywhere you touch your hat when you put it on (because the residue are still on your fingers).
3) Same as above, but with certain hair care products.
Basically do a search and analysis of any time any chemical touches your hat. The main culprits would be any peroxide, chlorine and the like keeping in mind that when dry, they may not affect your hat, but when wet or sweating will eventually discolor. Also keep in mind that it usually takes time sitting on the hat to eventually change the color. So if you rinse them off right away and wash them, it might minimize the discoloration when you do get chemicals on them.