The only way to permanently assign a symbol to a color is to create your own palette. You can have it just "loadable" or you can assign it to the default.
If you don't use all the colors in the palette for a given pattern, you can use the palette tool to remove all colors that are not being used. The colors included will still have the same symbols as those colors use on other patterns.
Symbols used to be defined just from the start of the symbol list but in recent versions have taken a different approach to try to keep from having too many similar symbols too close as the colors are often assigned by color groupings.