Re-do the winch wiring as something is more than likely connected wrong. Something is either grounding but sounds more like you tried to draw way too much current through that wire that melted.
Did you actually tap into any of the ATV wiring when installing the winch? If not, you may have just pinched something or done some other inadvertent damage to existing wiring when you installed the winch.The wire that melted was to the wheeler itself, nothing for the winch. I’m not sure it something got grounded out to which causes a flow of electricity back to the wire that the wheeler is hooked up to or what.