As already noted, it was a very DIY culture so everyone pretty much played around some houserules based off some base set of rules they'd either made up or purchased from some other people doing this in their basement. Most scenarios would need specific rules, and it was expected that you'd put those in place as a group. As noted, this wasn't so much competitive as revisiting historical battles and later experiencing the scenarios. Of note, Chainmail specifically mentioned that it didn't have rules that covered all situations and that people using it would need to make rulings as a group to cover situations when necessary. RPG's fit right into the DIY ethos of mini's wargaming. It isn't unique.How common were house rules in the wargaming community at the time? I feel that for most types of games what's in the book is considered canon and probably mandatory, and RPGs are a notable exception.
This did run counter to chit-n-hex wargames that tend to follow written rules closer, but those also weren't nearly as DIY in approach in general.
FWIW, most common household games tend to houserule. Lots of people's weekly card games have various options or houserules that are used. Almost everyone houserules Monopoly, and Risk, and a plethora of other common games.