I can offer how my system handles it, and maybe it can spark some ideas for you
In my system, you have three types of skills.
A Skill is anything, like you mentioned above: Unarmed Combat, Sword Combat, etc. (10 Ranks, which provide a bonus to rolls).
A Knowledge Skill costs 2x as much as a regular Skill, and provides a bonus to specific actions. A simple example might be "Unarmed Defense" granting up to +5 when defending while Unarmed.
A Professional Skill is a group of 3 to 6 regular Skills that must be defined when the Professional Skill is taken. Additionally a Professional skill grants one Knowledge Skill.
When increasing a Professional Skill Rank (3x cost of a regular Skill), all of it's subsidiary skills increase to that rank. You can rank up the subsidiary skills individually as well.
If you have at least 3 regular Skills and a Knowledge Skill which are untied to any Professional Skills, you can combine them into a Professional Skill which has a rank equal to the lowest of the skills combined together (you also get refunded excess skill points to be spent as you like).
When performing an action which you have no skill for, if you have a skill which is similar, you can use half of the ranks of the similar skill. (1/2 Pistol Skill when using a Rifle unskilled). This also applies to offhand actions. To get better offhand rolls you have to skill specifically for the off hand. (1/2 Sword skill when using Sword in off hand).
Mmm I also forgot to mention that while you have to define at least 3 Skills for a Professional Skill, those skills are meant to paint a picture of what it is the Professional Skill encompasses, and you are not entirely limited to those skills. So if your professional Skill is "Rogue" you can use it just like a regular skill to do anything that is deemed Rogue-like -- Or you might have a Professional Skill of "Hunter" and you are versed in everything in relation to Hunting.
My recommendation would be to go with combat styles, with investment in a particular style potentially resulting in access to new abilities, or bonuses to specific weapons or tactical manuevers. This could be something as generic and simple as "grappler" or "swordsman" or "marksman," but has the potential to become really flavorful if you are willing to pin your system down to a defined setting.
+1 on this. It can really help tie combat back into the reality of the world so your Viking types might be limited to Berserker Dual-axe or Spear+shield as they grow up, rather than some other styles, or maybe they get bonuses to learn those because they are dominant in that culture. They could learn another style but perhaps not quite so well. It can expand to provide other situational bonuses so Farmer style might learn how to wield a pitchfork or torch with bonuses when in a mob, and penalties when on their own. Hunter might learn to be very accurate with ranged combat in a 1:1 but maybe less so when multiple target are in the vicinity?
So I'm trying to think of ways to differentiate combat skills besides weapon categories, and I see why weapon categories keep defaulting to popularity. My best thought is to differentiate skills by function - an Attack Skill, a Parry skill, a Grappling skill, and so on. My ideal system is one wherein allocating your skill points genuinely feels like building your character's fighting style, and can organically account for archetypes like "big bruiser who's slow and hits hard" "little quick guy" and so on.
I find it helpful to determine what the role of combat is in your system, and how you perceive violence taking place in the game. If you were to imagine a fight taking place in your game, would it look like this:
or more like this:
In one, violence is a way of showing off how awesome the characters are. They show off to one another, they reference famous fencing masters that they've studied, they willfully choose to fight with their non-dominant hand to make the resulting fight more challenging, and the fight ends with one guy getting (non-lethally) clubbed in the head out of respect for his artistry with a sword. Characters might have styles of fighting akin to what's been discussed above, each with weaknesses and strengths that show off what's awesome about them.
In the other, violence is a tool for settling scores. Fighting isn't a time for witty banter, it's a time for staring your opponent down until your hands shake. The more experienced duelist is playing with him, but it's not a "I'm testing your skill" sort of deal. It looks, and feels like a cat playing with it's food before it goes in for the kill. Characters might have distinct ratings in Attack, Riposte, Evade, and Grapple and it's a game of move-and counter move to determine who comes out on top, with dice based partially on a character's combat skill and partially on their willpower: something that gets eroded every time you suffer a wound or a setback. When the lesser duelist gets his hand cut as he fails to grab his foes smallsword, he pulls back flinching and screaming, the first real noise of the bout.
I could give other examples: say a Rambo movie vs. the first 10 minutes of Saving Private Ryan but you get the idea. Nailing down the answer to that question will help you decide what role combat plays in your game, how important that role is, and thus how you should structure combat skills.
Related to that is answering the question of "should characters have combat skills". In some games, combat skills might be a rarity (the Warren, a game about intelligent rabbits ala Watership down) is a good example of this: only one special ability called Tooth and Claw gives a Rabbit the ability to make a roll to hurt another animal: otherwise it's up to the GM (and the fiction) to determine how it turns out. In other games, it might not make sense to have combat skills for the opposite reason: if characters are assumed to be combat-capable and dangerous then it might not make sense to rank their combat skills numerically. It's assumed people are dangerous: what would define a character's roll in combat in such a system is how they hurt people, the benefits and drawbacks of that approach, and why they hurt people, rather than "can I hurt people?"
We did just this thought experiment as we made Les combattantes (to find something more to the later but could have elements of the further). I think you should aim at what the feel of the game is about. What do you spend time doing as you fight. Count wounds, score, or choose what to do next. Is it more a skill of the character or of the player. Is it focused on the gameplay or the storytellng parts.
The thing that struck me is how much you want your characters to be able to specialise / train into a particular equipment set vs. the 'I've looted an item and have no idea how to use it' scene that sometimes occurs.
And does it matter if you're wrapping a grenade launcher, crossbow and throwing knife into one skill (if it works in the context of your game then all good)?
I'm building something at the moment and have opted for the skill-tree approach with the base skills being quite generic ... e.g. firearm, blade, bow ... with additional sub-skills that allow specialisms like Crossbow.
My thought was that level 1 of the base skill was the portable 'seen one gun seen 'em all' / I read the manual level of knowledge, and in combat, all the skills in an applicable branch come to bare: firearms 3, pistol 1 gives the +4 skills bonus for the automatic pistol that the character is holding.
This has worked out OK in basic play-testing, but as others say, it all depends on how important you want this element to be and how prescribed vs. freeform.
My game isn't combat heavy, but I share your view. I have three weapons abilities, tied to attributes:
Ranged - DEX
Melee - AGI
Unarmed - STR.
I don't currently have specialisms, but if I did, I'd have a point buy system whereby you'd have 0-3 skill points spread order 3-4 types of weapons in each category. So for Ranged weapons you'd have a DEX of 4 and a +2 in crossbow, +0 in Lance and +1 in polearm say.
My ideal system is one wherein allocating your skill points genuinely feels like building your character's fighting style, and can organically account for archetypes like "big bruiser who's slow and hits hard" "little quick guy" and so on.
If that is your goal, skill pattern might be a particularly hard way to do it. Skills provide little design space. You just get better numbers. And they provide very few imaginary reference points. You can only go by the skill names, whether your number is higher than other people, and maybe compare it to a reference chart. You might have it easier by having characters accumulate gifts, which can each have a unique effect and description.