To say it right away: The picture is not fair, because Gandalf is a Maiar, not a DnD wizard. With this clarification out of the way, let’s go:

The relationship between wizards and weapons is complicated.

Modern sensibilities see nothing wrong with a wizard in arms. Shooting a gun is easy enough so that toddlers can shoot their moms in a supermarket. Therefore it is obviously common in modern games like Unknown Armies, Kult or Call and Trail of Cthulhu or in the various World of Darkness settings to have characters who can sling a spell or a 10mm at the same time.

In Fantasy gaming, that is different. It is standard to say that all a magic user can competently wield is either a stick or a dagger.

Why? Balance!

Game Balance

From the earliest days the questions was: How do we balance characters so they all can contribute and none are inferior. In modern Urban Fantasy games that is no problem because

a) everyone is special, either a mage, or a vampire, or a chosen one, or whatever, and
b) a spell is about as strong as a hand grenade, so even if you can’t hex, you can at least lob a frag into a wizard’s face.

Balance through modernity.

In GURPS, balance was accomplished with the point-build: Every character had 100 points, and whatever he was, he was the equal of other 100 point characters.

Balance through choice.

Dungeons & Dragons as a class-based system had to restrict the powers of one class where another was supposed to shine so that the bottom line would be a diverse team of specialists in their various fields.

One such specialist was the fighter, who excelled in slaying people with steel. So it was necessary that the magic user would have trouble in that regard, just as the fighter could not turn his foes into toads.

Hence the restriction of magic users to only use a dagger, and to refrain from wearing armor, and the cleric from using swords. Both restrictions sound silly when you think them through, and all attempts at building some logic around them (“Bishop Odo!” “No spilling blood!” “Too much time spent with books!”) sound hollow.
Because they are hollow.
Because the real reason always was, and forever will be, not truly anything logical, but a meta-reason: game balance.

(Just like the level caps for non-humans. No, they were not based in the hateful racism of a cabal of monstrous humanocentric gatekeepers who hate elves. They were put in to prevent a game where everyone always just played Elves. Darkvision and uncanny perception OR higher levels, not both.)

Subtle

Over the decades, editions have opened up some restrictions and invented others. The bottom line is still a quest for game balance, if better hidden. The latest push to divorce certain advantages and disadvantages from the fantasy races they are associated with. Who says I can’t be a tall and musical dwarf arcanist, a smart peaceful orc scribe, or a 7 feet tall human with hairy feet who lives in a hole in the ground? Nobody, if you silently shave off all that makes a so-called “race” special, and, bottom line, have just a bunch of randos who call themselves “elf” or “human” or “giant”, but are, in fact, all the same.

That is a bland but very simple solution, because it is subtle enough to go almost unnoticed, and most of the time it won’t matter anyway. Everyone (or almost everyone) is happy, and game balance is preserved.

Everyone is equal: the 5e way

In the same vein, modern 5e characters, akin to Urban Fantasy settings, almost all can do magic. Every class in D&D 5e has spells – some more than others – and to offset this, some have heavier armor proficiencies. The magic user has even more spells, but to offset this he has no proficiency with swords. He can use them, just not as well as others; just like others can cast spells, just not as well as the wizard. Except for the sorcerer and the warlock, but everyone has some slightly different shenanigans that come out the same at the bottom line.

These slight differences still chafe for many players, so they can be filed off with some magic rings etc, and in the end you have enough magic users with armor and fighters with spells so you have to read the top of the character sheet where the class is spelled out if you want to know what a character is actually supposed to be.

Subtle. Bland. Simple. Everyone glad, game balance is preserved.

Vanilla, though. You can now say “I play an Elf”, but really, you are just Joe Random with pointy ears and with some spells, while Erin who says “I play a Cleric” is just Joe Random without pointy ears. Or if he plays an Elven Cleric he has the pointy ears, just like you.

If everyone is special, no-one is special.

OSR

Of course, some people like to play their old games, or games like the old games, and in the old games an orc is still an orc, an elf an elf, and magic user a magic user. And suddenly they can’t hurl cantrips around and are stuck with a dagger or a stick when their spells of the day run out.

So, just as ever, the battle rages on: A wizard-player who is out of spells starts arguing and wants just one more spell, and why is Vancian Magic so silly anyway, can’t we just mana-pool like in computer games? And can’t I just use a bow? Harry Miller from school can shoot a bow, and he is such a nerd! If Harry the chess nerd can do archery, surely my character Millicent the Magnificent can also nock an arrow?

And thus, house rules. But house rules are rarely elegant. They are conceived in a moment, to fix a specific problem, and they can cause other problems down the line, tilt the game balance, and derail the system. Then you need another house rule to fix the damage, and suddenly you are like a computer programmer who implements an ever growing cascade of workarounds to keep the system running with Johnny the Fighter moaning how his character is now useless. Or you work hard enough to make a new system, playtests and all. The line between house rules and systems is just as fuzzy as between blogs and online media.

HD

Some choose to take a page out of Original DnD, where every weapon did the same damage, regardless of size, length, or form.

A fresher idea is that every character does character damage instead of weapon damage, namely, as much damage as the character’s HD-size is. A warrior does d8 damage, no matter if his hairy hands wield an axe, a sword, or a potted plant. A magic user does d4 damage, even with a two-handed sword. A fighter does d8 damage even with a beer keg.

Game balance: preserved. And again everyone can be special but it does not matter and actually secretly no-one is special.

Rules variants

The OSR is full of derivates of B/X. One of those is BFRPG. In its section “Dealing with Players”, it posits to allow magic users to swing a sword or shoot an arrow, just at a modifier of -5.
Easy peasy.
They even can wear armor, just cannot cast spells when doing so.
In other words, in BFRPG you can have your spell, and once you have spent it, you can, in theory, put on the armor you have in a pack on your mule, gird a sword, and be a slightly inept warrior who does not hit a lot of targets but supports his friends when the going gets rough.
Easy peasy.

Rough & wild

A rougher way to preserve game balance is to kick it in the nuts: the way of Lamentations of the Flame Princess. This system does not carefully balance out the various classes, instead, it drives them apart to reach the same outcome. It takes what makes one class special, and dials it up to 11, making the similarities, like wearing armor, less important. A Lamentations magic user is a Magic User writ large, and Lamentations fighter is a Fighter with a vengeance.

In the case of the Magic User that means he can cast horrendous spells that tear holes into the fabric of reality itself AND wear chainmail, and he can shoot a musket, and he can ride a horse and drink sharp schnapps and set farmsteads aflame like the best and worst of brigands.

Yet the power to shoot a musket is insignificant next to the power to drown the whole planet. And it remains the power of the fighter to become better and better and better at shooting that musket, while everyone else has other strengths.

Only one thing: when the Magic User casts his spell, he has to do a very particular dance of precise motions, wave his arms in a special manner, and chant — he must effectively stand still, and he can only be lightly encumbered at that point. And now he is an easy target.

Some you win, some you lose: You can, as magic user, shoot guns and stab with a sword. But casting spells is dangerous while the blades are out and you should probably don’t do it if you have no-one to protect you.

That makes it rough. Not bland. But it keeps game balance nevertheless.

It makes it a choice.

Choice

As always in Roleplaying Games, it comes down to choice. Do I go west or east? Do I pick up a dagger or an axe? Do keep my head down or do I stick it out?

Can a Magic User swing an axe? Sure, do it. The GM will figure out what happens and how competent or embarrassing it will look. Can a Cleric spill blood with a sword? Try a mace in real life and see if that is so much cleaner. The GM can decide how the gods react to a sword-wielding cleric. Maybe one god likes it while another gets angry. The important part is not “…but on page 15 it says…“, the important part is that it makes consistent sense in the game and does not overpower one class over another.

As long as none of the characters becomes a Mary Sue that dominates the whole story, being the best at everything, everyone can contribute to the success of the group.

So if the question comes up, talk it over and pick a route to go: HD damage, equal damage, or go with RAW old school DnD and limit the Wizard to a kris.

Come to a decision you all (at the table, not in the world) can accept.

Then play.