Some basic RPG design (Late night thoughts)
Building (a traditional) RPG in short:
1) Pick genre/theme - these are the most important for
your rules set. "Realistic" is one option, so is "high fantasy",
"mystery" and "horror", among others. What kind of game people are going
to play will be affected by what they expect.
2) Basic randomizer. Dice: D20, d10, d6, pick any basic die or dice. Decide if
you want dicepool (better skill - more dice) or set amount of dice (1,
2, etc). Adding or roll-under/roll-over mechanic. Other randomizers:
Cards work (1/52 chance of pulling any single card, 1/4 of pulling a
card of a particular typ, i e aces), so do roulette wheels. The kind of
randomizer isn't important. The job of the randomizer is, ultimately, to
resolve conflicts between GM/Players on what happens. Make sure your
system is somewhat consistent and predictable. Put in conflict
resolution here (for armwrestling etc)
Skills (if you have those)
- make your own or steal a different games. Make skills "everyone
needs" either something everyone gets some stat in, or divide them up.
Try to catch skills which are too broad (i e give too much use for what
they cost) or too narrow (noone ever grabs "librarian"). Free skills is a
possibility, so is taxing the skills everyone gets more heavily -
alternately removing them.
3) Add a system/rules as necessary for your genre. If you're making a game where trading is the most important part, some kind of economy structure could be good. Maybe some trading minigame. For horror - actually, it might be a good idea to skip the traditional horror mechanic (i e fear/insanity). You want players to act/react on horror, not force the PCs to act a certain way.
Whatever the case, make sure your mechanics never work contrary to what you want to do in the game. See Trail of Cthulhu - not finding the clues should not be an issue in a detective game, piecing them together is the important part. One rule of thumb I've heard is "never add rules for stuff that is fun in and of itself". Which might be true, and why social rules usually are just there for those people who want to roll rather than roleplay getting past the guard.
4) Do you really need a combat system? As in, is combat a really important part of your game? If so, make combat system. For pulpy adventure, rules light works better than rules heavy, while a tactical faux-WW2 RPG might work better with more detailed stuff. If you don't need a detailed combat system but need a combat system, add in stuff like health, wounds, armour etc, remove as necessary. If you don't need one, just go with basic conflict resolution.
5) Revise, add and remove as necessary. Maybe you don't really need a randomizer? Maybe your game doesn't have a traditional-style GM. Just go with whatever you feel works.
Obviously, much more can be said on the subject. But it's 1:20 am here.