In the Mars Effect mod I noticed it has custom research buttons, how do you add/make these?
Take a look at the current research buttons, they're all titled RESEARCHSUBJECT__. All you really have to do is make your own .entity file (copy/modify an existing is the easiest, obviously

), give it the position on the tree you want (also defined in the entity file), and then add the research to the player(tech/phase/psi).entity file where it lists all the topics that race can research.
If you did everything properly, it's as easy as that

The only part I need help with is the random chance. Is it possible and if so what coding line is needed?
It is possible, but it's tricky and I doubt anyone can explain it clearly

The basic premise is the ability file calls a buff file. That buff file contains a periodic action that "onChance" calls a second buff file, where your actual buff effects are defined.
For an example of this, look at AbilityFighterBlink.entity and its associated buff files

This is the ability for the Vasari carrier capital that has a chance to make its fighters "blink" towards their target.