.deb and .dmg can be easily extracted. The former is just an `ar` archive containing tarballs, which you can (and should) extract to read the install scripts. (.dmg specifics escape me, since I only dealt with them one time, years ago.)
Binary code isn't inscrutable. Some good tools for this are, among many, many more, IDA, Hopper, and radare2. How long this takes depends on what your goals are, how comprehensive you are, and the program complexity. I don't think I've yet spent years on one project, fortunately, but the months-long efforts, for undoing some once-prominent copyright protection systems, were pretty brutal. Smaller programs have taken me just several hours to appropriately examine.