Well, it is a piece of HW, not SW so the question is a bit off I think

Or to be precise, the modem can be a separate chip or part of the SoC (built on the same chip as the main CPU)
The modem baseband part runs its own firmware, which is a RTOS variant (or rather, very dedicated O&M loop with some very simple OS operations...)
This firmware is very much propriertary SW owned and developed by the chip vendor and is not opened.
There does exist some limited (meaning, just 2G functionality) open-source implementation of the GERAN stack but even that only supports some very old modems not an longer on the market.