There were a couple of types of Start-o-matic, I think the main difference was DC or AC supply. The typical AC "lighting plant" (the most commonly seen Start-o-Matic for the 5/1 or 6/1 engine) was a 2.5kw alternator attached to the engine (which had "heavy" style flywheels, rather than the more common spoked design). A solenoid attached to the back of the engine operated the decompression lifter and the rack stop, and was normally held (by springs) in the "off" position. Via a control board, 24v batteries were connected to the lighting ring and the engine. When a load was detected (i.e. a light switch turned on), the control board switched 24v onto the starter circuit, which turned the alternator into a starter motor. Simultaneously, the decompression solenoid operated which allowed the fuel rack to open & the exhaust valve to operate as normal. Assuming all was well, the engine would then fire up, and start to "overspin" the alternator, which then switched to generating mode. This generated both 110v or 240v AC single phase, and sufficient DC voltage to recharge the batteries.
The system remained running until the last load on the circuit went off (i.e. the last light turned out), at which point the solenoid was released, shutting off the fuel and operating the decompression lifter, shutting the engine down. It then went back to waiting for a load to appear on the circuit.
Once the start cycle was started by a load, it would continue until the engine started to generate (even if the load was immediately removed); it would also only motor for a certain period of time, e.g. if the engine had no fuel, so it didn't flatten the batteries or burn out the starter circuit. This was all achieved using bi-metallic strips and other black magic... no computers back then!
The DC SOMs are slightly different in that the motor/dynamo are directly coupled to the engine, the engine itself is normally a CD or possibly a 9/1 (JP1) model rather than the CS we know & love, and they often had an additional "solenoid" (more of a motor really) attached to the compression changeover valve, so once started they could automatically switch over to low compression.