I doubt you can get what you want without a much more sophisticated circuit. Basically the devices you have contain a resistor and a small LED which ideally lights up fully at 5 VDC, but will still light up weakly down to about 1.5 to 2 VDC. But of course it only lights and conducts current when the voltage is "forward" for that diode, and is completely dark with near-zero current flow when the voltage is inverted, as any AC supply will do. The LED's each must be rated to withstand the Peak Inverse Voltage in the circuit, which will be about 175 V at the peak of a sine wave. So each LED actually pulses on and off 60 times per second,so fast you don't see the pulsing light. The resistor in series with the LED is sized so that, at the maximum current flowing through the LED / resistor serial pair at the moment of 175 V forward voltage, the peak current flow though the LED (whatever that is, known to the manufacturer) is such that the Voltage drop across the LED is 5 V, and across the resistor is about 170 V. At lower forward voltages in the sine wave the forward current through the pair is lower and produces less light, but still not NO light until the forward voltage across the LED drops down to less than 2 V. At that point the current is much lower, so the drop across the resistor is also lower. but still, if the ratio of voltages is about 5/175, then at 1.5 V across the LED, the total forward voltage drop across the pair is about 50 V, with VERY little current flow. So, any small induced currents flowing in that circuit of "dead" supply leg through the light unit to Neutral may be sufficient to produce a small amount of visible light. In a household power supply system there are enough connected devices and parallel wires carrying current to cause very small induced currents in adjacent wires, so achieving real zero voltage on both supply line legs when one is still operating fully is almost impossible.
Since you have these units installed downstream of the isolation switch / incoming breaker, I expect that if you shut that off so BOTH legs are "dead" then both LED's will go dark. However, if you wanted to ensure that one LED will go dark when the other leg is fully operating, you;d need a very different circuit for each LED that can really shut off ALL possible current flow at even very small forward voltages that originate in induced small current flows. I don't know a simple way to do that. You best path forward may be simply to accept that "Off" is signalled by LOW brightness, not NO brightness.