Neem oil affects insects in many different, ingenious and subtle ways

How neem oil messes with the insects' brains and bodies

Neem oil has many complex active ingredients. Rather than being simple poisons, those ingredients are similar to the hormones that insects produce. Insects take up the neem oil ingredients just like natural hormones.
Neem enters the system and blocks the real hormones from working properly. Insects "forget" to eat, to mate, or they stop laying eggs. Some forget that they can fly. If eggs are produced they don't hatch, or the larvae don't moult.
Obviously insects that are too confused to eat or breed will not survive. The population eventually plummets, and they disappear. The cycle is broken.
How precisely it works is difficult for scientists to find out. There are too many different active substances in neem oil, and every insect species reacts differently to neem insecticide.
Neem oil does not hurt beneficial insects. Only chewing and sucking insects are affected. It is certainly fascinating.
Like real hormones, neem oil insecticide works at very low concentrations, in the parts per million range. A little neem oil goes a long way.
But this is not something that happens over night. People spray neem oil as insecticide, and expect everything to die instantly, because that's what they are used to from chemical poisons. When that does not happen they conclude neem insecticide does not work.
It does work! Give it time to work. It's a much smarter way to deal with insect pests than to just kill everything.

How neem oil deters chewing and sucking insects

There is a nice story that demonstrates how grasshoppers react to neem oil insecticide. It goes something like this:
Someone did an experiment. It involved two jars, two leaves, and two grasshoppers. One leaf was sprayed with a chemical insecticide, and one with neem oil. The two grasshoppers were put in the two jars, with one leaf each.
The first grasshopper ate the leaf and died almost instantly. The grasshopper with the neem oil covered leaf did not touch the leaf and lived. At least for a few days. Eventually it starved to death.
What would you prefer? A poisonous half eaten lettuce, or an organic, untouched lettuce? It's a no brainer, isn't it?
Neem stops insects from eating the plants.
Part of this action is due to to the hormone like action of neem oil that I explained above. Insects "forget" to eat after they've been in contact with even traces of neem oil.
But it is also the presence, the mere hint of a smell of neem oil, that seems to be enough to keep leaf eating insects away. Neem oil can be very powerful as an anti-feedant and insect repellent.
This anti-feedant property is one of the most often advertised and lauded properties of neem oil insecticide. However, the hormonal effects I described above are even stronger.
Neem oil as an insect deterrent works well against grasshoppers and leafhoppers, but all other insect pests are controlled mostly through the hormone action.
The subtlety of the hormonal effects, and the fact that they may take days or weeks to manifest, makes people overlook them. Ill informed gardeners seek instant gratification, i.e. lots of dead insects immediately, rather than a balanced environment in the long run.
It's a shame, because the hormonal effect is where the real power of neem oil lies. It's the key to neem oil being an effective insecticide and good for the environment at the same time. It's also important to understand this effect to use neem oil insecticide correctly.

Neem oil works from inside the plant

Many insecticides break down quickly. They wash away with rain, or when irrigating, or the sunlight destroys them. You either have to spray all the time, or you have to spray something that's so stable that it stays around forever. That means the chemical builds up everywhere and eventually poisons everything, including you.
Neem oil breaks down very quickly, too. It is especially susceptible to UV light. But neem oil is also a systemic insecticide. That means you can pour it on the soil (not pure neem oil of course, you use a dilution or extract) and the plants absorb it. They take it up into their tissue, and it works from the inside. A leaf hopper may take a couple of bites, but that's it.
However, this does not work for all insect species. The neem ingredients accumulate in the tissues deeper inside the plant. The phloem, the outermost layer, contains hardly any. A tiny aphid feeds from the phloem, it can not penetrate deep enough to get a dose of neem. But any leaf hoppers, grass hoppers or similar chomping insects will be incapacitated quickly.
People eat neem leaves to cleanse the blood, stimulate the liver, and boost the immune system. So we certainly don't need to worry about a bit of neem inside our lettuce leaves. To me this is a much more attractive option than having poisonous foulicides build up in my garden.

Neem oil suffocates insects

Many gardeners use white oil (plain mineral oil) or even olive oil to combat soft bodied insects like aphids, thrips or whitefly. The oil coats the bugs and they suffocate. Neem oil insecticide does that as well. But it's more like a little bonus on top of everything else it does.
It can be a hazard, though. Of course there is no difference between suffocating good or bad bugs. Oil suffocates anything. So this aspect can harm beneficial insects!

Neem oil and beneficial insects

Neem is non toxic for beneficial insects. The main reason is that insects need to ingest the neem oil to be affected, and beneficial insects don't eat your plants. But you can still kill beneficial insects if you smother them with neem oil, so please be careful.
Beneficial insects are most active during the day. The best time to spray neem insecticide is very early in the morning, so the spray can dry before the good insects become active. Also a good time is the late afternoon or evening. Once the spray has dried it does not harm your bees, ladybugs, lacewings, predatory mites and wasps etc.