I don't claim to understand fully the various steps in this train of thought, and I doubt that there's anybody who understands all of them fully, but here goes…
Science
Crude oil is a mixture of various different types of hydrocarbon molecules. The smallest/lightest are CH4 (methane gas), there is no precise chemical formula for petrol, which "constitutes the largest fraction of product obtained per barrel of crude oil. The hydrocarbons in gasoline have a chain length of between 4 and 12 carbons." Next is diesel, which "consists of hydrocarbons of a chain length between eight and 21 carbon atoms", and so on, all the way down to the heavy stuff used for heating oil
Heating oil "is one of the “left-over” products of crude refining. It is often less pure than other refined products, containing a broader range of hydrocarbons. Because of its contaminants, fuel oil has a high flash point and is more prone to autoignition. It also produces more pollutants when burned."
Technology
There are different methods of splitting up crude oil into its constituent parts, known as refining, which in turn means a combination of distilling, fractionating and cracking, different kinds of crude oil have different mixes of the various kinds of hydrocarbon atom and different methods of refining produce different results etc.
But by and large, for every three or four litres of petrol, you get one litre of diesel.
The environment
As a rule of thumb, the lighter the original molecules, the cleaner the stuff burns, so methane burns away to CO2 (carbon dioxide) and H20 (water) which are more or less harmless and the heavier stuff like heating oil churns out the most soot. Diesel molecules are heavier than petrol molecules, so generate more soot or 'particulates' in the modern jargon, which is unhealthy to breathe in as is widely documented.
The Kuznets Curve
Initially, new industrial processes tend to be very polluting. The polluters don't care and governments in developing countries are loath to tax pollution because economic progress always comes first. With the benefit of hindsight, the UK was a developing country for these purposes until fifty years ago; having got a fair bit of industrialisation behind it, PR China is now at the cusp where the population are starting to care more about air quality than just economic growth (the Kuznets Curve).
Taxes on pollution
We would therefore expect the tax on diesel to be higher than on petrol. As a result of a complete policy foul-up, diesel duty used to be lower than fuel duty in the UK, but once they realised that they had messed up the rates were aligned (it's a tax on road use rather than a tax on pollution). Interestingly, in Germany, apparently because of the haulage and farming lobby, diesel duty is lower and diesel is about twenty percent cheaper than petrol.
Capitalism and waste
Initially, factories and oil extractors don't care about recycling or minimising waste. As the economy grows, raw materials become more scarce and labour and power become more expensive. As technology develops, people find ways of reducing waste or putting by-products to some sort of productive use.
A few examples: the average miles per gallon of cars on the road has doubled over the past fifty years and the life expectancy of cars has doubled too. Bauxite is plentiful, but it takes huge amounts of electricity to turn it into aluminium, so recycling rates for aluminium are approaching 100% in developed countries (melting down cans for re-use uses one-tenth as much electricity as refining bauxite). Even two-thirds of steel is recycled. When trees were plentiful, sawdust was just discarded or burned; nowadays, a lot of it is used to make chipboard or mdf, which is a higher value use than just burning it for heat, and so on.
Efficient use of by-products
So what would happen if everybody got a 'conscience' and deciding to switch from diesel to petrol, assuming they don't want to run their cars on vegetable oil (there isn't enough to that to go round anyway)?
That would push up the price of petrol and mean that a larger amount of crude oil has to be refined, but they would still be left over with one part diesel for every three or four parts petrol.
Having gone to all that bother extracting and refining, it makes no sense to throw the diesel away. So the price of diesel would fall relative to petrol and some people would then ignore their 'conscience' and stay with diesel anyway, thus restoring the three- or four-to-one ratio between petrol and diesel use.
The economy is dynamic
There are no wild swings in the relative price and quantity of diesel vs petrol, because it is a dynamic process where the two opposing effects constantly cancel each other out.
Taxes on pollution (2)
To a large extent, it is futile taxing diesel more heavily than petrol even though the usual suspects are crying out for it. The tax is borne by the end consumer and assuming the end consumer is prepared to pay roughly the same for either, if petrol duty is cut and diesel duty increased, all that means is that the pre-tax price of petrol will increase relative to the pre-tax price of diesel by an equal and opposite amount to the tax differential.
The German example mentioned above runs slightly counter to this. The article explains that the pump price saving is largely cancelled out by the fact that diesel cars are more expensive to buy and the annual road tax is higher, maybe chuck in the fact that most people prefer petrol engines.
