I won't get into why SegWit was originally implemented since that's been covered over and over by many different people, so I'll accept your opinion here that it was created for the purpose of scaling. I see two primary flaws in your reasoning:

The first flaw is assuming that Bitcoin is solely meant for "financial transactions" and that Ordinals are "non-financial transactions". If we can agree that Bitcoin is an economic settlement network, which I believe is an accurate description of what it ultimately is (without getting into censorship resistance, decentralization, etc.), then all activity on the network becomes "financial" since each transaction is effectively paying for settlement in the form of confirmations. Since Ordinals are paying for and receiving settlement, they are valid economic activity on the network and therefore "financial".

The second flaw is stating that Ordinals "are currently spamming layer 1 to such a degree that it's less usable for financial transactions". Since we've already described how Ordinals are valid "financial transactions" and that they are paying an acceptable fee for settlement, they can't reasonably be considered to be "spamming". Ordinal users currently have a stronger preference for confirmations (settlement) than other users. Other users, whether it be for standard P2P transactions, coinjoining, entering or leaving a second layer, etc., still have the ability to express their preference for confirmations by setting a higher fee on their own transactions. These other transactions may also be using SegWit for scaling purposes and receiving confirmations based on their fee.

Since non-Ordinal SegWit transactions have the same opportunity for settlement within Bitcoin, I believe your conclusion is invalid. Bitcoin will be used by people you don't like, for purposes you don't agree with. Its rules are in place to enforce an equal playing field for all users, regardless of their intentions, so long as they follow the rules. Ordinal transactions are following the rules of the network, otherwise their transactions wouldn't be propagated across the network to be included in blocks in the first place, nor would blocks that did include them be accepted by the network.