The bulk of technological innovation is about increasing efficiency, that is, doing more with less. A common example comes from the originator of a counterintuitive rule of thumb about how this affects the amount of demand for the good:
The issue has been re-examined by modern economists studying consumption rebound effects from improved energy efficiency. In addition to reducing the amount needed for a given use, improved efficiency also lowers the relative cost of using a resource, which increases the quantity demanded. This counteracts (to some extent) the reduction in use from improved efficiency. Additionally, improved efficiency increases real incomes and accelerates economic growth, further increasing the demand for resources. The Jevons effect occurs when the effect from increased demand predominates, and the improved efficiency results in a faster rate of resource utilization.[6]
If AI reduces the cost of producing text, it will ultimately increase the amount of demand for text. The long term net effect is going to be an increase in quantity and quality of text.
The fundamental nature of the AI is similar to a compression algorithm. It starts by finding the recurring sequences in the data, and collecting them into a list called a dictionary. The compressor then maps the symbols found in the recurring sequences to (large) hash values, and then creates a state transition map that specifies the symbols and their subsequent symbols, and then with this transition map, sorta like assembly instructions, symbols are copied out of the dictionary according to the sequencing codes and voila, a substantial amount of repeating, meaningless data is removed and our document is now a lot smaller.
The AI does the same thing, generates a dictionary, and then using those groovy big number things, 256 bit hashes, usually, to encode the sequence of the occurrance of the symbol in the document collection the model is being made for. Using some form of cryptographic filter, like Bloom filters or Galois Coded Sets, the hashes that represent the symbol are mashed up with the hashes of the prior symbol in the original document(s).
Then you can recursively decompose the paths into repeating paths, and repeating segments of paths, and all this relational information is printed into the symbol's document key.
The prompts you give the generators are parameters, and it generates a recognition matrix and dictionary out of the prompt, and this is then applied during the process of stepping through the dictionary that the parameters select from the text.
The reason why the outputs are "wrong" or "woke" or "nazi" is because the model stores everything, and stuff that humans can easily recognise but not easily explain, what is just below the surface is also found in the broader structure of the text, and has hidden or implicit meanings that may be contrary to what is generally believed.
After examining this subject, of "what exactly is a GPT?" I realised that it clearly must be a cryptographic state machine log that basically acts like a navigator from one symbol to the next.
I'm very interested in it now for two reasons:
-
it can be trained on code and generate code which is often valid and usually semantically correct.
-
it could be modified such that rather than simply creating a little map to freestyle riff off an input, it actually can be used with a given starting state value, and designate the first symbol to start with, and reconstitute the document that it was made from.
I'm going to get my hands on a reasonable GPT and start tinkering with it to do code generation and refactoring. Maybe build a few bucket list apps - programming language (modded from Go), text editor/terminal/file manager (inspired by Acme from Plan 9 and Oberon 2), operating system kernel - using the ideas from the Exokernel and Nokernel and other more recent projects. I may yet even already apply it to working with the Indra code base before long.
I dreamed of building a semantic map/recognition system like GPT back in 1999. I didn't know about cryptographic filters so my idea was missing the idea about how the connections between things was encoded.
I'd very much like to experiment with how to use GPT as a text compression system. First, create a model out of a metric buttload of randomly selected content. Then make a model that is a diff from this big model to the document's model. With the correct starting parameter and first symbol, the GPT should regenerate the exact text the model was created with, like a decompression of a text file.
Imagine if it was possible to compress the terabytes of text on Wikipedia down to a mere 50-100mb. Bookmarks are a symbol and a state hash. If you give it that parameter you effectively seek to that point in the document.
I'm a bit more excited about AI today than normal. To me it's always been a bit of a fraught subject and the Skynet doomsday of self replicating murder robots and creepy movies like Ex Machina and the expression "uncanny valley", and the Siri/Alexa/Cortana chatbots that are just spying on me so they can fool me better next time. I'm a touch typist and have an unusual custom mechanical keyboard and text processing is the bulk of my work as a programmer. GPT will make it a lot easier to do precise and correct bulk changes, scaffold stubs for common patterns and intelligently reformat, rename, sort, whatever I like.
Just need to find a GPT that I can train.