I’m doing it with LLMs or I’m not doing it at all.
📜 google scholar | ✉️ [email protected] |
My primary interest is in generative models, how they work and how we can get them to generate text and other media that communicate with humans is useful and novel ways. Lately, I’ve been thinking about how language models fit the definition of complex systems, systems in which we understand the low-level components (neurons) but can’t explain or even fully describe the high-level behaviors (e.g., in-context learning) as they emerge with more data and parameters. In the spirit of complex systems, I want to create a taxonomy of model behavior, analogous to the periodic table of elements in Chemistry, which hardly explains complex chemical processes in its own right, but gives a description of elementary components and their interactions that can be used to build-up more complex hypotheses. Currently, we rely on benchmark performance or vague intuitive descriptions to pin-point specific phenomena, which means most hypotheses rely on imprecise vocabulary that won’t stand the test of time.
In the short-term, I’m interested in thinking how we can map out what models can and can’t do, which I believe will naturally relate to long-form generation. It is incredibly difficult to evaluate long-form generation rigorously, and it is hard to show long-form generations in power point slides, which has made coordinating the issues in long-form generation difficult for the academic community. In the medium-term, I think we need to tackle the non-objective aspects of language, as almost all communication is open to interpretation, relying instead on the pragmatic attempt at cooperation to bridge this gap. Focusing on easy-to-evaluate aspects of language doesn’t do it justice. Perhaps looking at indirect evaluation, where we evaluate what generated language can be used for, rather than whether it is “correct” can help move researchers in that direction. My long-term goal is to create discursive machines.