Exploring AI

The 9th International Workshop on Computational Linguistics for Uralic Languages, or more familiarly IWCLUL, was held at Metropolia. The event brought a large group of international researchers to the Arabia campus, where they presented their language technology research related to Uralic languages, which are languages related to Finnish. The challenge of being endangered Of the Uralic languages, only Finnish, Estonian and Hungarian are majority languages with official state support. The other Uralic languages are more or less endangered. The number of speakers varies from Meadow Mari with 360,000 speakers and Erzya with 300,000 speakers, to Skolt Sámi with 300 speakers and Ume Sámi with just 5 speakers. Some languages no longer have native speakers at all. However, hope is not lost even for these languages, as Valts Ernštreits, the director of the Livonian Institute, often remarks: "Every time the last speaker of Livonian is believed to have died, a new last speaker emerges from some cottage." Jack Rueter reminded us of the importance of popular culture also in the context of endangered languages. Modern language technology requires a lot of data, which makes AI development for smaller languages more challenging. Often, there is little to no data available, and it tends to have significant variation. Spelling rules are often not as clearly defined or deeply ingrained in the speakers’ habits as they are for major languages. Large language models sparked discussion. Large language models like ChatGPT do not currently support any small Uralic language. However, researchers have devised methods to elicit responses from these models by carefully crafting prompts. In addition to my own presentation, both Flammie Pirinen and Niko Partanen reported the results of their research. IWCLUL was organized through volunteer efforts. In the picture, Lev Kharlashkin is inviting the next speaker to the stage. The problem with large language models, even for Finnish, Estonian and Hungarian, is that they split words into smaller units, tokens, based on the English language. Iaroslav Chelombitko and Aleksei Dorkin had proposed solutions for this issue. Metropolia's values on display The work done at Metropolia in the fields of sustainable development and artificial intelligence was also highlighted at the event. Melany Macías presented our research, in which AI learns to predict sustainable development goals in Finnish based on English-language data. Melany Macías presented the accuracy of predicting sustainable development goals.

An ordinary user interacts with large language models, like ChatGPT, by writing prompts through the user interface. In addition to this, large language models offer another functionality for technically skilled users – the creation of embeddings based on text. But what exactly are these embeddings, and what are they used for? Meaning of text in vectors When a large language model is given some text to embed, it produces a vector as a result. A vector is a list of numbers that may not be immediately interpretable to the human eye, but it enables the exploration of the text's meaning through mathematical methods. These vectors produced by the language model are called embeddings. UralicNLP Python library provides tools for embedding text using different language models. Here is an example of how text can be embedded with OpenAI's model using UralicNLP. from uralicNLP.llm import get_llmllm = get_llm("chatgpt", "VAIHDA TÄHÄN API-AVAIMESI", model="text-embedding-3-small")llm.embed("Teksti, jonka haluat upottaa")>>[-0.1803697, 1.1973963, 0.5283669, 1.5049516, -0.27077377...] As seen in the example, the result of an embedding is a list of numbers. These numbers represent the meaning of the text and can be used to compare the similarity of texts through mathematical methods. What are the benefits of embeddings? With embeddings, large volumes of text can be stored in a vector database for quick retrieval. This means database searches are based on meaning rather than character strings. The most common use case for such vector databases currently is the RAG model. RAG stands for Retrieval-Augmented Generation, which refers to a process where a large language model is provided with not just the user prompt but also source material to help generate a response. Retrieving the source material involves using embeddings to find documents relevant to the user’s input from a vector database. For example, Metropolia’s own Mikro-Mikko operates based on this principle. Embeddings can also be used to automatically group text documents into clusters of similar texts. This can be done with UralicNLP as follows. from uralicNLP.llm import get_llmfrom uralicNLP import semanticsllm = get_llm("chatgpt", "VAIHDA TÄHÄN API-AVAIMESI", model="text-embedding-3-small")texts = [“koirat on hauskoja”, “autot ajaa nopeasti”, “kissat leikkii keskenään”, “rekat ajaa kaupungista toiseen”]semantics.cluster(texts, llm)>>[[“koirat on hauskoja”, “kissat leikkii keskenään”], [“autot ajaa nopeasti”, “rekat ajaa kaupungista toiseen”]] The result is that texts are grouped into clusters of similar texts using embeddings and calculating their similarity. Does the model matter when embedding? Embeddings can be generated using both commercial large language models and open-source language models. When choosing a model, it’s important to remember that embeddings are not compatible across models. For example, you cannot create some embeddings with OpenAI's GPT-4 and others with an open-source LLaMA model and expect them to work together. Each model has learned its own representation of meaning from its training data, so the numerical content of the embeddings varies between models. When choosing a model, it's important to consider the cost of the model, the languages it supports, and its context window. Larger models can accommodate a large amount of text within the context window, allowing for a single embedding of an entire text. Smaller models require the text to be split into segments. This technical limitation can be significant depending on how the embeddings are intended to be used. Not all models support all languages. If a language model produces poor Finnish responses to prompts, it likely does not understand Finnish very well. Consequently, embeddings generated for Finnish text may not capture the meaning accurately enough.

Generative AI models like ChatGPT and Midjourney are capable of producing creative text and images faster than humans. So, why do we still need poets if ChatGPT can generate as many poems as one could ever request? And what about illustrators? Midjourney can produce stunning images even in the hands of an amateur prompt writer. Is it really the case that human creativity can be replaced by machines? What is Creativity? The relationship between human and computer creativity has been studied for a long time. Even long before the era of ChatGPT, researchers were developing AI models that could generate stories, music, and images. This field of study is called computational creativity. The interplay between human and machine creativity has long captivated researchers in this domain. Boden is one of the theorists in computational creativity. According to her, creativity can be divided into two categories: exploratory creativity and transformational creativity. This distinction is crucial when we aim to understand the limits of computational creativity. Artificial Intelligence Exploring Creative Possibilities Currently, all major AI models are capable of exploratory creativity. This means that AI models operate within a defined creative search space, exploring the possibilities within it. For example, if we consider an image made up of 512x512 pixels, where each pixel must represent one of a predetermined set of colors from a color palette, it is clear that the number of possible images is finite. An AI model capable of exploratory creativity can thus generate images only within these rules — these rules define and limit the creative search space. AI exploring suitable pandas from a constrained search space. The images were created by Khalid Alnajjar. The situation is even more constrained in reality. No generative AI model can produce all possible images that could exist in a 512x512 size. AI's functionality is also limited by the data it has been trained on. If you ask an AI for an image of a panda, it will not generate all possible panda images, but only those that align with the understanding of a panda it has learned from the data. In creativity, boundaries are made to be broken Exploratory creativity is clearly confined to a specific search space. But what kind of creativity is shackled by limitations? After all, boundaries are made to be broken! Boundary-breaking creativity is transformational creativity, as it transforms the boundaries of the search space itself. If I am given an A4 sheet of paper and tasked with drawing a house, my creativity is confined to creating a two-dimensional image. As a human, I can pick up the paper and fold it into a house — this is transformational creativity, as I have altered the search space. The third dimension enables the creation of entirely different houses from the A4 paper compared to the original two-dimensional sheet. AI is not yet capable of breaking boundaries. None of the popular AI tools can transform their search space and decide, for example, that instead of creating an image with pixels, they will paint it with a brush. Nor can they spontaneously try a new style that hasn't been taught to them in the training data. The role of humans remains essential AI can produce impressive and creative outputs, but it remains bound by its patterns. The creativity of AI stays within the constraints of its task and training data, and it cannot generate anything beyond the frameworks it has been given. This does not mean that AI is not creative or that it is inherently poor at creative tasks. It simply means that its creativity has not yet reached the level of human creativity. Therefore, there will still be a need for human creativity in the future, at least until we develop a categorically different kind of AI.