c't Working with AI (eBook)

Picture: Moritz Reichartz

Live on the bleeding edge

Pina Merkert

It‘s great that you can now play around with gigantic language models on your own computer. However, the models are all still very new and the software to run them is subject to constant change. The GGML format has already gone through three versions this year and it can easily happen that the data format of a downloaded model does not match the latest version of llama.cpp after all.

This software is opening up to other architectures (especially in order to support LLaMA-1 and -2 at the same time) by switching to the GGML variant GGUF. For the Falcon models not based on LLaMA, however, there has long been a fork called ggllm.cpp, for which it is not clear whether GGUF will make it obsolete.

These are just examples of the fast-moving nature of the entire software infrastructure. The fact that libraries work with new CUDA versions is to a certain extent a matter of luck. New Torch versions can have an impact on libraries that do not use Torch at all. Wrappers and helpers are often written for a specific version and then no longer maintained because developers have switched to a different base model.

At the moment, you can neither rely on stable APIs nor on long-term support. Libraries and frontends usually only work together if the developers have worked with precisely this combination. However, information on what works together is difficult to find and almost never documented.

During our research, we wasted days trying to compile Torch for CUDA 12.2, compile llama.cpp with the Visual Studio compiler and use the Python binding to llama.cpp with its low-level API for a browser-based chat. Most of the time something worked, but not everything, and we had to stop the research because we didn't want to force anyone to patch code with their own patches that wasn't yet ready for a reasonably stress-free use.

Feel free to try out the models mentioned here. It's fun to experiment and you'll get a feel for where the language model AI community is heading. However, we do not recommend investing too much time in individual experiments. For 99 percent of users, it is worth waiting until the software has matured and is no longer somewhere between a technical preview and an alpha version.

No PyTorch, no training

Apart from Google, pretty much all research groups use the PyTorch framework to structure and train their models. LLaMA and all post-trained variants such as Alpaca and Guanaco are Torch models. Unfortunately, the installation of Torch is not trivial because you have to install different libraries in advance depending on the hardware. For example, it is particularly difficult to achieve full hardware acceleration with PyTorch on Apple's M1 and M2 chips.

In addition, neural networks can only be trained if the parameters are available in a numerical format that can also represent very small changes to the values. This is necessary because training proceeds in tiny, but many steps. Anyone who has trained networks themselves knows that the training quickly becomes unstable if the learning rate is set too high, which ultimately results in steps that are too large.

If you estimate how much memory 7, 13, 40 or 75 billion parameters occupy if each number is 16 bits long (training is now usually done in bfloat16 format instead of float32 ), you will find that the graphics memory and often even the entire main memory of the PC overflow in no time at all. In practice, this means that most models cannot be trained or fine-tuned on the PC (Parameter Efficient Fine Tuning – PEFT is possible, but the trick leaves the original parameters untouched).

The sum of these difficulties is also the reason why we had previously only run the ‘small’ Falcon model with 7 billion parameters in Google Colab, i.e. in the cloud (see following article).

For inferencing, i.e. using an already trained model, the software does not have to be able to display small steps. If you multiply the neuron activations by numbers significantly greater than 1, you can even continue calculating with integer weights. Both CPUs and GPUs calculate integers with fewer clock cycles and if they are just a few bits long, more of them fit next to each other in the registers of the vector units. Converting a model so that it calculates with short integers is called quantization [2]. For the price of being untrainable, quantized models can be calculated much faster and on less complex hardware and consume much less memory.

The developers of llama.cpp describe the program as a playground for the development of the GGML library. GGML is both a quantization algorithm and a data format for packing the parameters of AI models together with meta-information about the architecture into a file. When quantizing, the need for such a data format quickly arises because the same neural network can handle data types of different lengths in different places and large models generate a lot of meta-information that has to be shared with the parameters anyway.