GPT local

A new project called Text Generation WebUI or OobaBooga as it is rather funnily known thanks to the name of its creator on GitHub is the latest LLM model in town Whatever the name it aims to simplify how LLMs are installed on the local computer similar to what Automatic 1111 did for Stable Diffusion truly democratising the technology So letrsquos get right into itRequirementsgtgt A computer with a Nvidia graphics card and more RAM on the card which translates to a betterperforming LLM You can technically run it on your processor as well but be prepared to wait a few hours for every responsegtgt Lots of storage space LLM models are at least a couple of GBs each and if you want a choice of a few LLMs be prepared to fill up your hard drive gtgt Access to the Internet to download the models and the software Once installed however the LLM will not need the Internet to functionDownload and installYou will need to go to OobaBoogarsquos Github httpsbitlysmdooba click on the green colour Code button and then click on Download Zip The entire file is around 20MB Copy this zip file to the place where you plan to use it and make sure there is a lot of space in the destination drive Unzip the file Go into the directory called textgenerationwebuimainThere should be one more directory with the same name Go into that directory as well If you are using Windows doubleclick on the start_windowsbat file This should start the installation process The program will ask you what graphics card you are running If you are running a Nvidia card select lsquoArsquo AMD GPUs are only supported on Linux installations If you donrsquot have a GPU select lsquoNrsquoIf you have selected lsquoArsquo for Nvidia you will have another choice to make It will ask you if you want to use CUDA 118 instead of 121 If you have an RTX or a GTX card select lsquoNrsquo If you have an older GPU select lsquoYrsquo If you are unsure select lsquoNrsquo It will start installing everything you need to run the program Brace yourself this is going to take a while Once done though you can immediately start up the web interface It should say ldquoRunning one local URL http1270017860Type the URL in your browserGetting the modelThe interface is useless without a model There are several models to choose from but Mistral 7B is currently my favourite model Letrsquos see how to install it First go to TheBlokersquos Hugging Face httpshuggingfacecoTheBlokeMistral7BInstructv01GPTQ Scroll down to ldquoHow to download including from branchesrdquoSelect the text ldquoTheBlokeMistral7BInstructv01GPTQrdquo Open your Textgenerationwebui window On top select the lsquoModelrsquo tab Go to the Download model or LoRA section on the right and paste the text Click on DownloadThis will take a while because the file is a couple of GBs You can use the same process to download different models including LLAMA Qwen Orca or Gorilla Read about models that suit your specific use case Itrsquos a bit of a rabbit hole but if you want something that works well stick to Mistral 7B InstructLoading the modelOnce downloaded On the left top section of the Model tab select the model you downloaded and press Load if the model you downloaded doesnrsquot show up Close the Command window running in the background and start the LLM up again by running the ldquostart_windowsbatrdquo file Make sure you try loading the model again and once it is loaded head to the Chat tab There you can ask all sorts of questions except those that show illegal or dangerous results Just like Chat GPT you canrsquot ask for instructions for making a bomb or about current affairsMaintenanceTextgenerationwebui is pretty selfsufficient and you can easily upgrade it to the newest version by running the ldquoupdate_windowsbatrdquo file in the same directory as the start file The update should be run every few weeks as there are new and exciting additions to the program This is however an upgrade tool only for the main program For a newer model you must explore TheBlokersquos hugging face and download a newer version there There are usually good descriptions for models but feel free to explore the parent page for each model to see if the model is a good fit for you