Skip to main content

Running Alpaca.cpp (LLaMA) on Android phone using Termux

Smartphone Termux tutorial AI
Table of Contents

中文版

Running LLaMA, a ChapGPT-like large language model released by Meta on Android phone locally.

I use antimatter15/alpaca.cpp, which is forked from ggerganov/llama.cpp.

Alpaca is the fine-tuned version of LLaMA which was released by Stanford University. The alpaca.cpp could run on CPU only mode.

1. Environment
#

  • Device:Xiaomi Pocophone F1
  • Android 13
  • SoC:Qualcomm Snapdragon 845
  • RAM: 6GB

2. Compile alpaca.cpp
#

Please install Termux first. Then choose one methods from below to compile Alpaca.cpp on your device.

Alpaca requires at leasts 4GB of RAM to run. If your device has RAM >= 8GB, you could run Alpaca directly in Termux or proot-distro (proot is slower). Devices with RAM < 8GB are not enough to run Alpaca 7B because there are always processes running in the background on Android OS. Termux may crash immediately on these devices.

Or you could root your phone and setup a chroot environment. Then mount a swapfile to get more RAM on your device.

2.1. Compile Alpaca.app in Termux
#

  1. Install these packages
pkg install clang wget git cmake
  1. Download Android NDK
wget https://github.com/lzhiyong/termux-ndk/releases/download/ndk-r23/android-ndk-r23c-aarch64.zip
unzip android-ndk-r23c-aarch64.zip
export NDK=~/android-ndk-r23c-aarch64
  1. Complile Alpaca.cpp and download the model
git clone https://github.com/rupeshs/alpaca.cpp.git
cd alpaca.cpp
mkdir build-android
cd build-android
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod ..
make -j8
wget https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/resolve/main/ggml-alpaca-7b-q4.bin
  1. Run it
./chat

2.2. In Chroot
#

  1. Install Chroot Ubuntu and log in to Ubuntu.

  2. If your devices has RAM lower than 8GB, it is recommened to mount a SWAP file.

dd if=/dev/zero of=/swapfile bs=1M count=8192 status=progress
chmod 0600 /swapfile
mkswap /swapfile
swapon /swapfile
  1. Install dependencies, compile the program, and download the model.
apt install build-essential wget
git clone https://github.com/antimatter15/alpaca.cpp
cd alpaca.cpp
make chat
wget https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/resolve/main/ggml-alpaca-7b-q4.bin
  1. Run it.
./chat

2.3. In Proot
#

  1. Install Proot Debian and log in to Debian.
proot-distro login debian --shared-tmp
  1. Install dependencies, compile the program, and download the model.
apt install build-essential wget
git clone https://github.com/antimatter15/alpaca.cpp
cd alpaca.cpp
make chat
wget https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/resolve/main/ggml-alpaca-7b-q4.bin
  1. Run it.
./chat

3. Usage
#

  1. Now you can start chattig with Alpaca.

  2. It will takes Alpaca 30 seconds to start answering your questions.

  3. What can you do with Termux? Well, this looks like the answers from Termux official website :)

Related

How to run Stable Diffusion on Termux on Android phone
Smartphone Termux tutorial Stable Diffusion AI Android Termux
Installing Anaconda on Android (aarch64) devices
Smartphone Termux tutorial Termux
An experiment of running postmarketOS on Android with Termux Proot
Smartphone Termux tutorial postmarketOS Termux

Leave a comment

Choose the comment system you like. Giscus for codes, Disqus for anonymous comments.

Please login to your Github account to leave a comment. You can post codes here for discussion. For images, please upload to Imgur and copy links. Your comments would be visible to everyone on Github Discussions.

This comment system is provided by Disqus, you may see forcing Disqus ADs on this page. Sometimes your comment may need to be reviewed, resulting in delayed display. Thank you for your understanding.