代码拉取完成，页面将自动刷新
Llama 2 is a large language model released by Meta in 2023, with parameters ranging from 7B to 70B. Compared to LLaMA, the training corpus of Llama 2 is 40% longer, and the context length has been upgraded from 2048 to 4096, allowing for understanding and generating longer texts.
# Install sqlite3
wget https://sqlite.org/2019/sqlite-autoconf-3290000.tar.gz
tar zxvf sqlite-autoconf-3290000.tar.gz
pushd sqlite-autoconf-3290000
./configure
make && make install
popd
# Reinstall Python
wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
tar xvf Python-3.7.9.tar.xz
pushd Python-3.7.9
./configure LDFLAGS="-L/usr/local/lib" CPPFLAGS="-I/usr/local/include" --prefix=/usr/bin
make && make install
cp /usr/bin/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-x86_64-linux-gnu.so /usr/local/lib/python3.7/lib-dynload/_sqlite3.so
popd
# Install Megatron-Deepspeed
bash build_megatron-deepspeed.sh && bash install_megatron-deepspeed.sh
cd dataset/
bash download_and_convert_dataset.sh
You can download huggingface llama2-7b pretrained model from here, and use below script to convert it.
cd checkpoints
bash convert_hf_2_meg.sh
cd examples/llama2
bash run_meg_llama2_7b_sft.sh
If the torchrun command cannot be found，you can execute:
ln -s /usr/local/corex-3.1.0/lib64/python3/dist-packages/bin/torchrun /usr/local/bin/
| GPUs | Toolbox | Model | Training speed |
|---|---|---|---|
| BI-V100 x8 | Megatron-DeepSpeed | LLaMA2-7B SFT | 1.146 samples/sec |
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。