About gpt2-ml

<img src="./.github/logo.svg" width="480"> # **GPT2** for Multiple Languages [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/imcaspar/gpt2-ml/blob/master/pretrained_model_demo.ipynb) [![GitHub](https://img.shields.io/github/license/imcaspar/gpt2-ml)](https://github.com/imcaspar/gpt2-ml) [![GitHub All Releases](https://img.shields.io/github/downloads/imcaspar/gpt2-ml/total)](https://github.com/imcaspar/gpt2-ml/releases) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/imcaspar/gpt2-ml/issues) [![GitHub stars](https://img.shields.io/github/stars/imcaspar/gpt2-ml?style=social)](https://github.com/imcaspar/gpt2-ml) [**中文说明**](./README_CN.md) | [**English**](./README.md) - [x] Simplifed GPT2 train scripts（based on Grover, supporting TPUs） - [x] Ported bert tokenizer, multilingual corpus compatible - [x] 1.5B GPT2 pretrained Chinese model ( ~15G corpus, 10w step ...

i

Published by

imcaspar

Visit View Profile

README.md

View on GitHub

GPT2 for Multiple Languages

中文说明 | English

[x] Simplifed GPT2 train scripts（based on Grover, supporting TPUs）
[x] Ported bert tokenizer, multilingual corpus compatible
[x] 1.5B GPT2 pretrained Chinese model ( ~15G corpus, 10w steps )
[x] Batteries-included Colab demo #
[x] 1.5B GPT2 pretrained Chinese model ( ~30G corpus, 22w steps )

Pretrained Model

Size	Language	Corpus	Vocab	Link1	Link2	SHA256
1.5B Params	Chinese	~30G	CLUE ( 8021 tokens )	Google Drive	Baidu Pan (ffz6)	e698cc97a7f5f706f84f58bb469d614e 51d3c0ce5f9ab9bf77e01e3fcb41d482
1.5B Params	Chinese	~15G	Bert ( 21128 tokens )	Google Drive	Baidu Pan (q9vr)	4a6e5124df8db7ac2bdd902e6191b807 a6983a7f5d09fb10ce011f9a073b183e

Corpus from THUCNews and nlp_chinese_corpus

Using Cloud TPU Pod v3-256 to train 22w steps

loss

Google Colab

With just 2 clicks (not including Colab auth process), the 1.5B pretrained Chinese model demo is ready to go:

[Colab Notebook]

Train

Disclaimer

The contents in this repository are for academic research purpose, and we do not provide any conclusive remarks.

Citation

@misc{GPT2-ML,
  author = {Zhibo Zhang},
  title = {GPT2-ML: GPT-2 for Multiple Languages},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/imcaspar/gpt2-ml}},
}