Jump to content

Wu Dao

From Wikipedia, the free encyclopedia
Ngộ đạo (Wu Dao)
Original author(s)Beijing Academy of Artificial Intelligence
Initial releaseJanuary 11, 2021
Websitewww.baai.ac.cn/english.html

Wu Dao(Chinese:Ngộ đạo;pinyin:wùdào;lit.'road to awareness')[1]is amultimodalartificial intelligencedeveloped by theBeijing Academy of Artificial Intelligence(BAAI).[2][3][4]Wu Dao 1.0 was first announced on January 11, 2021;[1][5]an improved version, Wu Dao 2.0, was announced on May 31.[6][5]It has been compared toGPT-3,[7]and is built on a similar architecture; in comparison, GPT-3 has 175 billionparameters[8][9]— variables and inputs within the machine learning model — while Wu Dao has 1.75 trillion parameters.[6][10]Wu Dao was trained on 4.9 terabytes of images and texts (which included 1.2 terabytes of Chinese text and 1.2 terabytes of English text),[6][11]while GPT-3 was trained on 45 terabytes of text data.[12]Yet, a growing body of work highlights the importance of increasing both data and parameters.[13]The chairman of BAAI said that Wu Dao was an attempt to "create the biggest, most powerful AI model possible".[8]Wu Dao 2.0, was called "the biggest language A.I. system yet".[4]It was interpreted by commenters as an attempt to "compete with the United States".[14][15].Notably, the type of architecture used for Wu Dao 2.0 is amixture-of-experts(MoE) model,[5]unlike GPT-3, which is a "dense" model:[16]while MoE models require much less computational power to train than dense models with the same numbers of parameters,[16]trillion-parameter MoE models have shown comparable performance to models that are hundreds of times smaller.[16]

Wu Dao's creators demonstrated its ability to performnatural language processingand image recognition, in addition to generation of text and images.[5]The model can not only write essays, poems and couplets in traditional Chinese, it can both generate alt text based on a static image and generate nearly photorealistic images based on natural language descriptions. Wu Dao also showed off its ability to power virtual idols (with a little help fromMicrosoft-spinoffXiaoice) and predict the 3D structures of proteins likeAlphaFold.[5]

History

[edit]

Wu Dao's development began in October 2020, several months after the May 2020 release ofGPT-3.[1]The first iteration of the model, Wu Dao 1.0, "initiated large-scale research projects"[17]via four related models.[18][17]

  • Wu Dao – Wen Yuan,a 2.6-billion-parameter pretrained language model, was designed for tasks like open-domain answering, sentiment analysis, and grammar correction.[17]
  • Wu Dao – Wen Lan,a 1-billion-parameter multimodal graphic model, was trained on 50 million image pairs to perform image captioning.[17]
  • Wu Dao – Wen Hui,an 11.3-billion-parameter generative language model, was designed for "essential problems in general artificial intelligence from a cognitive perspective";Syncedsays that it can "generate poetry, make videos, draw pictures, retrieve text, perform complex reasoning, etc".[17]
  • Wu Dao – Wen Su,based on Google's BERT language model and trained on the 100-gigabyte UNIPARC database (as well as thousands of gene sequences), was designed for biomolecular structure prediction and protein folding tasks.[17]

WuDao Corpora

[edit]

WuDao Corpora (also written as WuDaoCorpora), as of version 2.0, was a large dataset constructed for training Wu Dao 2.0. It contains 3 terabytes of text scraped from web data, 90 terabytes of graphical data (incorporating 630 million text/image pairs), and 181 gigabytes of Chinese dialogue (incorporating 1.4 billion dialogue rounds).[19]Wu Dao 2.0 was trained using FastMoE, a variant of the mixture of experts architecture published byGoogle.[18]TheNextWebsaid in June 2021 that "details as to exactly how Wu Dao was trained, what was in its various datasets, and what practical applications it can be used for remain scarce".[8]OpenAI's policy director called Wu Dao an example of "model diffusion", a neologism describing a situation in which multiple entities develop models similar to OpenAI's.[20]

References

[edit]
  1. ^abcTrí nguyên viện nghiên cứu (January 11, 2021)."Mặt hướng nhận tri, trí nguyên viện nghiên cứu liên hợp nhiều gia đơn vị tuyên bố siêu đại quy mô kiểu mới dự huấn luyện mô hình" ngộ đạo · văn hối "".
  2. ^Bailey, Ronald (June 11, 2021)."Amazing New Chinese A.I.-Powered Language Model Wu Dao 2.0 Unveiled".Reason.
  3. ^"Record-breaking AI model WuDao 2.0 unveiled in Beijing".China.org.cn.June 3, 2021.
  4. ^ab"Can A.I. help investors find the next hot technology? A number of companies are betting on it".Fortune.
  5. ^abcdeTarantola, A. (June 2, 2021)."China's gigantic multi-modal AI is no one-trick pony".Engadget.
  6. ^abcFeng, Coco (June 2, 2021)."Beijing-funded AI language model tops Google and OpenAI in raw numbers".South China Morning Post.
  7. ^"WuDao 2.0, l'intelligenza artificiale cinese più potente di OpenAI e Google".lantidiplomatico.it.6 June 2021.
  8. ^abcGreene, Tristan (June 3, 2021)."China's 'Wu Dao' AI is 10X bigger than GPT-3, and it can sing".TNW | Neural.
  9. ^Mott, Nathaniel (June 3, 2021)."China Says WuDao 2.0 AI Is an Even Better Conversationalist than OpenAI, Google".Tom's Hardware.
  10. ^"Wu Dao 2.0, el modelo de IA multimodal de aprendizaje profundo más grande del mundo".El Comercio.June 3, 2021.
  11. ^Heikkilä, Melissa (June 9, 2021)."Meet Wu Dao 2.0, the Chinese AI model making the West sweat".Politico.
  12. ^B. Brown, Tom (2020). "Language Models are Few-Shot Learners".arXiv:2005.14165[cs.CL].
  13. ^Hoffmann, Jordan (2022). "Training Compute-Optimal Large Language Models".arXiv:2203.15556[cs.CL].
  14. ^"Китайская нейросеть WuDao 2.0 с 1,75 трлн параметров превосходит аналоги от Google и OpenAI".3DNews - Daily Digital Digest.
  15. ^Mertens, Jennifer (June 5, 2021)."La Chine développe une IA pour concurrencer les États-Unis".
  16. ^abcArtetxe, Mikel (2021). "Efficient Large Scale Language Modeling with Mixtures of Experts".arXiv:2112.10684[cs.CL].
  17. ^abcdef"China's GPT-3? BAAI Introduces Superscale Intelligence Model 'Wu Dao 1.0' | Synced".syncedreview.March 23, 2021.
  18. ^abSchreiner, Maximilian (13 June 2021)."Sprach-KI aus China: Wu Dao 2.0 ist zehnmal größer als GPT-3".
  19. ^"resource".resource.wudaoai.cn.
  20. ^Wiggers, Kyle (June 4, 2021)."AI Weekly: China's massive multimodal model highlights AI research gap".VentureBeat.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wu_Dao&oldid=1234299896"