alibaba

Open Source

scalable-form-platform

# scalable-form-platform 基于动态表单协议（[JSON Schema](http://json-schema.org/)）的表单解决方案，提供了基于 [ant-design](https://github.com/ant-design/ant-design) 和 [ant-design-mobile](https://github.com/ant-design/ant-design-mobile) 的前端表单渲染引擎sdk、可视化表单编辑器和可独立部署镜像。 A solution for editing and publish dynamic web forms with visual editor, providing react components for rendering web forms from [JSON schema](http://json-schema.org/) using [ant-design](https://github.com/rjsf-team/react-jsonschema-form) or [ant-design-mobile](https://github.com/ant-design/ant-design-mobile), an visual editor component to edit dynamic form json schema and an server library helping you build an dynamic form system.   [](https://hub.docker.com/r/lvshuncn/scalable-form-platform)     查看文档 [https://scalable-form-platform.github.io/#/](https://scalable-form-platform.github.io/#/) <p align="center"> <a href="https://scalable-form-platform.github.io/" target="_blank"> <img max-width="1440" src="https://img.alicdn.com/tfs/TB1MnB9z7Y2gK0jSZFgXXc5OFXa-1440-900.png" /> </a> </p> ## ✨ 特性/features - 🛳 企业级解决方案和高稳定性 - 📦 开箱即用的动态表单渲染sdk - ⚙️ 完整的开发工具支持 - 🗄 服务端支持和用于管理表单的可独立部署站点 <br /> - 🛳 Enterprise-class Solutions for web form - 📦 A set of high-quality react components for rendering web forms from JSON schema out of the box - ⚙️ Whole package of development tools - 🗄 Server support and independently deployable sites <br /> - 更多特性请查看文档 [https://scalable-form-platform.github.io/#/](https://scalable-form-platform.github.io/#/) ## 📦 包管理/Packages 我们使用[Lerna](https://github.com/lerna/lerna)来进行包管理，所以本仓库会发布多个包到npm，包括： This repository is a monorepo that we manage using [Lerna](https://github.com/lerna/lerna). That means that we actually publish several packages to npm from the same codebase, including: | Package | NPMVersions | Documents | Description | | ---- | ---- | ---- | ---- | | scalable-form-antd | [](https://www.npmjs.com/package/scalable-form-antd) | [](https://scalable-form-platform.github.io/#/zh/scalable_form_antd) | 基于react-jsonschema-form，结合ant-design的动态表单渲染sdk | | scalable-form-antd-mobile | [](https://www.npmjs.com/package/scalable-form-antd-mobile)|[](https://scalable-form-platform.github.io/#/zh/scalable_form_antd_mobile)| 基于react-jsonschema-form，结合ant-design-mobile的动态表单渲染sdk，适用于移动端的渲染sdk | | scalable-form-editor | [](https://www.npmjs.com/package/scalable-form-editor) | [](https://scalable-form-platform.github.io/#/zh/scalable-form-editor)| 表单可视化编辑器，可视化编排表单，产出scalable-form-antd和scalable-form-antd-mobile可用的schema | | scalable-form-server | [](https://www.npmjs.com/package/scalable-form-server) | [](https://scalable-form-platform.github.io/#/zh/%E4%BD%BF%E7%94%A8Node)| 服务端sdk，用户可以基于scalable-form-server保存表单配置。服务端sdk提供一个可用的表单站点，提供表单编排，表单管理，表单投放，数据回流分析的能力。 | ## ⛳️ 快速上手 我们从一个例子快速开始，在这个例子中，我们会渲染一个表单，表单支持用户填写自己的名字（name字段）。 ### 使用scalable-form-antd渲染表单 > [scalable-form-antd](https://www.npmjs.com/package/scalable-form-antd)会根据表单描述协议（JSONSchema），使用ant-design组件渲染表单。 ```bash npm i scalable-form-antd -S ``` 使用scalable-form-antd，我们需要针对表单需求写一下表单描述（schema），并且将schema作为scalable-form-antd的props传入。 > 您可能觉得写这个schema很繁琐，不过放心，Scalable Form的一大创新就是支持使用可视化的编排组件 [scalable-form-editor](https://www.npmjs.com/package/scalable-form-editor) 来搭建生成这个schema，完全不需要您手动书写，下文中，您会了解如何使用这个editor。 ```jsx harmony import React from "react"; import "./styles.css"; import ScalableForm from "scalable-form-antd"; // 这个例子，使用scalable-form-antd渲染了一个表单 export default class FormExample extends React.PureComponent { constructor(props) { super(props); this.state = { formData: { name: "" } }; // 规则jsonSchema，用于描述表单字段信息，包括字段类型，长度等 this.jsonSchema = { title: "Scalable Form render sdk with Ant Design components", description: "", type: "object", required: ["name"], properties: { name: { type: "string", title: "名称", default: "", maxLength: 15 } } }; // 规则uiSchema，用于描述表单UI组件信息，包括输入框的placeholder等字段 this.uiSchema = { name: { "ui:help": '关于"名称"字段的帮助说明', "ui:options": { placeholder: "请输入名称" } } }; } // 用户输入表单数据变化时的回调，会返回用户当前填写的表单数据formData handleChanged = (formData) => { console.log("ScalableForm Changed!", formData); this.setState({ formData: { ...formData } }); }; // 用户点击提交时的回调，会返回用户填写的表单数据formData handleSubmit = (formData) => { console.log("ScalableForm Submitted!", formData); }; render() { return ( <div className="scalable-form-demo-element"> <ScalableForm jsonSchema={this.jsonSchema} uiSchema={this.uiSchema} formData={this.state.formData} onChange={this.handleChanged} onSubmit={this.handleSubmit} /> </div> ); } } ``` 您可以在 [codesandbox](https://codesandbox.io/s/late-bird-xform-antd-x6qoo?from-embed) 查看这个例子的演示。<br /> 您可以访问访问[scalable-form-antd文档](https://scalable-form-platform.github.io/#/zh/scalable_form_antd)了解更多信息 <img src="https://img.alicdn.com/tfs/TB1kDsPAeH2gK0jSZJnXXaT1FXa-2554-696.png" /> > 您也可以使用[scalable-form-antd-mobile](https://www.npmjs.com/package/scalable-form-antd-mobile)在移动端渲染表单，[点击这里](https://codesandbox.io/s/late-bird-xform-antd-mobile-qspcg)在codesandbox中查看移动端渲染表单的例子，您也可以[在这里](https://scalable-form-platform.github.io/#/zh/%E7%A7%BB%E5%8A%A8%E7%AB%AF%E6%B8%B2%E6%9F%93)查看更多scalable-form-antd-mobile的文档。 ### 使用scalable-form-editor可视化编排表单 ```bash npm i scalable-form-editor -S ``` [scalable-form-editor](https://www.npmjs.com/package/scalable-form-editor)是可视化编排表单的前端组件，您可以通过scalable-form-editor快速搭建您的表单。 ```javascript import React from "react"; import "./styles.css"; import ScalableFormEditor from "scalable-form-editor"; class FormEditorExample extends Component { constructor(...args) { super(...args); this.state = { uiSchema: {}, jsonSchema: {}, bizData: {}, sequence: [] formData: {} }; } handleSubmit = (formCode, {jsonSchema,uiSchema, formData, bizData, sequence}) => { console.log('submit by editor', jsonSchema, uiSchema, formData, bizData, sequence); }; render() { return ( <ScalableFormEditor jsonSchema={this.state.jsonSchema} uiSchema={this.state.uiSchema} formData={this.state.formData} bizData={this.state.bizData} sequence={this.state.sequence} onSubmit={this.handleSubmit} /> ); } } ``` 您可以在 [codesandbox](https://codesandbox.io/s/late-bird-xform-antd-x6qoo?from-embed) 查看scalable-form-editor的演示，[访问这里](https://scalable-form-platform.github.io/#/zh/scalable-form-editor)查看scalable-form-editor的更多文档  ### 使用Scalable Form站点 实际业务使用中，Scalable Form整体上由三个主要部分组成，表单渲染端，表单配置端，表单存储的服务端。 - 表单配置端可以通过可视化的方式来编排产出渲染端可用的表单配置，并支持用户填写表单。 - 渲染端使用配置端编排出来的表单配置，渲染为可用的表单。 - 服务端用于存储表单配置。不仅如此，服务端还可以串联起整个Scalable Form的使用场景，提供配置站点（权限控制，场景配置，自定义组件配置），表单维护管理，表单填写页面，表单数据存储展示与简单的数据分析能力。 通过docker镜像的方式，你可以很方便地在本地或者服务器部署一个Scalable Form服务器。 你可以从[docker hub](https://hub.docker.com/repository/docker/lvshuncn/scalable-form-platform/general)搜索到Scalable Form服务器的docker镜像 如果你在本地测试，你可以执行以下命令通过`scalable-form-platform`镜像开启一个Scalable Form服务器，端口绑定3000 ``` docker run -d -p 3000:3000 lvshuncn/scalable-form-platform ``` 开启后，访问`http://localhost:3000/xform/admin`即可使用Scalable Form独立站点 > 默认情况下docker中的Scalable Form会开启demo模式，demo模式仅用于快速演示，服务器以内存数据库的方式存储数据，重启后数据会清空，不要在生产环境下使用。生产环境使用方法请[查看文档](https://scalable-form-platform.github.io/#/zh/%E4%BD%BF%E7%94%A8dockerr) <p align="center"> <a href="https://scalable-form-platform.github.io/" target="_blank"> <img max-width="1440" src="https://img.alicdn.com/tfs/TB1MnB9z7Y2gK0jSZFgXXc5OFXa-1440-900.png" /> </a> </p> ## 🖥 浏览器支持/Environment Support * Modern browsers and Internet Explorer 9+ (with [polyfills](https://ant.design/docs/react/getting-started#Compatibility)) | [<img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/edge/edge_48x48.png" alt="IE / Edge" width="24px" height="24px" />](http://godban.github.io/browsers-support-badges/)</br>IE / Edge | [<img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/firefox/firefox_48x48.png" alt="Firefox" width="24px" height="24px" />](http://godban.github.io/browsers-support-badges/)</br>Firefox | [<img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/chrome/chrome_48x48.png" alt="Chrome" width="24px" height="24px" />](http://godban.github.io/browsers-support-badges/)</br>Chrome | [<img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/safari/safari_48x48.png" alt="Safari" width="24px" height="24px" />](http://godban.github.io/browsers-support-badges/)</br>Safari | [<img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/opera/opera_48x48.png" alt="Opera" width="24px" height="24px" />](http://godban.github.io/browsers-support-badges/)</br>Opera | | --------- | --------- | --------- | --------- | --------- | | IE9, IE10, IE11, Edge| last 2 versions| last 2 versions| last 2 versions| last 2 versions|

JavaScript Libraries & Components Form & Survey Builders

119 Github Stars

Read more

Open Source

rex-design

# ReX Design For OS **开发中，敬请期待** [](https://www.npmjs.org/package/@rexd/core) 基于 React 的多端自适应组件库 ## 开发 安装依赖 ```bash yarn ``` 运行示例 ```bash yarn start:story ``` ## 测试 ```bash yarn test ``` 自定义传参，只运行个别目录 ```bash yarn test src/utils --watch ``` 详细命令行参数参考：https://jestjs.io/docs/cli ## 提交代码 提交代码需要编写符合约定的 commit message 规范，格式如下： ``` git commit -m <type>(<scope>): <message> ``` 其中 type 允许的类型包括： - feat: (new feature for the user, not a new feature for build script) - fix: (bug fix for the user, not a fix to a build script) - docs: (changes to the documentation) - style: (formatting, missing semi colons, etc; no production code change) - refactor: (refactoring production code, eg. renaming a variable) - test: (adding missing tests, refactoring tests; no production code change) - chore: (updating grunt tasks etc; no production code change) 其中 scope 表示作用的组件，必须使用 Pascal-Case，例如 - Button - DatePicker 最后，message 为您此次提交的描述。例如： ```bash git commit -m feat(DatePicker): add onChange ``` ## Pull Request 流程 请参考：[如何优雅地在 github 上贡献代码](https://segmentfault.com/a/1190000000736629) 1. 克隆仓库 2. 创建分支 3. 提交代码 4. 合并修改 `git rebase master` 5. 发起 MR ## 社区分享 - 2020.12, D2 前端技术大会[《盒马中后台跨端方案探索》](https://mp.weixin.qq.com/s/-1PpjJyKiA63SifRbr5swg) - 2021.06, ReX Design 跨端组件测试版发布 [《盒马跨端设计系统 ReX Design For OS》](https://mp.weixin.qq.com/s/aRGiDMAqWMiCWqy0s5be7A) - 2021.07, ReX Design 设计理念分享 [《ReX Design for OS 面向多因子的跨端设计系统》](https://mp.weixin.qq.com/s/Nt3s7NdaH_8rc6JbJpX3_w) ## 致谢 感谢下述开源项目，部分功能实现参考了这些项目中的代码实现。 - [Alibaba Fusion Design](https://github.com/alibaba-fusion/next) - [Material UI](https://material-ui.com/) ## License This project is licensed under the terms of the MIT license.

Design Systems & Tokens

129 Github Stars

Read more

Open Source

flutter-go

# FlutterGo 感谢大家一直以来对 FlutterGo 的支持与反馈。 但是 **由于团队内部组织调整变更**，很遗憾的宣布 FlutterGo 目前**暂停维护**。 但是后期我们会再升级迭代 FlutterGo 新版本，也同时希望更多Flutter爱好者参与加入。 - 简历投递 邮箱：[email protected] - 钉钉扫码直接连接: - <img src="https://gw.alicdn.com/tfs/TB1VXgXYFT7gK0jSZFpXXaTkpXa-716-914.png" width=278 height=357>

Education & Learning Mobile Development

23.7K Github Stars

Read more

Open Source

fish-redux

<p align="center"><img src="https://img.alicdn.com/tfs/TB1r74NJyLaK1RjSZFxXXamPFXa-1024-1024.png" align="center" width="175"></p> <h1 align="center">Fish Redux</h1> [](https://travis-ci.org/alibaba/fish-redux) [](https://pub.dartlang.org/packages/fish_redux) [](https://codecov.io/gh/alibaba/fish-redux) ## What is Fish Redux ? Fish Redux is an assembled flutter application framework based on Redux state management. It is suitable for building medium and large applications. It has four characteristics: > 1. Functional Programming > 2. Predictable state container > 3. Pluggable componentization > 4. Non-destructive performance ## Architecture diagram <img src="https://img.alicdn.com/tfs/TB1pkhoJr2pK1RjSZFsXXaNlXXa-1004-1370.png" width="500px" height="680px"> ## Installation [Go](https://pub.dartlang.org/packages/fish_redux#-installing-tab-) ## Documentation Language: [English](doc/README.md) | [中文简体](doc/README-cn.md) ## Examples - [todo list](example) - a simple todo list demo. - run it: ``` cd ./example flutter create . flutter run ``` ## What's the difference between 'Fish Redux' and 'Redux' ? - [answers](doc/concept/what's-the-diiference.md) ## Plugins ### Code Template - [Fish Redux Template For Android Studio](https://github.com/BakerJQ/FishReduxTemplateForAS), by BakerJQ. - [Fish Redux Template For VSCode](https://github.com/huangjianke/fish-redux-template), by huangjianke. ### Dev-Tools - Redux Inspector (using [Flutter Debugger](https://github.com/blankapp/flutter-debugger) and [flipperkit_fish_redux_middleware](https://pub.dartlang.org/packages/flipperkit_fish_redux_middleware)) for Fish Redux apps, by [JianyingLi](https://github.com/lijy91) ## License - Fish Redux is released under the Apache 2.0 license. See [LICENSE](LICENSE) for details. ## 关于我们 阿里巴巴-闲鱼技术是国内最早也是最大规模线上运行Flutter的团队。 我们在公众号中为你精选了Flutter独家干货，全面而深入。 内容包括：Flutter的接入、规模化应用、引擎探秘、工程体系、创新技术等教程和开源信息。 **架构／服务端／客户端／前端／质量工程师 在公众号中投递简历，名额不限哦** 欢迎来闲鱼做一个好奇、幸福、有影响力的程序员，简历投递：[email protected] 订阅地址 <img src="https://img.alicdn.com/tfs/TB17Ki5XubviK0jSZFNXXaApXXa-656-656.png" width="328px" height="328px"> [For English](https://twitter.com/xianyutech "For English")

Developer Tools Mobile Development

7.3K Github Stars

Read more

Open Source

flutter_boost

<p align="center"> <img src="flutter_boost.png"> <b></b><br> <a href="README_CN.md">中文文档</a> <a href="https://zhuanlan.zhihu.com/p/362662962">中文介绍</a> </p> # Release Note ## [ohos] New Platform flutter_boost has been fully adapted for the ohos platform. Please refer to item 10 of the following document to understand the related conventions before integrating： <a href="Frequently Asked Question.md">FAQ</a> ## 4.6.5 PS：Null-safety is already supported. - Flutter SDK upgrades do not require Boost upgrades - Simplify the architecture - Simplify the interface - Unified design of double-end interface - Solved the Top Issue - Android does not need to distinguish between AndroidX and Support # FlutterBoost A next-generation Flutter-Native hybrid solution. FlutterBoost is a Flutter plugin which enables hybrid integration of Flutter for your existing native apps with minimum efforts. The philosophy of FlutterBoost is to use Flutter as easy as using a WebView. Managing Native pages and Flutter pages at the same time is non-trivial in an existing App. FlutterBoost takes care of page resolution for you. The only thing you need to care about is the name of the page(usually could be an URL).  <a name="bf647454"></a> # Prerequisites 1. Before proceeding, you need to integrate Flutter into your existing project. 2. The Flutter SDK version supported by Boost 3.0 is >= 1.22 # Getting Started ## Add a dependency in you Flutter project. Open you pubspec.yaml and add the following line to dependencies: ``` yaml flutter_boost: git: url: 'https://github.com/alibaba/flutter_boost.git' ref: '4.6.5' ``` # Version Notes - For Flutter SDK 3.0 and above, use `4.0.1+`. - For Flutter SDK below 3.0, use `v3.0-release.2` or earlier versions. - The null safety versions supporting Flutter SDK 2.5.x are `3.1.0+`. - The versions supporting Flutter SDK 3.16.x are `5.0.0+`. - The versions supporting HarmonyOS are `[4.5.0, 5.0.0)`. # Usage - [Detailed Integration Steps](https://github.com/alibaba/flutter_boost/blob/master/docs/install.md) - [Basic Routing API](https://github.com/alibaba/flutter_boost/blob/master/docs/routeAPI.md) - [API for Page Lifecycle](https://github.com/alibaba/flutter_boost/blob/master/docs/lifecycle.md) - [Custom API for Sending Cross-Platform Events](https://github.com/alibaba/flutter_boost/blob/master/docs/event.md) # Contribution - [How to File an Issue to Us](https://github.com/alibaba/flutter_boost/blob/master/docs/issue.md) - [How to Submit a PR to Us](https://github.com/alibaba/flutter_boost/blob/master/docs/pr.md) # FAQ please read this document: <a href="Frequently Asked Question.md">FAQ</a> # License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details ## About Us Alibaba-Xianyu Tech is one of the earliest and largest teams running Flutter on a large scale online in China. In our official WeChat account, we have carefully selected exclusive Flutter content for you, both comprehensive and in-depth. The content includes tutorials and open-source information on Flutter integration, large-scale applications, engine exploration, engineering systems, and innovative technologies. **Architects / Backend Engineers / Client-side Engineers / Frontend Developers / Algorithm Engineers / Quality Engineers - submit your resumes through our WeChat account, there is no limit to the number of positions.** We welcome you to join Xianyu and become a curious, happy, and influential programmer. To send your resume, please email: `[email protected]` Subscribe at: <img src="https://img.alicdn.com/tfs/TB17Ki5XubviK0jSZFNXXaApXXa-656-656.png" width="328px" height="328px"> [For English](https://twitter.com/xianyutech "For English")

Mobile Development

7.2K Github Stars

Read more

Open Source

formily

English | [简体中文](./README.zh-cn.md) <p align="center"> <img src="https://img.alicdn.com/tfs/TB1fHhZu4D1gK0jSZFyXXciOVXa-2500-1200.png"> <img src="https://img.shields.io/npm/dt/@formily/core"/> <img src="https://img.shields.io/npm/dm/@formily/core"/> <a href="https://www.npmjs.com/package/@formily/core"><img src="https://img.shields.io/npm/v/@formily/core.svg"></a> <a href="https://codecov.io/gh/alibaba/formily"> <img src="https://codecov.io/gh/alibaba/formily/branch/formily_next/graph/badge.svg?token=3V9RU8Wh9d"/> </a> <img alt="PRs Welcome" src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg"/> <a href="https://github.com/actions-cool/issues-helper"> <img src="https://img.shields.io/badge/using-issues--helper-blueviolet"/> </a> <a href="https://app.netlify.com/sites/formily/deploys"><img src="https://api.netlify.com/api/v1/badges/7145918b-9cb5-47f8-8a42-111969e232ef/deploy-status"/></a> </p> --- ## Background In React, the whole tree rendering performance problem of the form is very obvious in the controlled mode. Especially for the scene of data linkage, it is easy to cause the page to be stuck. To solve this problem, we have distributed the management of the state of each form field, which significantly improves the performance of the form operations. At the same time, we deeply integrate the JSON Schema protocol to help you solve the problem of back-end driven form rendering quickly. ## Features - 🖼 Designable, You can quickly develop forms at low cost through [Form Builder](https://designable-antd.formilyjs.org/). - 🚀 High performance, fields managed independently, rather rerender the whole tree. - 💡 Integrated Alibaba Fusion and Ant Design components are guaranteed to work out of the box. - 🎨 JSON Schema applied for BackEnd. JSchema applied for FrontEnd. Two paradigms can be converted to each other. - 🏅 Side effects are managed independently, making form data linkages easier than ever before. - 🌯 Override most complicated form layout use cases. ## Form Builder  ## WebSite 2.0 https://formilyjs.org 1.0 https://v1.formilyjs.org ## Community - [formilyjs](https://github.com/formilyjs) - [designable](https://github.com/alibaba/designable) - [icejs](https://github.com/alibaba/ice) ## How to contribute? - [Contribute document](https://formilyjs.org/zh-CN/guide/contribution) ## Contributors This project exists thanks to all the people who contribute. <a href="https://github.com/alibaba/formily/graphs/contributors"><img src="https://contrib.rocks/image?repo=alibaba/formily" /></a> ## LICENSE Formily is open source software licensed as [MIT](https://github.com/alibaba/formily/blob/master/LICENSE.md).

JavaScript Libraries & Components Mobile Development

12.5K Github Stars

Read more

Open Source

LuaViewSDK

# Welcome to LuaViewSDK LuaViewSDK is a cross-platform framework which is designed to build native, dynamic and swift user interfaces. It's based on Lua VM and chooses lua as it's script language. ## Why Lua * Lua is a efficient and swift language. A single Lua VM costs 200k to 300k memorys only in iOS * Lua is widly used and it's easy to bind to third party frameworks * LuaView API is nature to native mobile developers, you guys don't have to understand what React.js or Vue.js or DOM is. ## Demos * Complex LuaViewSDK-rendered native pages <img src='https://gw.alicdn.com/tfs/TB1RzmbQpXXXXXAapXXXXXXXXXX-1198-931.png'/> * Simple games  ## Try Playground App - iOS download [View in App Store](https://itunes.apple.com/cn/app/luaviewplayground/id1216114838?mt=8)  - Android download  ## How to use LuaViewSDK - [User Guide](https://alibaba.github.io/LuaViewSDK/guide.html) - [API manual](https://alibaba.github.io/LuaViewSDK/apimenu.html) ## Developers - [Xicheng Dong]([email protected]) - [Songtao Chen]([email protected]) ## License [MIT](https://github.com/alibaba/LuaViewSDK/blob/master/LICENSE.txt) ## Communities [Github Wiki](https://github.com/alibaba/LuaViewSDK/wiki) QQ group：539262083 Twitter：[@luaview](https://twitter.com/luaview) ## Who we are We are from Tmall tech team. LuaViewSDK has been widely used in Taobao app, Tmall app and Juhuasuan App nearly two years. And DAU of pages that are developed by LuaViewSDK is over 10 million. --- # 欢迎来到LuaViewSDK LuaViewSDK是一个跨iOS和Android平台的界面动态化解决方案，使用同一套Lua脚本在iOS和Android平台构建原生页面，同时兼具了H5的灵活性和Native的高性能 ## 为什么选择Lua？ - Lua是一个高效灵活的语言，在iOS上单个Lua虚拟机仅占用200k到300k - Lua已经被广泛应用于对于性能要求和灵活性要求都非常高的场合，比如游戏 - Lua可以非常方便的绑定各类底层库 - Lua的API对于native开发人员来说非常自然，不用再去理解各类前端开发概念 ## Demo * LuaViewSDK渲染的Native的复杂业务页面 <img src='https://gw.alicdn.com/tfs/TB1RzmbQpXXXXXAapXXXXXXXXXX-1198-931.png'/> * 小游戏  ## 试用Playground App - iOS download [在App Store中查看](https://itunes.apple.com/cn/app/luaviewplayground/id1216114838?mt=8)  - Android download  ## 如何使用LuaViewSDK - [使用说明](https://alibaba.github.io/LuaViewSDK/guide.html) - [接口文档](https://alibaba.github.io/LuaViewSDK/apimenu.html) ## 开发者 - [董希成]([email protected]) - [陈松涛]([email protected]) ## 许可证 [MIT](https://github.com/alibaba/LuaViewSDK/blob/master/LICENSE.txt) ## 社区 [Github Wiki](https://github.com/alibaba/LuaViewSDK/wiki) QQ群：539262083 Twitter：[@luaview](https://twitter.com/luaview)

Mobile Development UI Design & Prototyping

3.7K Github Stars

Read more

Open Source

G3D

<p align="center"> <a href="https://alibaba.github.io/G3D/"><img src ="https://alibaba.github.io/G3D/assets/logo-g3d.png" /></a> </p> <p align="center"> <a href="https://travis-ci.org/alibaba/G3D"><img src ="https://travis-ci.org/alibaba/G3D.svg?branch=master" /></a> <a href="https://codecov.io/github/alibaba/g3d?branch=master"><img src="https://img.shields.io/codecov/c/github/alibaba/g3d/master.svg"></a> <a href="https://www.npmjs.com/search?q=g3d"><img src="https://img.shields.io/npm/v/g3d.svg?style=flat"></a> </p> # G3D G3D is a javascript 3D engine based on WebGL. Compared with other WebGL 3D engines, G3D is a pure render engine, which means it depends on no DOM APIs but just a canvas(or canvas-like) object. This feature makes it possible that G3D works not only in browser but also in some hybrid environments like GCanvas (Weex or ReactNative). See [website](https://alibaba.github.io/G3D/) for more information. # G3D G3D 是一款基于 WebGL 的 javascript 3D 绘图引擎。与其他的 WebGL 3D 引擎相比，G3D 是更加「纯粹」的渲染引擎，也就是说，它完全不依赖任何 DOM API，而是仅仅依赖一个 canvas 对象（或者类 canvas 对象）。该特性使得 G3D 不仅能够运行在浏览器环境中，也能够运行在一些非浏览器的 js 终端环境下（hybrid 环境），比如 GCanvas 环境（基于 Weex 或 ReactNative）。 访问[网站](https://alibaba.github.io/G3D/)以了解更多信息。

JavaScript Libraries & Components 3D Modeling & Animation

1.1K Github Stars

Read more

Open Source

MNN

 --- [](LICENSE.txt) [](https://mnn-docs.readthedocs.io/en/latest/) [](README_CN.md) [](README_TW.md) [](README_JP.md) [](http://www.mnn.zone) [](https://zread.ai/alibaba/MNN) [](./apps/Android/MnnLlmChat/README.md) [](./apps/Android/Mnn3dAvatar/README.md) [](./apps/sana/README.md) ## News 🔥 - [2026/03/05] Support Qwen3.5 Series. <p align="center"> <img width="15%" alt="Icon" src="https://meta.alicdn.com/data/mnn/assets/qwen35_1.jpg" style="margin: 0 10px;"> <img width="15%" alt="Icon" src="https://meta.alicdn.com/data/mnn/assets/qwen35_2.jpg" style="margin: 0 10px;"> <img width="15%" alt="Icon" src="https://meta.alicdn.com/data/mnn/assets/qwen35_3.jpg" style="margin: 0 10px;"> </p> - [2026/02/13] MNN-Sana-Edit-V2 is now available at [apps](./apps/sana/README.md), offering cartoon-style photo editing based on Sana. <p align="center"> <img width="80%" alt="Icon" src="https://meta.alicdn.com/data/mnn/assets/sana_show_case.jpg" style="margin: 0 10px;"> </p> <details> <summary> History News </summary> - [2025/10/16] Support Qwen3-VL Series. - [2025/06/11] New App MNN TaoAvatar released, you can talk with 3DAvatar offline with LLM, ASR, TTS, A2BS and NNR models all run local on your device!! [MNN TaoAvatar](./apps/Android/Mnn3dAvatar/README.md) <p align="center"> <img width="20%" alt="Icon" src="https://meta.alicdn.com/data/mnn/avatar/avatar_demo.gif" style="margin: 0 10px;"> </p> - [2025/05/12] android app support qwen2.5 omni 3b and 7b [MNN Chat App](./apps/Android/MnnLlmChat/README.md#releases). <p align="center"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_home_new.jpg" style="margin: 0 10px;"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_sound_new.jpg" style="margin: 0 10px;"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_image_new.jpg" style="margin: 0 10px;"> </p> - [2025/04/30] android app support qwen3 and dark mode [MNN Chat App](./apps/Android/MnnLlmChat/README.md#releases). <p align="center"> <img width="20%" alt="Icon" src="https://meta.alicdn.com/data/mnn/qwen_3.gif" style="margin: 0 10px;"> </p> - [2025/02/18] iOS multimodal LLM App is released [MNN LLM iOS](./apps/iOS/MNNLLMChat/README.md). <p align="center"> <img width="20%" alt="Icon" src="./apps/iOS/MNNLLMChat/assets/introduction.gif" style="margin: 0 10px;"> </p> - [2025/02/11] android app support for [deepseek r1 1.5b](./project/android/apps/MnnLlmApp/README.md#version-021). <p align="center"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/deepseek_support.gif" style="margin: 0 10px;"> </p> - [2025/01/23] We released our full multimodal LLM Android App:[MNN-LLM-Android](./apps/Android/MnnLlmChat/README.md). including text-to-text, image-to-text, audio-to-text, and text-to-image generation. <p align="center"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_home_new.jpg" style="margin: 0 10px;"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_diffusion_new.jpg" style="margin: 0 10px;"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_sound_new.jpg" style="margin: 0 10px;"> <img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_image_new.jpg" style="margin: 0 10px;"> </p> </details> ## Intro MNN is a highly efficient and lightweight deep learning framework. It supports inference and training of deep learning models and has industry-leading performance for inference and training on-device. At present, MNN has been integrated into more than 30 apps of Alibaba Inc, such as Taobao, Tmall, Youku, DingTalk, Xianyu, etc., covering more than 70 usage scenarios such as live broadcast, short video capture, search recommendation, product searching by image, interactive marketing, equity distribution, security risk control. In addition, MNN is also used on embedded devices, such as IoT. [MNN-LLM](./transformers/README.md) is a large language model runtime solution developed based on the MNN engine. The mission of this project is to deploy LLM models locally on everyone's platforms(Mobile Phone/PC/IOT). It supports popular large language models such as Qianwen, Baichuan, Zhipu, LLAMA, and others. [MNN-LLM User guide](https://mnn-docs.readthedocs.io/en/latest/transformers/llm.html) [MNN-Diffusion](https://github.com/alibaba/MNN/tree/master/transformers/diffusion) is a stable diffusion model runtime solution developed based on the MNN engine. The mission of this project is to deploy stable diffusion models locally on everyone's platforms. [MNN-Diffusion User guide](https://mnn-docs.readthedocs.io/en/latest/transformers/diffusion.html)  Inside Alibaba, [MNN](https://mp.weixin.qq.com/s/5I1ISpx8lQqvCS8tGd6EJw) works as the basic module of the compute container in the [Walle](https://mp.weixin.qq.com/s/qpeCETty0BqqNJV9CMJafA) System, the first end-to-end, general-purpose, and large-scale production system for device-cloud collaborative machine learning, which has been published in the top system conference OSDI’22. The key design principles of MNN and the extensive benchmark testing results (vs. TensorFlow, TensorFlow Lite, PyTorch, PyTorch Mobile, TVM) can be found in the OSDI paper. The scripts and instructions for benchmark testing are put in the path “/benchmark”. If MNN or the design of Walle helps your research or production use, please cite our OSDI paper as follows: @inproceedings {proc:osdi22:walle, author = {Chengfei Lv and Chaoyue Niu and Renjie Gu and Xiaotang Jiang and Zhaode Wang and Bin Liu and Ziqi Wu and Qiulin Yao and Congyu Huang and Panos Huang and Tao Huang and Hui Shu and Jinde Song and Bin Zou and Peng Lan and Guohuan Xu and Fei Wu and Shaojie Tang and Fan Wu and Guihai Chen}, title = {Walle: An {End-to-End}, {General-Purpose}, and {Large-Scale} Production System for {Device-Cloud} Collaborative Machine Learning}, booktitle = {16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)}, year = {2022}, isbn = {978-1-939133-28-1}, address = {Carlsbad, CA}, pages = {249--265}, url = {https://www.usenix.org/conference/osdi22/presentation/lv}, publisher = {USENIX Association}, month = jul, } ## Documentation and Workbench MNN's docs are in place in [Read the docs](https://mnn-docs.readthedocs.io/en/latest). You can also read docs/README to build docs's html. MNN Workbench could be downloaded from [MNN's homepage](http://www.mnn.zone), which provides pretrained models, visualized training tools, and one-click deployment of models to devices. ## Key Features ### Lightweight - Optimized for devices, no dependencies, can be easily deployed to mobile devices and a variety of embedded devices. - iOS platform: static library size will full option for armv7+arm64 platforms is about 12MB, size increase of linked executables is about 2M. - Android platform: core so size is about 800KB (armv7a - c++_shared). - Using MNN_BUILD_MINI can reduce package size by about 25%, with a limit of fixed model input size - Support FP16 / Int8 quantize, can reduce model size 50%-70% ### Versatility - Supports `Tensorflow`, `Caffe`, `ONNX`,`Torchscripts` and supports common neural networks such as `CNN`, `RNN`, `GAN`, `Transformer`. - Supports AI model with multi-inputs or multi-outputs, every kind of dimension format, dynamic inputs, controlflow. - MNN supports approximate full OPs used for the AI Model. The converter supports 178 `Tensorflow` OPs, 52 `Caffe` OPs, 163 `Torchscripts` OPs, 158 `ONNX` OPs. - Supports iOS 8.0+, Android 4.3+, and embedded devices with POSIX interface. - Supports hybrid computing on multiple devices. Currently supports CPU and GPU. ### High performance - Implements core computing with lots of optimized assembly code to make full use of the ARM / x64 CPU. - Use Metal / OpenCL / Vulkan to support GPU inference on mobile. - Use CUDA and tensorcore to support NVIDIA GPU for better performance - Convolution and transposition convolution algorithms are efficient and stable. The Winograd convolution algorithm is widely used to better symmetric convolutions such as 3x3,4x4,5x5,6x6,7x7. - Twice speed increase for the new architecture ARM v8.2 with FP16 half-precision calculation support. 2.5 faster to use sdot for ARM v8.2 and VNNI. ### Ease of use - Support use MNN's OP to do numerical calculating like numpy. - Support lightweight image process module like OpenCV, which is only 100k. - Support build model and train it on PC / mobile. - MNN Python API helps ML engineers to easily use MNN to infer, train, and process images, without dipping their toes in C++ code. The Architecture / Precision MNN supported is shown below: - S ：Support and work well, deeply optimized, recommend to use - A ：Support and work well, can use - B ：Support but has bug or not optimized, no recommend to use - C ：Not Support | Architecture / Precision | | Normal | FP16 | BF16 | Int8 | | --- | --- | --- | --- | --- | --- | | CPU | Native | B | C | B | B | | | x86/x64-SSE4.1 | A | C | C | A | | | x86/x64-AVX2 | S | C | C | A | | | x86/x64-AVX512 | S | C | C | S | | | ARMv7a | S | S (ARMv8.2) | S | S | | | ARMv8 | S | S (ARMv8.2) | S(ARMv8.6) | S | | GPU | OpenCL | A | S | C | S | | | Vulkan | A | A | C | A | | | Metal | A | S | C | S | | | CUDA | A | S | C | A | | NPU | CoreML | A | C | C | C | | | HIAI | A | C | C | C | | | NNAPI | B | B | C | B | | | QNN | C | B | C | C | ## Tools Base on MNN (Tensor compute engine), we provided a series of tools for inference, train and general computation. - MNN-Converter: Convert other models to MNN models for inference, such as Tensorflow(lite), Caffe, ONNX, Torchscripts. And do graph optimization to reduce computation. - MNN-Compress: Compress model to reduce size and increase performance / speed - MNN-Express: Support model with controlflow, use MNN's OP to do general-purpose computing. - MNN-CV: An OpenCV-like library, but based on MNN and then much more lightweight. - MNN-Train: Support train MNN model. ## How to Discuss and Get Help From the MNN Community The group discussions are predominantly Chinese. But we welcome and will help English speakers. Dingtalk discussion groups: Group #4 (Available): 160170007549 Group #3 (Full) Group #2 (Full): 23350225 Group #1 (Full): 23329087 ## Historical Paper The preliminary version of MNN, as mobile inference engine and with the focus on manual optimization, has also been published in MLSys 2020. Please cite the paper, if MNN previously helped your research: @inproceedings{alibaba2020mnn, author = {Jiang, Xiaotang and Wang, Huan and Chen, Yiliu and Wu, Ziqi and Wang, Lichuan and Zou, Bin and Yang, Yafeng and Cui, Zongyang and Cai, Yu and Yu, Tianhang and Lv, Chengfei and Wu, Zhihua}, title = {MNN: A Universal and Efficient Inference Engine}, booktitle = {MLSys}, year = {2020} } ## License Apache 2.0 ## Acknowledgement MNN participants: Taobao Technology Department, Search Engineering Team, DAMO Team, Youku and other Alibaba Group employees. MNN refers to the following projects: - [Caffe](https://github.com/BVLC/caffe) - [flatbuffer](https://github.com/google/flatbuffers) - [gemmlowp](https://github.com/google/gemmlowp) - [Google Vulkan demo](http://www.github.com/googlesamples/android-vulkan-tutorials) - [Halide](https://github.com/halide/Halide) - [Mace](https://github.com/XiaoMi/mace) - [ONNX](https://github.com/onnx/onnx) - [protobuffer](https://github.com/protocolbuffers/protobuf) - [skia](https://github.com/google/skia) - [Tensorflow](https://github.com/tensorflow/tensorflow) - [ncnn](https://github.com/Tencent/ncnn) - [paddle-mobile](https://github.com/PaddlePaddle/paddle-mobile) - [stb](https://github.com/nothings/stb) - [rapidjson](https://github.com/Tencent/rapidjson) - [pybind11](https://github.com/pybind/pybind11) - [pytorch](https://github.com/pytorch/pytorch) - [bolt](https://github.com/huawei-noah/bolt) - [libyuv](https://chromium.googlesource.com/libyuv/libyuv) - [libjpeg](https://github.com/libjpeg-turbo/libjpeg-turbo) - [opencv](https://github.com/opencv/opencv) - [onnxruntime](https://github.com/microsoft/onnxruntime)

AI & Machine Learning ML Frameworks

15.5K Github Stars

Read more

Open Source

EasyNLP

<p align="center"> <br> <img src="https://cdn.nlark.com/yuque/0/2022/png/2480469/1649317417481-d20971cd-cd4f-4e29-8587-c342a128b762.png" width="200"/> <br> <p> <p align="center"> <b> EasyNLP is a Comprehensive and Easy-to-use NLP Toolkit </b> </p> <div align="center"> [](https://www.yuque.com/easyx/easynlp/iobg30) [](https://dsw-dev.data.aliyun.com/#/?fileUrl=https://raw.githubusercontent.com/alibaba/EasyTransfer/master/examples/easytransfer-quick_start.ipynb&fileName=easytransfer-quick_start.ipynb) [](https://github.com/alibaba/EasyNLP/issues) [](https://GitHub.com/alibaba/EasyNLP/pull/) [](https://GitHub.com/alibaba/EasyNLP/commit/) [](http://makeapullrequest.com) </div> # EasyNLP [中文介绍](https://github.com/alibaba/EasyNLP/blob/master/README.cn.md) EasyNLP is an easy-to-use NLP development and application toolkit in PyTorch, first released inside Alibaba in 2021. It is built with scalable distributed training strategies and supports a comprehensive suite of NLP algorithms for various NLP applications. EasyNLP integrates knowledge distillation and few-shot learning for landing large pre-trained models, together with various popular multi-modality pre-trained models. It provides a unified framework of model training, inference, and deployment for real-world applications. It has powered more than 10 BUs and more than 20 business scenarios within the Alibaba group. It is seamlessly integrated to [Platform of AI (PAI)](https://www.aliyun.com/product/bigdata/product/learn) products, including PAI-DSW for development, PAI-DLC for cloud-native training, PAI-EAS for serving, and PAI-Designer for zero-code model training. # Main Features - **Easy to use and highly customizable:** In addition to providing easy-to-use and concise commands to call cutting-edge models, it also abstracts certain custom modules such as AppZoo and ModelZoo to make it easy to build NLP applications. It is equipped with the PAI PyTorch distributed training framework TorchAccelerator to speed up distributed training. - **Compatible with open-source libraries:** EasyNLP has APIs to support the training of models from Huggingface/Transformers with the PAI distributed framework. It also supports the pre-trained models in [EasyTransfer](https://github.com/alibaba/EasyTransfer) ModelZoo. - **Knowledge-injected pre-training:** The PAI team has a lot of research on knowledge-injected pre-training, and builds a knowledge-injected model that wins first place in the CCF knowledge pre-training competition. EasyNLP integrates these cutting-edge knowledge pre-trained models, including DKPLM and KGBERT. - **Landing large pre-trained models:** EasyNLP provides few-shot learning capabilities, allowing users to finetune large models with only a few samples to achieve good results. At the same time, it provides knowledge distillation functions to help quickly distill large models to a small and efficient model to facilitate online deployment. - **Multi-modality pre-trained models:** EasyNLP is not about NLP only. It also supports various popular multi-modality pre-trained models to support vision-language tasks that require visual knowledge. For example, it is equipped with CLIP-style models for text-image matching and DALLE-style models for text-to-image generation. # Technical Articles We have a series of technical articles on the functionalities of EasyNLP. - [BeautifulPrompt：PAI推出自研Prompt美化器，赋能AIGC一键出美图](https://zhuanlan.zhihu.com/p/636546340) - [PAI-Diffusion中文模型全面升级，海量高清艺术大图一键生成](https://zhuanlan.zhihu.com/p/632031092) - [EasyNLP集成K-Global Pointer算法，支持中文信息抽取](https://zhuanlan.zhihu.com/p/608560954) - [阿里云PAI-Diffusion功能再升级，全链路支持模型调优，平均推理速度提升75%以上](https://zhuanlan.zhihu.com/p/604483551) - [PAI-Diffusion模型来了！阿里云机器学习团队带您徜徉中文艺术海洋](https://zhuanlan.zhihu.com/p/590020134) - [模型精度再被提升，统一跨任务小样本学习算法 UPT 给出解法!](https://zhuanlan.zhihu.com/p/590611518) - [Span抽取和元学习能碰撞出怎样的新火花，小样本实体识别来告诉你!](https://zhuanlan.zhihu.com/p/590297824) - [算法 KECP 被顶会 EMNLP 收录，极少训练数据就能实现机器阅读理解](https://zhuanlan.zhihu.com/p/590024650) - [当大火的文图生成模型遇见知识图谱，AI画像趋近于真实世界](https://zhuanlan.zhihu.com/p/581870071) - [EasyNLP发布融合语言学和事实知识的中文预训练模型CKBERT](https://zhuanlan.zhihu.com/p/574853281) - [EasyNLP带你实现中英文机器阅读理解](https://zhuanlan.zhihu.com/p/568890245) - [跨模态学习能力再升级，EasyNLP电商文图检索效果刷新SOTA](https://zhuanlan.zhihu.com/p/568512230) - [EasyNLP玩转文本摘要（新闻标题）生成](https://zhuanlan.zhihu.com/p/566607127) - [中文稀疏GPT大模型落地 — 通往低成本&高性能多任务通用自然语言理解的关键里程碑](https://zhuanlan.zhihu.com/p/561320982) - [EasyNLP集成K-BERT算法，借助知识图谱实现更优Finetune](https://zhuanlan.zhihu.com/p/553816104) - [EasyNLP中文文图生成模型带你秒变艺术家](https://zhuanlan.zhihu.com/p/547063102) - [面向长代码序列的Transformer模型优化方法，提升长代码场景性能](https://zhuanlan.zhihu.com/p/540060701) - [EasyNLP带你玩转CLIP图文检索](https://zhuanlan.zhihu.com/p/528476134) - [阿里云机器学习PAI开源中文NLP算法框架EasyNLP，助力NLP大模型落地](https://zhuanlan.zhihu.com/p/505785399) - [预训练知识度量比赛夺冠！阿里云PAI发布知识预训练工具](https://zhuanlan.zhihu.com/p/449487792) # Installation You can setup from the source： ```bash $ git clone https://github.com/alibaba/EasyNLP.git $ cd EasyNLP $ python setup.py install ``` This repo is tested on Python 3.6, PyTorch >= 1.8. # Quick Start Now let's show how to use just a few lines of code to build a text classification model based on BERT. ```python from easynlp.appzoo import ClassificationDataset from easynlp.appzoo import get_application_model, get_application_evaluator from easynlp.core import Trainer from easynlp.utils import initialize_easynlp, get_args from easynlp.utils.global_vars import parse_user_defined_parameters from easynlp.utils import get_pretrain_model_path initialize_easynlp() args = get_args() user_defined_parameters = parse_user_defined_parameters(args.user_defined_parameters) pretrained_model_name_or_path = get_pretrain_model_path(user_defined_parameters.get('pretrain_model_name_or_path', None)) train_dataset = ClassificationDataset( pretrained_model_name_or_path=pretrained_model_name_or_path, data_file=args.tables.split(",")[0], max_seq_length=args.sequence_length, input_schema=args.input_schema, first_sequence=args.first_sequence, second_sequence=args.second_sequence, label_name=args.label_name, label_enumerate_values=args.label_enumerate_values, user_defined_parameters=user_defined_parameters, is_training=True) valid_dataset = ClassificationDataset( pretrained_model_name_or_path=pretrained_model_name_or_path, data_file=args.tables.split(",")[-1], max_seq_length=args.sequence_length, input_schema=args.input_schema, first_sequence=args.first_sequence, second_sequence=args.second_sequence, label_name=args.label_name, label_enumerate_values=args.label_enumerate_values, user_defined_parameters=user_defined_parameters, is_training=False) model = get_application_model(app_name=args.app_name, pretrained_model_name_or_path=pretrained_model_name_or_path, num_labels=len(valid_dataset.label_enumerate_values), user_defined_parameters=user_defined_parameters) trainer = Trainer(model=model, train_dataset=train_dataset,user_defined_parameters=user_defined_parameters, evaluator=get_application_evaluator(app_name=args.app_name, valid_dataset=valid_dataset,user_defined_parameters=user_defined_parameters, eval_batch_size=args.micro_batch_size)) trainer.train() ``` The complete example can be found [here](https://github.com/alibaba/EasyNLP/blob/master/examples/appzoo_tutorials/sequence_classification/bert_classify/run_train_eval_predict_user_defined_local.sh). You can also use AppZoo Command Line Tools to quickly train an App model. Take text classification on SST-2 dataset as an example. First you can download the [train.tsv](http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/classification/train.tsv), and [dev.tsv](http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/classification/dev.tsv), then start training: ```bash $ easynlp \ --mode=train \ --worker_gpu=1 \ --tables=train.tsv,dev.tsv \ --input_schema=label:str:1,sid1:str:1,sid2:str:1,sent1:str:1,sent2:str:1 \ --first_sequence=sent1 \ --label_name=label \ --label_enumerate_values=0,1 \ --checkpoint_dir=./classification_model \ --epoch_num=1 \ --sequence_length=128 \ --app_name=text_classify \ --user_defined_parameters='pretrain_model_name_or_path=bert-small-uncased' ``` And then predict: ```bash $ easynlp \ --mode=predict \ --tables=dev.tsv \ --outputs=dev.pred.tsv \ --input_schema=label:str:1,sid1:str:1,sid2:str:1,sent1:str:1,sent2:str:1 \ --output_schema=predictions,probabilities,logits,output \ --append_cols=label \ --first_sequence=sent1 \ --checkpoint_path=./classification_model \ --app_name=text_classify ``` To learn more about the usage of AppZoo, please refer to our [documentation](https://www.yuque.com/easyx/easynlp/kkhkai). # ModelZoo EasyNLP currently provides the following models in ModelZoo: 1. PAI-BERT-zh (from Alibaba PAI): pre-trained BERT models with a large Chinese corpus. 2. DKPLM (from Alibaba PAI): released with the paper [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://arxiv.org/pdf/2112.01047.pdf) by Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang, Xiaofeng He and Jun Huang. 3. KGBERT (from Alibaba Damo Academy & PAI): pre-train BERT models with knowledge graph embeddings injected. 4. BERT (from Google): released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://aclanthology.org/N19-1423.pdf) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 5. RoBERTa (from Facebook): released with the paper [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/pdf/1907.11692.pdf) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. 6. Chinese RoBERTa (from HFL): the Chinese version of RoBERTa. 7. MacBERT (from HFL): released with the paper [Revisiting Pre-trained Models for Chinese Natural Language Processing](https://aclanthology.org/2020.findings-emnlp.58.pdf) by Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang and Guoping Hu. 8. WOBERT (from ZhuiyiTechnology): the word-based BERT for the Chinese language. 9. FashionBERT (from Alibaba PAI & ICBU): in progress. 10. GEEP (from Alibaba PAI): in progress. 11. Mengzi (from Langboat): released with the paper [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://arxiv.org/pdf/2110.06696.pdf) by Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang and Ming Zhou. 12. Erlangshen (from IDEA): released from the [repo](https://github.com/IDEA-CCNL/Fengshenbang-LM). Please refer to this [readme](https://github.com/alibaba/EasyNLP/blob/master/easynlp/modelzoo/README.md) for the usage of these models in EasyNLP. Meanwhile, EasyNLP supports to load pretrained models from Huggingface/Transformers, please refer to [this tutorial](https://www.yuque.com/easyx/easynlp/qmq8wh) for details. # EasyNLP Goes Multi-modal EasyNLP also supports various popular multi-modality pre-trained models to support vision-language tasks that require visual knowledge. For example, it is equipped with CLIP-style models for text-image matching and DALLE-style models for text-to-image generation. 1. [Text-image Matching](https://github.com/alibaba/EasyNLP/blob/master/examples/clip_retrieval/run_clip_local.sh) 2. [Text-to-image Generation](https://github.com/alibaba/EasyNLP/blob/master/examples/text2image_generation/run_appzoo_cli_local.sh) 3. [Image-to-text Generation](https://github.com/alibaba/EasyNLP/blob/master/examples/image2text_generation/run_appzoo_cli_local_clip.sh) # Landing Large Pre-trained Models EasyNLP provide few-shot learning and knowledge distillation to help land large pre-trained models. 1. [PET](https://github.com/alibaba/EasyNLP/blob/master/examples/fewshot_learning/run_fewshot_pet.sh) (from LMU Munich and Sulzer GmbH): released with the paper [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://aclanthology.org/2021.eacl-main.20.pdf) by Timo Schick and Hinrich Schutze. We have made some slight modifications to make the algorithm suitable for the Chinese language. 2. [P-Tuning](https://github.com/alibaba/EasyNLP/blob/master/examples/fewshot_learning/run_fewshot_ptuning.sh) (from Tsinghua University, Beijing Academy of AI, MIT and Recurrent AI, Ltd.): released with the paper [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf) by Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang and Jie Tang. We have made some slight modifications to make the algorithm suitable for the Chinese language. 3. [CP-Tuning](https://github.com/alibaba/EasyNLP/blob/master/examples/fewshot_learning/run_fewshot_cpt.sh) (from Alibaba PAI): released with the paper [Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning](https://arxiv.org/pdf/2204.00166.pdf) by Ziyun Xu, Chengyu Wang, Minghui Qiu, Fuli Luo, Runxin Xu, Songfang Huang and Jun Huang. 4. [Vanilla KD](https://github.com/alibaba/EasyNLP/tree/master/examples/knowledge_distillation) (from Alibaba PAI): distilling the logits of large BERT-style models to smaller ones. 5. [Meta KD](https://github.com/alibaba/EasyNLP/tree/master/examples/knowledge_distillation) (from Alibaba PAI): released with the paper [Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains](https://aclanthology.org/2021.acl-long.236.pdf) by Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li and Jun Huang. 6. [Data Augmentation](https://github.com/alibaba/EasyNLP/tree/master/examples/knowledge_distillation/test_data_aug.sh) (from Alibaba PAI): augmentating the data based on the MLM head of pre-trained language models. # [CLUE Benchmark](https://www.cluebenchmarks.com/) EasyNLP provides [a simple toolkit](https://github.com/alibaba/EasyNLP/tree/master/benchmarks/clue) to benchmark clue datasets. You can simply use just this command to benchmark CLUE dataset. ```bash # Format: bash run_clue.sh device_id train/predict dataset # e.g.: bash run_clue.sh 0 train csl ``` We've tested chiese bert and roberta modelson the datasets, the results of dev set are: (1) bert-base-chinese: | Task | AFQMC | CMNLI | CSL | IFLYTEK | OCNLI | TNEWS | WSC | |------|--------|--------|--------|---------|--------|--------|--------| | P | 72.17% | 75.74% | 80.93% | 60.22% | 78.31% | 57.52% | 75.33% | | F1 | 52.96% | 75.74% | 81.71% | 60.22% | 78.30% | 57.52% | 80.82% | (2) chinese-roberta-wwm-ext: | Task | AFQMC | CMNLI | CSL | IFLYTEK | OCNLI | TNEWS | WSC | |------|--------|--------|--------|---------|--------|--------|--------| | P | 73.10% | 80.75% | 80.07% | 60.98% | 80.75% | 57.93% | 86.84% | | F1 | 56.04% | 80.75% | 81.50% | 60.98% | 80.75% | 57.93% | 89.58% | Here is the detailed [CLUE benchmark example](https://github.com/alibaba/EasyNLP/tree/master/benchmarks/clue). # Tutorials - [自定义文本分类示例](https://www.yuque.com/easyx/easynlp/ds35qn) - [QuickStart-文本分类](https://www.yuque.com/easyx/easynlp/rxne07) - [QuickStart-PAI DSW](https://www.yuque.com/easyx/easynlp/gvat1o) - [QuickStart-MaxCompute/ODPS数据](https://www.yuque.com/easyx/easynlp/vgwe7f) - [AppZoo-文本向量化](https://www.yuque.com/easyx/easynlp/ts4czl) - [AppZoo-文本分类/匹配](https://www.yuque.com/easyx/easynlp/vgbopy) - [AppZoo-序列标注](https://www.yuque.com/easyx/easynlp/qkwqmb) - [AppZoo-GEEP文本分类](https://www.yuque.com/easyx/easynlp/lepm0q) - [AppZoo-文本生成](https://www.yuque.com/easyx/easynlp/svde6x) - [基础预训练实践](https://www.yuque.com/easyx/easynlp/lm1a5t) - [知识预训练实践](https://www.yuque.com/easyx/easynlp/za7ywp) - [知识蒸馏实践](https://www.yuque.com/easyx/easynlp/ffu6p9) - [跨任务知识蒸馏实践](https://www.yuque.com/easyx/easynlp/izbfqt) - [小样本学习实践](https://www.yuque.com/easyx/easynlp/ochmnf) - [Rapidformer模型训练加速实践](https://www.yuque.com/easyx/easynlp/bi6nzc) - API docs: [http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/easynlp/easynlp_docs/html/index.html](http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/easynlp/easynlp_docs/html/index.html) # License This project is licensed under the [Apache License (Version 2.0)](https://github.com/alibaba/EasyNLP/blob/master/LICENSE). This toolkit also contains some code modified from other repos under other open-source licenses. See the [NOTICE](https://github.com/alibaba/EasyNLP/blob/master/NOTICE) file for more information. # ChangeLog - EasyNLP v0.0.3 was released in 01/04/2022. Please refer to [tag_v0.0.3](https://github.com/alibaba/EasyNLP/releases/tag/v0.0.3) for more details and history. # Contact Us Scan the following QR codes to join Dingtalk discussion group. The group discussions are mostly in Chinese, but English is also welcomed. <img src="https://cdn.nlark.com/yuque/0/2022/png/2480469/1649324662278-fe178523-6b14-4eff-8f50-7abbf468f751.png?x-oss-process=image%2Fresize%2Cw_357%2Climit_0" width="300"/> # Reference - DKPLM: https://paperswithcode.com/paper/dkplm-decomposable-knowledge-enhanced-pre - MetaKD: https://paperswithcode.com/paper/meta-kd-a-meta-knowledge-distillation - CP-Tuning: https://paperswithcode.com/paper/making-pre-trained-language-models-end-to-end-1 - FashionBERT: https://paperswithcode.com/paper/fashionbert-text-and-image-matching-with We have [an arxiv paper](https://paperswithcode.com/paper/easynlp-a-comprehensive-and-easy-to-use) for you to cite for the EasyNLP library: ``` @article{easynlp, doi = {10.48550/ARXIV.2205.00258}, url = {https://arxiv.org/abs/2205.00258}, author = {Wang, Chengyu and Qiu, Minghui and Zhang, Taolin and Liu, Tingting and Li, Lei and Wang, Jianing and Wang, Ming and Huang, Jun and Lin, Wei}, title = {EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing}, publisher = {arXiv}, year = {2022} } ```

ML Frameworks

2.2K Github Stars

Read more

Open Source

AliceMind

# AliceMind #### AliceMind: ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab This repository provides pre-trained encoder-decoder models and its related optimization techniques developed by Alibaba's MinD (Machine IntelligeNce of Damo) Lab. The family of AliceMind: * Pre-trained Models: * Release the first multimodal large language model for enhancing LLM and MLLM through modal collaboration: [mPLUG-Owl2](https://github.com/X-PLUG/mPLUG-Owl/tree/main/mPLUG-Owl2)(```CVPR 2024```) * Release the first ocr-free multimodal large language model for universal document understanding: [mPLUG-DocOwl](https://github.com/X-PLUG/mPLUG-DocOwl)(```EMNLP 2023```) * Release the first and largest public Chinese Video-language pretraining dataset and benchmarks: [Youku-mPLUG](https://github.com/X-PLUG/Youku-mPLUG), and the Chinese video large language model named [mPLUG-video](https://github.com/X-PLUG/Youku-mPLUG) * A new training paradigm with a **modularized design** for large multi-modal language models: [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl) * Large-scale Chinese open-domain dialogue system for digital human: [ChatPLUG](https://github.com/X-PLUG/ChatPLUG) * A Modularized Multi-modal Foundation Model Across Text, Image and Video: [mPLUG-2](https://github.com/X-PLUG/mPLUG-2)(```ICML 2023```) * Large-scale vision-language understanding and generation model: [mPLUG](https://github.com/alibaba/AliceMind/tree/main/mPLUG)(```EMNLP 2022```) * Large-scale chinese understanding and generation model: [PLUG](https://github.com/alibaba/AliceMind/tree/main/PLUG) * Pre-training table model: [SDCUP](https://github.com/alibaba/AliceMind/tree/main/SDCUP) (```Under Review```) * Chinese language understanding model with multi-granularity inputs: [LatticeBERT](https://github.com/alibaba/AliceMind/tree/main/LatticeBERT) (```NAACL 2021```) * Structural language model: [StructuralLM](https://github.com/alibaba/AliceMind/tree/main/StructuralLM) (```ACL 2021```) * Cross-modal language model: [StructVBERT](https://github.com/alibaba/AliceMind/tree/main/StructVBERT) (```CVPR 2020 VQA Challenge Runner-up```) * Cross-lingual language model: [VECO](https://github.com/alibaba/AliceMind/tree/main/VECO) (```ACL 2021```) * Generative language model: [PALM](https://github.com/alibaba/AliceMind/tree/main/PALM) (```EMNLP 2020```) * Language understanding model: [StructBERT](https://github.com/alibaba/AliceMind/tree/main/StructBERT) (```ICLR 2020```) * Fine-tuning Methods: * Parameter-Efficient Sparsity methods [PST](https://github.com/alibaba/AliceMind/tree/main/S4/PST) (```IJCAI 2022```) * Effective and generalizable fine-tuning method [ChildTuning](https://github.com/alibaba/AliceMind/tree/main/ChildTuning) (```EMNLP 2021```) * Model Compression: * Language model compression methods [ContrastivePruning](https://github.com/alibaba/AliceMind/tree/main/ContrastivePruning) (```AAAI 2022```) ## News - **November 9, 2023: [mPLUG-Owl2](https://github.com/X-PLUG/mPLUG-Owl/tree/main/mPLUG-Owl2), the first multimodal large language model for enhancing LLM and MLLM through modal collaboration, were accepted by CVPR 2024.** - **July 7, 2023: [mPLUG-DocOwl](https://github.com/X-PLUG/mPLUG-DocOwl), the first ocr-free multimodal large language model for universal document understanding, were accepted by EMNLP 2023.** - **June 8, 2023: [Youku-mPLUG](https://github.com/X-PLUG/Youku-mPLUG), release the first and largest public Chinese Video-language pretraining dataset and benchmarks, and the Chinese video large language model named [mPLUG-video](https://github.com/X-PLUG/Youku-mPLUG).** - **April 27, 2023: [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl), a new training paradigm with a modularized design for large multi-modal language models released.** - **April 25, 2023: [mPLUG-2](https://github.com/X-PLUG/mPLUG-2) were accepted by ICML 2023.** - **April 16, 2023: [ChatPLUG](https://github.com/X-PLUG/ChatPLUG), the Chinese open-domain dialogue system for digital human applications released.** - October, 2022: [mPLUG](https://arxiv.org/abs/2205.12005) were accepted by EMNLP 2022. - May, 2022: [PST](https://arxiv.org/abs/2205.11005) were accepted by IJCAI 2022. - April, 2022: The SOFA modeling toolkit released which supports models&techs standard code and the direct use of them in transformers! - December, 2021: [ContrastivePruning](https://github.com/alibaba/AliceMind/tree/main/ContrastivePruning) were accepted by AAAI 2022. - October, 2021: [ChildTuning](https://arxiv.org/abs/2109.05687) were accepted by EMNLP 2021. - September, 2021: The first Chinese pre-training table model [SDCUP](https://arxiv.org/abs/2103.04399) released! - May, 2021: [VECO](https://arxiv.org/abs/2010.16046) and [StructuralLM](https://arxiv.org/abs/2105.11210) were accepted by ACL 2021. - March, 2021: AliceMind released! ## Pre-trained Models - [**mPLUG-Owl**](https://github.com/X-PLUG/mPLUG-Owl) (April 27, 2023): a new training paradigm with a modularized design for large multi-modal language models. Learns visual knowledge while support multi-turn conversation consisting of different modalities. Observed abilities such as multi-image correlation and scene text understanding, vision-based document comprehension. Release a visually-related instruction evaluation set OwlEval. [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://arxiv.org/abs/2304.14178) - [**ChatPLUG**](https://github.com/X-PLUG/ChatPLUG) (April 16, 2023): a Chinese open-domain dialogue system for digital human applications that instruction finetunes on a wide range of dialogue tasks in a unified internet-augmented format. Different from other open-domain dialogue models that focus on large-scale pre-training and scaling up model size or dialogue corpus, we aim to build a powerful and practical dialogue system for digital human with diverse skills and good multi-task generalization by internet-augmented instruction tuning. [ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human](https://arxiv.org/abs/2304.07849) - [**mPLUG**](mPLUG) (September 1, 2022): large-scale pre-trained model for vision-language **understanding and generation**. mPLUG is pre-trained end-to-end on large scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image-captioning, image-text retrieval, visual grounding and visual question answering. [mPLUG: Effective Multi-Modal Learning by Cross-Modal Skip Connections](https://arxiv.org/abs/2205.12005)(```EMNLP 2022```) - [**PLUG**](PLUG) (September 1, 2022): large-scale chinese pre-trained model for **understanding and generation**. PLUG (27B) is a large-scale chinese pre-training model for language understanding and generation. The training of PLUG is two-stage, the first stage is a 24-layer [StructBERT](https://arxiv.org/abs/1908.04577) encoder, and the second stage is a 24-6-layer [PALM](https://arxiv.org/pdf/2004.07159.pdf?fbclid=IwAR0BNl1IzR5bhcuEbyfNw2UN7MApHFoFP3BN40FKkW8x3bqolK_HilU293I) encoder-decoder. - [**SDCUP**](SDCUP) (September 6, 2021): pre-trained models for **table understanding**. We design a schema dependency pre-training objective to impose the desired inductive bias into the learned representations for table pre-training. We further propose a schema-aware curriculum learning approach to alleviate the impact of noise and learn effectively from the pre-training data in an easy-to-hard manner. The experiment results on SQUALL and Spider demonstrate the effectiveness of our pre-training objective and curriculum in comparison to a variety of baselines. "[SDCUP: Schema Dependency Enhanced Curriculum Pre-Training for Table Semantic Parsing]()" (```Under Review```) - [**LatticeBERT**](LatticeBERT) (March 15, 2021): we propose a novel pre-training paradigm for Chinese — Lattice-BERT which explicitly incorporates word representations with those of characters, thus can model a sentence in a multi-granularity manner. "[Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://arxiv.org/abs/2104.07204)" (`NAACL 2021`) - [**StructuralLM**](StructuralLM) (March 15, 2021): pre-trained models for **document-image understanding**. We propose a new pre-training approach, StructuralLM, to jointly leverage cell and layout information from scanned documents. The pre-trained StructuralLM achieves new state-of-the-art results in different types of downstream tasks. "[StructuralLM: Structural Pre-training for Form Understanding](https://arxiv.org/abs/2105.11210)" (```ACL 2021```) - [**StructVBERT**](StructVBERT) (March 15, 2021): pre-trained models for **vision-language understanding**. We propose a new single-stream visual-linguistic pre-training scheme by leveraging multi-stage progressive pre-training and multi-task learning. StructVBERT obtained the 2020 VQA Challenge Runner-up award, and SOTA result on VQA 2020 public Test-standard benchmark (June 2020). "[Talk Slides](StructVBERT/StructVBERT-talk.pdf)" (```CVPR 2020 VQA Challenge Runner-up```). - [**VECO v0**](VECO) (March 15, 2021): pre-trained models for **cross-lingual (x) natural language understanding (x-NLU) and generation (x-NLG)**. VECO (v0) achieves the **new SOTA results** on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1~2 BLEU. “[VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation](https://arxiv.org/abs/2010.16046)" (```ACL 2021```) - [**PALM**](PALM) (March 15, 2021): pre-trained models for **natural language generation (NLG)**. We propose a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. It achieves new SOTA results in several downstream tasks. "[PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation](https://arxiv.org/abs/2004.07159)" (```EMNLP 2020```) - [**StructBERT**](StructBERT) (March 15, 2021): pre-trained models for **natural language understanding (NLU)**. We extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. "[StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://arxiv.org/abs/1908.04577)" (```ICLR 2020```) ## Fine-tuning Methods - [**ChildTuning**](ChildTuning) (October 25, 2021): To mitigate the overfitting problem and improve generalization for fine-tuning large-scale PLMs, we propose a **straightforward yet effective fine-tuning technique**, ChildTuning, which only updates the child network during fine-tuning via strategically masking out the gradients of the non-child network. “[Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://arxiv.org/abs/2109.05687)" (```EMNLP 2021```) ## Model Compression - [**ContrastivePruning**](ContrastivePruning) (December 17, 2021): ContrAstive Pruning (CAP) is a **general pruning framework under the pre-training and fine-tuning paradigm**, which aims at maintaining both task-specific and task-agnostic knowledge during pruning. CAP is designed as a general framework, compatible with both structured and unstructured pruning. Unified in contrastive learning, CAP encourage the pruned model to learn from the pre-trained model, the snapshots (intermediate models during pruning), and the fine-tuned model, respectively. “[From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression](https://arxiv.org/abs/2112.07198)" (```AAAI 2022```) - [**PST**](S4/PST) (May 23, 2022): Parameter-efficient Sparse Training (PST) is to reduce the number of trainable parameters during sparse-aware training in downstream tasks. It combines the data-free and data-driven criteria to efficiently and accurately measures the importance of weights, and investigates the intrinsic redundancy of data-driven weight importance and derive two obvious characteristics i.e., low-rankness and structuredness, which therefore makes the sparse training resource-efficient and parameter-efficient. “[Parameter-Efficient Sparsity for Large Language Models Fine-Tuning](https://arxiv.org/abs/2205.11005)" (```IJCAI 2022```) ## Modeling toolkit - [**SOFA**](SOFA/README.md) SOFA aims to faciliate easy use and distribution of the pretrained language models from Alibaba DAMO Academy AliceMind project. In addition, detail examples in the project make it simple for any end-user to access those models. ## Contact Information **AliceMind Official Website**: [https://nlp.aliyun.com/portal#/alice](https://nlp.aliyun.com/portal#/alice) **AliceMind Open Platform**: [https://alicemind.aliyuncs.com](https://alicemind.aliyuncs.com/#/home) Please submit a GitHub issue if you have want help or have issues using ALICE. For more information, you can join the ``AliceMind Users Group`` on DingTalk to contact us. The number of the DingTalk group is 35738533. For other business communications, please contact [email protected] ## *License* AliceMind is released under the [Apache 2.0 license](LICENSE). ``` Copyright 1999-2020 Alibaba Group Holding Ltd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at the following link. http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ```

ML Frameworks

2K Github Stars

Read more

Open Source

zvec

<p align="right"> English | <a href="./README_CN.md">中文</a> </p> <div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://zvec.oss-cn-hongkong.aliyuncs.com/logo/github_log_2.svg" /> <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/logo/github_logo_1.svg" width="400" alt="zvec logo" /> </picture> </div> <p align="center"> <a href="https://codecov.io/github/alibaba/zvec"><img src="https://codecov.io/github/alibaba/zvec/graph/badge.svg?token=O81CT45B66" alt="Code Coverage"/></a> <a href="https://github.com/alibaba/zvec/actions/workflows/01-ci-pipeline.yml"><img src="https://github.com/alibaba/zvec/actions/workflows/01-ci-pipeline.yml/badge.svg?branch=main" alt="Main"/></a> <a href="https://github.com/alibaba/zvec/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"/></a> <a href="https://pypi.org/project/zvec/"><img src="https://img.shields.io/pypi/v/zvec.svg" alt="PyPI Release"/></a> <a href="https://pypi.org/project/zvec/"><img src="https://img.shields.io/badge/python-3.10%20~%203.14-blue.svg" alt="Python Versions"/></a> <a href="https://www.npmjs.com/package/@zvec/zvec"><img src="https://img.shields.io/npm/v/@zvec/zvec.svg" alt="npm Release"/></a> </p> <p align="center"> <a href="https://trendshift.io/repositories/20830" target="_blank"><img src="https://trendshift.io/api/badge/repositories/20830" alt="alibaba%2Fzvec | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> </p> <p align="center"> <a href="https://zvec.org/en/docs/db/quickstart/">🚀 <strong>Quickstart</strong> </a> | <a href="https://zvec.org/en/">🏠 <strong>Home</strong> </a> | <a href="https://zvec.org/en/docs/db/">📚 <strong>Docs</strong> </a> | <a href="https://zvec.org/en/docs/db/benchmarks/">📊 <strong>Benchmarks</strong> </a> | <a href="https://deepwiki.com/alibaba/zvec">🔎 <strong>DeepWiki</strong> </a> | <a href="https://discord.gg/rKddFBBu9z">🎮 <strong>Discord</strong> </a> | <a href="https://x.com/ZvecAI">🐦 <strong>X (Twitter)</strong> </a> </p> **Zvec** is an open-source, in-process vector database — lightweight, lightning-fast, and designed to embed directly into applications. Battle-tested within Alibaba Group, it delivers production-grade, low-latency and scalable similarity search with minimal setup. > [!Important] > 🚀 **v0.4.0 (May 9, 2026)** > > - **Dart/Flutter SDK**: Published the official [zvec](https://github.com/zvec-ai/zvec-dart) Flutter package with FFI bindings. Supports Android (arm64-v8a) and iOS (arm64) — no manual native compilation required. > - **iOS Build Support**: Added support for building on iOS platforms, expanding cross-platform coverage. > - **Enlarged topK Limit**: Relaxed the upper bound on topK to support larger-scale recall scenarios. > - **Bug Fixes**: SQ8 quantizer recall drop; Windows path handling; sparse vector index ordering. > > 👉 [Read the Release Notes](https://github.com/alibaba/zvec/releases/tag/v0.4.0) | [View Roadmap 📍](https://github.com/alibaba/zvec/issues/309) ## 💫 Features - **Blazing Fast**: Searches billions of vectors in milliseconds. - **Simple, Just Works**: [Install](#-installation) and start searching in seconds. Pure local, no servers, no config, no fuss. - **Dense + Sparse Vectors**: Work with both dense and sparse embeddings, with native support for multi-vector queries in a single call. - **Hybrid Search**: Combine semantic similarity with structured filters for precise results. - **Durable Storage**: Write-ahead logging (WAL) guarantees persistence — data is never lost, even on process crash or power failure. - **Concurrent Access**: Multiple processes can read the same collection simultaneously; writes are single-process exclusive. - **Runs Anywhere**: As an in-process library, Zvec runs wherever your code runs — notebooks, servers, CLI tools, or even edge devices. ## 📦 Installation ### [Python](https://pypi.org/project/zvec/) **Requirements**: Python 3.10 - 3.14 ```bash pip install zvec ``` ### [Node.js](https://www.npmjs.com/package/@zvec/zvec) ```bash npm install @zvec/zvec ``` ### ✅ Supported Platforms - Linux (x86_64, ARM64) - macOS (ARM64) - Windows (x86_64) ### 🛠️ Building from Source If you prefer to build Zvec from source, please check the [Building from Source](https://zvec.org/en/docs/db/build/) guide. ## ⚡ One-Minute Example ```python import zvec # Define collection schema schema = zvec.CollectionSchema( name="example", vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4), ) # Create collection collection = zvec.create_and_open(path="./zvec_example", schema=schema) # Insert documents collection.insert([ zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}), zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}), ]) # Search by vector similarity results = collection.query( zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]), topk=10 ) # Results: list of {'id': str, 'score': float, ...}, sorted by relevance print(results) ``` ## 📈 Performance at Scale Zvec delivers exceptional speed and efficiency, making it ideal for demanding production workloads. <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qps_10M.svg" width="800" alt="Zvec Performance Benchmarks" /> For detailed benchmark methodology, configurations, and complete results, please see our [Benchmarks documentation](https://zvec.org/en/docs/db/benchmarks/). ## 🤝 Join Our Community <div align="center"> <div align="center"> | 💬 DingTalk | 📱 WeChat | 🎮 Discord | X (Twitter) | | :---: | :---: | :---: | :---: | | <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/dingding.png" width="150" alt="DingTalk QR Code"/> | <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/wechat.png?v=6" width="150" alt="WeChat QR Code"/> | [](https://discord.gg/rKddFBBu9z) | [](<https://x.com/ZvecAI>) | | Scan to join | Scan to join | Click to join | Click to follow | </div> </div> ## ❤️ Contributing We welcome and appreciate contributions from the community! Whether you're fixing a bug, adding a feature, or improving documentation, your help makes Zvec better for everyone. Check out our [Contributing Guide](./CONTRIBUTING.md) to get started!

Database Vector Databases

9.8K Github Stars

Read more

Open Source

EasyCV

<div align="center"> [](https://pypi.org/project/pai-easycv/) [](https://easy-cv.readthedocs.io/en/latest/) [](https://github.com/open-mmlab/mmdetection/blob/master/LICENSE) [](https://github.com/alibaba/EasyCV/issues) [](https://GitHub.com/alibaba/EasyCV/pull/) [](https://GitHub.com/alibaba/EasyCV/commit/) <!-- [](https://GitHub.com/alibaba/EasyCV/graphs/contributors/) --> <!-- [](http://makeapullrequest.com) --> </div> # EasyCV English | [简体中文](README_zh-CN.md) ## Introduction EasyCV is an all-in-one computer vision toolbox based on PyTorch, mainly focuses on self-supervised learning, transformer based models, and major CV tasks including image classification, metric-learning, object detection, pose estimation, and so on. ### Major features - **SOTA SSL Algorithms** EasyCV provides state-of-the-art algorithms in self-supervised learning based on contrastive learning such as SimCLR, MoCO V2, Swav, DINO, and also MAE based on masked image modeling. We also provide standard benchmarking tools for ssl model evaluation. - **Vision Transformers** EasyCV aims to provide an easy way to use the off-the-shelf SOTA transformer models trained either using supervised learning or self-supervised learning, such as ViT, Swin Transformer, and DETR Series. More models will be added in the future. In addition, we support all the pretrained models from [timm](https://github.com/rwightman/pytorch-image-models). - **Functionality & Extensibility** In addition to SSL, EasyCV also supports image classification, object detection, metric learning, and more areas will be supported in the future. Although covering different areas, EasyCV decomposes the framework into different components such as dataset, model and running hook, making it easy to add new components and combining it with existing modules. EasyCV provides simple and comprehensive interface for inference. Additionally, all models are supported on [PAI-EAS](https://help.aliyun.com/document_detail/113696.html), which can be easily deployed as online service and support automatic scaling and service monitoring. - **Efficiency** EasyCV supports multi-gpu and multi-worker training. EasyCV uses [DALI](https://github.com/NVIDIA/DALI) to accelerate data io and preprocessing process, and uses [TorchAccelerator](https://github.com/alibaba/EasyCV/tree/master/docs/source/tutorials/torchacc.md) and fp16 to accelerate training process. For inference optimization, EasyCV exports model using jit script, which can be optimized by [PAI-Blade](https://help.aliyun.com/document_detail/205134.html) ## What's New [🔥 2023.05.09] * 09/05/2023 EasyCV v0.11.0 was released. - Support EasyCV as a plug-in for [modelscope](https://github.com/modelscope/modelscope. [🔥 2023.03.06] * 06/03/2023 EasyCV v0.10.0 was released. - Add segmentation model STDC - Add skeleton based video recognition model STGCN - Support ReID and Multi-len MOT [🔥 2023.01.17] * 17/01/2023 EasyCV v0.9.0 was released. - Support Single-lens MOT - Support video recognition (X3D, SWIN-video) [🔥 2022.12.02] * 02/12/2022 EasyCV v0.8.0 was released. - bevformer-base NDS increased by 0.8 on nuscenes val, training speed increased by 10%, and inference speed increased by 40%. - Support Objects365 pretrain and Adding the DINO++ model can achieve an accuracy of 63.4mAP at a model scale of 200M(Under the same scale, the accuracy is the best). [🔥 2022.08.31] We have released our YOLOX-PAI that achieves SOTA results within 40~50 mAP (less than 1ms). And we also provide a convenient and fast export/predictor api for end2end object detection. To get a quick start of YOLOX-PAI, click [here](docs/source/tutorials/yolox.md)! * 31/08/2022 EasyCV v0.6.0 was released. - Release YOLOX-PAI which achieves SOTA results within 40~50 mAP (less than 1ms) - Add detection algo DINO which achieves 58.5 mAP on COCO - Add mask2former algo - Releases imagenet1k, imagenet22k, coco, lvis, voc2012 data with BaiduDisk to accelerate downloading Please refer to [change_log.md](docs/source/change_log.md) for more details and history. ## Technical Articles We have a series of technical articles on the functionalities of EasyCV. * [EasyCV开源｜开箱即用的视觉自监督+Transformer算法库](https://zhuanlan.zhihu.com/p/505219993) * [MAE自监督算法介绍和基于EasyCV的复现](https://zhuanlan.zhihu.com/p/515859470) * [基于EasyCV复现ViTDet：单层特征超越FPN](https://zhuanlan.zhihu.com/p/528733299) * [基于EasyCV复现DETR和DAB-DETR，Object Query的正确打开方式](https://zhuanlan.zhihu.com/p/543129581) * [YOLOX-PAI: 加速YOLOX, 比YOLOv6更快更强](https://zhuanlan.zhihu.com/p/560597953) * [EasyCV带你复现更好更快的自监督算法-FastConvMAE](https://zhuanlan.zhihu.com/p/566988235) * [EasyCV DataHub 提供多领域视觉数据集下载，助力模型生产](https://zhuanlan.zhihu.com/p/572593950) * [使用EasyCV Mask2Former轻松实现图像分割](https://zhuanlan.zhihu.com/p/583831421) ## Installation Please refer to the installation section in [quick_start.md](docs/source/quick_start.md) for installation. ## Get Started Please refer to [quick_start.md](docs/source/quick_start.md) for quick start. We also provides tutorials for more usages. * [self-supervised learning](docs/source/tutorials/ssl.md) * [image classification](docs/source/tutorials/cls.md) * [metric learning](docs/source/tutorials/metric_learning.md) * [object detection with yolox-pai](docs/source/tutorials/yolox.md) * [model compression with yolox](docs/source/tutorials/compression.md) * [using torchacc](docs/source/tutorials/torchacc.md) * [file io for local and oss files](docs/source/tutorials/file.md) * [using mmdetection model in EasyCV](docs/source/tutorials/mmdet_models_usage_guide.md) * [batch prediction tools](docs/source/tutorials/predict.md) notebook * [self-supervised learning](docs/source/tutorials/EasyCV图像自监督训练-MAE.ipynb) * [image classification](docs/source/tutorials/EasyCV图像分类resnet50.ipynb) * [object detection with yolox-pai](docs/source/tutorials/EasyCV图像检测YoloX.ipynb) * [metric learning](docs/source/tutorials/EasyCV度量学习resnet50.ipynb) ## Model Zoo <div align="center"> <b>Architectures</b> </div> <table align="center"> <tbody> <tr align="center"> <td> <b>Self-Supervised Learning</b> </td> <td> <b>Image Classification</b> </td> <td> <b>Object Detection</b> </td> <td> <b>Segmentation</b> </td> <td> <b>Object Detection 3D</b> </td> </tr> <tr valign="top"> <td> <ul> <li><a href="configs/selfsup/byol">BYOL (NeurIPS'2020)</a></li> <li><a href="configs/selfsup/dino">DINO (ICCV'2021)</a></li> <li><a href="configs/selfsup/mixco">MiXCo (NeurIPS'2020)</a></li> <li><a href="configs/selfsup/moby">MoBY (ArXiv'2021)</a></li> <li><a href="configs/selfsup/mocov2">MoCov2 (ArXiv'2020)</a></li> <li><a href="configs/selfsup/simclr">SimCLR (ICML'2020)</a></li> <li><a href="configs/selfsup/swav">SwAV (NeurIPS'2020)</a></li> <li><a href="configs/selfsup/mae">MAE (CVPR'2022)</a></li> <li><a href="configs/selfsup/fast_convmae">FastConvMAE (ArXiv'2022)</a></li> </ul> </td> <td> <ul> <li><a href="configs/classification/imagenet/resnet">ResNet (CVPR'2016)</a></li> <li><a href="configs/classification/imagenet/resnext">ResNeXt (CVPR'2017)</a></li> <li><a href="configs/classification/imagenet/hrnet">HRNet (CVPR'2019)</a></li> <li><a href="configs/classification/imagenet/vit">ViT (ICLR'2021)</a></li> <li><a href="configs/classification/imagenet/swint">SwinT (ICCV'2021)</a></li> <li><a href="configs/classification/imagenet/efficientformer">EfficientFormer (ArXiv'2022)</a></li> <li><a href="configs/classification/imagenet/timm/deit">DeiT (ICML'2021)</a></li> <li><a href="configs/classification/imagenet/timm/xcit">XCiT (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/timm/tnt">TNT (NeurIPS'2021)</a></li> <li><a href="configs/classification/imagenet/timm/convit">ConViT (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/timm/cait">CaiT (ICCV'2021)</a></li> <li><a href="configs/classification/imagenet/timm/levit">LeViT (ICCV'2021)</a></li> <li><a href="configs/classification/imagenet/timm/convnext">ConvNeXt (CVPR'2022)</a></li> <li><a href="configs/classification/imagenet/timm/resmlp">ResMLP (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/timm/coat">CoaT (ICCV'2021)</a></li> <li><a href="configs/classification/imagenet/timm/convmixer">ConvMixer (ICLR'2022)</a></li> <li><a href="configs/classification/imagenet/timm/mlp-mixer">MLP-Mixer (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/timm/nest">NesT (AAAI'2022)</a></li> <li><a href="configs/classification/imagenet/timm/pit">PiT (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/timm/twins">Twins (NeurIPS'2021)</a></li> <li><a href="configs/classification/imagenet/timm/shuffle_transformer">Shuffle Transformer (ArXiv'2021)</a></li> <li><a href="configs/classification/imagenet/deitiii">DeiT III (ECCV'2022)</a></li> <li><a href="configs/classification/imagenet/deit">Hydra Attention (2022)</a></li> </ul> </td> <td> <ul> <li><a href="configs/detection/fcos">FCOS (ICCV'2019)</a></li> <li><a href="configs/detection/yolox">YOLOX (ArXiv'2021)</a></li> <li><a href="configs/detection/yolox">YOLOX-PAI (ArXiv'2022)</a></li> <li><a href="configs/detection/detr">DETR (ECCV'2020)</a></li> <li><a href="configs/detection/dab_detr">DAB-DETR (ICLR'2022)</a></li> <li><a href="configs/detection/dab_detr">DN-DETR (CVPR'2022)</a></li> <li><a href="configs/detection/dino">DINO (ArXiv'2022)</a></li> </ul> </td> <td> </ul> <li><b>Instance Segmentation</b></li> <ul> <ul> <li><a href="configs/detection/mask_rcnn">Mask R-CNN (ICCV'2017)</a></li> <li><a href="configs/detection/vitdet">ViTDet (ArXiv'2022)</a></li> <li><a href="configs/segmentation/mask2former">Mask2Former (CVPR'2022)</a></li> </ul> </ul> </ul> <li><b>Semantic Segmentation</b></li> <ul> <ul> <li><a href="configs/segmentation/fcn">FCN (CVPR'2015)</a></li> <li><a href="configs/segmentation/upernet">UperNet (ECCV'2018)</a></li> </ul> </ul> </ul> <li><b>Panoptic Segmentation</b></li> <ul> <ul> <li><a href="configs/segmentation/mask2former">Mask2Former (CVPR'2022)</a></li> </ul> </ul> </ul> </td> <td> <ul> <li><a href="configs/detection3d/bevformer">BEVFormer (ECCV'2022)</a></li> </ul> </td> </tr> </td> </tr> </tbody> </table> Please refer to the following model zoo for more details. - [self-supervised learning model zoo](docs/source/model_zoo_ssl.md) - [classification model zoo](docs/source/model_zoo_cls.md) - [detection model zoo](docs/source/model_zoo_det.md) - [detection3d model zoo](docs/source/model_zoo_det3d.md) - [segmentation model zoo](docs/source/model_zoo_seg.md) - [pose model zoo](docs/source/model_zoo_pose.md) ## Data Hub EasyCV have collected dataset info for different scenarios, making it easy for users to finetune or evaluate models in EasyCV model zoo. Please refer to [data_hub.md](docs/source/data_hub.md). ## License This project is licensed under the [Apache License (Version 2.0)](LICENSE). This toolkit also contains various third-party components and some code modified from other repos under other open source licenses. See the [NOTICE](NOTICE) file for more information. ## Contact This repo is currently maintained by PAI-CV team, you can contact us by * Dingding group number: 41783266 * Email: [email protected] ### Enterprise Service If you need EasyCV enterprise service support, or purchase cloud product services, you can contact us by DingDing Group. 

Developer Tools ML Frameworks

2K Github Stars

Read more

Open Source

feathub

FeatHub is a stream-batch unified feature store that simplifies feature development, deployment, monitoring, and sharing for machine learning applications. - [Introduction](#introduction) - [Core Benefits](#core-benefits) - [What you can do with FeatHub](#what-you-can-do-with-feathub) - [Architecture Overview](#architecture-overview) - [Supported Compute Engines](#supported-compute-engines) - [FeatHub SDK Highlights](#feathub-sdk-highlights) - [User Guide](#user-guide) * [Prerequisites](#prerequisites) * [Install FeatHub Nightly Build](#install-feathub-nightly-build) * [Quickstart](#quickstart) * [Examples](#examples) - [Developer Guide](#developer-guide) - [Roadmap](#roadmap) - [Contact Us](#contact-us) - [Additional Resources](#additional-resources) ## Introduction FeatHub is an open-source feature store designed to simplify the development and deployment of machine learning models. It supports feature ETL and provides an easy-to-use Python SDK that abstracts away the complexities of point-in-time correctness needed to avoid training-serving skew. With FeatHub, data scientists can speed up the feature deployment process and optimize feature ETL by automatically compiling declarative feature definitions into performant distributed ETL jobs using state-of-the-art computation engines of their choice, such as Flink or Spark. Checkout [Documentation](docs/content) for guidance on compute engines, connectors, expression language, and more. ## Core Benefits Similar to other feature stores, FeatHub provides the following core benefits: - **Simplified feature development**: The Pythonic [FeatHub SDK](docs/content/feathub-sdk) makes it easy to develop features without worrying about point-in-time correctness. This helps to avoid training-serving skew, which can negatively impact the accuracy of machine learning models. - **Faster feature deployment**: FeatHub automatically compiles user-specified declarative feature definitions into performant distributed ETL jobs using state-of-the-art computation engines, such as Flink or Spark. This speeds up the feature deployment process and eliminates the need for data engineers to re-write Python programs into distributed stream or batch processing jobs. - **Performant feature generation**: FeatHub offers a range of [built-in optimizations](docs/content/deep-dive/optimizations.md) that leverage commonly observed feature ETL job patterns. These optimizations are automatically applied to ETL jobs compiled from the declarative feature definitions, much like how SQL optimizations are applied. - **Facilitated feature sharing**: FeatHub allows developers to register and query feature definitions in a persistent [feature registry](docs/content/registries). This capability reduces the duplication of data engineering efforts and the resource cost of feature generation by allowing developers in the organization to share and re-use existing feature definitions and datasets. In addition to the above benefits, FeatHub provides several architectural benefits compared to other feature stores, including: - **Real-time feature generation**: FeatHub supports real-time feature generation using [Apache Flink](docs/content/engines/flink.md) as the stream computation engine with milli-second latency. This provides better performance than other open-source feature stores that only support feature generation using Apache Spark. - **Assisted feature monitoring**: FeatHub provides [built-in metrics](docs/content/metric-stores) to monitor the quality of features and alert users to issues such as feature drift. This helps to improve the accuracy and reliability of machine learning models. - **Stream-batch unified computation**: FeatHub allows for consistent feature computation across offline, nearline, and online stacks using [Apache Flink](docs/content/engines/flink.md) for real-time features with low latency, [Apache Spark](docs/content/engines/spark.md) for offline features with high throughput, and FeatureService for computing features online when the request is received. - **Extensible framework**: FeatHub's Python SDK is decoupled from the APIs of the underlying computation engines, providing flexibility and avoiding lock-in. This allows for the support of additional computation engines in the future. For example, FeatHub supports [Local Processor](docs/content/engines/local.md) that is implemented using Pandas library, in addition to its support for Apache Flink and Apache Spark. Usability is a crucial factor that sets feature store projects apart. Our SDK is designed to be **Pythonic**, **declarative**, intuitive, and highly expressive to support all the necessary feature transformations. We understand that a feature store's success depends on its usability as it directly affects developers' productivity. Check out the [FeatHub SDK Highlights](#feathub-sdk-highlights) section below to learn more about the exceptional usability of our SDK. <!-- TODO: provide examples showing the advantage of python SDK over SQL. --> ## What you can do with FeatHub With FeatHub, you can: - **Define new features**: Define features as the result of applying expressions, aggregations, and cross-table joins on existing features, all with point-in-time correctness. - **Read and write features data**: Read and write feature data into a variety of offline, nearline, and online [storage systems](docs/content/connectors) for both offline training and online serving. - **Backfill features data**: Process historical data with the given time range and/or keys to backfill feature data, whic - **Run experiments**: Run experiments on the local machine using LocalProcessor without connecting to Apache Flink or Apache Spark cluster. Then deploy the FeatHub program in a distributed Apache Flink or Apache Spark cluster by changing the program configuration. ## Architecture Overview The architecture of FeatHub and its key components are shown in the figure below. <img src="docs/static/img/architecture_1.png" width="50%" height="auto"> The workflow of defining, computing, and serving features using FeatHub is illustrated in the figure below. <img src="docs/static/img/architecture_2.png" width="70%" height="auto"> See [Basic Concepts](docs/content/basic-concepts.md) for more details about the key components in FeatHub. ## Supported Compute Engines FeatHub supports the following compute engines to execute feature ETL pipeline: - [Apache Flink 1.16](docs/content/engines/flink.md) - [Aapche Spark 3.3](docs/content/engines/spark.md) - [Local Processor](docs/content/engines/local.md) ## FeatHub SDK Highlights The following examples demonstrate how to define a variety of features concisely using FeatHub SDK. See [FeatHub SDK](docs/content/feathub-sdk) for more details. See [NYC Taxi Demo](docs/examples/nyc_taxi.ipynb) to learn more about how to define, generate and serve features using FeatHub SDK. - Define features via table joins with point-in-time correctness ```python f_price = Feature( name="price", transform=JoinTransform( table_name="price_update_events", feature_name="price" ), keys=["item_id"], ) ``` - Define over-window aggregation features: ```python f_total_payment_last_two_minutes = Feature( name="total_payment_last_two_minutes", transform=OverWindowTransform( expr="item_count * price", agg_func="SUM", window_size=timedelta(minutes=2), group_by_keys=["user_id"] ) ) ``` - Define sliding-window aggregation features: ```python f_total_payment_last_two_minutes = Feature( name="total_payment_last_two_minutes", transform=SlidingWindowTransform( expr="item_count * price", agg_func="SUM", window_size=timedelta(minutes=2), step_size=timedelta(minutes=1), group_by_keys=["user_id"] ) ) ``` - Define features via built-in functions and the FeatHub expression language: ```python f_trip_time_duration = Feature( name="f_trip_time_duration", transform="UNIX_TIMESTAMP(taxi_dropoff_datetime) - UNIX_TIMESTAMP(taxi_pickup_datetime)", ) ``` - Define a feature via Python UDF: ```python f_lower_case_name = Feature( name="lower_case_name", dtype=types.String, transform=PythonUdfTransform(lambda row: row["name"].lower()), ) ``` <!-- TODO: Add SqlFeatureView. --> ## User Guide Checkout [Documentation](docs/content) for guidance on compute engines, connectors, expression language, and more. ### Prerequisites You need the following to run FeatHub installed using pip: - Unix-like operating system (e.g. Linux, Mac OS X) - Python 3.7/3.8/3.9 ### Install FeatHub Nightly Build To install the nightly version of FeatHub and the corresponding extra requirements based on the compute engine you plan to use, run one of the following commands: ```bash # Run the following command if you plan to run FeatHub using a local process $ python -m pip install --upgrade feathub-nightly # Run the following command if you plan to use Apache Flink cluster $ python -m pip install --upgrade "feathub-nightly[flink]" # Run the following command if you plan to use Apache Spark cluster, or to use # Spark-supported storage in a local process. $ python -m pip install --upgrade "feathub-nightly[spark]" ``` ### Quickstart #### Quickstart using Local Processor Execute the following command to compute features defined in [nyc_taxi.py](python/feathub/examples/nyc_taxi.py) in the given Python process. ```bash $ python python/feathub/examples/nyc_taxi.py ``` #### Quickstart using Flink Processor You can use the following quickstart guides to compute features in a Flink cluster with different deployment modes: - [Flink Processor Session Mode Quickstart](docs/content/quickstarts/flink-session-mode.md) - [Flink Processor Cli Mode Quickstart](docs/content/quickstarts/flink-cli-mode.md) #### Quickstart using Spark Processor You can use the following quickstart guides to compute features in a standalone Spark cluster. - [Spark Processor Client Mode Quickstart](docs/content/quickstarts/spark-client-mode.md) ### Examples The following examples can be run on Google Colab. | Name | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | [NYC Taxi Demo](./docs/examples/nyc_taxi.ipynb) | Quickstart notebook that demonstrates how to define, extract, transform and materialize features with NYC taxi-fare prediction sample data. | | [Feature Embedding Demo](./docs/examples/feature_embedding.ipynb) | FeatHub UDF example showing how to define and use feature embedding with a pre-trained Transformer model and hotel review sample data. | | [Fraud Detection Demo](./docs/examples/fraud_detection.ipynb) | An example to demonstrate usage with multiple data sources such as user account and transaction data. | Examples in this [this](https://github.com/flink-extended/feathub-examples) repo can be run using docker-compose. ## Developer Guide ### Prerequisites You need the following to build FeatHub from source: - Unix-like operating system (e.g. Linux, Mac OS X) - x86_64 architecture - Python 3.7/3.8/3.9 - Java 8 - Maven >= 3.1.1 ### Install Development Dependencies 1. Install the required Python libraries. ```bash $ python -m pip install -r python/dev-requirements.txt ``` 2. Start docker engine and pull the required images. ```bash $ docker image pull redis:latest $ docker image pull confluentinc/cp-kafka:5.4.3 ``` 3. Increase open file limit to be at least 1024. ```bash $ ulimit -n 1024 ``` ### Build and Install FeatHub from Source <!-- TODO: Add instruction to install "./python[all]" after the dependency confliction in PyFlink and PySpark is resolved. --> ```bash $ mvn clean package -DskipTests -f ./java $ python -m pip install "./python[flink]" $ python -m pip install "./python[spark]" ``` ### Run Tests Please execute the following commands under Feathub's root folder to run tests. ```bash $ mvn clean package -f ./java $ pytest --tb=line -W ignore::DeprecationWarning ./python ``` While the commands above cover most of Feathub's tests, some FlinkProcessor's python tests, such as tests related to Parquet format, have been ignored by default as they require a Hadoop environment to function correctly. In order to run these tests, please install Hadoop on your local machine and set up environment variables as follows before executing the commands above. ```bash export FEATHUB_TEST_HADOOP_CLASSPATH=`hadoop classpath` ``` You may refer to [Flink's document for Hive connector](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/#supported-hive-versions) for supported Hadoop & Hive versions. ### Format Code Style FeatHub uses the following tools to maintain code quality: - [Black](https://black.readthedocs.io/en/stable/index.html) to format Python code - [flake8](https://flake8.pycqa.org/en/latest/) to check Python code style - [mypy](https://mypy.readthedocs.io/en/stable/) to check type annotation Before uploading pull requests (PRs) for review, format codes, check code style, and check type annotations using the following commands: ```bash # Format python code $ python -m black ./python # Check python code style $ python -m flake8 --config=python/setup.cfg ./python # Check python type annotation $ python -m mypy --config-file python/setup.cfg ./python ``` ## Roadmap Here is a list of key features that we plan to support: - [x] Support all FeatureView transformations with FlinkProcessor - [x] Support all FeatureView transformations with LocalProcessor - [x] Support all FeatureView transformations with SparkProcessor - [x] Support common online and offline feature storages (e.g. Kafka, Redis, Hive, MySQL) - [x] Support persisting feature metadata in MySQL - [x] Support exporting pre-defined and user-defined feature metrics to Prometheus - [ ] Support online transformation with feature service - [ ] Support feature metadata exploration (e.g. definition, lineage, metrics) with FeatHub UI ## Contact Us Chinese-speaking users are recommended to join the following DingTalk group for questions and discussion. You need to join the "Apache Flink China" DingTalk organization via [this](https://wx-in-i.dingtalk.com/invite-page/weixin.html?bizSource=____source____&corpId=ding82d2a9eeaf9e30ff35c2f4657eb6378f&inviteCode=zmC5CSqct5jEXoi) link first in order to join the following DingTalk Group. <img src="docs/static/img/dingtalk.png" width="20%" height="auto"> English-speaking users can use this [invitation link](https://join.slack.com/t/feathubworkspace/shared_invite/zt-1ik9wk0xe-MoMEotpCEYvRRc3ulpvg2Q) to join our [Slack channel](https://feathub.slack.com/) for questions and discussion. We are actively looking for user feedback and contributors from the community. Please feel free to create pull requests and open Github issues for feedback and feature requests. Come join us! ## Additional Resources - [Documentation](docs/content): Our documentation provides guidance on compute engines, connectors, expression language, and more. Check it out if you need help getting started or want to learn more about FeatHub. - [FeatHub Examples](https://github.com/flink-extended/feathub-examples): This repository provides a wide variety of FeatHub demos that can be executed using Docker Compose. It's a great resource if you want to try out FeatHub and see what it can do. - Tech Talks and Articles - DataFun 2023 ([slides](https://www.slideshare.net/DongLin1/feathubdatafun2023pptx)) - Flink Forward Asia 2022 ([slides](https://www.slideshare.net/DongLin1/feathub), [video](https://www.bilibili.com/video/BV1714y1E7fQ/?spm_id_from=333.337.search-card.all.click), [article](https://mp.weixin.qq.com/s/ZFKRNaQODe0LwRT1nlwZgA))

Database ML Frameworks Data Pipelines & ETL

349 Github Stars

Read more

Open Source

nacos

<img src="doc/Nacos_Logo.png" width="50%" syt height="50%" /> # Nacos: Dynamic *Na*ming and *Co*nfiguration *S*ervice [](https://gitter.im/alibaba/nacos?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [](https://www.apache.org/licenses/LICENSE-2.0.html) [](https://github.com/alibaba/nacos/actions/workflows/ci.yml) [](https://app.codecov.io/gh/alibaba/nacos/tree/develop) [](https://opensource.alibaba.com/contribution_leaderboard/details?projectValue=nacos) [](https://deepwiki.com/alibaba/nacos) ------- ## What does it do Nacos (official site: [nacos.io](https://nacos.io)) is an easy-to-use platform designed for dynamic service discovery and configuration and service management. It helps you to build cloud native applications and microservices platform easily. Service is a first-class citizen in Nacos. Nacos supports almost all type of services，for example，[Dubbo/gRPC service](https://nacos.io/docs/latest/ecology/use-nacos-with-dubbo/), [Spring Cloud RESTFul service](https://nacos.io/docs/latest/ecology/use-nacos-with-spring-cloud/) or [Kubernetes service](https://nacos.io/docs/latest/quickstart/quick-start-kubernetes/). Nacos provides four major functions. * **Service Discovery and Service Health Check** Nacos makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. Nacos also provides real-time health checks of services to prevent sending requests to unhealthy hosts or service instances. * **Dynamic Configuration Management** Dynamic Configuration Service allows you to manage configurations of all services in a centralized and dynamic manner across all environments. Nacos eliminates the need to redeploy applications and services when configurations are updated, which makes configuration changes more efficient and agile. * **Dynamic DNS Service** Nacos supports weighted routing, making it easier for you to implement mid-tier load balancing, flexible routing policies, flow control, and simple DNS resolution services in the production environment within your data center. It helps you to implement DNS-based service discovery easily and prevent applications from coupling to vendor-specific service discovery APIs. * **Service and MetaData Management** Nacos provides an easy-to-use service dashboard to help you manage your services metadata, configuration, kubernetes DNS, service health and metrics statistics. ## Quick Start It is super easy to get started with your first project. ### Deploying Nacos on cloud You can deploy Nacos on cloud, which is the easiest and most convenient way to start Nacos. Use the following [Nacos deployment guide](https://cn.aliyun.com/product/aliware/mse?spm=nacos-website.topbar.0.0.0) to see more information and deploy a stable and out-of-the-box Nacos server. ### Start by the provided startup package #### Step 1: Download the binary package You can download the package from the [latest stable release](https://github.com/alibaba/nacos/releases). Take release `nacos-server-1.0.0.zip` for example: ```sh unzip nacos-server-1.0.0.zip cd nacos/bin ``` #### Step 2: Start Server On the **Linux/Unix/Mac** platform, run the following command to start server with standalone mode: ```sh sh startup.sh -m standalone ``` On the **Windows** platform, run the following command to start server with standalone mode. Alternatively, you can also double-click the `startup.cmd` to run NacosServer. ``` startup.cmd -m standalone ``` For more details, see [quick-start.](https://nacos.io/docs/latest/quickstart/quick-start/) ## Quick start for other open-source projects: * [Quick start with Nacos command and console](https://nacos.io/docs/latest/quickstart/quick-start/) * [Quick start with dubbo](https://nacos.io/docs/latest/ecology/use-nacos-with-dubbo/) * [Quick start with spring cloud](https://nacos.io/docs/latest/ecology/use-nacos-with-spring-cloud/) * [Quick start with kubernetes](https://nacos.io/docs/latest/quickstart/quick-start-kubernetes/) ## Documentation You can view the full documentation from the [Nacos website](https://nacos.io/docs/latest/overview/). You can also read this online eBook from the [NACOS ARCHITECTURE & PRINCIPLES](https://nacos.io/docs/ebook/kbyo6n/). All the latest and long-term notice can also be found here from [GitHub notice issue](https://github.com/alibaba/nacos/labels/notice). ## Contributing Contributors are welcomed to join Nacos project. Please check [CONTRIBUTING](./CONTRIBUTING.md) about how to contribute to this project. ### How can I contribute? * Take a look at issues with tags marked [`good first issue`](https://github.com/alibaba/nacos/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) or [`contribution welcome`](https://github.com/alibaba/nacos/issues?q=is%3Aopen+is%3Aissue+label%3A%22contribution+welcome%22). * Answer questions on [issues](https://github.com/alibaba/nacos/issues). * Fix bugs reported on [issues](https://github.com/alibaba/nacos/issues), and send us a pull request. * Review the existing [pull request](https://github.com/alibaba/nacos/pulls). * Improve the [website](https://github.com/nacos-group/nacos-group.github.io), typically we need * blog post * translation on documentation * use cases around the integration of Nacos in enterprise systems. ## Other Related Project Repositories * [nacos-spring-project](https://github.com/nacos-group/nacos-spring-project) provides the integration functionality for Spring. * [nacos-group](https://github.com/nacos-group) is the repository that hosts the eco tools for Nacos, such as SDK, synchronization tool, etc. * [spring-cloud-alibaba](https://github.com/spring-cloud-incubator/spring-cloud-alibaba) provides the one-stop solution for application development over Alibaba middleware which includes Nacos. ## Contact * [Gitter](https://gitter.im/alibaba/nacos): Nacos's IM tool for community messaging, collaboration and discovery. * [Twitter](https://twitter.com/nacos2): Follow along for latest nacos news on Twitter. * [Weibo](https://weibo.com/u/6574374908): Follow along for latest nacos news on Weibo (Twitter of China version). * [Nacos Segmentfault](https://segmentfault.com/t/nacos): Get latest notice and prompt help from Segmentfault. * Email Group: * [email protected]: Nacos usage general discussion. * [email protected]: Nacos developer discussion (APIs, feature design, etc). * [email protected]: Commits notice, very high frequency. * Join us from DingDing(Group 1: 21708933(full), Group 2: 30438813(full), Group 3: 31222241(full), Group 4: 12810027056). ### DingDing Group QR Code  ### DingDing MCP Group QR Code  ### WeChat Group QR Code  ## Enterprise Service If you need Nacos enterprise service support, or purchase cloud product services, you can join the discussion by scanning the following DingTalk group. It can also be directly activated and used through the microservice engine (MSE) provided by Alibaba Cloud. https://cn.aliyun.com/product/aliware/mse?spm=nacos-website.topbar.0.0.0 <img src="https://img.alicdn.com/imgextra/i3/O1CN01RTfN7q1KUzX4TcH08_!!6000000001168-2-tps-864-814.png" width="500"> ## Download - [Nacos Official Website](https://nacos.io/download/nacos-server) - [GitHub Release](https://github.com/alibaba/nacos/releases) ## Who is using These are only part of the companies using Nacos, for reference only. If you are using Nacos, please [add your company here](https://github.com/alibaba/nacos/issues/273) to tell us your scenario to make Nacos better. <table> <tr> <td><img src="https://data.alibabagroup.com/ecms-files/886024452/296d05a1-c52a-4f5e-abf2-0d49d4c0d6b3.png" alt="Alibaba Group" width="180" height="120"></td> <td><img src="https://a.msstatic.com/huya/main/img/logo.png" alt="虎牙直播" width="180" height="120"></td> <td><img src="https://v.icbc.com.cn/userfiles/Resources/ICBC/shouye/images/2017/logo.png" alt="ICBC" width="180" height="120"></td> <td><img src="https://pic2.iqiyipic.com/lequ/20220422/e7fe69c75e2541f2a931c9e538e2ab9d.jpg" alt="爱奇艺" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1pwi9EwHqK1RjSZJnXXbNLpXa-479-59.png" alt="平安科技" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1MZWSEzDpK1RjSZFrXXa78VXa-269-69.png" alt="华夏信财" width="180" height="120"></td> <td><img src="https://www.urwork.cn/public/images/ui/logo.png" alt="优客工场" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1ebu.EAvoK1RjSZFwXXciCFXa-224-80.png" alt="贝壳找房" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1lxu7EBLoK1RjSZFuXXXn0XXa-409-74.png" alt="瑞安农村商业银行" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1L16eEzTpK1RjSZKPXXa3UpXa-302-50.png" alt="司法大数据" width="180" height="120"></td> <td><img src="https://www.souyidai.com/www-style/images/logo.gif" alt="搜易贷" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1OigyDyLaK1RjSZFxXXamPFXa-168-70.png" alt="平行云" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1gJ4vIhTpK1RjSZR0XXbEwXXa-462-60.jpg" alt="甘肃紫光" width="180" height="120"></td> <td><img src="http://www.seaskylight.com/cn/uploadfiles/image/logo.png" alt="海云天" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1DZWSEzDpK1RjSZFrXXa78VXa-240-62.png" alt="Acmedcare+" width="180" height="120"></td> <td><img src="https://14605854.s21i.faiusr.com/4/ABUIABAEGAAg4OvkzwUo8b-qlwUwxQ449gM!300x300.png" alt="北京天合互联信息有限公司" width="180" height="120"></td> </tr> <tr> <td><img src="http://www.mwclg.com/static-resource/front/images/home/img_logo_nav.png" alt="上海密尔克卫化工" width="180" height="120"></td> <td><img src="https://www.synwe.com/logo-full.png" alt="大连新唯" width="180" height="120"></td> <td><img src="https://user-images.githubusercontent.com/10215557/51593180-7563af00-1f2c-11e9-95b1-ec2c645d6a0b.png" alt="立思辰" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1zWW2EpYqK1RjSZLeXXbXppXa-262-81.png" alt="东家" width="180" height="120"></td> </tr> <tr> <td><img src="http://www.sh-guiyao.com/images/logo.jpg" alt="上海克垚" width="180" height="120"></td> <td><img src="http://www.lckjep.com:80//theme/img/logoTop.png" alt="联采科技" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1G216EsbpK1RjSZFyXXX_qFXa-325-53.jpg" alt="南京28研究所" width="180" height="120"></td> <td><img src="https://p1.ifengimg.com/auto/image/2017/0922/auto_logo.png" alt="凤凰网-汽车" width="180" height="120"></td> </tr> <tr> <td><img src="http://www.sinochemitech.com/zhxx/lib/images/-logo.png" alt="中化信息" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1DXerNgDqK1RjSZSyXXaxEVXa-333-103.png" alt="一点车" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1VfOANgHqK1RjSZFPXXcwapXa-313-40.png" alt="明传无线" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1lvCyNhTpK1RjSZFMXXbG_VXa-130-60.png" alt="妙优车" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1kY9qNgTqK1RjSZPhXXXfOFXa-120-50.png" alt="蜂巢" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1G.GBNbrpK1RjSZTEXXcWAVXa-234-65.png" alt="华存数据" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1qsurNgDqK1RjSZSyXXaxEVXa-300-90.png" alt="数云" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB13aywNhTpK1RjSZR0XXbEwXXa-98-38.png" alt="广通软件" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1xqmBNjTpK1RjSZKPXXa3UpXa-162-70.png" alt="菜菜" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB18DmINcfpK1RjSZFOXXa6nFXa-200-200.png" alt="科蓝公司" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB15uqANXzqK1RjSZFoXXbfcXXa-188-86.png" alt="浩鲸" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1mvmyNkvoK1RjSZPfXXXPKFXa-238-46.png" alt="未名天日语" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1PSWsNmrqK1RjSZK9XXXyypXa-195-130.jpg" alt="金联创" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1k1qzNbvpK1RjSZFqXXcXUVXa-160-69.png" alt="同窗链" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1HdyvNmzqK1RjSZFLXXcn2XXa-143-143.jpg" alt="顺能" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1UdaGNgHqK1RjSZJnXXbNLpXa-277-62.png" alt="百世快递" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB17OqENbrpK1RjSZTEXXcWAVXa-240-113.jpg" alt="汽车之家" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1q71ANkvoK1RjSZPfXXXPKFXa-257-104.png" alt="鲸打卡" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1UzuyNhTpK1RjSZR0XXbEwXXa-201-86.jpg" alt="时代光华" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB19RCANgHqK1RjSZFPXXcwapXa-180-180.jpg" alt="康美" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1iCGyNb2pK1RjSZFsXXaNlXXa-143-143.jpg" alt="环球易购" width="180" height="120"></td> <td><img src="https://avatars0.githubusercontent.com/u/16344119?s=200&v=4" alt="Nepxion" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1aUe5EpzqK1RjSZSgXXcpAVXa-248-124.png" alt="chigua" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1H9O5EAvoK1RjSZFNXXcxMVXa-221-221.jpg" alt="宅无限" width="180" height="120"></td> </tr> <tr> <td><img src="https://img.alicdn.com/tfs/TB1rNq4EwHqK1RjSZFgXXa7JXXa-200-200.jpg" alt="天阙" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1CRAxDxYaK1RjSZFnXXa80pXa-190-190.jpg" alt="联合永道" width="180" height="120"></td> <td><img src="https://img.alicdn.com/tfs/TB1.q14ErrpK1RjSZTEXXcWAVXa-219-219.jpg" alt="明源云" width="180" height="120"></td> <td><img src="https://www.daocloud.io/static/Logo-Light.png" alt="DaoCloud" width="180" height="120"></td> </tr> <tr> <td><img src="https://www.meicai.cn/img/logo.9210b6eb.jpg" alt="美菜" width="180" height="120"></td> <td><img src="https://img5.tianyancha.com/logo/lll/3aad34039972b57e70874df8c919ae8b.png@!f_200x200" alt="松格科技" width="180" height="120"></td> <td><img src="https://www.jsic-tech.com/Public/uploads/20191206/5de9b9baac696.jpg" alt="集萃智能" width="180" height="120"></td> <td><img src="https://www.wuuxiang.com/theme/images/common/logo1.png" alt="吾享" width="180" height="120"></td> </tr> <tr> <td><img src="http://www.tpson.cn/static/upload/image/20230111/1673427385140440.png" alt="拓深科技" width="180" height="120"></td> <td><img src="https://www.sunline.cn/u_file/fileUpload/2021-06/25/2021062586431.png" alt="长亮科技" width="180" height="120"></td> <td><img src="http://pmt2f499f.pic44.websiteonline.cn/upload/wv0c.png" alt="深圳易停车库" width="180" height="120"></td> <td><img src="http://www.dragonwake.cn/static/css/default/img/logo.png" alt="武汉日创科技" width="180" height="120"></td> </tr> <tr> <td><img src="https://i4im-web.oss-cn-shanghai.aliyuncs.com/images/logo.png" alt="易管智能" width="180" height="120"></td> <td><img src="https://www.yunzhangfang.com/assets/img/logo.4096cf52.png" alt="云帐房" width="180" height="120"></td> <td><img src="https://www.sinocare.com/sannuo/templates/web/img/bocweb-logo.svg" alt="三诺生物" width="180" height="120"></td> <td></td> </tr> <tr> <td>郑州山水</td> <td>知氏教育</td> <td></td> <td></td> </tr> </table>

DevOps & Infrastructure AI Agents

33K Github Stars

Read more

Open Source

page-agent

# Page Agent <picture> <source media="(prefers-color-scheme: dark)" srcset="https://img.alicdn.com/imgextra/i4/O1CN01qKig1P1FnhpFKNdi6_!!6000000000532-2-tps-1280-256.png"> <img alt="Page Agent Banner" src="https://img.alicdn.com/imgextra/i1/O1CN01NCMKXj1Gn4tkFTsxf_!!6000000000666-2-tps-1280-256.png"> </picture> [](https://opensource.org/licenses/MIT) [](http://www.typescriptlang.org/) [](https://bundlephobia.com/package/page-agent) [](https://www.npmjs.com/package/page-agent) [](https://github.com/alibaba/page-agent) The GUI Agent Living in Your Webpage. Control web interfaces with natural language. 🌐 **English** | [中文](./docs/README-zh.md) <a href="https://alibaba.github.io/page-agent/" target="_blank"><b>🚀 Demo</b></a> | <a href="https://alibaba.github.io/page-agent/docs/introduction/overview" target="_blank"><b>📖 Docs</b></a> | <a href="https://news.ycombinator.com/item?id=47264138" target="_blank"><b>📢 HN Discussion</b></a> | <a href="https://x.com/simonluvramen" target="_blank"><b>𝕏 Follow on X</b></a> <video id="demo-video" src="https://github.com/user-attachments/assets/a1f2eae2-13fb-4aae-98cf-a3fc1620a6c2" controls crossorigin muted></video> --- ## ✨ Features - **🎯 Easy integration** - No need for `browser extension` / `python` / `headless browser`. - Just in-page javascript. Everything happens in your web page. - **📖 Text-based DOM manipulation** - No screenshots. No multi-modal LLMs or special permissions needed. - **🧠 Bring your own LLMs** - **🐙 Optional [chrome extension](https://alibaba.github.io/page-agent/docs/features/chrome-extension) for multi-page tasks.** - And an [MCP Server (Beta)](https://alibaba.github.io/page-agent/docs/features/mcp-server) to control it from outside ## 💡 Use Cases - **SaaS AI Copilot** — Ship an AI copilot in your product in lines of code. No backend rewrite. - **Smart Form Filling** — Turn 20-click workflows into one sentence. Perfect for ERP, CRM, and admin systems. - **Accessibility** — Make any web app accessible through natural language. Voice commands, screen readers, zero barrier. - **Multi-page Agent** — Extend your own web agent's reach across browser tabs [chrome extension](https://alibaba.github.io/page-agent/docs/features/chrome-extension). - **MCP** - Allow your agent clients to control your browser. ## 🚀 Quick Start ### One-line integration Fastest way to try PageAgent with our free Demo LLM: ```html <script src="{URL}" crossorigin="true"></script> ``` > **⚠️ For technical evaluation only.** This demo CDN uses our free [testing LLM API](https://alibaba.github.io/page-agent/docs/features/models#free-testing-api). By using it, you agree to its [terms](https://github.com/alibaba/page-agent/blob/main/docs/terms-and-privacy.md). | Mirrors | URL | | ------- | ---------------------------------------------------------------------------------- | | Global | https://cdn.jsdelivr.net/npm/[email protected]/dist/iife/page-agent.demo.js | | China | https://registry.npmmirror.com/page-agent/1.9.0/files/dist/iife/page-agent.demo.js | Add `?autoInit=false` to load the script without creating the demo agent automatically. You can then instantiate it with `new window.PageAgent(...)`. ### NPM Installation ```bash npm install page-agent ``` ```javascript import { PageAgent } from 'page-agent' const agent = new PageAgent({ model: 'qwen3.5-plus', baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1', apiKey: 'YOUR_API_KEY', language: 'en-US', }) await agent.execute('Click the login button') ``` For more programmatic usage, see [📖 Documentations](https://alibaba.github.io/page-agent/docs/introduction/overview). ## 🌟 Awesome Page Agent Built something cool with PageAgent? Add it here! Open a PR to share your project. > These are community projects — not maintained or endorsed by us. Use at your own discretion. | Project | Description | | -------- | ----------------------------------------------------------- | | _Yours?_ | [Open a PR](https://github.com/alibaba/page-agent/pulls) 🙌 | ## 🤝 Contributing We welcome contributions from the community! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines and [docs/developer-guide.md](docs/developer-guide.md) for local development workflows. Please read the [maintainer's note](https://github.com/alibaba/page-agent/issues/349) on principles and current state. Contributions generated entirely by **bots or AI** without substantial human involvement will **not be accepted**. ## ⚖️ License [MIT License](LICENSE) ## 👏 Acknowledgments This project builds upon the excellent work of **[`browser-use`](https://github.com/browser-use/browser-use)**. `PageAgent` is designed for **client-side web enhancement**, not server-side automation. ``` DOM processing components and prompt are derived from browser-use: Browser Use <https://github.com/browser-use/browser-use> Copyright (c) 2024 Gregor Zunic Licensed under the MIT License We gratefully acknowledge the browser-use project and its contributors for their excellent work on web automation and DOM interaction patterns that helped make this project possible. ``` --- **⭐ Star this repo if you find PageAgent helpful!**

AI Agents Browser Automation

18.4K Github Stars

Read more

Open Source

ChatUI

<h1 align="center"> <a href="https://chatui.io/"> <img width="109" height="28" src="https://gw.alicdn.com/tfs/TB1uYH4QoY1gK0jSZFMXXaWcVXa-218-56.svg" alt="ChatUI"> </a> </h1> <p align="center">The UI design language and React library for Conversational UI</p> <p align="center">Website：<a href="https://chatui.io/" target="_blank">https://chatui.io</a></p> <div align="center"> [](https://github.com/alibaba/ChatUI/blob/master/LICENSE) [](https://www.npmjs.com/package/@chatui/core) [](https://www.npmjs.com/package/@chatui/core) [](https://unpkg.com/@chatui/[email protected]/dist/index.js) [](https://cdn.jsdelivr.net/npm/@chatui/core) </div> <p align="center"> <img width="750" src="https://gw.alicdn.com/tfs/TB1WTl.lQ9l0K4jSZFKXXXFjpXa-1500-833.jpg"> </p> English | [简体中文](./README.zh-CN.md) ## Features - 😎 **Best Practices**: The best practice for chat interaction based on our experience of Alime Chatbot - 🛡 **TypeScript**: Written in TypeScript with predictable static types - 📱 **Responsive**: Responsive design to adapt automatically to whatever device - ♿ **Accessibility**: Accessibility support and get the certification from Accessibility Research Association - 🎨 **Theming**: Powerful theme customization in every detail - 🌍 **International**: Internationalization support for dozens of languages ## Environment Support - Modern browsers (support [CSS Variables](https://caniuse.com/css-variables)) - Internet Explorer 11 (with [polyfills](https://stackoverflow.com/questions/57020976/polyfills-in-2019-for-ie11) and [CSS Variables Polyfill](https://github.com/nuxodin/ie11CustomProperties) / [css-vars-ponyfill](https://github.com/jhildenbiddle/css-vars-ponyfill)) | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/edge/edge_48x48.png" alt="Edge" width="24px" height="24px" /><br>Edge | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/firefox/firefox_48x48.png" alt="Firefox" width="24px" height="24px" /><br>Firefox | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/chrome/chrome_48x48.png" alt="Chrome" width="24px" height="24px" /><br>Chrome | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/safari/safari_48x48.png" alt="Safari" width="24px" height="24px" /><br>Safari | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/safari-ios/safari-ios_48x48.png" alt="iOS Safari" width="24px" height="24px" /><br>iOS Safari | <img src="https://raw.githubusercontent.com/alrra/browser-logos/master/src/android-webview/android-webview_48x48.png" alt="Android WebView" width="24px" height="24px" /><br>Android WebView | | --- | --- | --- | --- | --- | --- | | 16+ | 31+ | 49+ | 9.1+ | 9.3+ | 6+ | ## Install ```bash npm install @chatui/core --save ``` ```bash yarn add @chatui/core ``` ## Usage ```jsx import Chat, { Bubble, useMessages } from '@chatui/core'; import '@chatui/core/dist/index.css'; const App = () => { const { messages, appendMsg, setTyping } = useMessages([]); function handleSend(type, val) { if (type === 'text' && val.trim()) { appendMsg({ type: 'text', content: { text: val }, position: 'right', }); setTyping(true); setTimeout(() => { appendMsg({ type: 'text', content: { text: 'Bala bala' }, }); }, 1000); } } function renderMessageContent(msg) { const { content } = msg; return <Bubble content={content.text} />; } return ( <Chat navbar={{ title: 'Assistant' }} messages={messages} renderMessageContent={renderMessageContent} onSend={handleSend} /> ); }; ``` [](https://codesandbox.io/s/chatui-demo-o6n3z?fontsize=14&hidenavigation=1&theme=dark) ### Development ```bash cd demo npm i npm run dev ``` ## Theme Visit [Customize Theme](https://chatui.io/docs/customize-theme) for detail ## Internationalization Visit [i18n](https://chatui.io/docs/i18n) for detail ## License MIT

CSS Frameworks & UI Kits JavaScript Libraries & Components Live Chat & Chatbots

4.4K Github Stars

Read more

Open Source

butterfly

<p align="center"> <a href="http://noonnightstorm.github.io"> <!-- <img width="900" src="http://img.alicdn.com/tfs/TB1TlngGFYqK1RjSZLeXXbXppXa-844-474.png"> --> </a> </p> <h3 align="center">一个基于JS的数据驱动的节点式编排组件库</h3> <div align="center"> [](https://dev.azure.com/noonnightstorm/butterfly/_build/latest?definitionId=1&branchName=master) [](https://circleci.com/gh/alibaba/butterfly) [](https://www.npmjs.org/package/butterfly-dag) [](http://npmjs.com/butterfly-dag) [](https://david-dm.org/alibaba/butterfly) [](https://david-dm.org/alibaba/butterfly?type=dev) </div> [English](./README.en-US.md) | 简体中文 ## ✨ 特性 * 丰富DEMO，开箱即用 * 全方位管理画布，开发者只需要更专注定制化的需求 * 利用DOM/REACT/VUE来定制元素；灵活性，可塑性，拓展性优秀 <p align="center"> <img width="900" src="https://img.alicdn.com/imgextra/i3/O1CN018CrqXz1KRK7Euhj6X_!!6000000001160-2-tps-1155-1081.png"> </p> ## 🚀DEMO ### 本地DEMO ``` git clone [email protected]:alibaba/butterfly.git npm install cd example npm install npm start ``` ### 线上DEMO [小蝴蝶官网](https://butterfly-dag.gitee.io/butterfly-dag/demo/analysis) ## 📦 安装 ``` npm install butterfly-dag ``` ## 🔨 快速上手 ### 引入方式 ``` // 完全版，内部包含jquery和lodash import {Canvas, Group, Node, Edge} from 'butterfly-dag'; import 'butterfly-dag/dist/index.css'; // 如果您引用的项目使用了jquery和lodash，为了缩小项目的体积，我们建议： import {Canvas, Group, Node, Edge} from 'butterfly-dag/pack/index.js'; import 'butterfly-dag/pack/index.css'; ``` ### 生成画布 ``` import {Canvas} from 'butterfly-dag'; let canvas = new Canvas({ root: dom, //canvas的根节点(必传) zoomable: true, //可缩放(可传) moveable: true, //可平移(可传) draggable: true, //节点可拖动(可传) }); canvas.draw({ groups: [], //分组信息 nodes: [], //节点信息 edges: [] // 连线信息 }) ``` ## 🔗 API文档 * *__!!! 3.x的API文档，请移步到__*[这里](https://github.com/alibaba/butterfly/blob/master/README.md); * [画布(Canvas)](./docs/zh-CN/canvas.md) * [节点组(Group)](./docs/zh-CN/group.md) * [节点(Node)](./docs/zh-CN/node.md) * [线(Edge)](./docs/zh-CN/edge.md) * [锚点(Endpoint)](./docs/zh-CN/endpoint.md) * [缩略图(Minimap)](./docs/zh-CN/minimap.md#endpoint-api) * [提示 & 菜单(tooltips & menu)](./docs/zh-CN/tooltip.md) * [布局(Layout)](./docs/zh-CN/layout.md) * 插件 * [箭头(arrow)](./docs/zh-CN/plugins-arrows.md) * [左侧画板(pannel)](./docs/zh-CN/plugins-pannel.md) * [快捷键(hotkey)](./docs/zh-CN/plugins-hotkey.md) * React & Vue支持 * [React butterfly组件支持](./docs/zh-CN/react.md) [不维护，推荐用原生小蝴蝶] * [Vue2 butterfly组件支持](./docs/zh-CN/vue.md) ## 🎨优秀案例 <p align="center"> <img width="900" src="https://img.alicdn.com/imgextra/i4/O1CN01d7WHVs1vkEDzWRRlW_!!6000000006210-2-tps-2400-8172.png"> </p> ## ⌨️垂直业务React拓展组件 * [数据/字段映射组件](https://github.com/aliyun/react-data-mapping): 适用于做数据字段映射，表字段映射，表格连线等业务 <p align="center"> <img width="49%" src="https://img.alicdn.com/imgextra/i4/O1CN012ecl7n25IsnZeXw1d_!!6000000007504-1-tps-595-411.gif"> <img width="49%" src="https://img.alicdn.com/imgextra/i2/O1CN017Gcu0Y1mbgIHcgqwr_!!6000000004973-1-tps-595-411.gif"> </p> <p align="center"> <img width="49%" src="https://img.alicdn.com/imgextra/i2/O1CN011xYzxM1ZenzfVE0Xq_!!6000000003220-1-tps-595-411.gif"> <img width="49%" src="https://img.alicdn.com/imgextra/i4/O1CN01Nt9rpo25y6NlRMUtR_!!6000000007594-1-tps-595-411.gif"> </p> * [表字段血缘/业务血缘](https://github.com/aliyun/react-lineage-dag): 适用于表级血缘,表字段级血缘,业务链路血缘等业务 <img width="98%" src="https://img.alicdn.com/imgextra/i4/O1CN01ou8wTq20SQv4AnedD_!!6000000006848-1-tps-1337-761.gif"> * [可视化建模图](https://github.com/aliyun/react-visual-modeling): 适用于UML，数据库建模，数据仓库建设等业务 <img width="98%" src="https://img.alicdn.com/imgextra/i4/O1CN01VZxfyl1pOLc15k7XM_!!6000000005350-1-tps-1665-829.gif"> * 调度编排图(doing) * [监控图](https://github.com/aliyun/react-monitor-dag): 适用于任务流，数据流等业务的状态展示 <img width="98%" src="https://img.alicdn.com/imgextra/i2/O1CN01eJigoL1gd9tjRSvdi_!!6000000004164-1-tps-1665-826.gif"> * Butterfly-Editor(doing) ## 🤝如何贡献 我们欢迎所有的贡献者，在成为贡献者之前，请先阅读[贡献指南](./docs/zh-CN/CONTRIBUTING.md)。 如果您已经了解，快来[Issus](https://github.com/alibaba/butterfly/issues)或[Pull requests](https://github.com/alibaba/butterfly/pulls)成为贡献者吧，让我们和小蝴蝶一起成长，一起变得更好、更棒！

Diagramming & Flowcharts

4.7K Github Stars

Read more