Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions malaya/.github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# These are supported funding model platforms

patreon: huseinzolkepli
custom: https://www.buymeacoffee.com/huseinzolkepli
8 changes: 8 additions & 0 deletions malaya/.github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: 2
updates:

# Maintain dependencies for pip in main package
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
28 changes: 28 additions & 0 deletions malaya/.github/workflows/run.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Run on Push to Master

on:
push:
branches:
- master

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.9"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install torch --index-url https://download.pytorch.org/whl/cpu

- name: Run a script
run: |
python -c "import malaya; model = malaya.jawi.huggingface(); model.generate(['helo'], to_lang = 'jawi')"
15 changes: 15 additions & 0 deletions malaya/.readthedocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
version: 2

build:
os: ubuntu-22.04
tools:
python: "3.9"

sphinx:
configuration: docs/conf.py

formats: []

python:
install:
- requirements: docs/requirements.txt
95 changes: 95 additions & 0 deletions malaya/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Contributing

Contributions are welcome and are greatly appreciated! Every little bit helps, and credit will always be given.

**Table of Contents**

- [Types of Contributions](#types-of-contributions)
- [Report Bugs](#report-bugs)
- [Fix Bugs](#fix-bugs)
- [Implement Features](#implement-features)
- [Dataset](#dataset)
- [Improve Documentation](#improve-documentation)
- [Submit Feedback](#submit-feedback)
- [Documentation](#documentation)
- [Local development environment](#local-development-environment)
- [Installation](#installation)
- [Pull Request Guidelines](#pull-request-guidelines)

# Types of Contributions

## Report Bugs

Report bugs through [Github issue](https://github.com/huseinzol05/Malaya/issues/new).

Please report relevant information and preferably code that exhibits the problem.

## Fix Bugs

Look through the [Github issue](https://github.com/huseinzol05/Malaya/issues/new) for bugs. Anything is open to whoever wants to implement it.

## Implement Features

Look through the [Github issue](https://github.com/huseinzol05/Malaya/issues/new) or [Malaya-project](https://github.com/huseinzol05/Malaya/projects/1) for features. Any unassigned `improvement` issue is open to whoever wants to implement it.

Remember, **100% Tensorflow (version 1.10 and above, not 2.0, yet), no Keras**.

## Dataset

Create a new issue in [Github issue](https://github.com/huseinzol05/Malaya/issues/new) related to your data including the data link or attached it there. If you want to improve current dataset we have, you can check at [Malaya-Dataset](https://github.com/huseinzol05/Malaya-Dataset).

Or, you can simply email your data if you do not want to expose the data to public. Malaya will not exposed your data, but we will exposed our trained models based on your data.

Thanks to,

1. [Fake news](https://github.com/huseinzol05/Malaya-Dataset#fake-news), contributed by [syazanihussin](https://github.com/syazanihussin/FLUX/tree/master/data)
2. [Speech voice](https://github.com/huseinzol05/Malaya-Dataset#tolong-sebut), contributed by [Khalil Nooh](https://www.linkedin.com/in/khalilnooh/)
3. [Speech voice](https://github.com/huseinzol05/Malaya-Dataset#tolong-sebut), contributed by [Mas Aisyah Ahmad](https://www.linkedin.com/in/mas-aisyah-ahmad-b46508a9/)
4. [Singlish text dump](https://github.com/huseinzol05/malaya-dataset#singlish-text), contributed by [brytjy](https://github.com/brytjy)
5. [Singapore news](https://github.com/huseinzol05/malaya-dataset#singapore-news), contributed by [brytjy](https://github.com/brytjy)

## Improve Documentation

Malaya could always use better documentation, might have some typos or uncorrect object names.

## Submit Feedback

The best way to send feedback is to open an issue on [Github issue](https://github.com/huseinzol05/Malaya/issues/new).

## Unit test

Test every possible program flow! You can check [unit tests here](https://github.com/huseinzol05/Malaya/tree/master/tests).

# Documentation

The latest API documentation is usually available [here](https://malaya.readthedocs.io/en/latest/index.html). To generate a local version,

```
pip install readthedocs
cd docs
bash generate_template.sh
```

# Local development environment

When you develop Malaya you can create local `virtualenv` with all requirements required by Malaya.

Advantage of local installation is that everything works locally, you do not have to enter Docker/container environment and you can easily debug the code locally. You can also have access to python `virtualenv` that contains all the necessary requirements and use it in your local IDE - this aids autocompletion, and running tests directly from within the IDE.

The disadvantage is that you have to maintain your dependencies and local environment consistent with other development environments that you have on your local machine.

It's also very difficult to make sure that your local environment is consistent with other environments. This can often lead to "works for me" syndrome. It's better to use the Docker Compose integration test environment in case you want reproducible environment consistent with other people.

## Installation

Install Python (3.6 and above) by using system-level package managers like yum, apt-get for Linux, or Homebrew for Mac OS at first.

# Pull Request Guidelines

Before you submit a pull request from your forked repo, check that it meets these guidelines:

1. The pull request should include step-by-step if the request is `improvement`.
2. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993), squash commits, and resolve all conflicts.
3. The pull request should work for Python 3.6 and above.
4. As Malaya grows as a project, we try to enforce a more consistent style and try to follow the Python
community guidelines. We currently enforce to use [BlackMamba](https://github.com/mohtar/blackmamba) for code standard.
21 changes: 21 additions & 0 deletions malaya/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2018 huseinzol05

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
66 changes: 66 additions & 0 deletions malaya/README-pypi.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
**Malaya** is a Natural-Language-Toolkit library for bahasa Malaysia, powered by PyTorch.

Documentation
--------------

Proper documentation is available at https://malaya.readthedocs.io/

Installing from the PyPI
----------------------------------

::

$ pip install malaya

It will automatically install all dependencies except for PyTorch. So you can choose your own PyTorch CPU / GPU version.

Only **Python >= 3.6.0**, and **PyTorch >= 1.10** are supported.

If you are a Windows user, make sure read https://malaya.readthedocs.io/en/latest/running-on-windows.html

Development Release
---------------------------------

Install from `master` branch,

::

$ pip install git+https://github.com/huseinzol05/malaya.git


We recommend to use **virtualenv** for development.

Documentation at https://malaya.readthedocs.io/en/latest/

Pretrained Models
------------------

Malaya also released Malaysian pretrained models, simply check at https://huggingface.co/mesolitica

References
-----------

If you use our software for research, please cite:

::

@misc{Malaya, Natural-Language-Toolkit library for bahasa Malaysia, powered by PyTorch,
author = {Husein, Zolkepli},
title = {Malaya},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/mesolitica/malaya}}
}

Acknowledgement
----------------

Thanks to `KeyReply <https://www.keyreply.com/>`_ for private V100s cloud and `Mesolitica <https://mesolitica.com/>`_ for private RTXs cloud to train Malaya models.

Also, thanks to `Tensorflow Research Cloud <https://www.tensorflow.org/tfrc>`_ for free TPUs access.

Contributing
----------------

Thank you for contributing this library, really helps a lot. Feel free to contact me to suggest me anything or want to contribute other kind of forms, we accept everything, not just code!
115 changes: 115 additions & 0 deletions malaya/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
.. raw:: html

<p align="center">
<a href="#readme">
<img alt="logo" width="40%" src="https://i.imgur.com/yi6jwST.png">
</a>
</p>
<p align="center">
<a href="https://pypi.python.org/pypi/malaya"><img alt="Pypi version" src="https://badge.fury.io/py/malaya.svg"></a>
<a href="https://pypi.python.org/pypi/malaya"><img alt="Python3 version" src="https://img.shields.io/pypi/pyversions/malaya.svg"></a>
<a href="https://github.com/huseinzol05/Malaya/blob/master/LICENSE"><img alt="MIT License" src="https://img.shields.io/github/license/huseinzol05/malaya.svg?color=blue"></a>
<a href="https://malaya.readthedocs.io/"><img alt="Documentation" src="https://readthedocs.org/projects/malaya/badge/?version=latest"></a>
<a href="https://pepy.tech/project/malaya"><img alt="total stats" src="https://static.pepy.tech/badge/malaya"></a>
<a href="https://pepy.tech/project/malaya"><img alt="download stats / month" src="https://static.pepy.tech/badge/malaya/month"></a>
<a href="https://discord.gg/J3aSWyMy9A"><img alt="discord" src="https://img.shields.io/badge/discord%20-Malaysia_AI-rgb(118,138,212).svg"></a>
</p>

=========

**Malaya** is a Natural-Language-Toolkit library for bahasa Malaysia, powered by PyTorch.

Documentation
--------------

Proper documentation is available at https://malaya.readthedocs.io/

Installing from the PyPI
----------------------------------

::

$ pip install malaya

It will automatically install all dependencies except for PyTorch. So you can choose your own PyTorch CPU / GPU version.

Only **Python >= 3.6.0**, and **PyTorch >= 1.10** are supported.

If you are a Windows user, make sure read https://malaya.readthedocs.io/en/latest/running-on-windows.html

Development Release
---------------------------------

Install from `master` branch,

::

$ pip install git+https://github.com/huseinzol05/malaya.git


We recommend to use **virtualenv** for development.

Documentation at https://malaya.readthedocs.io/en/latest/

Pretrained Models
------------------

Malaya also released Malaysian pretrained models, simply check at https://huggingface.co/mesolitica

References
-----------

If you use our software for research, please cite:

::

@misc{Malaya, Natural-Language-Toolkit library for bahasa Malaysia, powered by PyTorch,
author = {Husein, Zolkepli},
title = {Malaya},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/mesolitica/malaya}}
}

Acknowledgement
----------------

Thanks to,

1. `KeyReply <https://www.keyreply.com/>`_ for private V100s cloud.

.. raw:: html

<a href="#readme">
<img alt="logo" width="20%" src="https://image4.owler.com/logo/keyreply_owler_20191024_163259_original.png">
</a>

2. `Nvidia <https://www.nvidia.com/en-us/>`_ for Azure credit.

.. raw:: html

<a href="#readme">
<img alt="logo" width="20%" src="https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/[email protected]">
</a>


3. `Tensorflow Research Cloud <https://www.tensorflow.org/tfrc>`_ for free TPUs access.

.. raw:: html

<a href="https://www.tensorflow.org/tfrc">
<img alt="logo" width="20%" src="https://2.bp.blogspot.com/-xojf3dn8Ngc/WRubNXxUZJI/AAAAAAAAB1A/0W7o1hR_n20QcWyXHXDI1OTo7vXBR8f7QCLcB/s400/image2.png">
</a>


Contributing
----------------

Thank you for contributing this library, really helps a lot. Feel free to contact me to suggest me anything or want to contribute other kind of forms, we accept everything, not just code!

.. raw:: html

<a href="#readme">
<img alt="logo" width="30%" src="https://contributors-img.firebaseapp.com/image?repo=huseinzol05/malaya">
</a>
Binary file added malaya/accuracy/abstractive-accuracy.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
29 changes: 29 additions & 0 deletions malaya/accuracy/abstractive-template.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
option = {
xAxis: {
type: 'category',
axisLabel: {
interval: 0,
rotate: 30
},
data: ['T5-small (122MB)', 'T5-base (448MB)']
},
yAxis: {
type: 'value',
min: 0,
max: 0.4
},
grid: {
bottom: 100
},
backgroundColor: 'rgb(252,252,252)',
series: [{
data: [0.33854, 0.34103],
type: 'bar',
label: {
normal: {
show: true,
position: 'top'
}
},
}]
};
Binary file added malaya/accuracy/constituency-accuracy.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading