Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![CI](https://github.com/fourMs/MGT-python/actions/workflows/ci.yml/badge.svg)](https://github.com/fourMs/MGT-python/actions/workflows/ci.yml)
[![Documentation](https://github.com/fourMs/MGT-python/actions/workflows/docs.yml/badge.svg)](https://fourms.github.io/MGT-python/)

The **Musical Gestures Toolbox for Python** is a comprehensive collection of tools for visualization and analysis of audio and video, with a focus on motion capture and musical gesture analysis.
The **Musical Gestures Toolbox for Python** is a collection of tools for visualization and analysis of audio and video, with a focus on motion capture and musical gesture analysis.

![MGT python demo](https://raw.githubusercontent.com/fourMs/MGT-python/master/musicalgestures/documentation/figures/promo/ipython_example.gif)

Expand Down Expand Up @@ -117,15 +117,18 @@ This toolbox builds upon years of research in musical gesture analysis:

## Citation

If you use MGT-python in your research, please cite:
If you use MGT-python in your research, please cite this article:

- Laczkó, B., & Jensenius, A. R. (2021). [Reflections on the Development of the Musical Gestures Toolbox for Python](http://urn.nb.no/URN:NBN:no-91935). *Proceedings of the Nordic Sound and Music Computing Conference*, Copenhagen.

```bibtex
@software{mgt_python,
title={Musical Gestures Toolbox for Python},
author={University of Oslo fourMs Lab},
url={https://fourms.github.io/MGT-python/},
version={1.3.2},
year={2024}
@inproceedings{laczkoReflectionsDevelopmentMusical2021,
title = {Reflections on the Development of the Musical Gestures Toolbox for Python},
author = {Laczkó, Bálint and Jensenius, Alexander Refsum},
booktitle = {Proceedings of the Nordic Sound and Music Computing Conference},
year = {2021},
address = {Copenhagen},
url = {http://urn.nb.no/URN:NBN:no-91935}
}
```

Expand Down
14 changes: 7 additions & 7 deletions docs/musicalgestures/_centerface.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,48 +12,48 @@

## CenterFace

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L7)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L8)

```python
class CenterFace(object):
def __init__(landmarks=True):
def __init__(landmarks=True, use_gpu=False):
```

### CenterFace().decode

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L53)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L62)

```python
def decode(heatmap, scale, offset, landmark, size, threshold=0.1):
```

### CenterFace().inference_opencv

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L21)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L30)

```python
def inference_opencv(img, threshold):
```

### CenterFace().nms

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L87)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L96)

```python
def nms(boxes, scores, nms_thresh):
```

### CenterFace().postprocess

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L35)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L44)

```python
def postprocess(heatmap, lms, offset, scale, threshold):
```

### CenterFace().transform

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L30)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_centerface.py#L39)

```python
def transform(h, w):
Expand Down
6 changes: 3 additions & 3 deletions docs/musicalgestures/_cropvideo.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

## async_subprocess

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L239)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L243)

```python
async def async_subprocess(command):
Expand Down Expand Up @@ -59,7 +59,7 @@ Helper function to find the area of motion in a video, using ffmpeg.

## manual_text_input

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L282)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L286)

```python
def manual_text_input():
Expand Down Expand Up @@ -104,7 +104,7 @@ Crops the video using ffmpeg.

## run_cropping_window

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L258)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_cropvideo.py#L262)

```python
def run_cropping_window(imgpath, scale_ratio, scaled_width, scaled_height):
Expand Down
8 changes: 4 additions & 4 deletions docs/musicalgestures/_flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,15 +77,15 @@ Renders a dense optical flow video of the input video file using `cv2.calcOptica

### Flow().get_acceleration

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L252)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L288)

```python
def get_acceleration(velocity, fps):
```

### Flow().get_velocity

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L262)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L298)

```python
def get_velocity(
Expand All @@ -101,7 +101,7 @@ def get_velocity(

### Flow().sparse

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L277)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L313)

```python
def sparse(
Expand Down Expand Up @@ -141,7 +141,7 @@ Renders a sparse optical flow video of the input video file using `cv2.calcOptic

### Flow().velocity_meters_per_second

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L270)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_flow.py#L306)

```python
def velocity_meters_per_second(
Expand Down
44 changes: 32 additions & 12 deletions docs/musicalgestures/_pose.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,13 @@
- [download_model](#download_model)
- [pose](#pose)

#### Attributes

- `MEDIAPIPE_POSE_CONNECTIONS` - MediaPipe Pose skeleton connections (pairs of landmark indices): `[(0, 1), (1, 2), (2, 3), (3, 7), (0, 4), (4, 5)...`

## download_model

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose.py#L370)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose.py#L582)

```python
def download_model(modeltype):
Expand Down Expand Up @@ -42,21 +46,37 @@ Outputs the predictions in a text file containing the normalized x and y coordin

Supports two backends:

- **MediaPipe** (`model='mediapipe'`): Uses Google's MediaPipe Pose which detects 33 landmarks entirely on CPU. Requires the optional `mediapipe` package (`pip install musicalgestures[pose]`). The model file (~8–28 MB) is auto-downloaded on first use and cached in `musicalgestures/models/`.
- **OpenPose** (`model='body_25'`, `'coco'`, or `'mpi'`): Uses Caffe-based OpenPose models. Model weights (~200 MB) are downloaded on first use.
* **MediaPipe** (``model='mediapipe'``): Uses Google's MediaPipe Pose which detects 33
landmarks entirely on CPU. Requires the optional ``mediapipe`` package
(``pip install musicalgestures[pose]``). On first use, the model file
(~8–28 MB) is downloaded automatically and cached in ``musicalgestures/models/``.
* **OpenPose** (``model='body_25'``, ``'coco'``, or ``'mpi'``): Uses Caffe-based OpenPose
models. Model weights (~200 MB) are downloaded on first use.

#### Arguments

- `model` *str, optional* - Pose model to use. `'mediapipe'` uses MediaPipe Pose (33 landmarks, model auto-downloaded on first use). `'body_25'` loads the OpenPose BODY_25 model (25 keypoints), `'mpi'` loads the MPII model (15 keypoints), `'coco'` loads the COCO model (18 keypoints). Defaults to 'body_25'.
- `device` *str, optional* - Sets the backend to use for the neural network ('cpu' or 'gpu'). Ignored when `model='mediapipe'` (MediaPipe always runs on CPU). Defaults to 'gpu'.
- `threshold` *float, optional* - The normalized confidence threshold that decides whether we keep or discard a predicted point. Discarded points get substituted with (0, 0) in the output data. Defaults to 0.1.
- `downsampling_factor` *int, optional* - Decides how much we downsample the video before we pass it to the neural network. Ignored when `model='mediapipe'`. Defaults to 2.
- `model` *str, optional* - Pose model to use. ``'mediapipe'`` uses MediaPipe Pose (33
landmarks, model auto-downloaded on first use). ``'body_25'`` loads the OpenPose BODY_25 model
(25 keypoints), ``'mpi'`` loads the MPII model (15 keypoints), ``'coco'`` loads
the COCO model (18 keypoints). Defaults to 'body_25'.
- `device` *str, optional* - Sets the backend to use for the neural network ('cpu' or 'gpu').
Ignored when ``model='mediapipe'`` (MediaPipe always runs on CPU). Defaults to 'gpu'.
- `threshold` *float, optional* - The normalized confidence threshold that decides whether we
keep or discard a predicted point. Discarded points get substituted with (0, 0) in the
output data. Defaults to 0.1.
- `downsampling_factor` *int, optional* - Decides how much we downsample the video before we
pass it to the neural network. Ignored when ``model='mediapipe'``. Defaults to 2.
- `save_data` *bool, optional* - Whether we save the predicted pose data to a file. Defaults to True.
- `data_format` *str, optional* - Specifies format of pose-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
- `save_video` *bool, optional* - Whether we save the video with the estimated pose overlaid on it. Defaults to True.
- `target_name_video` *str, optional* - Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).
- `target_name_data` *str, optional* - Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).
- `overwrite` *bool, optional* - Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to False.
- `data_format` *str, optional* - Specifies format of pose-data. Accepted values are 'csv', 'tsv'
and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
- `save_video` *bool, optional* - Whether we save the video with the estimated pose overlaid on it.
Defaults to True.
- `target_name_video` *str, optional* - Target output name for the video. Defaults to None (which
assumes that the input filename with the suffix "_pose" should be used).
- `target_name_data` *str, optional* - Target output name for the data. Defaults to None (which
assumes that the input filename with the suffix "_pose" should be used).
- `overwrite` *bool, optional* - Whether to allow overwriting existing files or to automatically
increment target filenames to avoid overwriting. Defaults to False.

#### Returns

Expand Down
20 changes: 10 additions & 10 deletions docs/musicalgestures/_pose_estimator.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This module provides:
* class `PoseEstimator` – an abstract base class (ABC) defining the common
interface that all pose backends must implement.
* class `MediaPipePoseEstimator` – a concrete backend powered by Google
MediaPipe Pose (33 landmarks, CPU-friendly, auto-downloads model on first use).
MediaPipe Pose (33 landmarks, CPU-friendly, zero model download).
* class `OpenPosePoseEstimator` – a thin wrapper around the legacy OpenPose /
Caffe-model implementation already present in :mod:[Pose](_pose.md#pose).

Expand Down Expand Up @@ -68,8 +68,8 @@ pip install musicalgestures[pose]
```

The first time you use a given complexity level the corresponding
`.task` model file (~8–28 MB) is downloaded from Google's model
storage and cached in `musicalgestures/models/`.
``.task`` model file (~8–28 MB) is downloaded from Google's model
storage and cached in ``musicalgestures/models/``.

Parameters
----------
Expand Down Expand Up @@ -97,7 +97,7 @@ Examples

### MediaPipePoseEstimator().close

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L290)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L360)

```python
def close() -> None:
Expand All @@ -107,7 +107,7 @@ Release MediaPipe resources.

### MediaPipePoseEstimator().landmark_names

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L255)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L323)

```python
@property
Expand All @@ -116,7 +116,7 @@ def landmark_names() -> list[str]:

### MediaPipePoseEstimator().predict_frame

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L259)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L327)

```python
def predict_frame(frame: np.ndarray) -> PoseEstimatorResult:
Expand All @@ -140,7 +140,7 @@ PoseEstimatorResult

## OpenPosePoseEstimator

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L303)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L373)

```python
class OpenPosePoseEstimator(PoseEstimator):
Expand Down Expand Up @@ -172,7 +172,7 @@ threshold:

### OpenPosePoseEstimator().landmark_names

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L330)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L400)

```python
@property
Expand All @@ -181,7 +181,7 @@ def landmark_names() -> list[str]:

### OpenPosePoseEstimator().predict_frame

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L335)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L405)

```python
def predict_frame(frame: np.ndarray) -> PoseEstimatorResult:
Expand Down Expand Up @@ -341,7 +341,7 @@ Return a plain dict representation.

## get_pose_estimator

[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L348)
[[find in source code]](https://github.com/fourMs/MGT-python/blob/master/musicalgestures/_pose_estimator.py#L418)

```python
def get_pose_estimator(
Expand Down
Loading
Loading