Add README with usage instructions and architecture overview
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
101
README.md
Normal file
101
README.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
# MetaVox Load Test
|
||||||
|
|
||||||
|
Ansible playbook voor het opzetten van een MetaVox loadtest omgeving op een Nextcloud server.
|
||||||
|
|
||||||
|
## Wat doet het?
|
||||||
|
|
||||||
|
- **50 teamfolders** aanmaken via de Groupfolders API
|
||||||
|
- **500.000 bestanden** (10.000 per teamfolder) direct op het filesystem
|
||||||
|
- **3-niveau mappenstructuur** (10 hoofdmappen × 3 submappen = 30 leaf folders)
|
||||||
|
- **100 metadata velddefinities** (10 teamfolder + 90 file-level) via MetaVox OCS API
|
||||||
|
- **~43 miljoen metadata records** direct via MySQL
|
||||||
|
- **7 redundante database indexes** droppen voor performance en diskruimte
|
||||||
|
|
||||||
|
## Waarom direct filesystem + MySQL?
|
||||||
|
|
||||||
|
| Methode | Snelheid | 500K bestanden |
|
||||||
|
|---------|----------|----------------|
|
||||||
|
| WebDAV upload | ~1.8 files/s | ~3 dagen |
|
||||||
|
| `occ files:scan` | ~2 files/s | ~3 dagen |
|
||||||
|
| **Direct filesystem + MySQL** | **~200 files/s** | **~40 minuten** |
|
||||||
|
|
||||||
|
| Methode | Snelheid | 43M metadata records |
|
||||||
|
|---------|----------|----------------------|
|
||||||
|
| MetaVox API | ~0.5 req/s (timeouts) | weken |
|
||||||
|
| **Direct MySQL insert** | **~6.000 rows/s** | **~2 uur** |
|
||||||
|
|
||||||
|
## Vereisten
|
||||||
|
|
||||||
|
- Nextcloud server met MetaVox en Groupfolders app geïnstalleerd
|
||||||
|
- Ansible (lokaal op de server, `ansible_connection: local`)
|
||||||
|
- Python 3 met `requests` module
|
||||||
|
- MySQL/MariaDB root toegang (zonder wachtwoord, via unix socket)
|
||||||
|
- Minimaal 15GB vrije schijfruimte
|
||||||
|
|
||||||
|
## Gebruik
|
||||||
|
|
||||||
|
### Deployen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone op de Nextcloud server
|
||||||
|
git clone https://gitea.rikdekker.nl/Samkey/metavox-loadtest.git
|
||||||
|
cd metavox-loadtest
|
||||||
|
|
||||||
|
# Pas server URL en credentials aan
|
||||||
|
vi group_vars/all.yml
|
||||||
|
|
||||||
|
# Draai het playbook
|
||||||
|
ansible-playbook -i inventory/hosts.yml site.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Specifieke fases draaien
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Alleen teamfolders aanmaken
|
||||||
|
ansible-playbook -i inventory/hosts.yml site.yml --tags teamfolders
|
||||||
|
|
||||||
|
# Alleen bestanden + DB registratie
|
||||||
|
ansible-playbook -i inventory/hosts.yml site.yml --tags files,db-register
|
||||||
|
|
||||||
|
# Alleen metadata records invoegen
|
||||||
|
ansible-playbook -i inventory/hosts.yml site.yml --tags metadata-records
|
||||||
|
```
|
||||||
|
|
||||||
|
### Opruimen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ansible-playbook -i inventory/hosts.yml cleanup.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fases
|
||||||
|
|
||||||
|
1. **Pre-check** — Nextcloud bereikbaarheid en schijfruimte controleren
|
||||||
|
2. **Teamfolders** — 50 teamfolders aanmaken via Groupfolders API
|
||||||
|
3. **Metadata velden** — 100 velddefinities aanmaken en toewijzen via MetaVox API
|
||||||
|
4. **Bestanden** — 500K `.txt` bestanden direct op filesystem schrijven
|
||||||
|
5. **DB registratie** — Bestanden registreren in `oc_filecache` via MySQL
|
||||||
|
6. **Mappenstructuur** — Submappen aanmaken en bestanden verdelen
|
||||||
|
7. **Drop indexes** — 7 redundante indexes verwijderen van `oc_metavox_file_gf_meta`
|
||||||
|
8. **Metadata records** — ~43M records invoegen in `oc_metavox_file_gf_meta` en `oc_metavox_gf_metadata`
|
||||||
|
|
||||||
|
## Database indexes
|
||||||
|
|
||||||
|
Na het droppen van 7 redundante indexes blijven er 4 over op `oc_metavox_file_gf_meta`:
|
||||||
|
|
||||||
|
| Index | Kolommen |
|
||||||
|
|-------|----------|
|
||||||
|
| PRIMARY | id |
|
||||||
|
| mf_file_gf_meta_unique | file_id, groupfolder_id, field_name |
|
||||||
|
| idx_file_gf_gf_lookup | groupfolder_id, field_name, field_value |
|
||||||
|
| idx_gf_file_meta_filter | field_name, field_value, groupfolder_id |
|
||||||
|
|
||||||
|
## Configuratie
|
||||||
|
|
||||||
|
Alle instellingen staan in `group_vars/all.yml`:
|
||||||
|
|
||||||
|
| Variabele | Default | Beschrijving |
|
||||||
|
|-----------|---------|-------------|
|
||||||
|
| `num_teamfolders` | 50 | Aantal teamfolders |
|
||||||
|
| `files_per_teamfolder` | 10000 | Bestanden per teamfolder |
|
||||||
|
| `sql_batch_size` | 2000 | Rijen per INSERT statement |
|
||||||
|
| `folder_structure` | 10×3 | Mappenstructuur definitie |
|
||||||
Reference in New Issue
Block a user