Menu Zamknij

„Praca domowa” SysAdmina – part 1: disk baselines

Historia pewnej rekrutacji

Jakiś czas temu zostałem zaproszony na proces rekrutacyjny na stanowisko SysAdmina przez pewną firmę i dostałem „pracę domową” składającą się z dwóch zadań mających sprawdzić moje praktyczne umiejętności i podejście do problemu. Firma okazała się turboniepoważna i mimo pozytywnego feedbacku od autora zadań kilka dni później, zasłaniając się „wewnętrznymi problemami” poprosiła mnie o chwilę cierpliwości. Kilkadziesiąt chwil później kontakt się urwał i po 3 tygodniach uznałem że czas o nich zapomnieć. Ale zadania zostały i warte są opublikowania. Nic nie wspomniano o zakazie publikacji rozwiązań, a teraz nie zamierzam o to pytać. Tak czy inaczej – jeśli dostaniecie od jakiejś firmy identyczne lub podobne zadania – uważajcie na nich.

Wstęp do zadań

Zadania były dwa i podzielę je na dwa posty. Z drobnego lenistwa rozwiązania zostawię w oryginalnej formie i języku w którym je przesłałem (czyli angielskim).

Na sam początek kilka słów o środowisku testowym, systemie prezentacji wyników i założeniach. Dla niecierpliwych – ładne wykresy są na końcu 😉

System

Hardware

  • VMware virtual machine hosted on Windows PC
  • 16GB RAM
  • 1x 8 core CPU (Intel Xeon E5540)
  • System disk – virtual SCSI 20GB ext4
  • hosted on LSI SAS card, RAID1, NTFS formatted 15k RPM hard drive

Software

  • OS – Ubuntu Server 18.04 LTS
  • installed using standard wizard
  • Database – PostgreSQL 10

Storage

I decided to have one VM setup for both exercises so data disk is optimized for PostgreSQL database. Data disk (`/home`) is hosted on same physical matrix as system disk. Binaries and test reports stored on different disk than main IO operations not to affect baselines/performance.

Data visualization

Methodology:

  • all scrips produce csv which is easily parsable by everything from awk, including quick Excel up to tools like R and plotly.js
  • plotly.js was decided as it provides interactive graphs in html+js form which is easily available to display everywhere; it allows pretty advanced plotting and data import from various sources like CSV files
  • I used it previously for big CSVs (20-30MB logs) without a problem on modern browsers.

Including JS code of each graph page in this document would be pointless but scaffold code is linked to show simplicity: https://codepen.io/plotly/pen/eNaErL

Basic concept of JS code

  • loadFile selects which data set to work on
  • Plotly.d3.csv loads CSV over network to memory
  • processData gets associative table, parses row by row and adds to new 1 dimensional tables and some additional computations like avg values are done
  • Plotly.newPlot fills HTML div with data series and layout config

Methodology and tricks common for both tasks

  • `###` denotes some omitted lines from verbose commands
  • `ntfy` which appears in long running tasks is alias using ntfy script that can send messages to different chat providers like slack or telegram; it’s quite useful to step away from terminal (in-browser notification) or computer (mobile phone notification)
  • https://pypi.org/project/ntfy/

Zadanie 1 – disk baselines

Pierwsze polegało na przeprowadzeniu testów obciążeniowych dysku i zebranie metryk wydajności w formie umożliwiającej łatwe zobrazowanie w formie wykresów.

Environment

apt install bonnie++ sysstat

First iteration – iostat-csv + R/GSheet

Data collection

Simple wrapper from internet

[email protected]:/home/sah# cd /var/sah/bin; git clone https://github.com/ymdysk/iostat-csv.git
[email protected]:/home/sah#

Simple test – `bonnie++` simple test (2x RAM size) on data disk, simple wrapper over `iostat -x` that ouputs csv run in parallel.

[email protected]:/home/sah# time bonnie++ -d /home/sah/tests/ -r 16192 -u daniel
Using uid:1000, gid:1000.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
sah 32G 203 99 28598 9 52906 14 441 99 156629 24 649.2 32
Latency 54332us 2904ms 302ms 31186us 78416us 209ms
Version 1.97 ------Sequential Create------ --------Random Create--------
sah -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17954 47 +++++ +++ 27306 51 21447 54 +++++ +++ 27218 48
Latency 867us 680us 1587us 566us 112us 1260us
1.97,1.97,sah,1,1537736827,32G,,203,99,28598,9,52906,14,441,99,156629,24,649.2,32,16,,,,,17954,47,+++++,+++,27306,51,21447,54,+++++,+++,27218,48,54332us,2904ms,302ms,31186us,78416us,209ms,867us,680us,1587us,566us,112us,1260us

real 34m24.300s
user 0m10.288s
sys 4m34.876s
[email protected]:/home/sah#
[email protected]:/home/daniel# /var/sah/bin/iostat-csv/iostat-csv.sh | tee -a /var/sah/task1_iter1.csv
Date,Time,%user,%nice,%system,%iowait,%steal,%idle,Device,r/s,w/s,rkB/s,wkB/s,rrqm/s,wrqm/s,%rrqm,%wrqm,r_await,w_await,aqu-sz,rareq-sz,wareq-sz,svctm,%util,Device,r/s,w/s,rkB/s,wkB/s,rrqm/s,wrqm/s,%rrqm,%wrqm,r_await,w_await,aqu-sz,rareq-sz,wareq-sz,svctm,%util,Device,r/s,w/s,rkB/s,wkB/s,rrqm/s,wrqm/s,%rrqm,%wrqm,r_await,w_await,aqu-sz,rareq-sz,wareq-sz,svctm,%util,Device,r/s,w/s,rkB/s,wkB/s,rrqm/s,wrqm/s,%rrqm,%wrqm,r_await,w_await,aqu-sz,rareq-sz,wareq-sz,svctm,%util
09/24/18,02:24:56,0.10,0.00,0.62,2.39,0.00,96.89,loop0,0.01,0.00,0.07,0.00,0.00,0.00,0.00,0.00,1.36,0.00,0.00,10.79,0.00,0.11,0.00,loop1,0.00,0.00,0.02,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,6.60,0.00,0.00,0.00,sda,1.18,1.03,34.43,32.15,0.09,2.44,7.07,70.40,15.24,181.59,0.20,29.16,31.35,8.08,1.78,sdb,31.94,4.30,4067.23,4259.74,0.11,0.70,0.34,14.02,2.39,3352.68,14.48,127.34,990.63,3.65,13.23
09/24/18,02:24:57,0.00,0.00,0.00,0.00,0.00,100.00,loop0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,loop1,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,sda,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,sdb,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
###...

Data parsing

[email protected]:/var/sah# tail -n2300 task1_iter1.csv > task1_iter1_stripped.csv
[email protected]:/var/sah# cat task1_iter1.csv | head -n1 | tr ',' '\n' | cat -n
## column numbers for awk
[email protected]:/var/sah# cat task1_iter1.csv | head -n 2300 | awk 'BEGIN {FS=",";OFS=","}{print $1,$2,$3,$5,$58,$59,$60,$61,$66,$67,$68}' > task1_iter1_stripped.csv
Date,Time,%user,%system,r/s,w/s,rkB/s,wkB/s,r_await,w_await,aqu-sz
09/24/18,02:24:56,0.10,0.62,31.94,4.30,4067.23,4259.74,2.39,3352.68,14.48
09/24/18,02:24:57,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:24:58,0.75,8.51,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:24:59,1.62,11.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:00,1.25,11.25,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:01,1.12,11.49,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:02,1.25,11.61,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:03,1.25,11.25,0.00,1.98,0.00,23.76,0.00,8.00,0.02
09/24/18,02:25:04,1.25,11.36,0.00,0.00,0.00,0.00,0.00,0.00,0.00
[email protected]:/var/sah# head task1_iter1_stripped.csv
Date,Time,%user,%system,r/s,w/s,rkB/s,wkB/s,r_await,w_await,aqu-sz
09/24/18,02:24:56,0.10,0.62,31.94,4.30,4067.23,4259.74,2.39,3352.68,14.48
09/24/18,02:24:57,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:24:58,0.75,8.51,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:24:59,1.62,11.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:00,1.25,11.25,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:01,1.12,11.49,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:02,1.25,11.61,0.00,0.00,0.00,0.00,0.00,0.00,0.00
09/24/18,02:25:03,1.25,11.25,0.00,1.98,0.00,23.76,0.00,8.00,0.02
09/24/18,02:25:04,1.25,11.36,0.00,0.00,0.00,0.00,0.00,0.00,0.00
[email protected]:/var/sah#

Data visualization

First iteration in R

#!/usr/bin/env Rscript
library(lattice)
d = read.csv("d:/sah/task1_iter1_stripped.csv", header = TRUE)

# crude all-in-one
f <- paste(paste(names(d[,-1,drop=FALSE]), collapse="+"), names(d[,2,drop=FALSE]), sep=" ~ ")
xyplot(as.formula(f), data=d, type='l', auto.key=list(space='right'))

R is not suitable for interactive analysis or any easy analysis at all with huge datasets that need every point to be shown.

Second iteration in GSheet, kind of disappointing – https://docs.google.com/spreadsheets/d/13QFwxlwpk5QKAj-fj3VX_o49nSNUIYdHEL54ijR_DiQ/edit?usp=sharing

Second iteration – custom watcher script and plotly.js

Custom script `statsWatcher.sh`

#!/bin/bash
dev=$1
out=$2
if [[ -z $dev || -z $out ]]; then
echo "usage: $0 diskToMonitor outputLogFilename"
exit 1
fi

echo "timestamp,mem_free,mem_buff,mem_cache,cpu_user,cpu_sys,cpu_iowait,r_s,w_s,rkB_s,wkB_s,r_await,w_await,avg_q,util_perc" >> $out

while [[ true ]]; do
date=`date +%s`
diskStats=`iostat -t -x $dev -y 1 1 | tail -n 2| head -n1 | awk '{print $2","$3","$4","$5","$10","$11","$12","$16}'`
#r/s,w/s,rkB/s,wkB/s,r_await,w_await,aqu-sz,%util
sysStats=`vmstat -w 1 2 | tail -n1 | awk '{print $4","$5","$6","$13","$14","$16}'`
#mem_free,mem_buff,mem_cache,cpu_user,cpu_sys,cpu_iowait
echo $date","$sysStats","$diskStats >> $out

#sleep 1
done

`sleep 1` is commented out since `iostat` and `vmstat` need to run for one second to collect current statistics. First output line is average value from boot time until now so it’s not valuable.

Simple test benchmark

Simple test was performed to see how data is displayed on graphs. It consisted of following commands executed on another machine:

thor:/# dd if=/dev/urandom of=/home/xxxx bs=1M count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.30361 s, 129 MB/s
thor:/# dd if=/dev/urandom of=/home/xxxx bs=1M count=100k
^C7253+0 records in
7252+0 records out
7604273152 bytes (7.6 GB, 7.1 GiB) copied, 69.5066 s, 109 MB/s
thor:/# dd of=/dev/urandom if=/home/xxxx bs=1M count=100k
7252+0 records in
7252+0 records out
7604273152 bytes (7.6 GB, 7.1 GiB) copied, 59.418 s, 128 MB/s
thor:/# dd of=/home/xxxx2 if=/home/xxxx bs=1M count=100k
7252+0 records in
7252+0 records out
7604273152 bytes (7.6 GB, 7.1 GiB) copied, 59.418 s, 128 MB/s
thor:/# dd of=/home/xxxx2 if=/home/xxxx bs=1M count=100k
7252+0 records in
7252+0 records out
7604273152 bytes (7.6 GB, 7.1 GiB) copied, 147.806 s, 51.4 MB/s
thor:/# rm /home/xxxx2 /home/xxxx
thor:/#

Result of running `./statsWatcher.sh /dev/sdb testLog.csv` (first 20 lines):

timestamp,mem_free,mem_buff,mem_cache,cpu_user,cpu_sys,cpu_iowait,r_s,w_s,rkB_s,wkB_s,r_await,w_await,avg_q,util_perc
1538404489,9254124,206608,1590424,0,0,0,0.00,1.00,0.00,0.50,0.00,4.00,0.00,0.40
1538404491,9257000,206608,1590452,0,0,0,0.00,52.00,0.00,712.50,0.00,2.08,0.14,14.00
1538404493,9256604,206608,1590452,0,0,0,0.00,15.00,0.00,57.00,0.00,2.93,0.04,2.00
1538404495,9256812,206608,1590452,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404497,9256788,206608,1590452,0,0,0,0.00,48.00,0.00,629.00,0.00,3.50,0.17,16.00
1538404499,9256812,206608,1590452,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404501,9256004,206608,1590452,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404503,9256948,206608,1590424,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404505,9257516,206608,1590424,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404507,9257104,206608,1590424,0,0,0,0.00,51.00,0.00,645.00,0.00,3.45,0.18,17.20
1538404509,9257252,206608,1590424,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404511,9257352,206608,1590448,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404513,9257428,206608,1590448,0,0,0,0.00,4.00,0.00,9.00,0.00,9.00,0.04,3.60
1538404515,9257396,206608,1590448,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404517,9263560,206608,1590444,0,0,0,0.00,49.00,0.00,637.00,0.00,3.67,0.18,17.60
1538404519,9263400,206608,1590444,0,0,0,0.00,3.00,0.00,5.00,0.00,12.00,0.04,3.60
1538404521,9264192,206608,1590420,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
1538404523,9264472,206608,1590416,0,0,0,0.00,3.00,0.00,5.00,0.00,9.33,0.03,2.80
1538404525,9265104,206608,1590416,0,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00

Graphs

Plotly.js was described in intro. Graphs created with final version of plotly html template: https://storage.dsinf.net/sah/task1_benchmarks/baselines.html#testLog

Real baselines

Target disk

I decided to add third virtual disk that can be moved to different physical disk on my workstation and reformat it at ease (`/home` has postgresql for task2).

Here’s setup of it under VMware:

  • virtual SCSI connected
  • independent (not affected by snapshots), persistent
  • 16GB (same as RAM)
  • preallocated
  • stored as single file on host filesystem
  • first stored on same disk as rootfs and `/home` (SAS 15k RPM HDD)
Test suite

It’ll run baselines collector for 15 seconds without load, then perform 3 tests of bonnie++ (-x3 on 15G output data sizew) and again collect 15 minutes of idle stats. 3 files are created – bonnie.csv with raw results, bonnie.html with html formatted data and baseline.csv with IO and CPU stats.

logname=filesystem_ext4journal; 
./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html

Testing various filesystems

I’ve chosen same filesystems as in Task2 for database storage selection.

ext4 journalled

[email protected]:/# mkfs.ext4 /dev/sdc1
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 4194048 4k blocks and 1048576 inodes
Filesystem UUID: 68d9ad29-e5ad-4040-b5a8-648b3d686da5
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
[email protected]:/# mount /dev/sdc1 /test/
[email protected]:/#

[email protected]:~/task1# logname=filesystem_ext4journal; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 9200
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

ext4 no journall

[email protected]:~/task1# mkfs.ext4 -O ^has_journal /dev/sdc1
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdc1 contains a ext4 file system
last mounted on /test on Mon Oct 1 16:14:47 2018
Proceed anyway? (y,N) y
Creating filesystem with 4194048 4k blocks and 1048576 inodes
Filesystem UUID: d9c66b94-4464-40cd-bc6d-5f6758bed555
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

[email protected]:~/task1# mount /dev/sdc1 /test
[email protected]:~/task1# logname=filesystem_ext4nojournal; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 15939
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

JFS

[email protected]:~/task1# umount /test/
[email protected]:~/task1# mkfs.jfs /dev/sdc1
mkfs.jfs version 1.1.15, 04-Mar-2011
Warning! All data on device /dev/sdc1 will be lost!

Continue? (Y/N) y
\

Format completed successfully.

16776192 kilobytes total disk space.
[email protected]:~/task1# mount /dev/sdc1 /test/
[email protected]:~/task1# logname=filesystem_jfs; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 22842
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

XFS

[email protected]:~/task1# umount /test/
[email protected]:~/task1# mkfs.xfs /dev/sdc1
mkfs.xfs: /dev/sdc1 appears to contain an existing filesystem (jfs).
mkfs.xfs: Use the -f option to force overwrite.
[email protected]:~/task1# mkfs.xfs -f /dev/sdc1
meta-data=/dev/sdc1 isize=512 agcount=4, agsize=1048512 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=4194048, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[email protected]:~/task1# mount /dev/sdc1 /test/
[email protected]:~/task1# logname=filesystem_xfs; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 30026
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

Testing various physical disks

The same ext4 formatted disk will be moved between following disks:

cheetah

  • SAS connected 15k RPM HDD in hardware RAID1
  • same as /home
  • results stored in physical_cheetah

SSD

  • SATA2 connected modern SSD drive in software RAID1
  • host OS disk
  • results stored in `physical_ssd`

HDD

  • SATA2 connected enterprise grade HDD WD Black disk
  • results stored in physical_hdd

Testing various block sizes on ext4

Those are only valid block sizes for ext4

1024

[email protected]:~/task1# umount /test/
[email protected]:~/task1# mkfs.ext4 -b 1024 /dev/sdc1
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdc1 contains a ext4 file system
last mounted on /test on Mon Oct 1 21:20:22 2018
Proceed anyway? (y,N) y
Creating filesystem with 16776192 1k blocks and 1048576 inodes
Filesystem UUID: aebf6ae7-2b1b-4918-bd54-41ded0ee9028
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409, 663553,
1024001, 1990657, 2809857, 5120001, 5971969

Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done

[email protected]:~/task1# mount /dev/sdc1 /test
[email protected]:~/task1# logname=ext4blocksize1024; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 51212
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

2048

[email protected]:~/task1# umount /test/
[email protected]:~/task1# mkfs.ext4 -b 2048 /dev/sdc1
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdc1 contains a ext4 file system
last mounted on /test on Mon Oct 1 20:50:58 2018
Proceed anyway? (y,N) y
Creating filesystem with 8388096 2k blocks and 1048576 inodes
Filesystem UUID: 1cd1df52-8d8b-4f92-8540-258ae88fc847
Superblock backups stored on blocks:
16384, 49152, 81920, 114688, 147456, 409600, 442368, 802816, 1327104,
2048000, 3981312, 5619712

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[email protected]:~/task1# mount /dev/sdc1 /test
[email protected]:~/task1# logname=ext4blocksize2048; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 44356
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

4096

[email protected]:~/task1# tune2fs -l /dev/sdc1 | grep -i block\ size
Block size: 4096
[email protected]:~/task1# cp filesystem_ext4journal.baseline.csv ext4blocksize4096.baseline.csv
[email protected]:~/task1# cp filesystem_ext4journal.bonnie.csv ext4blocksize4096.bonnie.csv
[email protected]:~/task1# cp filesystem_ext4journal.bonnie.html ext4blocksize4096.bonnie.html

8192

[email protected]:~/task1# umount /test/
[email protected]:~/task1# mkfs.ext4 -b 8129 /dev/sdc1
Warning: blocksize 8129 not usable on most systems.
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdc1 contains a ext4 file system
last mounted on Mon Oct 1 20:26:17 2018
Proceed anyway? (y,N) y
Creating filesystem with 4194048 4k blocks and 1048576 inodes
Filesystem UUID: 90417a62-0277-4cc8-92b5-4ac9efdaef49
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

[email protected]:~/task1# mount /dev/sdc1 /test/
[email protected]:~/task1# logname=ext4blocksize8192; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s15G -r7.5G -x3 -q > $logname.bonnie.csv; sleep 15; kill -9 `jobs -p`; bon_csv2html $logname.bonnie.csv > $logname.bonnie.html
[1] 37635
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

Testing various thread count

Tested 2 threads of bonnie++

[email protected]:~/task1# logname=threads_2; ./statsWatcher.sh /dev/sdc $logname.baseline.csv & sleep 15; bonnie++ -u root -d /test/ -s7G -r3.5G -x3 -q > $logname.1.bonnie.csv & bonnie++ -u root -d /test/ -s7G -r3.5G -x3 -q > $logname.2.bonnie.csv; sleep 1m; kill -9 `jobs -p`; bon_csv2html $logname.1.bonnie.csv > $logname.bonnie.html; bon_csv2html $logname.2.bonnie.csv >> $logname.bonnie.html
[1] 90591
[2] 90669
[2]+ Done bonnie++ -u root -d /test/ -s7G -r3.5G -x3 -q > $logname.1.bonnie.csv
[1]+ Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[email protected]:~/task1#

Baselines of ETL (task2)

5 tests were performed for baselining purposes:

data download & unpacking it – etl_dataload

[email protected]:/home/csv# ./statsWatcher.sh /dev/sdb etl_dataload.csv & sleep 15; for i in `seq -f "%05g" 0 29`; do wget https://XXXXXXXX.s3.amazonaws.com/hive_csv_altus_gz/part-$i.gz; done; sleep 30; for i in `seq -f "%05g" 0 29`; do gzip -d part-$i.gz; done; sleep 15; kill -9 `jobs -p`; ntfy -b telegram send "done"
[1] 97612
[1]+ Killed ./statsWatcher.sh /dev/sdb etl_dataload.csv
[email protected]:/home/csv#

1.2GB chunks with 1 threads – etl_1_1

[email protected]:/home/csv# timepse "truncate table extract"
TRUNCATE TABLE
4.88
[email protected]:/home/csv# logname=etl_1_1; ./statsWatcher.sh /dev/sdb $logname.baseline.csv & sleep 15; ./multipleFilesLoader 1 $logname.log.csv; sleep 15; kill -9 `jobs -p`
[2] 126089
[1]- Killed ./statsWatcher.sh /dev/sdc $logname.baseline.csv
[2]+ Killed ./statsWatcher.sh /dev/sdb $logname.baseline.csv
[email protected]:/home/csv#

1.2GB chunks with 4 threads- etl_1_4

[email protected]:/home/csv# timepse "truncate table extract"
TRUNCATE TABLE
3.12
[email protected]:/home/csv# logname=etl_1_4; ./statsWatcher.sh /dev/sdb $logname.baseline.csv & sleep 15; ./multipleFilesLoader 4 $logname.log.csv; sleep 15; kill -9 `jobs -p`
[2] 17749
[1]- Killed ./statsWatcher.sh /dev/sdb $logname.baseline.csv
[2]+ Killed ./statsWatcher.sh /dev/sdb $logname.baseline.csv
[email protected]:/home/csv#

2.4GB chunks with 8 threads – etl_2_8

[email protected]:/home/csv# ls -alh part-*
-rw-r--r-- 1 root root 2.4G Oct 2 18:28 part-0_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:32 part-10_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:32 part-12_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:33 part-14_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:34 part-16_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:34 part-18_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:35 part-20_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:29 part-2_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:36 part-22_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:36 part-24_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:37 part-26_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:38 part-28_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:30 part-4_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:30 part-6_2
-rw-r--r-- 1 root root 2.4G Oct 2 18:31 part-8_2
[email protected]:/home/csv# timepse "truncate table extract"
TRUNCATE TABLE
1.25
[email protected]:/home/csv# logname=etl_2_8; ./statsWatcher.sh /dev/sdb $logname.baseline.csv & sleep 15; ./multipleFilesLoader 4 $logname.log.csv; sleep 15; kill -9 `jobs -p`
[1] 39851
[1]+ Killed ./statsWatcher.sh /dev/sdb $logname.baseline.csv
[email protected]:/home/csv#

pg_bulkload parallel – etl_pgbulkload

[email protected]:/home/csv# timepse "truncate table extract"
TRUNCATE TABLE
1.21
[email protected]:/home/csv# logname=etl_pgbulkload; ./statsWatcher.sh /dev/sdb $logname.baseline.csv & sleep 15; for f in part-*; do /home/pg_bulkload/bin/pg_bulkload -d sah -o

As with bonnie++ they were run with 15 seconds overlap of idle state.

Results

DELIMITER=\t' -i $f import_alldata.ctl; done; sleep 15; kill -9 `jobs -p`
[1] 50658
[1]+ Killed ./statsWatcher.sh /dev/sdb $logname.baseline.csv
[email protected]:/home/csv#

As with bonnie++ they were run with 15 seconds overlap of idle state.

Results

1 Komentarz

  1. Pingback:„Praca domowa” SysAdmina – part 2: ETL, czyli ładowanie dużej ilości danych do bazy danych – blog.dsinf.net

Leave a Reply