Machine Learning Primer, C. Gros, WS 24/25 (2024)

Claudius Gros, WS 2024/25

Institut für theoretische Physik
Goethe-University Frankfurt a.M.

history

• Unix $\to$ Linux
  • 1960s/70s: Unix (AT&T) secure, multi-user
  • 1991: Linus kernel (Linus Torval) open source
  • 1990s Gnu libaries and untilities added
    (Free Software Foundation)
• typing, not clicking
  :: you know what you do (hopefully)
• scripts
• ...

Linux directory structure

• cd directory: change to directory
• cd / : change to root
• cd &nbsp: change to home
• cd ..: one up
• cd . : to current directory
• ls : list directory content

user@pc:~$ cd /user@pc:/$ cd varuser@pc:/var$ 

mkdir - create directory

• mkdir directory: create directory
• mkdir -p foo/test/abc/123/bar:
  create directory foo/test/abc/123/bar and all parents if necessary

user@pc:~$ mkdir testuser@pc:~$ lstest

rmdir – remove directory

• rmdir directory: delete directory
• only works on empty directories

user@pc:~$ rmdir testuser@pc:~$ rmdir downloadsrmdir: failed to remove `downloads': Directory not emptyuser@pc:~$ 

recursive removal

• rm -r directory/
  delete directory and all subdirectories

more – show file on screen

• more file: show file on screen
• move with ↑ and ↓
• exit with Q
• more /proc/cpuinfo : show processor informations

Linux file names

• Hallo.dat and hallo.dat are distinct:
  capital sensitive
• data.really-important-stuff: arbitrary file extentions
• avoid empty spaces

mv – move/rename files

• mv file1 file2: rename file1 to file2
• mv file directory/: move file to directory/
• mv myImages/*.jpg /home/user/otherUser/yourImages/:
  move all jpg files in the directory myImages/
  to the directory /home/user/otherUser/yourImages/

user@pc:~$ mv foo.txt bar.txtuser@pc:~$ mv bar.txt downloadsuser@pc:~$

rm – remove files or directories

• rm file: delete file
• rm directory: delete directory
• rm -r directory: recursive delete directory and all containing files
• rm image*: delete all files starting with image
• rm -r *: remove all data you ever created

user@pc:~$ rm downloads/bar.txt user@pc:~$ rm -r downloads/user@pc:~$ 

cat – concatenate files and print to standard output

• cat file: print contents of file
• cat file1 file2 > file3: concatenate the files file1 and file2
  and print output to file3
• cat aaa* > aaa.all: concatenate all files starting with aaa

user@pc:~$ cat file1first line of file1second line of file1user@pc:~$ cat file2first line of file2second line of file2user@pc:~$ cat file* > alluser@pc:~$ cat allfirst line of file1second line of file1first line of file2second line of file2

hidden files:
files starting with a . are hidden

• used for setting user preferences/configuration for applications
• .vimrc settings for vi/vim editor
• .eclipse settings for the Eclipse development environment
• ...

user@pc:~/TEST$ ls -ltotal 32-rw-rw-r-- 1 user user 1775 2012-03-09 17:58 HashMapMapDemo.class-rw-rw-r-- 1 user user 2019 2012-03-09 17:59 HashMapMapDemo.java-rw-rw-r-- 1 user user 20961 2012-03-09 15:45 hashMap.svguser@pc:~/TEST$ ls -latotal 56drwxrwxr-x 2 user user 4096 2012-03-12 09:57 .drwxr-xr-x 153 user user 12288 2012-03-12 09:56 ..-rw-rw-r-- 1 user user 1775 2012-03-09 17:58 HashMapMapDemo.class-rw-rw-r-- 1 user user 2019 2012-03-09 17:59 HashMapMapDemo.java-rw-rw-r-- 1 user user 20961 2012-03-09 15:45 hashMap.svg-rw-rw-r-- 1 user user 3 2012-03-12 09:56 .I-am-a-hidden-file

ls – list directory contents

• ls -l: long listing format
• ls -a: list all files
• ls -h: human readable
• ls *dat: all files ending with 'dat':
  use wildcards

convert – audio, video and text formats

• man convert: for options
• convert -delay 50 frame*.gif -loop 0 animated.gif
  converts frame01.gif, frame02.gif, .., into an animated gif
• convert test.gif frame%04d.gif
  extract frames from animated gif as size-4 integers
• the basis of all crawlers: wget :: downloading files and entire webpages
• image viewer: eog ('eye of gnome')

wget https://upload.wikimedia.org/wikipedia/commons/3/30/Jud-morote-gari.gifeog Jud-morote-gari.gifmv Jud* test.gifconvert test.gif frame%04d.gifconvert -delay 50 frame*.gif -loop 0 animated.gifeog animated.gif

ps – report a snapshot of the current processes

• ps: list current processes
• ps aux: list all processes

user@pc:~$ ps auxUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 4020 476 ? Ss Feb20 1:08 /sbin/initroot 2 0.0 0.0 0 0 ? S Feb20 0:00 [kthreadd]root 3 0.0 0.0 0 0 ? S Feb20 0:02 [migration/0]root 4 0.0 0.0 0 0 ? S Feb20 0:23 [ksoftirqd/0]root 5 0.0 0.0 0 0 ? S Feb20 0:00 [watchdog/0]root 6 0.0 0.0 0 0 ? S Feb20 0:04 [migration/1]root 7 0.0 0.0 0 0 ? S Feb20 1:04 [ksoftirqd/1]1635 4650 0.0 0.0 92248 2752 ? S Apr21 0:00 /usr/lib/eclipse/eclipse -vm /usr/lib/jvm/java-gcj/bin/java -install /usr/lib/eclipse1635 4656 0.7 9.4 1021416 381456 ? Sl Apr21 29:27 /usr/lib/jvm/java-gcj/bin/java -Djava.library.path=/usr/lib/jni -Dgnu.gcj.precompileddaemon 4715 0.0 0.0 8084 524 ? Ss Feb20 0:00 /sbin/portmaproot 4776 0.0 0.0 740 36 ? Ss Feb20 0:37 /usr/local/bin/netbeep eth0statd 4813 0.0 0.0 14416 816 ? Ss Feb20 0:00 /sbin/rpc.statd -p 2051 -o 20521635 17246 0.0 0.1 17052 7332 pts/1 Ss 09:13 0:00 bashuser@pc:~$ 

kill – send a signal to a process

• kill pid: send signal TERM to process with id pid
• kill -9 pid: kill process with id pid
• killall firefox: kill all processes with process name firefox

user@pc:~$ ps -lF S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD0 S 1467 8100 8095 0 80 0 - 8641 wait pts/0 00:00:02 bash0 S 1467 16732 8100 1 80 0 - 133912 poll_s pts/0 00:00:00 kwrite0 R 1467 16797 8100 0 80 0 - 5796 - pts/0 00:00:00 ps0 S 1467 26446 1 0 80 0 - 221014 futex_ pts/0 00:02:11 soffice.binuser@pc:~$ kill 16732

| - output of one command as input of next

• sort filename: sort the file filename
• grep expression: extract all lines containing the expression from output
• create files with content with echo

echo 'Ralph JunghansAnna RosenkranzJoachim NeumayerAnna Furlger' > names.txt

user@pc:~$ more names.txt Ralph JunghansAnna RosenkranzJoachim NeumayerAnna Furlgeruser@pc:~$ more names.txt | grep "Anna" | sort > names.annauser@pc:~$ more names.anna Anna FurlgerAnna Rosenkranz

wc - counting lines/words/characters

user@pc:~$ more names.txt | grep AnnaAnna RosenkranzAnna Furlgeruser@pc:~ more names.txt | grep Anna | wc 2 4 29user@pc:~$ wc names.txt 4 8 61 names.txt

use the editor you feel at ease with

• emacs, kwrite, .. possible editors
• vi/vim: standard editor
  keyboard controlled - no need to use the mouse

i		start inserting modus
R		start replacing modus
o		start insert in next line
a		start inserting at end of current line
esc		exit editing modi
u		undo last insertion or change
r		replace current char
x		delete char
7x		delete seven chars
dd		delete line
8dd		delete eight line
y		copy (yank) current line to buffer
9y		copy (yank) nine lines to buffer
p		insert (paste) buffer
j		move cursor a line down
k		move cursor a line up
h		move cursor to the left
l		move cursor to the right
:w		save (write) file
:q		exit (quit) vi
:wq		write and exit
:111		goto line 111
:1,11d		delete lines 1 through 11
:1,$s/error/correct/		substitute error by correct throught the entire file
:10,20s/yes/no/g		substitution from line 10 to twenty, also multiple times per line (g)

using university computers from home

• ssh remote-host: establish a connection to remote host
  itp.uni-frankfurt.de
  th.physik.uni-frankfurt.de
  
• scp remote-host-path .: transfer data from remote host to current directory
• scp ./filename remote-host-path : transfer data from current directory to remote host
• g++ : compile a C++ program
• time... : measuring the time needed to excute a program
• nohup .a/out & : starting a program which runs in the background (&)
  and which continues to be executed after logging-out (nohup)

user@pc:~$ ssh -l gros karlstad.th.physik.uni-frankfurt.degros@karlstad.th.physik.uni-frankfurt.de's password: gros@karlstad:~$ cd TEST/gros@karlstad:~$ javac Test.java gros@karlstad:~$ nohup time java Test test.old test.new &[1] 2176gros@karlstad:~$ nohup: ignoring input and appending output to `nohup.out'gros@karlstad:~$ exitlogoutConnection to karlstad.th.physik.uni-frankfurt.de closed.user@pc:~$ scp gros@karlstad.th.physik.uni-frankfurt.de:TEST/nohup.out .gros@karlstad.th.physik.uni-frankfurt.de's password: nohup.out 100% 326 0.3KB/s 00:00 user@pc:~$ more nohup.out # starting Test.main()# scanning the file test.old# scanning the file test.new 1000 lines in test.old 1000 lines in test.new 84 lines in test.output0.15user 0.02system 0:00.21elapsed 77%CPU (0avgtext+0avgdata 77248maxresident)k888inputs+80outputs (6major+5261minor)pagefaults 0swaps

• ping remote-host: checking availability and connection quality

user@pc:~$ ping 141.2.247.163PING 141.2.247.163 (141.2.247.163) 56(84) bytes of data.64 bytes from 141.2.247.163: icmp_req=1 ttl=64 time=4.99 ms64 bytes from 141.2.247.163: icmp_req=2 ttl=64 time=0.132 ms64 bytes from 141.2.247.163: icmp_req=3 ttl=64 time=0.263 ms64 bytes from 141.2.247.163: icmp_req=4 ttl=64 time=0.299 ms

Directory Operations

Editors

FileOperations

FileManipulation

Info and Documentation

Data Compression

Miscellaneous

ProcessManagement

SystemInfos

RemoteConnections

automated execution of commands

• chmod +x: make file executable (x)

user@pc:~$ vi test.shuser@pc:~$ chmod -x test.sh

• echo : print message to screen
• sleep 8: sleep for 8 seconds

#! /bin/bash i=0N=2while test $i -le $N do a=`expr 50 \\* $i` b=`expr $a + 1` more "inputFile_$i.dat" | grep "Anna" > "outFile_$b.out" echo "ich gehe jetzt für $i Sekunden schlafen " $i sleep $i; i=`expr $i + 1`done cat outFile_*.out > outFile.all

• create input files

echo 'Ralph JunghansAnna RosenkranzJoachim NeumayerAnna Furlger' > inputFile_0.datecho 'Keine Anna hier,oder etwa doch?' > inputFile_1.datecho 'Hier ist nun wirklichkeine anna zu finden.' > inputFile_2.dat

• run script

user@pc:~$ lsinputFile_0.dat inputFile_1.dat inputFile_2.dat test.shuser@pc:~$ ./test.sh ich gehe jetzt für 0 Sekunden schlafen 0ich gehe jetzt für 1 Sekunden schlafen 1ich gehe jetzt für 2 Sekunden schlafen 2user@pc:~$ lsinputFile_0.dat inputFile_2.dat outFile_1.out outFile.allinputFile_1.dat outFile_101.out outFile_51.out test.sh

• check output files

user@pc:~$ more inputFile_*.dat::::::::::::::inputFile_0.dat::::::::::::::Ralph JunghansAnna RosenkranzJoachim NeumayerAnna Furlger::::::::::::::inputFile_1.dat::::::::::::::Keine Anna hier,oder etwa doch?::::::::::::::inputFile_2.dat::::::::::::::Hier ist nun wirklich keine anna zu finden.user@pc:~$ more outFile.all Anna RosenkranzAnna FurlgerKeine Anna hier,

a very large selection of free utilities

• gnuplot datafile
  `fast and dirty' visualization of xy, xyz or multicolumn data files
• xmgrace datafile
  more elaborate plotting of xy, xyz or multicolumn data files
• gimp
  advanced editor for bitmap images
• inkscape
  advanced editor for scalable vector graphics
• alpine
  console mail program
• libreoffice
  OpenOffice (LibreOffice) package
• dot / neato / circo
  Graphviz graph ploting utilities
• awk
  awk is a lightweight text manipulation utility