Listing posts

Displaying posts 1 - 5 of 333 in total
fish | friendly interactive shell attachment
mouse 3063 · person cloud · link
Last update
2024-09-19
2024
09-19
« — »

Resources

Extensions

Prompt power-up

  • decompress fish_config.7z attachment in ~/.config/fish/
  • curl -L https://get.oh-my.fish | fish
  • omf install bobthefish
  • download and install Meslo font from nerd-fonts
  • set MesloLGMDZ Nerd Font Mono font for your terminal
  • put set -g theme_nerd_fonts yes in your ~/.config/fish/config.fish
  • customize fish_greeting.fish and fish_right_prompt.fish in ~/.config/fish/functions/

On cygwin under windows you can install DejaVu Sans Mono ttf powerline font, and set fish as the default shell.

In the attached file there is a simple prompt with git support.

Commands

  • fish_update_completions -- scan man pages for completion tips
  • env VAR1=xxx VAR2=yyy command arg1 arg2 ... -- run command with env vars

~~~ * ~~~

Docker howto attachment
Last update
2024-09-17
2024
09-17
« — »

Installation on debian

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# check system compatibility
modprobe configs # loads /proc/config.gz
wget -q -O - https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh | \
  bash | tee docker-check.txt

# install docker: key, repo, packages
apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

# amd64 - x64
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker-ce.list
# armhf - x32 / raspberry pi / raspbian
echo "deb [arch=armhf] https://download.docker.com/linux/raspbian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker-ce.list

apt-get update && apt-get install docker-ce

# allow user to use docker
usermod -aG docker username

# test installation
docker version
docker info

# run a simple test image
docker run hello-world

See also post install for troubleshooting dns/network/remote access.

On raspberry pi just use curl -sSL https://get.docker.com | sh (repo not working).

Configure daemon

1
2
3
4
5
mkdir -p        /path/to/data
chown root.root /path/to/data
chmod 711       /path/to/data
echo '{ "data-root": "/path/to/data" }' > /etc/docker/daemon.json
systemctl restart docker
1
echo '{ "log-driver": "local" }' > /etc/docker/daemon.json

Creating an image (ref, best practices)

1
2
3
4
5
6
7
8
9
10
11
12
touch Dockerfile # and fill it
docker build -t test-myimg . # create the image with a tag

# test run image
docker run -p 4000:80    test-myimg
docker run -it test-myimg /bin/bash

# run image detached/on background
docker run -p 4000:80 -d --name tmi test-myimg
docker container ls -a
docker container stop <container_id>
docker container start -i tmi # restart container

Interact (ref)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# run interactive shell into debian image (temporary)
docker run --name prova --rm -it debian /bin/bash 

# run interactive shell into debian image
docker run -it debian /bin/bash 

apt-get update

apt-get install -y dialog nano ncdu
apt-get install -y locales

localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
echo "LANG=en_US.utf8" >> /etc/environment

rm -rf /var/lib/apt/lists/*

docker commit e2b7329257ba myimg:v1

docker run --rm -it myimg:v1 /bin/bash

# run a command in a running container
docker exec -ti a123098734e bash -il

docker stop a123098734e
docker kill a123098734e

Save & restore

1
2
3
4
5
6
7
8
9
10
# dump image
docker save imgname | gzip > imgname.tgz
zcat imgname.tgz | docker load

# dump container
docker create --name=mytemp imgname
docker export mytemp | gzip > imgname-container.tgz

# flatten image layers (losing Dockerfile) from a container
docker export <id> | docker import - imgname:tag

Registry - Image repository

1
2
3
4
5
# push image to gitlab registry
docker login registry.gitlab.com
docker tag test-myimg registry.gitlab.com/username/repo:tag # add new tag...
docker rmi test-myimg # ...and remove the old tag
docker push registry.gitlab.com/username/repo:tag

Tips

1
2
3
# remove untagged image -- https://stackoverflow.com/a/33913711/13231285
docker images --digests
docker image rm image-name@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxx

DockerHub official base images links: debian, ruby, rails, redis, nginx.

Available free registry services:

Name # Priv/Pub Notes
gitlab inf/ND 1 prj x registry
treescale inf/inf max 500 pulls & 50GB
canister 20/ND very good service
docker hub 1/inf perfect

Running arm image on x86

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# https://ownyourbits.com/2018/06/27/running-and-building-arm-docker-containers-in-x86/
apt-get install qemu-user-static

docker run \
  -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static \
  -e LANG=en_US.utf8 -ti --name myarmimg arm32v7/debian:wheezy

[...]

docker commit myarmimg myarmimg

docker container prune -f

docker run \
  -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static \
  -ti --rm --name myarmimg \
  myarmimg /bin/bash -il

Composer (ref, dl) - Services

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# docker-compose.yml
version: "3"
services:
  web:
    image: username/repo:tag
    deploy:
      replicas: 5
      resources:
        limits:
          cpus: "0.1"
          memory: 50M
      restart_policy:
        condition: on-failure
    ports:
      - "4000:80"
    networks:
      - webnet
networks:
  webnet:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# install docker-compose
curl -L  -o /usr/local/bin/docker-compose https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m`
chmod 755 /usr/local/bin/docker-compose

docker swarm init

docker stack deploy --with-registry-auth -c docker-compose.yml getstartedlab
docker service ls
docker service ps getstartedlab_web # or docker stack ps getstartedlab

# change the yml file and restart service
docker stack deploy --with-registry-auth -c docker-compose.yml getstartedlab
docker service ps getstartedlab_web
docker container prune -f

# stop & destroy service
docker stack rm getstartedlab
docker container prune -f

# leave the swarm
docker swarm leave --force

Machine (ref, dl) - SWARM/Provisioning

Remember to update the host firewall: open port 2376 and do not apply rate limits on port 22.

On the fish shell you can install the useful omf plugin-docker-machine to easily select the current machine.

Without an official supported driver we can use the generic one. Install docker-ce on your worker nodes and then in your swarm manager host:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# install docker-machine
curl -L -o /usr/local/bin/docker-machine https://github.com/docker/machine/releases/download/v0.16.1/docker-machine-`uname -s`-`uname -m`
chmod 755 /usr/local/bin/docker-machine

# setup each VMs (this creates and shares the certificates for a secure
# connetion between your client and the daemon runnig on the server)
ssh-copy-id -i ~/.ssh/id_rsa user@ww.xx.yy.zz
docker-machine create --driver generic --generic-ssh-key ~/.ssh/id_rsa \
  --generic-ip-address=ww.xx.yy.zz myvm1

ssh-copy-id -i ~/.ssh/id_rsa user@ww.xx.yy.kk
docker-machine create --driver generic --generic-ssh-key ~/.ssh/id_rsa \
  --generic-ip-address=ww.xx.yy.kk myvm2

docker-machine ls

# run a command via ssh in a VM
docker-machine ssh myvm1 "ls -l"                 # use internal SSH lib
docker-machine --native-ssh ssh myvm1 "bash -il" # use system SSH lib

# set env to run all docker commands remotely on a VM
eval $(docker-machine env myvm1) # on bash
docker-machine use myvm1         # on fish + omf plugin-docker-machine

# set VM1 to be a swarm manager
docker-machine use myvm1
docker swarm init # --advertise-addr ww.xx.yy.zz
docker swarm join-token worker # get token for adding worker nodes

# set VM2 to join the swarm as a worker
docker-machine use myvm2
docker swarm join --token SWMTKN-xxx ww.xx.yy.zz:2377

# check cluster status on your local machine...
docker-machine ls
# ...or on the manager node
docker-machine use myvm1
docker node ls

# locally login on your registry...
docker-machine unset
docker login registry.gitlab.com
# ...then deploy the app on the swarm manager
docker-machine use myvm1
docker stack deploy --with-registry-auth -c docker-compose.yml getstartedlab
docker service ls
docker service ps getstartedlab_web

# access cluster from any VM's IP
curl http://ww.xx.yy.zz:4000
curl http://ww.xx.yy.kk:4000

# eventually re-run "docker stack deploy ..." to apply changes

# undo app deployment
docker-machine use myvm1
docker stack rm getstartedlab

# remove the swarm
docker-machine ssh myvm2 "docker swarm leave"
docker-machine ssh myvm1 "docker swarm leave --force"

Stack / Deploy application

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# docker-compose.yml
version: "3"
services:
  web:
    image: username/repo:tag
    deploy:
      replicas: 5
      restart_policy:
        condition: on-failure
      resources:
        limits:
          cpus: "0.1"
          memory: 50M
    ports:
      - "80:80"
    networks:
      - webnet
  visualizer:
    image: dockersamples/visualizer:stable
    ports:
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    deploy:
      placement:
        constraints: [node.role == manager]
    networks:
      - webnet
  redis:
    image: redis
    ports:
      - "6379:6379"
    volumes:
      - "/home/docker/data:/data"
    deploy:
      placement:
        constraints: [node.role == manager]
    command: redis-server --appendonly yes
    networks:
      - webnet
networks:
  webnet:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
docker-machine use myvm1
docker-machine ssh myvm1 "mkdir ./data" # create redis data folder

# run stack / deploy app
docker stack deploy -c docker-compose.yml getstartedlab
docker stack ps getstartedlab

# show deployed services and restart one
docker service ls
docker service update --force getstartedlab_web

firefox http://<myvm1-ip>:8080/ # docker visualizer
redis-cli -h <myvm1-ip>         # interact with redis

docker stack rm getstartedlab

Init process to reap zombies and forward signals

  • single process: tini (use docker run --init or init: true in docker-compose.yml)
  • multiprocess: s6 and s6-overlay
  • init systems comparison

SWARM managers

Container-Host user remapping

You can map container users to the host ones for greater security.

  • put myuser:100000:65536 (start:length) in /etc/subuid and /etc/subgid, this defines the mapping id range 100000-165535 available to the host user myuser
  • configure docker daemon to use the remapping specified for myuser:

    1
    2
    echo '{ "userns-remap": "myuser" }' > daemon.json
    systemctl restart docker
    

    note that all images will reside in a /var/lib/docker subfolder named after myuser ids

  • now all your container user/group ids will be mapped to 100000+id on the host

You can write up to 5 ranges in sub* files for each user, in this example we set identical ids for users 0-999 and map ids >=1000 to id+1:

1
2
myuser:0:1000
myuser:1001:65536

UFW Firewall interactions

Docker bypasses UFW rules and published ports can be accessed from outside.

See a solution involving DOCKER-USER and ufw-user-forward/ufw-user-input chains.

Dockerizing Rails

  • docker-rails-base -- preinstalled gems, multi stage, multi image, uses onbuild triggers
  • dockerfile-rails -- Dockerfile extracted from Rails 7.1 by fly.io
  • Kamal -- formerly MRSK, DHH solution, deploy web apps anywhere with zero downtime, guide posts

Terms:

  • service = containers that only runs one/same image,
  • task = a single container running in a service,
  • swarm = a cluster of machines running Docker,
  • stack = a group of interrelated services orchestrated and scalable, defining and coordinating the functionality of an entire application.

Source: install, install@raspi, tutorial, overview, manage app data, config. daemon, config. containers,

Source for user mapping: docker docs, jujens.eu, ilya-bystrov

Useful tips: cleanup, network host mode for nginx to get client real IP, limit ram/cpu usage, docker system prune -a -f to remove all cache files

See also: thread swarm gui, docker swarm rocks


~~~ * ~~~

Free uncensored DNS
mouse 13908 · person cloud · link
Last update
2024-09-16
2024
09-16
« — »
Name IPs Origin Notes
DNS.watch 84.200.69.80
84.200.70.40
EU / Munich, Germany no logs, DNSSEC
FreeDNS 37.235.1.174
37.235.1.177
EU / Wien, Austria no logs
Cloudflare 1.1.1.1
1.0.0.1
1dot1dot1dot1.cloudflare-dns.com
HK / Honk Kong
AU / Australia
logs lasts 24h
does not log IPs
DNScrypt.eu 77.66.84.233:443 EU / Koebenhavn, Denmark DNSCrypt only1
DNScrypt.eu 176.56.237.171:443 EU / Nuland, Netherlands DNSCrypt only1
UncensoredDNS 91.239.100.100
89.233.43.71
EU / Copenhagen, Denmark no logs, AnyCast
OpenNIC many * depends on server

Source: greycoder

Android app: OverrideDNS


~~~ * ~~~

Chromium personalizations
mouse 1299 · person cloud · link
Last update
2024-09-12
2024
09-12
«chromium/google chrome/vivaldi/webkit engine based browsers
apps/addons/plugins fix»

Add-ons

developer mode

Go to chrome://extensions/ / vivaldi://extensions/ and toggle developer mode switch.

Vivaldi

Command line options

1
2
3
4
5
6
7
8
9
# https://www.ghacks.net/2017/02/13/how-to-speed-up-the-vivaldi-web-browser/
# optimized command for raspberry pi
/usr/bin/vivaldi \
  --process-per-site \
  --enable-low-res-tiling \
  --enable-low-end-device-mode \
  --disk-cache-size=104857600 \
  --disk-cache-dir=$TMPD \
  "$@"

Delete undeletable cookies

If you can't delete them from Settings > Privacy and Security > Cookies:

  • Open Site Settings (click on the lock in the address field and selecting the last menu item)
  • Go back and filter the desired site
  • Delete cookies and data storage

Fix passwords not syncing

If you observe the following error in vivaldi://sync

1
2
Error: CleanupPasswordStore@components/password_manager/core/browser/sync/password_sync_bridge.cc:1067,
datatype error was encountered: Failed to get encryption key during database cleanup.
  1. close Vivaldi
  2. rm -f ~/.config/vivaldi/Default/Login\ Data*
  3. launch Vivaldi and the sync error in vivaldi://sync should have vanished

Video DRM/Widevine on Vivaldi

1
find ~/.config/vivaldi -type d -regex ".+\(GPU\|Graphite\|Shader\|Dawn\)Cache" -exec rm -rf "{}" +

obsolete instructions

See: old script, vivaldi forum, test video, another test page

1
2
apt install libwidevinecdm0
echo '{"Path":"/opt/WidevineCdm"}' > ~/.config/vivaldi/WidevineCdm/latest-component-updated-widevine-cdm

~~~ * ~~~

Multimedia scripts
mouse 1504 · person cloud · link
Last update
2024-09-05
2024
09-05
«scripts per organizzare i files multimediali a suoceri e moglie»

MOGLIE / ANDROID


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# ----- FOTO -------------------------------------------------------------------
# resize f.jpg to _f.jpg
ruby -e '%w{fileutils shellwords}.each{|l| require l}; lista=Dir["**/*"].grep(/\.jpg/i).sort; lista.each_with_index{|f,i| '\
'puts "#{((i+1)/lista.size.to_f*100).to_i}%\t#{f}"; next if File.basename(f).start_with?("_"); system %Q[ '\
'vips jpegsave #{f.shellescape} /tmp/rz.jpg --Q 80 && '\
'jhead -autorot /tmp/rz.jpg && '\
'mv /tmp/rz.jpg #{File.dirname(f).shellescape}/_#{File.basename(f).shellescape} && '\
'rm -f #{f.shellescape} || echo ERRORE ] }'
# rename by exif/file date
ruby -e 'lista=Dir["**/*"].grep(/\.jpg/i).sort; lista.each_with_index{|f,i| '\
'print "#{((i+1)/lista.size.to_f*100).to_i}%\t"; '\
'system %Q[jhead -n"%Y-%m-%d_%H-%M-%S" "#{f}"] }'
## OR rename _f.jpg to f.jpg
#ruby -e 'lista=Dir["**/*"].grep(/\.jpg/i).sort; lista.each_with_index{|f,i| '\
#'puts "#{((i+1)/lista.size.to_f*100).to_i}%\t#{f}"; next unless File.basename(f).start_with?("_"); '\
#'File.rename(f, File.join(File.dirname(f), File.basename(f).sub(/^_/, ""))) }'

# crea sottocartelle foto e video (per ogni cartella avente files video)
ruby -e 'Dir["**/*"].grep_v(/\.jpg/i).map{|f| File.dirname(f) if File.file?(f) }.compact.sort.uniq.each{|d| '\
'Dir.chdir(d){ files = Dir["*"]; system "mkdir -p foto video"; '\
'files.each{|f| dst = (f =~ /\.jpg/i ? :foto : :video); File.rename(f, "#{dst}/#{f}") } } }'

# zippa cartelle di sole foto
ruby -e 'require "shellwords"; Dir["**/*"].grep(/\.jpg/i).map{|f| File.dirname(f) }.compact.sort.uniq.each{|d| '\
'puts d; Dir.chdir(File.join d, ".."){ n=File.basename(d); system "zip -r #{n.shellescape}.zip #{n.shellescape} && rm -rf #{n.shellescape}" } }'


# ----- AUDIO ------------------------------------------------------------------
# converti video neri in MP3  =>  ffmpeg -i in.mp4 -ab 128k out.mp3
mdkir audio && cd audio
for i in *.mp4; ffmpeg -i $i -ab 128k (string replace mp4 mp3 $i); end # fish
#for i in *.mp4; do ffmpeg -i "$i" -ab 128k "${i%.*}.mp3"; done # bash
rm -f *.mp4


# ----- VIDEO ------------------------------------------------------------------
# converti video MP4 in AVI/h264
for i in (find -type f -iname "*.mp4" -printf '%s\t%p\n' | sort -n | cut -f 2)
  echo $i
  vid2all -l -v 2048 -a 192 --no-expand --no-resize $i /mnt/ramd/out.avi \
    && mv /mnt/ramd/out.avi (string replace -i mp4 avi $i) \
    || touch (string replace -i mp4 ERR $i)
end

find -type f -name "*.ERR"

# sposta tutti gli MP4
for i in (find -type f -iname "*.mp4"|sed -r 's/^..//'); mv -i $i (echo $i|sed -r 's/ /_/g ; s/\//--/g'); end

SUOCERI / WINDOWS


external tools

comprimi_media

1
2
3
4
5
6
7
8
9
10
11
#!/bin/env ruby
dirs = %x[ find -mindepth 1 -maxdepth 1 -type d ].split("\n").sort

dirs.each do |d|
  puts "===== CARTELLA [ #{d} ] ====="
  print 'foto : '; system 'ls|egrep -i "(jpe*g|png|heic)$"|wc -l', chdir: d
  print 'video: '; system 'ls|egrep -i "(mp4|mov|avi|mpeg|hevc)$"|wc -l', chdir: d
  system 'comprimi_media.single', chdir: d
end

puts "\n\nFINE"

comprimi_media.single

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#!/bin/env ruby

%w{ shellwords FileUtils progressbar }.each{|l| require l }

pb_opts = { progress_mark: '#', remainder_mark: '_', length: 79,
            starting_at: 0, format: '%t: %J
[%B] %e'
} all_files = Dir['*'].sort all_files.grep(/\.(aae)$/i).each{|f| FileUtils.rm_f f } re_images = /\.(jpe*g|png|heic)$/i files = all_files.grep(re_images) if files.size > 0 puts '' FileUtils.mkdir_p '_fatti' FileUtils.mkdir_p 'foto' pb = ProgressBar.create pb_opts.merge(title: "#{files.size} immagini", total: files.size) files.each do |f| pb.log " => #{f}" fout = f.sub(re_images, '.jpg') system %Q| convert -quality 80 #{f.shellescape} foto/#{fout.shellescape} | FileUtils.mv f, "_fatti/#{f}" if $?.to_i == 0 pb.increment end end re_videos = /\.(mp4|mov|avi|mpeg|hevc)$/i files = all_files.grep(re_videos) if files.size > 0 puts '' FileUtils.mkdir_p '_fatti' FileUtils.mkdir_p 'video' pb = ProgressBar.create pb_opts.merge(title: "#{files.size} video", total: files.size) files.each do |f| pb.log " => #{f}" fout = f.sub(re_videos, '.mp4') # https://superuser.com/questions/326629/how-can-i-make-ffmpeg-be-quieter-less-verbose system %Q| ffmpeg -hide_banner -loglevel error -stats -i #{f.shellescape} -c:v libx264 -c:a libmp3lame -b:a 128k video/#{fout.shellescape} | FileUtils.mv f, "_fatti/#{f}" if $?.to_i == 0 pb.increment end end if Dir['*'].reject{|f| File.directory? f }.size == 0 # elinmina fatti se tutto e' andato a buon fine FileUtils.rm_rf '_fatti' # elimina l'unica cartella presente dopo averne spostato i files nel parent if Dir['*'].size == 1 dir = Dir['*'].first system "mv -i * ..", chdir: dir FileUtils.rm_rf dir if $?.to_i == 0 end end # mostra evntuali cartelle _fatti ancora presenti dirs = %x[ find -mindepth 1 -type d -name _fatti ].split("\n").sort.map{|d| File.dirname d[2,300] } if dirs.size > 0 puts "Cartelle da controllare:" puts dirs end

misc ruby

1
2
3
4
5
# rename_foto_by_ts
ruby -e 'Dir["**/*"].grep(/\.jpg/i).sort.each{|f| system %Q[jhead -n"%Y-%m-%d_%H-%M-%S" "#{f}"] }'

# enumerate files
Dir['*'].sort.each_with_index{|f, i| File.rename f, "#{'%04d' % (i+1)}.mp4"}; nil

misc shell

1
2
3
4
5
# PDF to JPG
ls *.pdf | sed -r 's/(.+).pdf/pdftoppm -jpeg \0 \L\1.jpg/' | sh

# lossless rotate JPG
jpegtran -rotate 90 in.jpg > out.jpg