knowledge š
Welcome to knowledge, a book by Joel JucĆ”. If youāre reading the online version, use the menu on the left side to browse the content.
What?
This book contains things I learned during my life, either while studying or living. Most of its contents are not properly organized. The reading experience is not linear, so feel free to jump directly to sections you feel more interested in.
Why?
I always wanted to write a book with things Iāve been learning during my life, but it just never happened, simply because I generally donāt have time and willingness to do it at the same time. š But I still want to share. So, Iāll be just writing down things, thoughts, techniques, etc., without too much effort on organization.
An unstructured version that gets published would be better than a well formatted one that never goes public.
How?
# build it for deploy
mdbook build
# serve it while you edit things
mdbook serve
License
āļø life
š£ļø langs
šØš³ chinese
Nihao, wo shi Joel! š
Yep ā I am learning Chinese. I never thought Iād put up some effort to learn it as well. But, China is becoming this amazing country, and Iād like to understand better what China is about: this ancient society which its History and Philosophy ages are measured in millennia, not centuries.
š©šŖ german
Willkommen, motherfuckers!
German is definitely super entirely very totally hard to learn! Really. But Iām stubborn! So letās figure out the basics.
Note: Iām making heay use of ChatGPT as my language instructor, but informations here might be entirely accurate. Please let me know:
joel at joeljuca dot com.
Verb tenses
German has the following tenses:
- PrƤsens (Present tense)
- PrƤteritum (Simple past tense)
- Perfekt (Present perfect tense)
- Plusquamperfekt (Past perfect tense)
- Futur I (Future tense)
- Futur II (Future perfect tense)
- Konjunktiv I (Subjunctive I)
- Konjunktiv II (Subjunctive II)
Each of these tenses has its own specific usage and forms, allowing you to express actions or events in various time frames and contexts. Keep in mind that some of these tenses, like the subjunctive forms, might be used more in written language or in formal contexts.
TOP 3 verbs: Sein, Haben, and Werden
When I asked ChatGPT for the TOP 3 most used verbs in the German language it came up with these three. Below are their conjugations.
Verb Sein (To Be)
- Ich bin (I am)
- Du bist (You are; informal singular)
- Er/Sie/Es ist (He/She/It is)
- Wir sind (We are)
- Ihr seid (You are; informal plural)
- Sie sind (They are; formal and plural āyouā)
Examples
- Ich bin Student (Iām a student)
- Du bist mein Freund (You are my friend)
- Er ist sehr intelligent (He is very intelligent)
- Wir sind glücklich (We are happy)
- Ihr seid geschwister (You are siblings)
- Sie sind Lehrer (They are teachers)
Sein is a crucial verb in German. Pretty much the most commonly used German verb. So, itās important to absolutely master it entirely, for real.
Verb Haben (To Have)
TBD.
Verb Werden (To Become)
TBD.
š ļø monday fix
Something I try to do every Monday: fix something in my life. A broken part of smt, a squeaky door or chair, a loose button in my car, smt wrong in my keyboard, whatever! Just pick smt and fix it. :)
Here, I just intend to log things Iāve fixed.
š¼ music
Iām back on learning music, so Iāll document here the stuff Iāve been learning.
Music Theory 101
Itās important to know what are the basic building blocks of music, right?
Music Notes
The basic building block of music is a note. They are represented by English charaters A-G (A, B, C, D, E, F, and G). We know them by the solfĆØge doāreāmiāfaāsolālaāsi, which translates to the characters C, D, E, F, G, A, and B (it starts on C because itās the equivalent to do, with D being the equivalent to re, and so on).
Notes can be modified by accidentals, creating intermediary notes between each one. The sharp sign (#) raises a note by a semitone (or half-step), while a flat (or bemolle) lowers note by a semitone. So, c C# would be a note that exists between C and D, while Eā would be a note that exists between D and E. Itās also possible to have a semitone represented by two possible accidentals - for instance, the C# one could also be represented by Dā. Since the sharp sign raises one semitone and bemolle lowers one semitone, the two representations are equivalent.
Note: there are some special notes that does not have intermediary semitones. The
BandC, and theEandF. These ones does not have semitones between them, so the semitonesB#,Cā,E#andFādoes not exist.
The full list of notes and sharp semitones would be:
C C# D D# E F F# G G# A A# B
The equivalent full list of notes and flat semitones would be:
C Dā D Eā E F Gā G Aā A Bā B
Scales and Tetrachords
In music, a scale is an ordered set of musical notes. One of the most common scales is probably the C Major: C, D, E, F, G, A and B. It has no sharps or flats, and can easily be played on piano by just hitting the white keys starting on C.
A tetrachord is a set of four notes separated by three intervals. Tetrachords are the basic building blocks of a major scale, and its notes are systematically spaced by the following rule: whole step, whole step, half step, with a whole step separating the tetrachord from the second one. This way you can draw any major scale by just starting from a given note and following the tetrachord spacing order.
Using the tetrachord schema, I could describe the logic behind the C major with:
C major (
C D E F G A B): the first tetrachord goes one whole step (full note) fromCtoD, one whole step fromDtoE, and a half step (semitone only) fromEtoF(remember, there are no semitones betweenEandF). Then one whole step to start the second tetrachord at noteG. Then, one whole step fromGtoA, one whole step fromAtoB, and one half step fromBtoC(again, there are no semitones betweenBandC).
Below are some other major scales Iām writing through the tetrachord system (Iām checking them on Wikipedia right after writing, and they are correct - yeah!):
- D major:
D E F# G A B C# D - E major:
E F# G# A B C# D# E - F major:
F G A A# C D E F - G major:
G A B C D E F# G - A major:
A B C# D E F# G# A - B major:
B C# D# E F# G# A# B
project management ā
Iāve tried many different setups to manage projects: stickies, kanban, many different apps and services, and I failed horribly at getting things done. Now I have a quite simple setup:
- I use Wunderlist as my main to-do app, with lists for personal and professional things;
- āI generally add all projects and tasks I must/should do/perform to my Wunderlist as lists, so it is common to find lists with dozens of items;
- āWunderlist has an interesting feature called smart lists that group items from multiple lists based on its due date. I use them to see tasks for today and the current week;
- āMy daily focus is always to get done all shit scheduled for today. Everything that canāt be accomplished today will be postponed, either for tomorrow or later, so I can focus on what I can do today;
- āSince Wunderlist helps me to focus on my daily goals I have lists for sanzonal projects, like Dsafio, Exercism and HoraExtraJP alongside all their tasks (e.g.: issues to solve, pull requests to review, or events to organize);
- āIntegration with my calendar helps me visualize my daily schedule together with compromises (planned working hours, meetings, as well as personal and professional appointments).
technology š»
Technology is revolutionizing the human life. Today, almost everything we do is now related to, or uses technology somehow. We communicate, work, chill, cook, date, fall in love, cheat, overcome, explore, learn, evolve, etc., all with technology. Since this thing has created such deep roots in our species, powering us to do things we never imagined before, it would be wise to think about it in order to, at least, try to understand where we currently are and where we are heading to, now as the coolest kids who ever lived on Earth.
š» code
asdf
asdf might fail to install some project versions with the following error:
Authenticity of checksum file can not be assured! Please be sure to check the README of asdf-nodejs in case you did not yet bootstrap trust. If you already did that then that is the point to become SUSPICIOUS! There must be a reason why this is failing. If you are installing an older NodeJS version you might need to import OpenPGP keys of previous release managers. Exiting.
It fails to verify the authenticity of packagesā signatures. A simple fix is to import some keys by executing a command provided by asdf itself:
~/.asdf/plugins/nodejs/bin/import-release-team-keyring
āRecent versions not showing up. WTF?ā
The asdf work alongside plugins that declares installers and manages them separatedly. So, in order to see the latest version of the respective binaries (node, ruby, etc.) you must update their respective plugins.
I ran on this problem today: I was trying to install Ruby 3.0.0 but it wasnāt showing up as option when I ran asdf list all ruby. The solution? Update the respective plugin:
asdf plugin update ruby
You can also update them all for greater good:
asdf plugin update -all
ci/cd āļø
Tips & tricks out of wisdom acquired from working on CI/CD.
gitlab š¦
Use file(s) hash(es) as cache keys
After spending a huge amount of time on it, I figured itās actually embedded into GitLab CI itself. In order to use file hashes, add them to cache:key:files:
# .gitlab-ci.yml
# Eg.: a global cache setup
cache:
key:
files:
- yarn.lock
Proper documentation available at https://gitlab.com/help/ci/yaml/README.md#cachekeyfiles
š» editors
Annotations about the code editors I use.
VS Code/Codium
The editor I use most of the time. Years of configuration tweaks and muscle memory over keyboard shortcuts!
Error āCannot find runtime ānodeā on PATH. Is ānodeā installed?ā
I had this error when I was trying to debug a Node.js project using VS Codeās built-in debugger. It happens because VS Code is unable to find node in your $PATH. No news here, but I both my Bash and Zsh configured to add /usr/local/bin to my $PATH, but no luck so far.
I never managed to fix this bug - but I found a workaround so good that this solution becomes unnecessary: running VS Code as a subprocess of your current shell.
In practice, itās quite simple actually. Just run VS Code from your command-line instead of opening it directly. Go to your project directory and run code from it. Ex.:
cd path/your/project
code .
It opens VS Code with your project directory already loaded - but the important thing is that, since you ran it from your current shell session, it will inherint your current $PATH, and node (or in my case, nodemon) will be there. š
Extension management from the command line
Itās possible to manage VS Code extensions through the command line, using the following commands:
code --extensions-dir <dir>
Set the root path for extensions.
code --list-extensions
List the installed extensions.
code --show-versions
Show versions of installed extensions, when using --list-extension.
code --install-extension (<extension-id> | <extension-vsix-path>)
Installs an extension.
code --uninstall-extension (<extension-id> | <extension-vsix-path>)
Uninstalls an extension.
code --enable-proposed-api (<extension-id>)
Enables proposed API features for extensions. Can receive one or more extension IDs to enable individually.
Read more at https://code.visualstudio.com/docs/editor/extension-gallery
Neovim
Tips and tricks for the most insane code editor out there.
ctrlp
A Vim/Neovim plugin to open files with Ctrl+P and fuzzy search.
Invoke it project-wide using Ctrl+P or :CtrlP, or from a specific directory w/ :CtrlP [directory name].
Once itās open:
<F5>to refresh the cache (get new files, rm deleted ones, etc.)<Ctrl>Dto switch between filepath-based and filename-based searching<Ctrl>Rto switch to regex-based searching
elixir š
The most pleasant programming language and runtime Iāve worked with through my entire career.
Installing Phoenix
Phoenix is the most used Elixir framework for web development. Below is a series of commands required to install the Phoenix project bootstrapper (mix phx.new), extracted from its official documentation:
# Ensure Hex is installed
mix local.hex
# Installs the project bootstrapper
mix archive.install hex phx_new
LiveView
In 2025, LV (short for LiveView) is a thing. In this section I list the things Iāve been systematically learning about it.
The bare minimal
A LiveView is an Elixir module that use Phoenix.LiveView and implements a fn render/1:
# file: lib/hello.ex
MyApp.Hello do
use Phoenix.LiveView
def render(assigns) do
~H"""
<p>Hello there!</p>
"""
end
end
The param assigns is a map. You wonāt use it directly; instead, youāll use special syntax, like @name, to use values coming in it.
š” It is best practice to suffix LV modules and files with āLiveā; so weād do
MyApp.HelloLiveinstead, in a file namedhello_live.ex.
Phoenix has some built-in facilities to integrate your LVs with Phoenixā default layouts, so instead of use Phoenix.LiveView directly, in Phoenix youād do use MyWeb, :live_view:
# file: lib/hello_live.ex
MyApp.HelloLive do
use Phoenix.LiveView
def render(assigns) do
~H"""
<p>Hello there!</p>
"""
end
end
To see it in action, youād have to set up a route in lib/my_app_web/router.ex:
defmodule MyAppWeb.Router do
use MyAppWeb, :router
# (...)
# Below is the important part: the main `scope`
scope "/", MyAppWeb do
pipe_through :browser
# (...)
# Live routes are set like this
live "/hi", HelloLive
# Live routes accepting params look like this
live "/hi/:name", HelloLive
end
# (...)
end
There it is. Youāll be able to see it in action at http://localhost:4000/hi.
erlang āļø
My adventures on this amazing distributed computing platform.
Erlang on macOS through asdf
Installing Erlang through asdf is probably the easiest way to get it up and running on macOS. Youāll need OpenSSL to run the installation, and the easiest way to get it is through Homebrew:
brew install openssl
Homebrewās OpenSSL formula is keg-only, and it means that even after installed it wonāt be globally available in your system. Thereās an useful note about it on asdf-erlangās README.md, but I ended up with a command that would use brew to get the path of the OpenSSL installation:
# first, export this Kerl-specific variable with the following content
export KERL_CONFIGURE_OPTIONS="--without-javac --with-ssl=$(brew --prefix openssl@1.1)"
# then, install the Erlang version you want (e.g.: 22.1.5)
asdf install erlang 22.1.5
These options will disable Java-related features and point to the correct OpenSSL paths.
Youāll find more information about setting up Erlang in your machine in the Setup chapter of the Adopting Erlang book (a must-read if youāre adopting Erlang just now).
go š¹
The new popular boy in the tech landscape.
fmt formatting opts
When using functions from the fmt package, there are some formatting codes one must use to print out values to the console. Eg.:
package main
type User struct {
Id int
Name string
}
func main() {
user := User{Id: 123, Name: "Joel"}
fmt.Println("User: %+v", user) // => {Id:123 Username:joeljuca}
}
At first sight, the question that comes to mind is: WTF is %+v?
Below I list these notations and WTF each of them are.
General
%v: the value in a default format when printing structs%+v: same as%vbut adds field names
%#v: a Go-syntax representation of the value%T: a Go-syntax representation of the type of the value%%: a literal percent sign; consumes no value
Boolean
%t: the word true or false
Integer
%b: base 2%c: the character represented by the corresponding Unicode code point%d: base 10%o: base 8%O: base 8 with 0o prefix%q: a single-quoted character literal safely escaped with Go syntax%x: base 16, with lower-case letters fora-f%X: base 16, with upper-case letters forA-F%U: Unicode format:U+1234; same asU+%04X
See: pkg.go.dev/fmt
javascript š
My [former] prefered language, with whom I have a love-hate relationship.
jest š
My new preferred testing framework (2020 edition).
Mock clearing/resetting/restoring WTF
There are three options to undo things in Jest mocks:
m.mockClear(): remove all mock data (eg:m.mock.callsandm.mock.instances)m.mockReset(): clear the mock (m.mockClear()) and removes mocked return values and implementationsm.mockRestore(): reset the mock (m.mockReset()) and restore the original (non-mocked) implementation
Useful links
PS: the
mvariable on the examples below is a mock function:const m = jest.fn(() => true);
mocha āļø
Not the fastest unit testing framework but ĀÆ\_(ć)_/ĀÆ.
nyan reporter
Add --reporter=nyan to your test script on package.json and your tests will be the most nyaned awesome omg-thats-really-cool ones in the world:
$ mocha --reporter=nyan
15 -_-_-_-_-_-_-_-__,------,
0 -_-_-_-_-_-_-_-__| /\_/\
0 -_-_-_-_-_-_-_-_~|_( ^ .^)
-_-_-_-_-_-_-_-_ "" ""
15 passing (77ms)
react āļø
The best overcomplicated JavaScript framework out there.
Hooks
React Hooks are this new way to handle state changes (which are now called side effects - or just effects) on React components. Class-based components are now dead and every React developer now thinks it hurts. God kills a kitten every time you start a new React component with export classā¦
So, Hooks. This is basically a weirdo replacement for lifecycle methods componentDidMount, componentDidUpdate, and componentWillUnmount:
import { useState } from "react";
const TogglePage = () => {
const [isOn, setState] = useState(false);
return (
<p>
<button onClick={(ev) => setState(!isOn)}>Toggle</button>
Current state: {isOn ? "On" : "Off"}
</p>
);
};
In the example above, we are using useState() to create a local state container with an initial value of false (Hooks doesnāt dictate the type of data that goes in it - itās up to you). It returns a tuple with two items: the first one is the state itself, and the second one is a function that updates the state containerās internal state.
The cool part is: whenever you update the state container, your functional component is re-rendered.
Notes
- Hooks are executed on every fucking render - both on mount and updates, so itās pretty easy to create memory leaks with it. Be careful, motherfucker!
- You can also return a function from it, which is then called when the component unmounts. These are called clean-up functions
- You can pass an array as second argument of
useState()to control when it should run, so whenever the hook is about to execute, it will compare the array with the last version of it. If it differs, your hook is executed again- You can have multiple items on this array - and whenever any of these items change, the hook (and the clean-up function) will execute again
- If you pass an empty array the hook will be executed once, during mount - and if thereās a clean-up function, itāll be executed once, during unmount (two empty arrays will never differ)
Lisp š½
Lisp (historically LISP; from List Processing) was a programming language created in 1958 by John McCarthy. Many dialects have been created over time, and today Lisp is a generic name to refer to all these cousin programming languages.
The most known Lisp dialects today (IIUC) are Clojure, Common Lisp, and Scheme.
Lisp might be mostly known for its heavy usage of parenthesis to enclose its S-expressions (parenthesis-enclosed hierarchical structures used to represent complex data structures as well as program statements itself).
Vocabulary and WTFs
- Datum
a single piece of data (eg.: a string, a number, etc; itās not a strict Lisp term, though itās frequently seem in many Lisp-related texts) - Atom
any data structure that is not a cons cell. Essentially, atoms are the fundamental building blocks that do not contain other LISP objects (numbers, symbols, strings, characters, keywords, and booleans are all atoms). The concept of atoms is useful for distinguishing between compound data structures (like lists) and simpler elements. For example, in an expression, you might want to check if a value is a simple element (atom) or a list - Symbol
a basic data type that represents an identifier or a name. Itās a fundamental building block in LISP, used to refer to various entities such as variables, functions, or special keywords - S-expression
is a syntax for for denoting nested, tree-structured code and data. S-expressions are nestable lists made of atoms and/or other S-expressions (aka.: symbolic expression, sexpr, or sexp; see: S-expression) - Cons cell
a two-item tuple thatās used to implement lists in Lisp. Items are historically named CAR and CDR (reads coulder), being CAR the one holding the actual value of the const cell, and CDR holding a reference to the next item in the list (when CDR is nil the list has reached its end; itās indeed a linked list, and cons cells are how they call its nodes)
Syntax and data types
How it works in practice.
Comments
They start with a semicolon:
; This is a comment
There are conventions for the number of semicolons to use:
- semicolon when right-aligned with code
- semicolons when block-aligned with code
- semicolons when top-level
Atoms
; A number atom
123
; A string atom
"Hello there"
Symbols
; Define a symbol `name` with value `"Joel"`
(setq name "Joel") ; => "Joel"
name ; => "Joel"
; Here, the symbol `name` is being used with quote - so, a quoted symbol
'name ; => NAME
S-expressions
An S-expression for a list of numbers:
(1 2 3 4 5)
An S-expression for a sum operation:
(+ 1 2) ; => 3
S-expressions can nest:
(+ 1 (+ 2 (+ 3 4))) ; => 10
lfe āļø
LFE (Lisp Flavoured Erlang; website, Wikipedia) is a Lisp dialect built on Erlang/BEAM.
oubiwann on webdev
I stumbled upon a discussion in the #web channel of LFEās Slack, where @oubiwann (Duncan McGreggor; GitHub, Twitter) was sharing lots of interesting advises on architectural matters for web softwares.
I just copy-pasted everything down below! Too much gold to loose it for Slack. š
oubiwann Apr 10th at 11:15 AM
I've done TONS of web development in LFE, both web front ends and REST APIs
oubiwann 2 days ago
I've done a tiny bit of web dev in Common Lisp and various Scheme dialects
oubiwann 2 days ago
but I've done a MASSIVE amount in Clojure (100s of 1000s of lines of code)
oubiwann 2 days ago
while the ecosystem for LFE is not anywhere as mature as what Clojure enjoys (luxuriates in, actually) I much prefer web dev in LFE
oubiwann 2 days ago
the language itself is so perfectly suited to it, supplemental libraries are not actually needed as much
oubiwann 2 days ago
side note: the best Clojure libraries I've used for web dev are not actually compojure and ring (though I did use the first initially, and all the ring libs for much of my time in the Clojure ecosystem), but rather what I used towards the end of my time at NASA: https://github.com/metosin/reitit
oubiwann 2 days ago
any future LFE work I do around HTTP / REST / routes macros will be 100% influenced by that library
oubiwann 11:16 AM
not much recently, though
11:16
regardless, LFE is truly fabulous for this type of thing (thanks to its BEAM inheritance and beautiful syntax(lessness))
11:17
what I recommend to people who want to develop services for eventual deployment is start with an OTP release right away -- don't wait!
11:17
I avoided that for years
11:17
and shouldn't have
11:18
pretty much everything I build now starts with rebar3 lfe new-release myapp
oubiwann Apr 10th at 11:20 AM
also: separate these things cleanly, in different modules:
1. routing
2. request transformations
3. business logic
4. database access (even if you start with something in-memory)
5. generation of responses
oubiwann 2 days ago
this has also been critical in my many management consulting efforts: walking into teams that were on the verge of technical and/or morale collapse, I always saw that these principles had not been followed
oubiwann 2 days ago
when mgmt or product introduced radical new changes, the whole codebase had to be touched, instead of just changing the bits of code that implemented the product-specific bits
oubiwann 2 days ago
morale always radically changed a) after I worked with them to refactor to proper separation of concerns, and b) upon receiving the next request from mgmt to make radical changes ....
oubiwann 2 days ago
they never can believe their eyes when these refactors help them make changes in minutes vs their previous months
oubiwann 2 days ago
back to the bullet points:
1. this is very simple, format and approach depends almost entirely upon the selected framework (if you have to roll your own, of course this will be much more involved!)
2. this should contain no HTTP response-related code (or any of the other things mentioned in the list); this is all about transforming all required data sent in the request (params, query strings, POSTed body, URL paths, etc.) into an application representation (e.g., one or more Erlang maps or LFE records) -- DON'T use the framework's (or the underlying HTTP lib's) data structures for this! Your app needs to be losely coupled to those (you need to be able to easily replace them with another, without having your application suffer from vendor lock-in!)
3. this is where all the crazy shit lives; don't let the insanity here impact any other part of your app -- no leaky abstractions!
4. I've found the best way to handle this one is do define an app-specific internal API for data access, and hide the underlying implementation details behind that (VERY useful when switching out backends / DBs)
5. this last one is often skipped -- don't fall into that trap! It's usually a small amount of code, but you want your responses, errors presented to consumers, etc., to be easy to test, maintain, and to get new features (all of which should happen without touching any other part of your project)
(edited)
oubiwann 2 days ago
think carefully about error propogation (edited)
oubiwann 2 days ago
I'd seriously recommend creating a single data structure representing both "results" and "errors" and making sure this is passed and transformed appropriately from the first bullet to the last (edited)
oubiwann 2 days ago
this would, of course, be affected by how you handle errors inside the app and how much you present to the final consumer of the app/API
oubiwann 2 days ago
you want to protect yourself from revealing too much of your app's internal workings to a public consumer or to bad actors that might take advantage of these errors to exploit potential vulnerabilities hinted at by the errors (edited)
oubiwann 11:21 AM
intermixing one or more of those has been the cause of much difficult-to-read code and origin of too many bugs
11:26
(not to mention slow-to-ship releases) (edited)
11:28
the tendency to say "oh, this is just a simple thing; I'll keep things separate in this module" is strong ... and by the time the code has evolved to something not-so-simple, it's too late (and many hard-to-see issues have already snuck into the code) (edited)
oubiwann 11:37 AM
incidentally, coding web projects in such a way also leads to much improved project delivery times -- I credit the accompanying clarity of thought (and thus code) which lends itself well to quickly iterating on logical, functional portions of a project
oubiwann 11:51 AM
two very important (and quick) reads when thinking about shipping software using the BEAM:
https://www.erlang-in-anger.com
https://adoptingerlang.org
don't let the names fool you! These are two of the best bits of principal engineer-level pieces of writing for software development and deployment; I'm constantly recommending them to non-Erlangers for reading
python š
Python (Wikipedia) is a general purpose programming language, one of the most famous PLs in the world, and definitely the lingua-franca of the data world (data science, machine learning, deep learning, etc.).
Vocab
- PyPI: Python Package Index. The website that hosts publicly available Python packages
- Virtual environment: a CLI trickery to overcome Pythonās inability to isolate project-specific dependencies
Basic syntax and data types
Good syntax and basic data types overview available in lxiym/python
Virtual environments
venv: itās a Python built-in module, a subset ofvirtualenv(another Python module, available on PyPi)
See: TIL: Python virtual environments and venv
ruby š
Ruby is awesome!
Command-line Twitter with t
Ruby has a powerful library called t (https://rubygems.org/gems/t). Unfortunatelly, it seems abandoned by its author, and currently presents a dependency issue, requiring you to install the version 6.1.0 twitter library, instead of 6.2.0:
gem install t
gem install twitter -v 6.1.0
gem uninstall twitter -v 6.2.0
shell
All things command-line wizardry.
One-line if, for, etc.
# one-line if statements
if true; then echo "Do something"; fi
# => Do something
if false; then echo "Do something"; fi
# =>
# one-line for statements
for i in hello there; do echo $i; done
# => hello
# => there
for i in $(seq 1 3); do echo $i; done
# => 1
# => 2
# => 3
Infinite loop
Youāll often need to run a command over and over again for some reason (mine right now is: I have to check if a file exists). There are obviously better options - but doing things through command-line ninjutsu is always funnier. So, one-line infinite loops in Shell (both sh and Bash):
while true; do echo "Doing some work" && sleep 1; done
# => Doing some work
# => Doing some work
# => Doing some work
# => ...
This will run echo "Hello!" forever, āsleepingā for one second between each execution.
š ļø ops
kubernetes āø
My adventures with Kubernetes.
š” FYI: āK8sā is short for Kubernetes.
Intro
TLDR: K8s is an orcherstration system for deployment and management of containers (e.g.: Docker).
Terms
- Kubectl
- Control Plane
- Node
- Node Process
- Volume
- Containerized App
- Pod
- Deployment
- Service
- ReplicaSet
- IP Address
- Cluster
apache kafka š»
Apache Kafka (GitHub mirror) is a powerful Java/JVM powered open-source streaming platform.
Apache Kafka in 6 minutes by James Cutajar
š„ļø unix
My notes on Unix (and Unix-variant) systems.
WTFs
The Unix world definitely have some WTFs.
FTPS vs SFTP
- FTPS: (FTP with SSL) is the FTP protocol with some SSL to improve security
- SFTP: (Secure FTP) is the FTP protocol on top of SSH
IMHO, SFTP is better - SSH is rock-solid, fckn secure (when well configured and used properly with strong algo/config key-pairs), and it has an optimized use of the network (aka.: it should be faster).
ffmpeg š¼
FFmpeg is a cross-platform tool to record, convert and stream audio and video. Itās the de-facto standard for everything related to video processing, etc. - and I use it to perform basic video editing from the command line (yep, itās kind of using a bazooka to kill a single cockroach - but itās practical and easy to use).
TL;DR
To convert video formats, run:
ffmpeg -i movie.mp4 movie.avi
To cut a movie from the moment 1min 23s up to the moment 23min 45s, run:
ffmpeg -ss 01:23 -to 23:45 -i ./movie.mp4 -c copy out.mp4
Usage
The ffmpeg command, at its most basic format, accepts an input file (with the parameter -i) and an output file (given as the last argument):
ffmpeg -i movie.mp4 movie.avi
Thatās all it needs to convert a MP4 movie to the AVI format. See the file extensions? ffmpeg understands that itās a convertion operation by relying on the file extensions youāre providing. Pretty clever, huh?!
Well, the things I do the most is: cutting a video from moment X to moment Y. I do it mostly to edit the recordings of the meetups I ran on Elug CE. With ffmpeg you can do it by using the parameters -ss <position>. It must be used before the parameter -i, so it affects the decoding of the input file. So, when using ffmpeg to cut videos, the order of these parameters does affect how ffmpeg works.
The value <position> must be a time duration specification
Hereās an example of how
ffmpeg -ss 01:23 -t 23:45 -i ./movie.mp4 -c copy out.mp4
Time duration specification
There are two syntaxes for expression time durations:
[-][<HH>:]<MM>:<SS>[.<m>...]
HH expresses the number of hours, MM the number of minutes for a maximum of 2 digits, and SS the number of seconds for a maximum of 2 digits. The m at the end expresses decimal value for SS.
So, if you got two numbers of two digits separated by a collon
[-]<S>+[.<m>...]
S expresses the number of seconds, with the optional decimal part m.
In both expressions, the optional - indicates negative duration.
Examples
55: 55 seconds0.2: 0.2 seconds200ms: 200 milliseconds (or 0.2s)200000us: 200000 microseconds, (or 0.2s)12:03:45: 12 hours, 03 minutes and 45 seconds23.189: 23.189 seconds
So, for a quick reference:
12:34: 12 minutes and 34 seconds01:23.567: 1 minute, 23 seconds, and 567 milliseconds12:34:56: 12 hours, 34 minutes, and 56 seconds12:34:56.789: 12 hours, 34 minutes, 56 seconds, and 789 milliseconds
-ss position (input/output) When used as an input option (before ā-iā), seeks in this input file to position. Note that in most formats it is not possible to seek exactly, so ffmpeg will seek to the closest seek point before position. When transcoding and -accurate_seek is enabled (the default), this extra segment between the seek point and position will be decoded and discarded. When doing stream copy or when -noaccurate_seek is used, it will be preserved.
When used as an output option (before an output url), decodes but discards input until the timestamps reach position.
position must be a time duration specification, see the Time duration section in the ffmpeg-utils(1) manual.
ffmpeg -ss 11:58 -t 12:30 -i ./elug-ce-meetup-5-eceba7abc25e241c41f43a898ea32dd49a9bf71b\ on\ 2020-11-04\ 23-20.mp4 -c copy out.mp4
position must be a time duration specification, see the Time duration section in the ffmpeg-utils(1) manual.
ffmpeg buddy: https://evanhahn.github.io/ffmpeg-buddy/
Video estabilization with FFmpeg and VidStab
- A guide by maxogden (and comments by Paul Irish)
- A follow up by Paul Irish: Video Stabilization With
ffmpegandVidStab
freebsd š
Iām giving FreeBSD a real try during this pandemic thing. Iām excited by the fact FreeBSD is extremely secure and stable, thus being a great option for servers. Also, thereās a jails thing - something like containers, that seems to be even powerful than LXC/Docker and I would like to explore its potential as a disposable desktop environment.
First experience: FreeBSD on GCP
I need some practical experience to learn things, so I launched a FreeBSD node (a shiny f1-micro one) on Google Cloud. Since it has a Always-Free Tier and I currently donāt have things running on GCP, it will cost me nothing (yay!). Iāll use it both to learn FreeBSD itself and to deploy some service to play with. Iāve been looking for an opportunity to deploy a Mastodon instance somewhere, just to play with - and it seemed like a great opportunity!
Creating the machine required me to find a FreeBSD option on GPCās Marketplace. Luckily, itās kind of an official one, made by the FreeBSD Release Engineering team. Iām glad thereās an official/supported option available, I wouldnāt consider building an image myself.
The next step now is to SSH into it. GCP provides a way to launch a browser window with a shell on it. I used this one to send my public key to ~/.ssh/authorized_keys, but even still I canāt SSH into it. No luck with it.
The solution ended up using the Google Cloud SDKās command-line gcloud (available in Homebrew as a cask), which you can install with the following command (macOS-specific):
brew cask install google-cloud-sdk
Then, you would run the following command to connect to the GCP machine:
gcloud compute ssh "<your-username>@<your-machine-name>" --project="<your-project>"
FreeBSD 12.1-RELEASE-p3 GENERIC
Welcome to FreeBSD!
Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums: https://forums.FreeBSD.org/
Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.
Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier
Edit /etc/motd to change this login announcement.
ZFS can display I/O statistics for a given pool using the iostat subcommand.
By default, it will display one line of current activity. To display stats
every 5 seconds run the following command (cancel with CTRL+C):
zpool iostat 5
To view individual disk activities, specify the -v parameter:
zpool iostat -v
Of course, both can be combined. For more options, see zpool(8).
-- Benedict Reuschling <bcr@FreeBSD.org>
It seems that after this āEdit /etc/motd (ā¦)ā line the content is dynamic, and shows interesting tips about FreeBSD every time you log in. Cool.
Package management with pkg
The package management solution of FreeBSD is Ports - or FreeBSD Ports to be exact. The name seems weird - but yeah, if it works fine I wonāt care about a weirdo name. Its main command-line interface is the command pkg, which seems quite similar to Debianās APT (apt):
# Search for packages with `pkg search`
pkg search htop
# Install a package with `pkg install`
pkg install htop
# See a list of commands with `pkg help`
pkg help
I thought it would be harder to learn Ports, but since it looks so similar to Debian I felt at home.
Service management
FreeBSD uses rc to manage services. Again, itās seems to be similar do Debianās, so nothing that much new here. To manage a service you use the service command, followed by the service name (eg: sshd), and finally a subcommand (eg: start, stop, restart, etc.):
# Start sshd
services sshd start
# Stop sshd
services sshd stop
# Restart sshd
services sshd restart
It would make more sense if you had something like service <subcommand> <service-name> - but again, if it works I can live with it.
ZFS over FreeBSD
Yep, I dove into ZFS, I gotta learn this thing. So, hereās what Iāve been learning so far:
ZFS concepts
- zpool
The most basic ZFS building block. A zpool is a group (one or more) of disk behaving as a single storage āthingā. Youāll need to create datasets and/or volumes (each described below) in your zpool in order to store files and/or raw data - dataset
The ZFS file system. It allows you to store directories and files in hierarchical order, just like any other file system. - volume
Itās a raw storage unit, useful to store raw data, like non-ZFS file systems like UFS, ext4 (Linux), or NTFS (Windows). IMHO, the great thing here is that itās possible to use non-featureful file systems from within ZFS and have some of its features (compression, snapshots, encryption, etc.) on it - vdev
I didnāt get this yet - snapshot
A point-in-time freezing state of how this dataset or volume looked like at the moment the snapshot was taken. Absolutely useful for backups!
FreeBSD Fridays: Introduction to ZFS by Dan Langille
The FreeBSD Handbook
Iām learning all this stuff by reading the FreeBSD Handbook, the official project documentation. In fact, Iām learning a ton of interesting stuff about UNIX systems - like permission tricks like setuid and setgid, sticky bits
I highly recommend reading the FreeBSD Handbook if youāre interested on learning more about UNIX and/or BSD! The FreeBSD documentation is really awesome and worth reading.
gpg š
GNU Privacy Guard - GNUās libre implementation of PGP.
Install
Iāve been using GPG Suite instead of plain gpg, so in macOS itās a matter of:
brew install gpg-suite
Usage (cheatsheet)
gpg --list-keys
Lists all GPG keys in your keychain.
gpg --list-secret-keys --keyid-format LONG
If youāre looking for your own keys, this is probably the command youāre looking for. It will list secret keys only, including key IDs in the output.
$ gpg --list-secret-keys --key-format LONG
$HOME/.gnupg/pubring.kbx
-----------------------------
sec rsa4096/AA9F6CB28B67BDA3 2022-02-10 [SC] [expires: 2025-01-01]
086760D4F317A99E271802B4AA9F6CB28B67BDA3
uid [ultimate] Joel JucĆ” <joelwallis@gmail.com>
ssb rsa4096/4A11F4331FF35AF9 2022-02-10 [E] [expires: 2025-01-01]
The output is read somehow like that:
$ gpg --list-secret-keys --key-format LONG
$HOME/.gnupg/pubring.kbx
-----------------------------
sec @{key-algorithm}/@{key-id} @{creation-date} [SC] [expires: @{expiration-date}]
@{fingerprint}
uid [ultimate] @{user-name} <@{user-email}>
ssb rsa4096/4A11F4331FF35AF9 2022-02-10 [E] [expires: 2025-01-01]
gpg --symmetric <filename>
Encrypts a file with a password, which will be asked from you in the process.
gpg --encrypt --sign --recipient <your-email> <filename>
Encrypts a file with a given GPG key.
linux š§
Creating boot USB sticks/microcards on macOS
A real quick-and-dirty way.
1st. Get an operating system image
You gotta get a *.img file somehow. Most distributions distribute them as compacted versions (tar, gzip, xz, etc.).
If all youāve got is a *.iso file, convert it to *.img using hdiutil:
hdiutil convert -format UDRW -o /output.img /path/to/your/file.iso
2nd. Prepare your removable
Format your removable as FAT32.
3rd. Now, find the physical address of your removable
Plug your removable into your macOS system and run:
diskutil list
Youāll see something like this:
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *100.0 GB disk0
1: EFI EFI 79.0 MB disk0s1
2: Apple_CoreStorage HD 99.0 GB disk0s2
3: Apple_Boot Recovery HD 650.0 MB disk0s3
/dev/disk1 (internal, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: Apple_HFS HD +98.0 GB disk1
Logical Volume on disk0s2
A1B2C3D4-E5F6-G7H8-J9K1-0L11M12N13P1
Unencrypted
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *2.0 GB disk2
1: DOS_FAT_32 SD 2.0 GB disk2s1
Now figure out which one is the physical address of your removable disk.
TL;DR: diskutil list lists all your disks, both physical and virtual. Youāll have to figure out which physical disk is your removable, but you can generally check it through the storage capacities (in the example, itās a microsd capable of 2GB storage).
4th. Copy the raw data to your disk
After identifying the physical address of your removable, run:
diskutil unmountDisk <your removable disk address>
According to the example above, it would be:
# diskutil unmountDisk /dev/disk2
Then use dd to copy the contents your ``*.img` file into your disk:
sudo dd if=path/to/your/file.img if=<your-removable-disk> bs=1ms
š” Use GNU
ddto get progress reportingThe
ddbinary that ships in macOS does not report progress, but the GNUās does. You can install it through Homebrewās pkgcoreutils(brew install coreutils).GNUās
ddwill then be available in your$PATHasgdd.
With GNU dd in place, you can run it just like you would do in dd, with an additional option, status=progress:
gdd bs=1M status=progress if=./your-image.img of=/dev/disk2
Additional resources:
Pacman
I just never remember these cmds of Pacman, really.
| Homebrew things | Pacman equivalents |
|---|---|
brew list | pacman -Q |
pacman -Qe (explicitly installed only; no deps) | |
brew outdated | pacman -Qu |
brew update | pacman -Sy |
brew search pkg | pacman -Ss pkg |
brew info pkg | pacman -Si <pkg> |
brew install pkg | pacman -Suy pkg |
brew uninstall pkg | pacman -Rs pkg |
brew cleanup | pacman -Scc |
| List orphan pkgs | pacman -Qdt |
| Autoremove orphans | pacman -Qdtq | pacman -Rns - |
š macos
Apple and/or macOS related wisdom.
Rebuilding Spotlightās cache
Whenever you find yourself struggling with Spotlight results being affected by past typos and having undesired first results for your typings, you can just reset the Spotlight cache.
To do it, go to System Preferences > Spotlight > Privacy, then add your disks to the search prevention list. It will force Spotlight to clear the cache of indexed items like docs, apps, etc. :-) After doing so, just remove the disks from the same list so the Spotlightās cache index can be rebuilt.
rsync š
ārsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison. ā
rsync website. rsync.samba.org
So, rsync. This is definitely one of my favorite piece of software in the world! It basically is a swiss-knife tool for anything related to syncing files across folders.
Using rsync
Using rsync is super simple! Hereās the command synopsis taken from its manual (man rsync):
NAME
rsync - a fast, versatile, remote (and local) file-copying tool
SYNOPSIS
Local:
rsync [OPTION...] SRC... [DEST]
Access via remote shell:
Pull:
rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push:
rsync [OPTION...] SRC... [USER@]HOST:DEST
Access via rsync daemon:
Pull:
rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push:
rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST)
Usages with just one SRC arg and no DEST arg will list the source files
instead of copying.
Youāll find a long list of options by reading its manual - but Iāll list here the post important ones:
-c: compare filesā checksums during sync-r: sync files and dirs recursively-l: copy symlinks as symlinks-n: dry run (run a trial; aka. without changes)-z: compress files during transfer (useful when syncing through network)--progress: show progress during transfer
So, a quick-and-dirty example to sync two folders would be like that:
# Assuming there's pen drive named "pen0" mounted in `/Volumes/pen0`, here's
# how you could sync files between a folder with the same name in your $HOME
# First, there must be a folder to be used as destiny (it's important)
mkdir /Volumes/pen0/my-files
rsync -cr --progress $HOME/my-files/ /Volumes/pen0/my-files/
# PS: the forward slashes in the end of every directory name are importants too
rsync in the real world
I used rsync to automate the deployment of WordPress-based projects in the past, back in my old days of PHP/WordPress. The techniques used were shown in a presentation held during a meetup of PHP-PB in April 2017. The ideas in this setup are still useful today:
https://speakerdeck.com/joeljuca/wordpress-at-meetup-php-pb
sbc š
SBCs (Single-board Computer) are cool. Hereās my experience with them.
Raspberry Pi 2 Model B
A 32-bit quad-core ARM Cortex-A7 processor⦠(To be continued)
šŖ¶ sqlite
SQLite is the most popular database in the world. Pretty much every fucking person in the world uses something that uses SQLite. However, itās generally considered a ātoy databaseā, often used for development purposes but not really considered as a possible solution to production usage.
In fact, itās easier to find people using ordinary data files (CSV, JSON, YAML, etc.) to store and organize larger amounts of data than to find people using SQLite for real-world scenarios. This is due to the fact that developers are often educated to use āproduction-grade databasesā like MySQL or Postgres instead of SQLite ā which is unfortunate, because all these three options are production-grade tools, capable of serving different types of products.
Lately, Iāve attempting to adopt SQLite for serious workloads. Iām building an app thatās intended for a wider audience, and Iāve adopted SQLite as my main database.
What a challenge! Letās see how far I can get with such embedded database.
Using SQLite
SQLite databases are stored in a single file. Apparently thereās no rule for how these are named, but Wikipediaās SQLite page lists the following file extensions: .sqlite, .sqlite3, .db, .db3, .s3db, and .sl3 (MIME type is application/vnd.sqlite3).
CLI work is made through a CLI tool called sqlite3. Eg.:
sqlite3 database.sqlite
- To create an empty database, just create an empty file:
touch database.sqlite - To delete a database, delete the file:
rm database.sqlite
Dot-commands
The most important one: .help Eg.:
$ sqlite
SQLite version 3.41.2 2023-03-22 11:56:21 [22/382]
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .help
.archive ... Manage SQL archives
.auth ON|OFF Show authorizer callbacks
.backup ?DB? FILE Backup DB (default "main") to FILE
.bail on|off Stop after hitting an error. Default OFF
.binary on|off Turn binary output on or off. Default OFF
.cd DIRECTORY Change the working directory to DIRECTORY
.changes on|off Show number of rows changed by SQL
.check GLOB Fail if output since .testcase does not match
.clone NEWDB Clone data into NEWDB from the existing database
.connection [close] [#] Open or close an auxiliary database connection
.databases List names and files of attached databases
.dbconfig ?op? ?val? List or change sqlite3_db_config() options
.dbinfo ?DB? Show status information about the database
.dump ?OBJECTS? Render database content as SQL
.echo on|off Turn command echo on or off
.eqp on|off|full|... Enable or disable automatic EXPLAIN QUERY PLAN
.excel Display the output of next command in spreadsheet
.exit ?CODE? Exit this program with return-code CODE
.expert EXPERIMENTAL. Suggest indexes for queries
.explain ?on|off|auto? Change the EXPLAIN formatting mode. Default: auto
.filectrl CMD ... Run various sqlite3_file_control() operations
.fullschema ?--indent? Show schema and the content of sqlite_stat tables
.headers on|off Turn display of headers on or off
.help ?-all? ?PATTERN? Show help text for PATTERN
.import FILE TABLE Import data from FILE into TABLE
.imposter INDEX TABLE Create imposter table TABLE on index INDEX
.indexes ?TABLE? Show names of indexes
.limit ?LIMIT? ?VAL? Display or change the value of an SQLITE_LIMIT
.lint OPTIONS Report potential schema issues.
.load FILE ?ENTRY? Load an extension library
.log FILE|off Turn logging on or off. FILE can be stderr/stdout
.mode MODE ?OPTIONS? Set output mode
.nonce STRING Suspend safe mode for one command if nonce matches
.nullvalue STRING Use STRING in place of NULL values
.once ?OPTIONS? ?FILE? Output for the next SQL command only to FILE
.open ?OPTIONS? ?FILE? Close existing database and reopen FILE
.output ?FILE? Send output to FILE or stdout if FILE is omitted
.parameter CMD ... Manage SQL parameter bindings
.print STRING... Print literal STRING
.progress N Invoke progress handler after every N opcodes
.prompt MAIN CONTINUE Replace the standard prompts
.quit Stop interpreting input stream, exit if primary.
.read FILE Read input from FILE or command output
.recover Recover as much data as possible from corrupt db.
.restore ?DB? FILE Restore content of DB (default "main") from FILE
.save ?OPTIONS? FILE Write database to FILE (an alias for .backup ...)
.scanstats on|off|est Turn sqlite3_stmt_scanstatus() metrics on or off
.schema ?PATTERN? Show the CREATE statements matching PATTERN
.selftest ?OPTIONS? Run tests defined in the SELFTEST table
.separator COL ?ROW? Change the column and row separators
.session ?NAME? CMD ... Create or control sessions
.sha3sum ... Compute a SHA3 hash of database content
.shell CMD ARGS... Run CMD ARGS... in a system shell
.show Show the current values for various settings
.stats ?ARG? Show stats or turn stats on or off
.system CMD ARGS... Run CMD ARGS... in a system shell
.tables ?TABLE? List names of tables matching LIKE pattern TABLE
.testcase NAME Begin redirecting output to 'testcase-out.txt'
.testctrl CMD ... Run various sqlite3_test_control() operations
.timeout MS Try opening locked tables for MS milliseconds
.timer on|off Turn SQL timer on or off
.trace ?OPTIONS? Output each SQL statement as it is run
.version Show source, library and compiler versions
.vfsinfo ?AUX? Information about the top-level VFS
.vfslist List all available VFSes
.vfsname ?AUX? Print the name of the VFS stack
.width NUM1 NUM2 ... Set minimum column widths for columnar output
sqlite>
Resources
SQLite is unlike most popular relational databases due to it being embedded instead of client-server.
- Mycelial YouTube channel
They develop extensions to SQLite for distributed use cases
SQLite and the N+1 Problem