There’s a faster and easier alternative to grep: ripgrep (rg).
TIL: split
I had a SQL file with 19M rows, one insert on each row, and I wanted to split it into smaller chunks. I found out that there is a split command which does exactly that:
split -l 100000 -d --additional-suffix=.sql mv-ids.sql mv-ids-
This takes mv-ids.sql file as an input and produces mv-ids-nn.sql files where each file has 100k lines. In my case nn goes from 00 to 89 and then from 9000 onwards. Good enough.
Learned today: UTF8 cannot store all of UTF8 in MySQL/MariaDB
You might think that using UTF8 encoding and collation in MySQL or MariaDB would solve the character encoding issues for good. Wrong! In MySQL and MariaDB UTF8 is an alias to UTF8MB3, which means it cannot store four-byte character sequences. Whose idea was that?
So, use UTF8MB4 instead. Sigh.
Learned today: Creating Grafana dashboard
I managed to create a new dashboard in our Grafana, to show the metrics collected from my services.
Learned today: How to provide Prometheus metrics in a Go program
Using https://godoc.org/github.com/prometheus/client_golang/prometheus I created a custom Collector that reads Stats from sql.DB and gives them out as Prometheus metrics.
Learned today: Why DNS failed in Docker swarm containers
Our hosts are in 10.x.x.x network, and the DNS server IP is 10.0.0.2. Now when we added an overlay network in Docker swarm without setting the subnet explicitly, Docker assigned the same 10.0.0.0/255 subnet to the overlay network. When the container tried to resolve a name, it was not able to talk to the 10.0.0.2 DNS server because IP packets got routed to the overlay network.
This was fixed by defining the subnets for overlay networks explicitly in another address range.
It was this GitHub issue that shed the light.
Learned today: Go templates
I built a library for loading configuration files that are actually Go templates, with some custom template functions for including values from other configuration files and loading secrets from files mounted by docker secrets.
If it turns out to be usable maybe I can open-source it some day, but for now it is proprietary.
Learned today: Apache strips Authorization header
TL;DR: CGIPassAuth On
The problem was that WordPress Rest API was not getting the HTTP basic auth information in the request. In this RunCloud setup there is first Nginx, which proxies to Apache, which proxies to FastCGI, which runs PHP ( I suppose the only reason Apache is there is to support .htaccess files).
First I suspected Nginx, because I’ve recently had issues with Nginx not always forwarding request headers to upstream host. I added proxy_pass_request_headers on there, but it made no difference.
I created a test script that prints out the Authorization header, and indeed it was not getting all the way to PHP. Some other headers, like User-Agent were getting through.
I checked the Apache access logs, and the username was not getting printed there, so I thought that Apache is not getting the header. I tried making the request directly to Apache, bypassing Nginx, but my test script still wasn’t getting the header. That lead me to suspect either Apache or PHP.
Finally I managed to find a Stack Overflow answer that lead me to the solution. I added CGIPassAuth On to .htaccess and bingo! Problem solved.
Learned today: How to use row_number with MySQL/MariaDB
I’m not going to explain it here, there are plenty of tutorials in the net. But I did find a possible bug in MariaDB and filed an issue.
I was trying to create a database migration in SQL, to add some events for entities that are saved in the database using event sourcing.
Solid
For a long, long time I’ve wanted to have a platform where I could store my own data wherever I want, share the data and allow applications to access it while retaining full control of who can access what. For years I’ve wanted to build such a service myself, but I’ve never got anywhere far with it.
But now, for the first time, there is a promising solution coming up. From the father of World Wide Web itself, Sir Tim Berner’s Lee: Solid.
I’ve only briefly looked at the docs, but the idea looks just like I want. I need to look at the implementation some day.