Sivers tech posts

How to sync Mac and Linux /home

2024-08-31T04:00:00Z

the problem:

My main computer is a Linux/BSD desktop, but I also use a Mac laptop for recording and travel. This created a problem keeping them in sync.

I don’t like that the MacOS user’s home has directories like Movies and Pictures that we can’t delete, or that ~/Library is filled with Mac-only crap I don’t want on my other computer. Also, why did Apple make it /Users/me and not /home/me?

the solution: synthetic.conf

We can use the fact that Mac has no /home/ to our advantage. In your terminal, type man synthetic.conf to read about it.

Assuming username “me” here, (meaning: replace “me”, below, with your actual username) I type, in the terminal:


sudo vi /etc/synthetic.conf

Then I add this one line, which has to be separated with a [tab]:


home        Users/me

Save, verify, then reboot.

/home/me = /Users/me/me

Make a new directory ~/me which will now be /home/me


mkdir ~/me
touch /home/me/hello
ls ~/me
ls /home/me/

Verify it’s working, that whatever you put into /Users/me/me/ is also now in /home/me/ and vice-versa. Then…

rsync

Of course you have to be able to SSH into your other computer, so if not, go figure that out first. I’ll assume that “desktop” reaches your Linux/BSD/other computer, probably via a nice entry in ~/.ssh/config

Then use free and lovely rsync to synchronize:


rsync -av --delete desktop:/home/me /home/

When it’s done, verify that everything from /home/me on your Linux/BSD computer is in /home/me on your Mac. Then move a file from your Mac Desktop or Documents into /home/me/ and copy it back:


rsync -av --delete /home/me desktop:/home/

When it’s done, verify that whatever you just added from your Mac is now on your Linux/BSD computer. I like to go back and forth a couple times to be absolutely sure it’s working correctly, to make sure I can rely on it.

I use these two commands so much that I made them one-line shell scripts in /home/me/bin/ so I can just type “gethome” when I first open the Mac, or “puthome” when I’m done on the Mac.

Note this also works very well for a remote copy of your home directory on an encrypted remote server as described in my “Tech Independence” page. In that case, the shell script has three steps:

SSH into the remote server, so you can type the password to mount the encrypted partition.
rsync -av --delete between them
SSH into the remote server to unmount the encrypted partition.

Whatever’s not in /home/me is Mac-only

This /home/me where you’ll keep everything of value — everything you share between computers. Whatever’s in your /Users/me directory but not in your /Users/me/me is Mac-only, and won’t be copied to your other computer. So whatever you have in the MacOS defaults of Desktop, Documents, Downloads, Library, Movies, Music, Pictures, and Public is only on the Mac. If you want to sync anything in there to your other computer, move it to /home/me/

I actually like this if I’m travelling with the Mac laptop and want to download some music or movies into ~/Music or ~/Movies but wouldn’t want to sync that 20GB of media to my other computer.

But most of all, I love having /home/me be the same on all computers, so I can be writing or coding on the Mac laptop on a train, then get home and type “puthome”, and get back to work on my desktop.

Another nice side-effect of keeping them in sync is if I accidently delete a file on one computer, I can pick up the other one to restore it. Or if the entire computer is stolen or destroyed, I can just curse a bit, then get back to work on the other.

How and why to make a /now page on your site

2024-05-18T04:00:00Z

Background

I used to wonder what my friend Benny Lewis was doing. He has a website and social media accounts, but neither gave an overview of what he’s doing now.

Then I realized some people might wonder the same about me. So in 2015, I made a /now page on my website, saying what I’d tell a friend I hadn’t seen in a year.

It has a nice side-effect of being a public declaration of priorities. It’s a good link to give people when saying no to invitations and distractions.

Word spread, and soon hundreds of people had a /now page on their personal website. So I made a site to showcase them all: nownownow.com — (a static site generated by PostgreSQL functions.) It currently has over 2300 people worldwide. This week I added browse by location and search.

Got a personal website? Add a /now page

The three main ingredients are:

a page, usually at URL /now, linked from your main menu, usually alongside /about
an overview of what’s going on with you — what you’d tell a friend you hadn’t seen in a year
the date it was last updated

WordPress instructions:

In the left menu, under “Pages”, click “Add New Page”. Then, where it says “Add title”, replace that with just three letters: now. That will ensure the URL is /now, and after it’s posted, you can change the title to “What I’m doing now” or whatever.

Wix instructions:

On the left, under “Site Menu”, click “+ Add Page”. Call it Now, next to your “About” page. Click the (…) to its right, then to “SEO basics”, to edit “URL slug” and make sure it’s just the three letters: now

No website yet? Use Bear

If you don’t have a personal website yet, I highly recommend Bear at BearBlog.dev. It’s so simple, clean, and free. The owner and creator, Herman in South Africa, runs it himself with great love. And no investors so no enshittification. He plans to keep it alive forever.

I believe in it so much that I told Herman I would be its godfather. If he ever can’t (or doesn’t want to) run it anymore, I will help run it, or fund a foundation to keep it alive.

To create a /now page on Bear, click “Pages”, then “New page”. Then, after it says “title:”, type just three letters: now. That will ensure the URL is /now, and after it’s posted, you can change the title to “What I’m doing now” or whatever.

Got a /now page? I’ll add you to nownownow.com

Once it’s live, just email me your URL, and I add it (by hand) to nownownow.com. (This is also a good time to say hello, if you haven’t yet. I read and reply to every email.)

How to learn JavaScript

2024-03-08T05:00:00Z

Since I mentioned that I learned JavaScript, people have asked me how and what I recommend. So here’s my experience and best advice for 2024.

Learn plain JavaScript

First, it’s important to learn plain JavaScript.

Don’t depend on a library of shortcuts. JavaScript will be around for many years, while libraries and frameworks come and go every year.

How to begin

Don’t just learn from little bits of tutorials, tips, or tricks online. Instead, you want a real foundation and solid understanding. Then all your future learning will be so much faster.

Start with the book: “Eloquent JavaScript”. It’s free to read there on his website. It’s deep and thorough. A great start-to-finish JavaScript tutorial. Relax. Focus. Do it in order. You’ll know more than most once you get to the end.

Do Free Code Camp. Someone who had 600 JavaScript learners said Free Code Camp had the best results. Do it at the same time as you’re reading books and articles. Learning sticks better when coming through different senses.

Make it stick

I highly recommend you write down every new thing you learn, ideally into a flashcard program, like this. It’s a lot to digest, so you’ll have to remind yourself of what you’ve learned, or you’ll immediately forget.

Avoid the shortcuts

When solving a problem, everyone will point you to some pre-made solution. “Use jQuery! Use React! Use this library and save yourself some typing!”

But no! Not yet! Do it the hard way. Solve everything yourself with plain JavaScript. It’s the best way to learn.

What next?

If you had a web app in mind, start building it now. After you wrestle with using plain “vanilla” JavaScript, then learn React to see the shortcuts it gives you.

If you want a good-paying job, you can get hired almost anywhere. Since you know the real JavaScript foundations more than most people, you’re very valuable. You can quickly learn whatever framework they’re using.

Any other suggestions? What worked for you?

Please share any other suggestions or experiences, here:

How I backup

2024-02-27T05:00:00Z

Some people have asked, so here is how I do my backups. It takes me about ten seconds per day and five minutes per month to maintain.

It works well for me, but I’m always open to suggestions. Just email me with any ideas or questions.

every-day documents (~40 GB)

Files I use and change every day: documents, emails, code, diary, ideas, website, accounts, etc.

I have a desktop and a laptop, so I keep this ~40 GB directory cloned with rsync every day or so between them. Whenever I turn on one computer, I sync it from the other.
Daily rsync to an encrypted Linux USB stick attached to the desktop.
Daily rsync to an encrypted MacOS USB stick attached to the laptop.
Daily rsync to an encrypted ZFS SSD inside the desktop.
Daily rsync to OpenBSD remote attached storage at vultr.com as described at my “Tech Independence” page.

I’ve written shortcuts for these rsync commands so it’s really as simple as me typing bkz or bkstick in the terminal as I’m working anyway. I do it a few times a day, especially if I’ve just made or saved something of value. And always right before I shut down the computer, which I do almost any time I step away from it for more than a few minutes. That’s why I say it’s like ten seconds a day, just typing that command occasionally.

keepsakes (~3 TB)

Rarely-accessed files I want to keep forever: videos and photos of my kid, music and film collection.

Three different external 4TB 2.5" drives, in three different formats: ZFS, OpenBSD, MacOS.
Every month or so, I connect them via USB cable, and rsync everything to each of them, bringing one down to a safe deposit box downtown, and taking out the one that was there from my last visit.
5TB at Hetzner Storage Box, I rsync remotely.
4TB at ZFS.rent, a ZFS encrypted clone.

So that’s five copies of my keepsakes in four different locations. Each one also has a recent copy of my every-day documents from above. So a few minutes a month to connect the USB drives, and I only do the safe deposit box when I’m going that way anyway, and only takes a couple minutes.

In the days of spinning-platter hard drives, they used to die surprisingly often, so I like the security of many copies. But it also helps to have some deliberately behind others, so on the rare case where I’ve deleted something, then rsynced my dailies and even my monthly, then realized I need that thing I deleted, then I know the drive in the safe deposit box has it since I haven’t updated that one in a couple months.

Want anonymity? Make a persona not a mystery.

2023-02-02T05:00:00Z

Because of my open inbox, I meet a lot of strangers. I love it. Almost everyone tells me who and where they are in the world. If they don’t, I wonder.

Am I talking with someone from Australia? Philippines? Brazil? Are they 20 or 60? Male or female? It doesn’t really matter, but the brain can’t help wondering. It’s human nature to want to know who’s speaking. If they don’t say, it creates a mystery.

Once people start wondering, they need to know. Mysteries are intriguing. They’re unsettling.

That’s a problem if you really want to be anonymous. If you defiantly refuse to say who you are, it can make people angry that you’re upsetting social reciprocity. You know who they are, but they don’t know who you are. It feels rude. An obsessive personality might make it their damn mission to figure out who you are! You don’t want that.

So for real anonymity, don’t create a mystery. Create a believable persona. Then nobody will wonder.

If you don’t want any attention, just pick a very common name like Mary Kim or Adam Johnson.

Use an AI face generator to create a completely believable face to match your new name. Download it once and use it everywhere. Run it through face aging software to use this same persona for the rest of your life.

Pick a city and say it’s your location, to avoid that question too.

For email, Mailbox.org is great, and doesn’t care who you are.

Create social media profiles with your new name, email, city, and face.

Nobody will wonder who you are if you answer that question. Instead of block and battle, deflect and settle. That’s better anonymity.

But if you want to be both anonymous and famous, pick a name that is rare but believable. Cool but not too cool. Tom Kahlo. Keaton Carolina. Miles Wenley. Pick a name that has the .com domain available, so you can really brand it.

Now you can create anything online freely, and nobody will doubt your identity. Create and post a back-story to answer (instead of avoid) the frequently asked questions. Then, instead of wondering who you really are, they can focus on what you’re really saying.

Static HTML comments

2022-10-08T04:00:00Z

If you have a static HTML website, but you want to include comments, here’s an interesting way to do it using PostgreSQL’s NOTIFY and LISTEN.

The big idea is to write the comments as static HTML, only when comments change, instead of doing a database query to display them every time. This prevents the “hug of death” if you get a burst of traffic.

I’ve been doing it this way for over six years, and it works great. Here’s the recipe, using Ruby as the glue, though you could use any scripting language.

PostgreSQL database table for comments
Ruby receives form posts, inserts into database
When comments change, PostgreSQL trigger sends NOTIFY
Ruby runs PostgreSQL LISTEN, exporting updated comments to HTML
JavaScript on static page includes HTML

PostgreSQL database table for comments

create table comments (
  id integer primary key generated by default as identity,
  uri text,
  created_at date default current_date,
  name text,
  email text,
  comment text
);
create index on comments(uri);

download code

Ruby receives form posts, inserts into database

Put this on any HTML page where you want comments:

<section id="comments"></section>
<script src="/comments.js"></script>

download code

Put this next code in your Nginx config, to send /comments to localhost.

location = /comments {
  proxy_pass http://127.0.0.1:4567;
}

download code

Ruby Sinatra receives form posts.

require 'pg'
require 'sinatra'
DB = PG::Connection.new(dbname: 'test', user: 'tester')

post '/comments' do
  DB.exec_params("insert into comments
    (uri, name, email, comment)
    values ($1, $2, $3, $4)",
    [params[:uri], params[:name], params[:email], params[:comment]])
  redirect to(request.env['HTTP_REFERER'])
end

download code

Run that in a terminal on the server, and it should default to listen on port 4567.

When comments change, PostgreSQL trigger sends NOTIFY

create function comments_changed() returns trigger as $$
begin
  perform pg_notify('comments_changed', new.uri);
  return new;
end;
$$ language plpgsql;
create trigger comments_changed after insert or update on comments
for each row execute procedure comments_changed();

download code

Load that function into the PostgreSQL database that has your comments table.

It sends the listener (below) a notification that comments for this URI have changed. Then the listener will re-output comments just for this URI, instead of all.

Ruby runs PostgreSQL LISTEN, exporting updated comments to HTML

Make a directory in your web root called /commentcache/, to hold the static comments.

Then keep this Ruby script running in a terminal to listen for database changes, and write the updated comments to disk as HTML.

require 'pg'
DB = PG::Connection.new(dbname: 'test', user: 'tester')
BASEDIR = '/var/www/htdocs/commentcache/' # directory in your web root

# a single comment list entry, used in ol map, below
def li(row)
  '<li><cite>%s (%s)</cite><p>%s</p></li>' %
    [row['name'], row['created_at'], row['comment']]
end

# top-level map of database rows into HTML list
def ol(rows)
  rows.inject('') {|html, row| html += li(row) ; html}
end

# write comments to disk for this URI
def save_comments(uri)
  rows = DB.exec_params("select name, created_at, comment
    from comments where uri = $1 order by id", [uri]).to_a
  File.open(BASEDIR + uri, 'w') do |f|
    f.puts ol(rows)
  end
end

# first write them all
DB.exec("select distinct(uri) from comments").each do |r|
  save_comments(r['uri'])
end

# listen for changes. re-write when changed
DB.exec('listen comments_changed')
while true do
  DB.wait_for_notify do |event, pid, uri|
    save_comments(uri)
  end
end

download code

JavaScript on static page includes current HTML when viewed

Use JavaScript to show the form to post a comment, and load the list of comments from the /commentcache/ path.

function showForm(uri) {
  document.getElementById('comments').innerHTML = `
<header><h1>Comments:</h1></header>
<form method="post" action="/comments">
<input type="hidden" name="uri" value="${uri}">
<label for="name">Your Name</label>
<input type="text" name="name" id="name" required>
<label for="email">Your Email</label>
<input type="email" name="email" id="email" required>
<label for="comment">Comment</label>
<textarea name="comment" id="comment" cols="80" rows="10" required></textarea>
<input type="submit" value="post comment">
</form>
<ol id="commentlist"></ol>`;
}

function getComments(uri) {
  try {
    const xhr = new XMLHttpRequest();
    xhr.open('get', '/commentcache/' + uri);
    xhr.send(null);
    xhr.onload = function() {
      if (xhr.status === 200) {
        document.getElementById('commentlist').innerHTML = xhr.responseText;
      }
    };
  } catch(e) { }
}

// /blog/topic/page.html uri = 'blog_topic_page.html' for filesystem
const uri = location.pathname.substring(1).replace(/\//g, '_');
showForm(uri);
getComments(uri);

download code

That’s all. I’ve simplified it a bit from my real usage, where I have constraints and checks that would have distracted from the core point of this example.

There are other ways to do it. The NOTIFY and LISTEN isn’t necessary. The Ruby Sinatra route that receives the posted comment could just write the HTML to disk immediately. But I have other scripts that delete and update comments, and I like how the combination of NOTIFY trigger and LISTEN script always keeps them updated on disk.

Another interesting approach would be to write the comments into each HTML file directly, instead of in a separate file, so you wouldn’t need JavaScript at all.

Optional upgrade: NOTIFY on delete

I simplified the PostgreSQL trigger for the example, but with a few more lines of code, you can use the same trigger to notify of deleted comments, too. The value of a deleted row is in “old”, whereas inserted and updated is in “new”, so we have to make a uri variable, and an if/then/else to know which to use.

create or replace function comments_changed() returns trigger as $$
declare
  uri text;
begin
  if tg_op = 'DELETE' then
    uri = old.uri;
  else
    uri = new.uri;
  end if;
  perform pg_notify('comments_changed', uri);
  return old;
end;
$$ language plpgsql;
create trigger comments_changed after insert or update or delete on comments
for each row execute procedure comments_changed();

download code

Find a good available .com domain

2022-06-08T04:00:00Z

If you need a new domain name, and you want a .com, and you don’t want to type random ideas into a registrar search, here’s a way to do it.

Download the list of all registered .com domains

First, apply for access to the zone file, using ICANN’s Centralized Zone Data Service (CZDS) at https://czds.icann.org/. It’s free, but takes a few days to get approved. Read more about it here.

Once approved, they email you a password to log in and download the file called com.txt.gz.

$ du -hs com.txt.gz # 4.6GB compressed
4.6G    com.txt.gz
$ gunzip com.txt.gz # uncompress and wait
$ du -hs com.txt
23.0G   com.txt     # 23 gigs uncompressed
$ wc -l com.txt
404261754 com.txt   # 404 million lines

Extract the unique names

com.txt has 404 million lines like this:

zombahomes.com. 172800  in      ns      ns2.tierra.net.
zombai.com.     172800  in      ns      ns1.parkingcrew.net.
zombai.com.     172800  in      ns      ns2.parkingcrew.net.
zombaid.com.    172800  in      ns      nsg1.namebrightdns.com.
zombaid.com.    172800  in      ns      nsg2.namebrightdns.com.
zombaimmo.com.  172800  in      ns      ns10.lwsdns.com.
zombaimmo.com.  172800  in      ns      ns11.lwsdns.com.
zombaimmo.com.  172800  in      ns      ns12.lwsdns.com.
zombaimmo.com.  172800  in      ns      ns17.lwsdns.com.
zombaio.com.    172800  in      ns      ns-1073.awsdns-06.org.

Domains usually have more than one entry. You need to extract the unique entries. And you only need the part before the “.com”.

Here’s a Ruby script that loops through com.txt, gets the part before .com, skips it if duplicate, and outputs it if unique.

domain = ''
File.open('com.txt', 'r') do |infile|
  File.open('domains.txt', 'w') do |outfile|
    while line = infile.gets
      temp = line[0...(line.index('.com'))]
      next if temp == domain
      domain = temp
      outfile.puts domain
    end
  end
end

download code

“domains.txt” should now be about 162 million lines - (about 2.2GB) - that look like this:

zombahomes
zombai
zombaid
zombaimmo
zombaio

Load it into SQLite, and index it.

$ sqlite3 domains.db 
sqlite> create table domains(domain text);
sqlite> .import "domains.txt" domains
sqlite> create index dd on domains(domain);

Find available dictionary words

If you’re on Mac, Linux, or BSD, you should have a dictionary of words at /usr/share/dict/words. See which of those words are available:

require 'sqlite3'
db = SQLite3::Database.new('domains.db')
query = db.prepare('select domain from domains where domain = ?')
File.readlines('/usr/share/dict/words').each do |word|
  rows = query.execute(word.downcase.strip)
  puts word unless rows.next
end

download code

Run that, and you’ll have a list of 93,000 dictionary words that are available with the .com extension. Congratulations! Go to porkbun.com (a great little registrar) to register yours.

You’ll find that some are not actually available because that “com.txt” file doesn’t list domains on hold, pending deletion, or without name servers.

Combine short dictionary words

If you are not excited that “electrotelethermometer.com” or “counterexcommunication.com” is available, maybe you would like a combination of two short words? Select only dictionary words up to four letters, then search for the combination.

require 'sqlite3'
words = File.readlines('/usr/share/dict/words').map(&:strip)
words.select! {|w| w.size <= 4}
db = SQLite3::Database.new('domains.db')
query = db.prepare('select domain from domains where domain = ?')
words.each do |word1|
  words.each do |word2|
    combo = (word1 + word2).downcase
    rows = query.execute(combo)
    puts combo unless rows.next
  end
end

download code

Narrow it down to good words

If you ran that last script, you’ll get tens of millions of available domains like “knabtuik.com” because there are many unknown, ugly, and useless short words.

So make a new file called “goodwords.txt” of only three and four letter words, using grep:

$ grep "^...$" /usr/share/dict/words >> goodwords.txt
$ grep "^....$" /usr/share/dict/words >> goodwords.txt

Edit that file by hand, deleting every word you would never want. (The less you keep, the better.) Then run that Ruby script again, combining just the good words:

require 'sqlite3'
words = File.readlines('goodwords.txt').map(&:strip)
db = SQLite3::Database.new('domains.db')
query = db.prepare('select domain from domains where domain = ?')
words.each do |word1|
  words.each do |word2|
    combo = (word1 + word2).downcase
    rows = query.execute(combo)
    puts combo unless rows.next
  end
end

download code

Much better, right? A little time consuming, but worth it. This is how I found the name of my new translation service, Inchword.

Need it super-short and nerdy?

One final hack is that there are tons of very-short .com domain names available in the format “letter-number-letter-number”. For example: q7r7.com or e3p3.com.

require 'sqlite3'
db = SQLite3::Database.new('domains.db')
query = db.prepare('select domain from domains where domain = ?')
('a'..'z').each do |a|
  ('0'..'9').each do |b|
    ('a'..'z').each do |c|
      ('0'..'9').each do |d|
        combo = a + b + c + d
        rows = query.execute(combo)
        puts combo unless rows.next
      end
    end
  end
end

download code

Short URLs: why and how

2022-05-08T04:00:00Z

If you make your own website, consider making short URLs.

This is not about a URL shortener. This is about making your original URLs short in the first place.

Instead of:
yoursite.com/blog/2022/05/08/short-urls-why-and-how.html
… consider just:
yoursite.com/short

Why?

Short URLs matter for a few reasons:

I can remember my URLs. I can avoid the search engine step. (No need to search when I already know the answer.) Which means…
I can type it or say it. Whether texting, answering an email, or talking to someone on the phone, I can say, “Go to sive.rs/ff for my talk about the first follower.” or “My newest book is at sive.rs/h.” I do this often, so having memorable URLs, easy to type in full, saves me a lot of searching.
They look nicer. They’re aesthetic. They show care. We should put something of beauty into the world, instead of creating digital pollution.
They remove the middle-man. With long URLs, people use those ugly social share buttons that promote (and further entrench) harmful social media sites, and add visual clutter to your site. Short URLs encourage people to copy and paste the URL directly, which lets them share it anywhere, instead of only the sites for which you have a share button.
They’re enough. Using 36 characters (a-z and 0-9):
2-character URLs give me 1296 (36²) unique combinations.
3-character URLs give me 46,656 (36³) unique combinations.
4-character URLs give me 1,679,616 (36⁴) unique combinations.
I don’t need more than that.

That doesn’t mean the shorter the better. If I can remember “/plaintext” easier than “/pt”, then that’s a better choice.

How?

Here’s how I do it: Save my HTML file as the URL name, with no extension. Instead of “hi.html”, I save it as “hi” in my public web root.

Then, assuming the Nginx web server, add this line to my http block:

default_type text/html;

That’s it! That serves my files without extensions as HTML.

If I ever switch to a different system or server, there is always a way to pull up your HTML using a short URL.

Write plain text files

2022-03-02T05:00:00Z

I write almost everything important in my life: thoughts, plans, notes, diaries, correspondence, code, articles, and entire books.

They are my extended memory — my noted self — my organized thoughts. I refer to them often. I search them, update them, and learn from them. I convert them into HTML to make websites, or LaTeX to make books.

My written words are my most precious asset. They are also a history of my life. That’s why I only use plain text files. They are the most reliable, flexible, and long-lasting option. Here’s why.

PORTABLE

I’ve brought my text files with me since 1990, from Mac to Windows to Linux to BSD, from PCs to laptops to tablets to Android to iOS to a tiny device the size of my thumb, and back again.

Every device, including ones long gone, and ones not invented yet, can read and edit plain text. Whether future virtual reality, or a chip you can implant in your earlobe, plain text will be there. Will Microsoft Word? Evernote? Notion? Maybe. Maybe not.

But plain text? Always. Everywhere.

UN-COMMERCIAL

Every few years a new company says you should use their special format. You have to pay them a monthly fee to use it — or keep all of your documents in their care. They offer some convenience or features, but at the cost of flexibility, portability, and independence.

When you store your writing in one company’s unique format, then you need that program to access it. Then the economy takes a turn, they go out of business, and your work is trapped in an unusable format.

You will outlive these companies. Your writing should outlive you. Depending on companies is not an option.

Plain text is un-commercial. It removes you from the world of subscriptions and hype. There will always be plenty of free, non-commercial software in the public domain for reading and editing text files.

OFFLINE

There are places and times when you can’t get online. Don’t depend on any tool that needs an internet connection.

There are great benefits to being intentionally offline and unreachable, to focus. It’s a super productivity boost. You need to be able to write, and have access to all your writing, during these times.

NO DEPENDENCIES

If you rely on Word, Evernote or Notion, for example, then you can’t work unless you have Word, Evernote, or Notion. You are helpless without them. You are dependent.

People tell me about more tools I could use in addition to my text files. But I don’t need or want anything else. Plain text files and a basic text editor are enough. This is everything you need for great thinking and writing. (A paper notebook and pencil are enough, too.)

If you only use plain text, you can work on any device, forever. The less you depend on, the better. Peace and focus come when you stop looking for more.

EASIEST TO CONVERT

Plain text can be converted into anything else.

HTML, Markdown, JSON, LaTeX, and many other standard formats, are just plain text. I’ve written four books and four hundred blog posts in plain text.

You can make your own personal formats in your plain text files. Maybe in each diary entry, the first two lines are like:

date: 2022-02-28
tags: where-to-live, kids, dog, anxious

Then it’s easy to use any little scripting language like Ruby, Python, or JavaScript to grab the date and tags, and use them for categorizing, sorting, renaming, archiving, or exporting.

Or if you don’t want to do it yourself, then it’s easy to find someone who can. Anyone who’s been programming for more than a week should be able to do it easily.

NEED HIERARCHY?

Use directories — also known as folders. These are also good for keeping your text together with other files like images and audio.

Documents/
Documents/Diary/
Documents/Diary/2022/
Documents/Diary/2022/2022-02-28.txt
Documents/Thoughts/
Documents/Thoughts/WhereToLive/
Documents/Thoughts/WhereToLive/2019-06-30.txt
Documents/Thoughts/WhereToLive/2020-01-18.txt
Documents/Ideas/
Documents/Ideas/MusicalChairs.txt
Documents/Ideas/NewHouse/
Documents/Ideas/NewHouse/Design/
Documents/Ideas/NewHouse/Design/entryway.jpg
Documents/Ideas/NewHouse/Design/roof.jpg
Documents/Ideas/NewHouse/Architect/
Documents/Ideas/NewHouse/Architect/JM_Lim.txt
Documents/Ideas/NewHouse/Architect/TPS_Inc.txt

NEED VISUALS OR GRAPHICS?

Need visual mind-mapping with circles and lines? Maybe you do. But maybe you don’t. Maybe it’s just another distraction, focusing on the tools instead of your thinking.

I love that plain text files have no formatting to tinker with. A tab key, SHIFT KEY, and vertical line breaks can go a long way, keeping you writing instead of formatting.

If you really need graphics, do your drawing using something else. Digital drawing into SVG files. Paper drawing, scanned into JPGs. Formats that aren’t owned by any company. Formats that will outlast you.

Keep your graphics files alongside your text files. But keep your text as plain text.

CONCLUSION

Reliable, flexible, portable, independent, and long-lasting. Plain text files will be readable by future generations, hundreds of years from now.

I especially enjoy the tranquility of their offline, non-commercial nature. They’re quiet. They’re focused. (As I aim to be.)

Generate unique random values directly in the database

2022-03-01T05:00:00Z

You often need to generate random strings, like for login cookies and unique entry codes.

These strings always need to be stored in the database. So make your life simpler by generating the random strings directly in the database. Here's a very handy PostgreSQL function to do it:

create function gen_random_bytes(int) returns bytea as
'$libdir/pgcrypto', 'pg_random_bytes' language c strict;

create function random_string(len int) returns text as $$
declare
  chars text[] = '{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z}';
  result text = '';
  i int = 0;
  rand bytea;
begin
  -- generate secure random bytes and convert them to a string of chars.
  rand = gen_random_bytes($1);
  for i in 0..len-1 loop
    -- rand indexing is zero-based, chars is 1-based.
    result = result || chars[1 + (get_byte(rand, i) % array_length(chars, 1))];
  end loop;
  return result;
end;
$$ language plpgsql;

download code

Give it a number: the length of the random string you want. It will return random alphanumeric text of that length.

select random_string(8);
random_string 
───────────────
 yBuXga02

select random_string(8);
 random_string 
───────────────
 eP3X7yqe

download code

The chance of random clash is very small. But you need to be completely sure that the new random string is unique — not already existing in that column in that table. So here's a function where you give it the string length, table name, and column name. It will return a random string confirmed to be unique — to not exist there already. It gets a random string, searches for it in that table and column, and if not found, returns it. Otherwise, if it is found, gets a new random string and loops back, trying again until not found.

-- return random string confirmed to not exist in given tablename.colname
create function unique_random(len int, _table text, _col text) returns text as $$
declare
  result text;
  numrows int;
begin
  result = random_string(len);
  loop
    execute format('select 1 from %I where %I = %L', _table, _col, result);
    get diagnostics numrows = row_count;
    if numrows = 0 then
      return result; 
    end if;
    result = random_string(len);
  end loop;
end;
$$ language plpgsql;

download code

I used to call functions like this using database triggers, called on any insert. But then I found out something surprisingly cool and so much simpler: You can call functions as default values directly in table definitions.

Look at this table, calling unique_random as its default value:

create table things (
  code char(8) primary key default unique_random(8, 'things', 'code'),
  name text
);

download code

So simple and clear! To use it, you just do a regular insert, and it generates the guaranteed-unique default value.

insert into things (name) values ('one') returning *;

   code   │ name 
──────────┼──────
 nRSXbVWQ │ one

insert into things (name) values ('two') returning *;

   code   │ name 
──────────┼──────
 EAS9wGcl │ two

download code

I've found this particularly handy for creating login cookies:

create table cookies (
  person_id int primary key,
  cookie char(32) unique default unique_random(32, 'cookies', 'cookie')
);

download code

Instead of having your client code, your JavaScript, Python, Ruby or whatever, generating the random code, it's extra-nice to have this in your database directly, not only because it's cleaner, but because it saves repeated calls between your client code and database, confirming uniqueness. One simple insert of the person_id returns the unique and already-saved random cookie string:

insert into cookies (person_id) values (1) returning *;

 person_id │              cookie              
───────────┼──────────────────────────────────
         1 │ 0P8Tp4wjXuTqCCh1NCR9XIom20z9IcYv

download code

Download the code at /code/rand1.sql.

Database triggers to clean text inputs

2022-03-01T05:00:00Z

Even smart people can accidently put bad data into a database.

We copy text from a web page, then paste it into a form, not realizing it also copied a space, tab, or newline character. Then your system thinks someone’s name is not “Jim” but “ Jim\n”.

You could use JavaScript to sanitize all form inputs, but what about when you import a CSV file, or get data from an API?

No, the best place for your data-cleaning functions is in the database itself. So no matter what code is inserting or updating, a database trigger will sanitize it before saving.

Here’s a PostgreSQL example. Let’s make two tables, people and emails, so you can see how one function can be used by many triggers.

create table people (
  id serial primary key,
  name text,
  code text
);

create table emails (
  id serial primary key,
  person_id integer not null references people(id),
  email text
);

download code

Let’s make two simple functions:

Remove all whitespace, then lowercase.
Remove unwanted whitespace characters like tab and newline. Replace them all with a single space. Then trim spaces from the front and end.

-- remove all whitespace, then lowercase it
create function lower_no_space(text) returns text as $$
  select lower(regexp_replace($1, '\s', '', 'g'));
$$ language sql;

-- replace all whitespace with single space, then trim start and end
create function no_extra_space(text) returns text as $$
  select btrim(regexp_replace($1, '\s+', ' ', 'g'));
$$ language sql;

download code

The function that removes all whitespace then lowercases, is good for email addresses and codes like checksums.

The function that removes extra whitespace then trims, is good for many things like names, addresses, email subjects, and anywhere that needs to retain inner spaces. So “ New \t Zealand \n” will be “New Zealand”.

Now make trigger functions that use your smaller re-usable cleaning functions. I find it best to make one trigger per table, sanitizing all fields on any insert or update.

create function clean_people() returns trigger as $$
begin
  new.name = no_extra_space(new.name);
  new.code = lower_no_space(new.code);
  return new;
end;
$$ language plpgsql;
create trigger clean_people
before insert or update on people
for each row execute procedure clean_people();

create function clean_emails() returns trigger as $$
begin
  new.email = lower_no_space(new.email);
  return new;
end;
$$ language plpgsql;
create trigger clean_emails
before insert or update on emails
for each row execute procedure clean_emails();

download code

The language is a little verbose, but oh well. The ugliness of the boilerplate code is made up for by the beautiful simplicity of having all of this handled in the database. It’s so nice to not have to sanitize form fields! Just toss the unwashed inputs at the database.

Here, let’s give it some dirty data, and watch it come out clean.

insert into people (name, code) values (e' \t \r \n Dr.  \n \r JM \t Lim \r\n', '   XX o Z ') returning *;
-- id │    name    │ code 
--────┼────────────┼──────
--  1 │ Dr. JM Lim │ xxoz

insert into emails (person_id, email) values (1, e' \r\n \t DR. L @ JM Lim . com \n') returning *;
-- id │ person_id │     email      
--────┼───────────┼────────────────
--  1 │         1 │ dr.l@jmlim.com

download code

This approach has made my code much simpler, and data much cleaner.

Download the full code example at /code/clean1.sql.

Also look at using triggers to ensure data integrity.

Database functions to wrap logic and SQL queries

2022-02-28T05:00:00Z

When you make a database-backed app, you have some functions that need to run multiple database queries.

For example, to move money from one account to another, you have to insert a negative amount into account #1, and a positive amount into account #2.

Usually you do in your main code: your JavaScript, Python, Ruby, or whatever.

But what if some future code bypasses your crucial “business logic” functions? New code could access the database directly, without going through your existing functions.

Or what if you need to rewrite some code in a new language? You’ll have a lot of data to rewrite if all this data logic was kept in the surrounding code.

I felt the pain of this at my last company, when we converted some old PHP code to Ruby. I had to rewrite so much of the logic.

In hindsight, data logic should be in the database itself.

Simple logic that’s always needed to update the data (like the money-moving example) should be kept in database functions. Then your surrounding code - your JavaScript, Python, Ruby, or whatever - can just call these database functions, and never need to be rewritten if you change languages to Swift, Kotlin, Elixir, or whatever.

Here’s a PostgreSQL example, from my previous post:

First, make three simple tables:

Items with prices.
Lineitems with quantities.
Invoices with the total price.

Create two example items, a $5 and a $9 item. And create invoice #1 for testing.

create table items (
  id serial primary key,
  price int not null check (price > 0)
);

create table invoices (
  id serial primary key,
  total int
);

create table lineitems (
  invoice_id int not null references invoices(id),
  item_id int not null references items(id),
  quantity int not null check (quantity > 0),
  primary key (invoice_id, item_id)
);

-- example data:
insert into items (price) values (5);
insert into items (price) values (9);
insert into invoices (total) values (0);

download code

If someone wants to add an item to their cart, you need to first see if it’s in their cart already. If it’s not in their cart, insert it. But if that Item is in their cart, you need to update it, to add the new quantity to their existing quantity.

So wrap all that logic in a simple function called cart_add.

create function cart_add(inv int, item int, quant int) returns void as $$
begin
  -- does this invoice + item combination already exist?
  perform 1 from lineitems
  where invoice_id = inv
  and item_id = item;
  if found then  -- yes? add this quantity
    update lineitems
    set quantity = quantity + quant
    where invoice_id = inv
    and item_id = item;
  else  -- no? insert
    insert into lineitems values (inv, item, quant);
  end if;
end;
$$ language plpgsql;

download code

Someone updates their cart, to change the quantity of a Lineitem. If they change the quantity to 2, 5, or even 1, no problem, just update the quantity. But what if they change the quantity to 0? You don’t want a Lineitem hanging around their cart with a quantity of 0. No, if the quantity is 0 or below, you want to delete that Lineitem.

So wrap all that logic in a simple function called cart_set.

-- update the quantity of an item in the cart
create function cart_set(inv int, item int, quant int) returns void as $$
begin
  if quant > 0 then
    update lineitems
    set quantity = quant
    where invoice_id = inv
    and item_id = item;
  else  -- quantity 0 or below? delete
    delete from lineitems
    where invoice_id = inv
    and item_id = item;
  end if;
end;
$$ language plpgsql;

download code

There, now this data logic is where it belongs: with the data itself.

Your JavaScript, Python, Ruby, or whatever can just call the functions, like this:

select cart_add(1, 1, 3);
select * from lineitems;

select cart_add(1, 2, 4);
select * from lineitems;

select cart_set(1, 2, 1);
select * from lineitems;

download code

Imagine if you did it like this for all of the important things you need to do in your database?

Then any code, in any language, could just call those functions, knowing the database itself will handle the logic. Keeping the data-logic where it should be: with the data.

Download the final example file here: /code/api01.sql.

More on this in future posts. Or until then, see my example on Github.

Database trigger recalculates totals, for data integrity

2022-02-27T05:00:00Z

When you make a database-backed app, you write code to ensure data integrity in related fields.

If an item is sold, you update the inventory. If someone withdraws money, you update their balance. You recalcuate the total and update the related field in the database whenever somebody makes a change.

Usually this is done in your main code: your JavaScript, Python, Ruby, Java, or whatever.

But what if some future code doesn’t use your currently-crucial function? A new back-end interface, API hook, or simple shell script might not use your current code. New code will access the database directly, not using your old code.

I encountered this exact situation at my last company, and felt the pain from inventory and even finances becoming wrong, all because some new code was accessing the database directly.

So that’s why I’m an evangelist now for how important it is to put your crucial code in the database itself. This is data logic (not “business logic”) and should be bound to the data. Database functions can be triggered, ensuring integrity, no matter what outside code is accessing it.

Here’s an example in PostgreSQL, using a shopping cart:

First, make three simple tables:

Items with prices.
Lineitems with quantities.
Invoices with the total price.

Create two example items, a $5 and a $9 item. And create invoice #1 for testing.

create table items (
  id serial primary key,
  price int not null check (price > 0)
);

create table invoices (
  id serial primary key,
  total int
);

create table lineitems (
  invoice_id int not null references invoices(id),
  item_id int not null references items(id),
  quantity int not null check (quantity > 0),
  primary key (invoice_id, item_id)
);

-- example data:
insert into items (price) values (5);
insert into items (price) values (9);
insert into invoices (total) values (0);

download code

Now, you want it to re-calculate the Invoice total whenever Lineitems are changed. That’s a special function called a trigger. First you create the function, then create a trigger that executes the function.

-- re-calculate the total of a lineitem's invoice
create function recalc() returns trigger as $$
begin
  -- update invoice using lineitems's invoice_id
  update invoices set total = (
    select sum(quantity * price)
    from lineitems
    join items on lineitems.item_id = items.id
    where invoice_id = new.invoice_id)
  where id = new.invoice_id;
  return new;
end;
$$ language plpgsql;
-- run this function after any change to lineitems
create trigger recalc
  after insert or update or delete on lineitems
  for each row execute procedure recalc();

download code

Test it by adding Lineitems, and updating their quantities. After each change, look at the Invoice. It works!

insert into lineitems (invoice_id, item_id, quantity) values (1, 1, 1);
insert into lineitems (invoice_id, item_id, quantity) values (1, 2, 1);
select * from lineitems; select * from invoices;

select 'Notice new total when you update quantity:' look;
update lineitems set quantity = 5 where invoice_id = 1 and item_id = 2;
select * from lineitems; select * from invoices;

select 'But when you delete a line? Oh no. Total is still $50' look;
delete from lineitems where invoice_id = 1 and item_id = 2;
select * from lineitems; select * from invoices;

-- Let's try again in recalc-4.sql
drop function recalc() cascade;

download code

But notice if you delete a Lineitem, it doesn’t update the Invoice! What the hell? Why not?

Ah, that’s because the function refers to a record variable called “new”, passed by the trigger on Lineitems, and uses it to get the invoice_id. But when when you delete a Lineitem, the record variable is called “old”, since it refers to a Lineitem that’s already been deleted. Damn.

You could make two separate triggers, one that uses “new”, called only on update and insert operations, and one that uses “old”, called only on deletes. But then you’d be duplicating the calculation of the total. So instead, just add one ugly if/then to assign either “new” or “old” to a variable called “r”, and use “r” instead. Here’s the updated function:

-- re-calculate the total of a lineitem's invoice
create function recalc() returns trigger as $$
declare
  r record;
begin
  -- use "new" lineitems record for insert/update, or "old" if delete
  if (tg_op = 'DELETE') then
    r = old;
  else
    r = new;
  end if;
  -- update invoice using lineitems(now "r")'s invoice_id
  update invoices set total = (
    select sum(quantity * price)
    from lineitems
    join items on lineitems.item_id = items.id
    where invoice_id = r.invoice_id)
  where id = r.invoice_id;
  -- must return incoming "new" or "old" record when done
  return r;
end;
$$ language plpgsql;
-- run this function after any change to lineitems
create trigger recalc
  after insert or update or delete on lineitems
  for each row execute procedure recalc();

download code

Now test adding, updating, and deleting, and notice they all update the Invoice total:

insert into lineitems (invoice_id, item_id, quantity) values (1, 2, 1);
select * from lineitems; select * from invoices;

select 'Notice new total when you update quantity:' look;
update lineitems set quantity = 5 where invoice_id = 1 and item_id = 2;
select * from lineitems; select * from invoices;

select 'Notice new total when you delete:' look;
delete from lineitems where invoice_id = 1 and item_id = 2;
select * from lineitems; select * from invoices;

download code

Cool? You get much more security and peace-of-mind, knowing that no matter what code updates your database, the related fields will be updated automatically.

Download the final example file here: /code/recalc.sql.

After you get this example working, let’s try another example of putting data logic in the database, by making functions for updating the cart.

Also look at using triggers to clean incoming data.

PostgreSQL example of self-contained stored procedures

2019-10-25T04:00:00Z

First, see my previous article about PostgreSQL functions at sive.rs/pg. That article gave tiny examples, but no finished working code.

This week, I wrote a shopping cart to sell my books directly from my own site.

So I took a couple extra hours today to put my code into public view, so anyone can play around with it. See github.com/sivers/store, to browse, download, and try it.

It’s a working self-contained shopping cart / store. It’s a very concrete example of using stored procedures to keep all the data logic together in one place. You can use it from JavaScript, Python, Ruby, or any language you want, since all the functionality is in the database itself. It works.

If you have any questions, or want to tell me how stupid I am for doing this, email me.

Digital pollution

2019-10-15T04:00:00Z

You couldn’t just roll down the street leaving huge piles of garbage everywhere you go, making life slower for everyone as they climb over your mountains of junk, just to get on with their life. You’d feel bad about it, right?

That’s how I feel about the digital things we put out into the world: websites, apps, and files.

I prefer coding everything by hand, because I don’t like the huge piles of garbage that the automated generators create. These programs that generate a website, app, or file for you spit out thousands of lines of unnecessary junk when really only 10 lines are needed. Then people wonder why their site is so slow, and they think it’s their phone or connection’s fault.

Yesterday I needed to make a little vector logo. Two lines and two triangles. I tried to use a couple different vector drawing programs but they saved it as hundreds of lines. I knew it could be simpler, so I read up on SVG and made exactly what I wanted:


<svg height="54" width="54">
<defs><style type="text/css"><![CDATA[line,polygon{stroke:black;stroke-width:4} polygon{fill:black}]]></style></defs>
<line x1="2" y1="2" x2="2" y2="52" />
<line x1="52" y1="2" x2="52" y2="52" />
<polygon points="2,2 27,27 2,27" />
<polygon points="52,2 27,27 52,27" />
</svg>

Much better! 95% smaller file size, and the joy of making something by hand instead of having it done for me. But I think my biggest joy is eliminating the digital pollution that the auto-generated one created. It makes everything faster, easier, and cleaner for anyone involved. 95% less junk over the wires.

Same thing with the EPUB file for my new book. Today I spent the day creating the EPUB’s XML and XHTML by hand, instead of using a generator. I love the manual control and again - 90% smaller file size.

This makes me unreasonably happy. It feels like cleaning up the neighborhood. Or at least my yard.

(And I love it when people notice how fast my site loads.)

Where we do and don’t want automation

2019-10-10T04:00:00Z

I used to use Gmail. But one day, as I typed my mother’s email address into the “To:” field, Google popped up a prompt asking if I also wanted to CC my uncle. That was so invasive and creepy that I deleted the account immediately and never used it again. I don’t want automated intelligence in my private email.

My friend lives in a home full of the smartest technology, and loves getting all the new smart things, but he drives a deliberately retro old car with no computer chips. He loves to tinker with his car, and wants to do any maintenance himself.

Another friend lives in a tech-free rustic cabin with no screens, but drives a Tesla.

I do everything on a broken old Linux laptop, using only the command line, usually offline, nothing in the cloud. I think it’s because I don’t want any outside automation or intelligence in the work that matters to me.

When software is described as “auto-”, “smart”, or “intelligent” it means that somebody else put their rules into it. But I don’t want my computer to do anything I didn’t explicitly tell it to do. It shouldn’t change what I typed unless I tell it to. It should never guess or predict what I want. I want full manual control.

On the other hand, I don’t mind if my phone does these things, because I don’t care about my phone much. And I would love a high-tech car, full of smart AI automation, because I’m not a car aficionado.

At first I thought that an expert at something won’t want assistance. But no, of course, auto-pilot for airline pilots, and IDEs for programmers.

So I think it comes down to:

how much of an expert you are at controlling this thing yourself
how much you still enjoy doing it
if you want the kind of assistance it provides

Any thoughts on this subject? I’d love to hear another point of view.

Have a private email account

2019-09-28T04:00:00Z

I used to like the internet. I thought it was cool, creative, wild, untamed, expressive, decentralized, and educational. I guess it was, back then, but now? I kinda hate most of what’s out there.

I don’t like social media, either. Staged photos, and people trying to get a reaction. Noise, hype, and drama. It makes me want to avoid the internet completely.

Then I thought about what I do like.

My email inbox is really nice. It’s only people I like, who are emailing me personally. No lists.

I highly recommend setting up a private email address.

Use Fastmail, Posteo, Mailbox, or any similar service where you actually pay $1-$3 per month and in return get a completely ad-free spam-free wonderful email experience.

Then never give this new private email to anyone except dear friends and family.

Let your old Gmail collect the junk. The people you really care about will use the new one, so you won’t need to check the old one much anymore.

It feels nice to have a notification mean something again — to only get one or two emails a day, and know that they are really for you. Or, if you don’t get any notifications, then nothing you really care about has arrived — so no need to check.

Most days I don’t look at the web or any apps. I just write, text friends, call friends, and check email. That’s enough.

OpenBSD : why and how

2018-04-20T04:00:00Z

The only operating system I use on my computers is not Mac, not Windows, and not even Linux. It’s OpenBSD, and I love it so much. So I figured I should say a little something about why, and how you can try it.

It’s probably not for you.

It’s not for beginners. Most people should use Ubuntu.

It’s not for people who want to click a button and have the computer hide the details from you.

If software bloat doesn’t bother you — if every new Mac/Windows/Linux release you say, “Bring on the features! The more the better!” — it’s not for you.

But if you’re experienced, like to “look under the hood”, and prefer software that does the minimum necessary, OpenBSD is for you.

What is it?

It’s like Linux, but has different goals.

It’s known for its focus on security. But, like a well-engineered house will also be earthquake-proof, you don’t have to be paranoid about earthquakes to appreciate great construction. To me, the security features are just a side-effect of great coding.

OpenBSD comes with a secure minimal firewall, webserver, mailserver, and an optional graphical desktop. So if all you want is a few of those things, you do the default install, tweak one config file, and you’re done.

Why OpenBSD instead of Linux?

It’s uncompromising. It’s not a people-pleaser or vendor-pleaser. Linux is in everything from Android phones to massive supercomputers, so has to include features for all of them. The OpenBSD developers say no to most things. Instead of trying to make it do more, they keep it focused on doing what it does with more security and reliability.

They review and remove code as often as they add. If something is unused, unmaintained, or unnecessary, they’ll axe it. If it’s unwieldy, they’ll make a small simple replacement. For examples, see doas, OpenSMTPD, httpd, and LibreSSL. This is great for security, too. The more code, the more chance of a bug that could compromise your entire computer. The less code, the better. Each new release seems to be getting leaner by removing old cruft. No other operating system does that.

Great documentation is a top priority. The built-in man pages are amazing. So if you’re stuck on anything, searching the man pages on your own computer is going to give you a better answer than searching Google. (This makes it nicer to work offline, too.)

The installers are amazing. The initial installation takes like five minutes. Hit [Enter] to the defaults, make your username and password, and it’s ready to go. Then the software installer is ideal, too. Just pkg_info to search for something and pkg_add to install it in seconds. (Which also installs all of its documentation, too.)

Everything is rock-solid and just works. Hardware I couldn’t get working in Linux just works on a first try with OpenBSD. And because they don’t stay cutting-edge, keeping a cautious pace, it keeps working and doesn’t break. The whole system is carefully planned and consistent, instead of a hodge-podge of bits and pieces.

It’s all free and run by helpful volunteers. If you searched ports, but some application you need is missing or out of date, just contact the maintainer and offer some assistance or money to help get it updated or added. I’ve donated $3850 to the developers to help improve the OpenBSD port of Node.js, Elixir, Erlang, Anki, Ledger, and Qutebrowser.

Also see Solene’s list of selling points. I agree with those.

Use the internet, not just companies

2018-02-12T05:00:00Z

I’ve been online since 1994, and seen so many companies come and go.

In the year 2000, the place to be was mp3.com. Every musician would keep all of their music and fans there. A few years later, it was gone — shut down — all music and fan lists deleted.

In 2005, it was MySpace. Again, musicians kept all of their music, photos, and fans there. A few years later, it was gone. Not shut down, but basically moot. There was no way to communicate with all of those people, because you didn’t have their direct contact info — you only had their MySpace inbox, which nobody checked anymore.

As I’m writing this now in 2018, it’s Facebook, YouTube, and Spotify. Just like with mp3.com and MySpace, people act like these websites are everything, and keep all of their music, photos, and fans there. By the time you read this, they might be gone.

Don’t depend on a company. They come and go. Think long-term. You’re going to be creating stuff, making fans, and building relationships for the rest of your life — much longer than these companies will last.

So have your own website. Instead of sending your fans to some company’s site, send them to yours. Get everyone’s direct contact information so you don’t have to go through a company to reach them.

Your website should be the definitive place to get everything you create. If you put your stuff on some company’s site, have it be secondary — a copy of the stuff that’s already on your site. That way you can use the popular networks without depending on them.

Only rely on open standards that aren’t owned by any company — like email and the web.

Email skills:

Go into your email settings, and make sure you have a signature. You need this because you’re going to be emailing people who have no idea who or where you are! Give them some context. Your signature should say who, what, and where, with a URL or two. For example:

--
Maya Danubé, fragrant jazz bass clarinet, New York City
http://mayadanube.com  me@mayadanube.com  (917)611-5310
Watch + listen: https://www.youtube.com/user/mayadanube
Friend me, baby: https://www.facebook.com/mayadanube

When you email people, write a descriptive subject. Never “hey” or “booking”. Try “Available June 6 for showcase?” or “introduction to photographer”. This is considerate. Now when your email is one of hundreds in an inbox, it will say exactly what is contained inside.

Make it as short as possible. The shorter your email, the more likely it will get a response. Be direct. Five sentences is ideal. If your email is too long, they are likely to procrastinate, and never get back to it.

Use short paragraphs. Leave plenty of space. Reading a screen is different from reading a book.

Web skills:

Know how to update your website. Don’t depend on someone else to do this for you. Know how to add new songs or videos, and how to make any changes.

Know your URLs. Telling someone to go search for you is like telling them to look up your phone number. Instead, know your exact URLs (yoursite.com, twitter.com/something, facebook.com/whatever) so you can give it to people directly. If you don’t, they’ll probably never bother to go search for you.

Know how to make an MP3. Give it a good filename like YOUR_NAME-Song_Title.mp3 (not mix7.mp3) Don’t use spaces in the filename. Edit the ID3 tags to put your full name and URL in the info, so whoever has this MP3 knows who it is and how to find you.

Sorry if these sound too basic to you. But you’d be surprised by how many people don’t know these skills, and so are silently handicapped when interacting with the world.

Should you learn programming? Yes.

2016-02-14T05:00:00Z

When I was 14 years old, my guitar teacher told me something important:

“You need to learn to sing. Because if you don’t, you’re always going to be at the mercy of some asshole singer.”

His point was about self-reliance. If you ever have any ideas, whether for songs or apps, you need to have some basic skills to turn those ideas into reality.

One of the most common things I hear from aspiring entrepreneurs is, “I have this idea for an app or site. But I’m not technical, so I need to find someone who can make it for me.”

I point them to my advice about how to hire a programmer, but most programmers are already busy and expensive.

Imagine if someone said, “I have this idea for a song. But I’m not musical, so I need to find someone who will write, perform, and record it for me.”

You’d probably advise them to just learn enough guitar or piano so they can play their song.

It’s like learning to drive or make dinner. You only need to learn enough so you’re not helpless.

So, yes, you should learn some programming. Basic HTML, CSS, and JavaScript are enough to start.

I recommend Head First HTML and CSS first, then Head First HTML5 Programming. Those are very fun, visual books where you need to see the detailed illustrations, so get the paper book or PDF.

If you prefer a course and community, use Free Code Camp. Ideally, do those books and this course at the same time, to really reinforce what you’re learning.

You could go through those books or courses in a few weeks, and you’d already know as much as half of the people that call themselves web developers.

It’s a really amazing feeling. The mystery is lifted. You’ll look at all websites in a new way. You’ll understand what’s going on behind the scenes. You’ll know how to do it yourself. It’s really empowering. (It’s definitely been the most rewarding thing I’ve ever learned.)

For an inspiring example see Jennifer Dewalt, who learned to code by building 180 websites in 180 days, with no previous experience.

Next, read my advice on learning JavaScript.

Simplify: move code into database functions

2015-05-04T04:00:00Z

If you are a web or API developer, programming code that uses an SQL database, this is for you.

I’ve found a very different and useful way to structure code. It’s made such a big difference for me that I had to share it here.

How things are

Most web development — whether custom or using frameworks like Rails, Django, Laravel, Sinatra, Flask, and Symfony — tends to work the same way:

At the core is a database, which is just the storage of data.
All intelligence is in Ruby/Python/PHP/JavaScript classes.

Why that’s bad

These norms have some dangerous implications:

Everything must go through these Ruby/Python/PHP/JavaScript classes — including shell scripts and other things not part of this application.
Nothing else may access the database directly, since doing so may break the rules defined by these surrounding classes.
The database is treated as dumb storage, even though the database is smart enough to have most of this logic built-in.
But if you add data logic into the database itself, it’s now duplicated, requiring changing in multiple places if the rules change.
These two systems — the database and its surrounding code — are coupled and dependent on each other.
If it’s ever advantageous to switch applications (say from a web app to mobile app, or Python to JavaScript), you’re going to have to re-write all of that data logic.

Simple vs complex

Please go watch this amazing 35-minute talk as soon as possible: Simplicity Matters by Rich Hickey.

Here are his important points for this article:

“Complex” is objective. It means many things tied together.
“Simple” is objective. It means one ingredient — the opposite of complex.
These are unrelated to “easy”. It is easy to install and bind yourself to something very complex (like ORM), and can be hard to build something simple.
Classes, models, and methods (OOP) are an unnecessary complication.
Information is simple, so don’t hide it behind a micro-language.
Work with values directly: hash/map of strings.
Since a JSON API — a hash/map of strings — is often the eventual interface, it’s even more reason to skip the abstractions and work with values directly.

Why this hit home for me

Databases outlive the applications that access them.

I’ve been using the same SQL database since 1997: same data, values, and SQL tables. But the code around it has changed so many times.

In 1997, I started in Perl. In 1998, I switched to PHP. In 2004, a rewrite in Rails. In 2007, back to PHP. In 2009, minimalist Ruby. In 2012, client-side JavaScript.

Each time I’d have to re-write all of the logic around the database: how to add a new person into the database, how to verify an invoice is correct, how to mark an order as paid, etc.

But that whole time, my trusty PostgreSQL database stayed the same.

Since most of this is data logic, not business logic, it should be in the database.

So I’m putting this data logic directly into PostgreSQL, since I plan to stay with it for many more years, but plan to keep experimenting with programming languages. (Nim, Elixir, Racket, Lua, whatever.)

How things could be

Web developers have been treating the database as dumb storage, but it’s actually quite smart.

It’s simple to have all of this intelligence in the database itself.

It’s complex to have it tied to surrounding outside code.

Once you put all of the intelligence directly into the database, then the outside code disappears!

Then the database is self-contained, and not tied to anything.

Your outside interface can switch to JavaScript, Haskell, Elixir or anything else with ease, because your core intelligence is all inside the database.

How to do it

Table constraints

The easiest place to start is constraints:


create table people (
  id serial primary key,
  name text not null constraint no_name check (length(name) > 0),
  email text unique constraint valid_email check (email ~ '\A\S+@\S+\.\S+\Z')
);
create table tags (
  person_id integer not null references people(id) on delete cascade,
  tag varchar(16) constraint tag_format check (tag ~ '\A[a-z0-9._-]+\Z')
);

Define what is considered valid/invalid data here.

In my people example above, it says name can’t be empty, email must match that pattern with “@” and “.” and no whitespace. Then it says tags.person_id has to exist in the people table, but if the person is deleted then delete the tags, too. And the tag has to fit that regexp pattern of lowercase letters, numbers, dot, underscore, dash.

It helps to name your constraints for later use in error catching.

Triggers

For things that happen before or after you alter data, use triggers:


create function clean() returns trigger as $$
begin
  new.name = btrim(regexp_replace(new.name, '\s+', ' ', 'g'));
  new.email = lower(regexp_replace(new.email, '\s', '', 'g'));
end;
$$ language plpgsql;
create trigger clean before insert or update of name, email on people
  for each row execute procedure clean();

This example cleans the input before it’s put into the database, in case someone accidentally put a space in their email address, or a line-break in their name.

Functions

Make little re-usable functions for things you’ll use often inside your code.


create function get_person(a_name text, a_email text) returns setof people as $$
begin
  if not exists (select 1 from people where email = a_email) then
    return query insert into people (name, email)
      values (a_name, a_email) returning people.*;
  else
    return query select * from people where email = a_email;
  end if;
end;
$$ language plpgsql;

That’s one I use often: Given someone’s name and email, if they’re not already in my database, add them. Then, either way, return the database info for this person.

Views for JSON

Instead of requiring outside code to convert your data into JSON, you can have the database create JSON directly.

For this, use views as JSON structure templates. Inside the view, use json_agg for nested values.


create view person_view as
  select *, (select json_agg(t) as tags from
    (select tag from tags where person_id=people.id) t)
  from people;

This will be used in the API functions, below:

API functions

These are the only functions your outside code will access.

They return only JSON.


create function update_password(p_id integer, nu_pass text, out js json) as $$
begin
  update people set password=crypt(nu_pass, gen_salt('bf', 8)) where id = p_id;
  js := row_to_json(r) from (select * from person_view where id = p_id) r;
end;
$$ language plpgsql;
create function people_with_tag(a_tag text, out js json) as $$
begin
  js := json_agg(r) from
    (select * from person_view where id in
      (select person_id from tags where tag = a_tag)) r;
end;
$$ language plpgsql;

No matter what you need to do with your database, the procedural languages built-in to PostgreSQL can do it.

PL/pgSQL is not the most beautiful language, but the simplicity of having everything in the database is worth it.

If you like JavaScript, check out the promising plv8.

Now, if you need a REST API:


require 'pg'
require 'sinatra'
DB = PG::Connection.new(dbconfig)
def qry(sql, params=[])
  @res = DB.exec_params('select js from ' + sql, params)
end
after do
  content_type 'application/json'
  body @res[0]['js']
end
get '/people' do
  qry('get_people()')
end
get %r{/people/([0-9]+)} do |id|
  qry('get_person($1)', [id])
end
put %r{/people/([0-9]+)} do |id|
  qry('update_password($1, $2)', [id, params[:password]])
end
get '/people/tagged' do
  qry('people_with_tag($1)', [params[:tag]])
end

Or if you need a client library:


require 'pg'
require 'json'
DB = PG::Connection.new(dbconfig)
def js(func, params=[])
  res = DB.exec_params('select js from ' + func, params)
  JSON.parse(res[0]['js'])
end
def people
  js('get_people()')
end
def person(id)
  js('get_person($1)', [id])
end
def update_password(id, newpass)
  js('update_password($1, $2)', [id, newpass])
end
def people_tagged(tag)
  js('people_with_tag($1)', [tag])
end

That’s it!

Now whether a REST API or client library, all it really has to do is pass the arguments into the database functions, and return the JSON.

I’m not trying to convince everyone to do things this way. But I hope you find it useful or at least interesting to consider.

Memorizing a programming language using spaced repetition software

2013-01-06T05:00:00Z

I’ve been doing this for a year, and it’s the most helpful learning technique I’ve found in 14 years of computer programming.

Background:

I’m an intermediate programmer. I didn’t go to school for it. I just learned by necessity because I started a website that just kept growing and growing, and I couldn’t afford to hire a programmer, so I picked up a few books on PHP, SQL, Linux, and Apache, learned just enough to make it work, then used that little knowledge for years.

But later, when I worked along side a real programmer, I was blown away by his vocabulary! All of these commands and functions just flowing effortlessly out of his fingers. We were using the same language, but he had memorized so much of it, that I felt like a child next to a university professor. I really wanted to get that kind of fluency.

It made me think about how much I’ve learned then immediately forgotten, over the years. I read books or articles about some useful feature, try it once, but then I get distracted, forget about it, and go about my normal way of doing things.

I wanted to deeply memorize the commands and techniques of the language, and not forget them, so that they stay at the forefront of my mind whenever I need them.

Spaced Repetition:

When you hear a new fact, it’s forgotten pretty quickly unless it’s brought back to the forefront of your mind repeatedly.

You can do this haphazardly by immersing yourself in a language, for example, where the new words you learn will be brought up by chance occasionally.

But memory research shows that the most effective and efficient time for a new fact to be remembered is right before you were about to forget it.

Say if you learn a new word in a foreign language, you’d want to practice it again a few minutes after hearing it, then a few hours, then the next day, then in 2 days, then 5 days, then 10 days, 3 weeks, 6 weeks, 3 months, 8 months, etc. After a while it’s basically permanently memorized with a rare reminder.

Spaced Repetition Software does this for you, so you can just give it a bunch of facts you want to remember, then have it quiz you once a day, and it manages the intervals based on your feedback. After each quiz question, if you say that one was easy, it won’t be introduced for a long time, but if you were stumped, then it’ll ask it again in a few minutes, until you’ve got it.

Go to apps.ankiweb.net and download Anki. It’s a free, open source, popular spaced repetition software program.

As for programming, you get where I’m going with this.

What if you could memorize everything about the programming language of your choice? Every command, every parameter, every function. Every solution to hundreds of the most common problems, all completely memorized at your fingertips? Imagine going through the documentation or a book, and permanently remembering every single thing in it?

Enough of the intro, let’s get to the HOW-TO:

First, learn!

Flash cards are for remembering what you’ve learned.

Before you create a flash card for something, you need to actually learn and understand it. Create the flash card after you’ve really understood.

(This is why it’s not that helpful to look at someone else’s deck. Those are just reminders.)

Convert Knowledge into Small Facts:

You’re going to be making a bunch of flash cards. Question on the front. Answer on the back.

If you were just using this to memorize foreign language vocabulary, then the formatting would be easy. The front would have a word or phrase, and the back would have its translation, and vice-versa.

But if you’re learning anything else, you’re going to have to put a little craft and creativity into making your own flash cards.

It takes some effort to read through paragraphs of stuff you want to remember, pick out the key facts, break them down into their smallest form, and turn them into questions for quizzing your future self.

Here are my best time-saving tips from a year of doing this:

Turn prose into code

If you’re reading a tutorial about programming, and come across a paragraph describing a feature.

“The add (+) operator... if only one operand is a string, the other operand is converted to a string and the result is the concatenation of the two strings.”

You test it out yourself, play around with it, and understand it. So you make a flashcard to remember it.

  var a = 5 + '5';
  // what is a?

  '55'
  If either side of + is a string, the other is
  converted to a string before adding like strings.

Try to trick your future self

Sometimes you learn a “gotcha” — a common mistake or surprising feature.

“If the new Array constructor is passed a single number, it creates an empty Array with a length of that number. Any other combination of arguments creates an Array of those arguments.”

You test it out yourself, play around with it, and understand it. Then make two flash cards to try to trick your future self.

  var a = new Array('5');
  // what is a?

  An array with one item, the string '5': ['5'];

... and then an almost-identical question ...

  var a = new Array(5);
  // what is a?

  An empty array with a length of 5.

When the program quizzes you, it will shuffle the cards, so that hopefully your examples will intentionally catch you by surprise.

You can also try to trick yourself with more complicated examples, to keep these gotchas fresh in your mind:

  var a = [20, 10, 5, 1];
  // what is a.sort()?

  [1, 10, 20, 5]
  // sort treats all values as strings

Don’t forget to quiz yourself on the solution, too:

  var a = [20, 10, 5, 1];
  // sort these in numeric order

  function compare(v1, v2) { return(v1 — v2); }
  a.sort(compare);

Save the cool tricks

If you find a cool trick you want to remember, turn it into the answer of a small challenge.

  var albums = [
    {name: 'Beatles', title: 'White Album', price: 15},
    {name: 'Zeppelin', title: 'II', price: 7}];
  // make this work:
  albums.sort(by('name'));
  albums.sort(by('title'));
  albums.sort(by('price'));

  function by(propName) {
    return function(obj1, obj2) {
      v1 = obj1[propName];
      v2 = obj2[propName];
      if (v1 < v2) { return -1; }
      else if (v1 > v2) { return 1; }
      else { return 0; }
    };
  }

Make the answer require multiple solutions

If there’s more than one way of doing something, and you want to remember both, make your future self come up with more than one solution, so you can keep both alternatives in mind.

  s = 'string like this'
  # In Ruby, show two ways to turn it into 'String Like This'

  s.split.map(&:capitalize).join(' ')
  s.gsub(/\b\S/) {|x| x.upcase}

Turn broad concepts into succinct examples

Say you just spent 20 minutes learning something that’s more conceptual, and not as much about remembering specific functions. Sometimes all you need is one succinct example to remind yourself of the concept.

  /(a(b)((c)d))/.match('abcd')
  # What will $1, $2, $3, $4 be?

  $1 = 'abcd'
  $2 = 'b'
  $3 = 'cd'
  $4 = 'c'

Another example:

  class C
    self
  end
  class D < C
  end
  d1 = D.new
  # which object is self?

  class D

Read “20 Rules of Formulating Knowledge” by Piotr Wozniak

The best advice on this stuff is an article called “20 Rules of Formulating Knowledge” by Piotr Wozniak. So please read that one.

Run Through it Daily

For most efficient results, turn on your spaced repetition software once a day. If you go too long without, you’ll screw up all the timings, and have to re-learn stuff you would have remembered.

You can remember thousands of these facts in only 20 minutes a day. I just make it a morning routine. Make a cup of boiling tea. Do my Anki. Drink my tea.

It’s fun when quizzing yourself to add a little adrenaline, and make yourself go as fast as you can.

It’s like a mental visit to the gym. A little intense 20 minutes a day is so worth it for the immediate and long-term results.

Add some new cards whenever you can, and you’ll be amazed that everything you saved stays fresh in your mind.

Conclusion

I’ve been doing this for a year, and it’s been a HUGE boost to my fluency. I highly recommend it, as you can tell.

Feel free to email me anytime to let me know how it goes for you.

The “Janki Method Refined” by Jack Kinsella is a great article on similar approach,and explains it even better than I did here.

You can also use Anki to learn all kinds of things.

“Augmenting Long-term Memory” by Michael Nielsen is an amazing article about using Anki for learning anything.

Read the interview with Piotr Wozniak at wired.com for a story about a guy taking this to the limit.

How to send personalized emails

2009-08-31T04:00:00Z

People often ask what I use to send out my personalized emails.

It’s a short little PHP script, run on the command-line, directly on your webserver. The script and simple instructions are below.

1. Keep your list in a spreadsheet

It’s important that you do it with email address in column A and name in column B like this:

2. Save as tab-delimited text

Under “Save As...”, every spreadsheet program has an option to save as plain text, with the columns separated with a “tab” character. For this example, name it list.txt. Upload it to your server.

3. Here’s the PHP script that does it

<?php
$from = "Willy Wonka <willy@wonka.com>";
foreach(file(’list.txt’) as $line)
    {
    list($email, $name) = explode("\t", $line);
    list($firstname) = explode(’ ’, $name);
    $subject = "Hi $firstname! The chocolate is all yours.";
    $body = "Hi $firstname — 
Those other kids are awful. You deserve it all. Come and get it.
--
Willy Wonka — willy@wonka.com — http://wonka.com
";
    mail($email, $subject, $body, $from);
    print "$email sent\n";
    }
?>

Look at it slowly, even if you don’t know PHP. It’s pretty self-explanatory.

It goes through each line of your list.txt file
Breaks each line into $email and $name, separated by tab (\t)
Breaks the name into words, taking the first one as $firstname
Merges $firstname into the subject and body of the mail
Emails it, using the customized $email, $subject and $body

Obviously, replace the $from, $subject, and $body for your own needs. Save it as mailer.php. Then upload it to your server.

4. Log in to the command-line on your server

This is the part your web-hosting company will have to tell you how to do. Using Terminal on Mac, or PuTTY on Windows, SSH into your server’s command-line.

5. Run it!

On the command-line, where the mailer.php and list.txt files are, just type…

php mailer.php

…and everyone will be sent a customized email.

A computer that would require kids to be creative to use it

2009-07-26T04:00:00Z

Talking with a parent yesterday about kids’ use of computers. She felt that her kids were still too passive with computers, just turning it on and playing games, surfing, or chatting — but not really creating or learning.

I had an idea for a computer that would require kids to be creative to use it:

What if there was a computer that did nothing by default, but had all the building blocks to make it do anything, with a little effort?

Say you turn it on, and there are just some puzzle pieces on the screen, and some “HOW TO” instructions.

Want to browse the web? You need to connect the internet-connection block, HTML parsing block, and user interface block. Bundle them up, type a command to name it anything you want, give it an icon, and now you have a web browser program on your desktop forever... and you made it!

Want to chat with friends? You need to connect the internet-connection block, message-sending block, message-receiving block, smiley-face-making block, and user interface block. Bundle it up, type a command to name it anything you want, give it an icon, and now you have a chat program.

Maybe you could go back and add more blocks to your chat program, like a video-camera interface or file-sending interface.

All of this could be as easy as dragging puzzle pieces together, but each piece could have options for customizing, though the best options would only be available by editing the (easy) code. This would get kids used to the idea of programming, and seeing real results.

My first computer was a Radio Shack TRS-80 in 1978. When you powered it up, it just gave you a black screen and a command prompt. It was up to you to learn how to type commands to make it do what you wanted. That sense of control and creativity would be useful again for kids.

7 reasons I switched back to PHP after 2 years on Rails

2007-09-23T04:00:00Z

This was originally posted on O’Reilly in September 2007. Re-posting it here for a permanent archive.

SUMMARY: I spent two years trying to make Rails do something it wasn’t meant to do, then realized my old abandoned language would do just fine if approached with my new Rails-gained wisdom.

BACKGROUND:

Back in January 2005, I announced that I was going to completely scrap over 100,000 lines of messy PHP code in my existing CD Baby (cdbaby.com) website, and rewrite the entire thing in Rails, from scratch.

I hired one of the best Rails programmers in the world (Jeremy Kemper), and we set off on this huge task with intensity. The first few months showed good progress, and Jeremy could not have been more amazing, twisting the deep inner guts of Rails to make it do things it was never intended to do.

But at every step, it seemed our needs clashed with Rails’ preferences. Like trying to turn a train into a boat. It’s possible with a lot of glue. But it’s damn hard, and makes you ask why you’re really doing this.

Two years later, after various setbacks, we were less than halfway done. (To be fair to Jeremy’s expertise: many setbacks were because of tech emergencies that pulled our attention to other internal projects that were not the rewrite itself.) The entire music distribution world had changed, and we were still working on the same goddamn rewrite. I said fuckit, and we abandoned the Rails rewrite. Jeremy took a job with 37 Signals, and that was that.

I didn’t abandon the rewrite idea, though. I just asked myself one important question:

“Is there anything Rails can do, that PHP can’t do?”

The answer is no.

I threw away 2 years of Rails code, and opened a new empty project.

Then in a mere two months, by myself, not even telling anyone I was doing this, using nothing but vim, and no frameworks, I rewrote CD Baby from scratch in PHP. Done! Launched! And it works amazingly well.

It’s the most beautiful PHP I’ve ever written, all wonderfully MVC and DRY, and I owe it all to Rails.

Inspired by Rails:

all logic is coming from the models, one per database table, like Martin Fowler’s Active Record pattern.
no requires or includes needed, thanks to __autoload
real MVC separation: controllers have no HTML or business-logic, and only use REST-approved HTTP.
all HTML coming from a cute and powerful templating system I whipped up in 80 lines, all multi-lingual and caching and everything
… and much more. In only 12,000 lines of code, including HTML templates. (Down from 90,000, before.)

Though I’m not saying other people should do what I’ve done, I thought I should share my reasons and lessons-learned, here:

7 reasons I switched back to PHP after 2 years on Rails:

#1 — “Is there anything Rails/Ruby can do that PHP can’t do? No.”

For 2 years, I thought Rails is genius, PHP is shit. Rails is powerful, PHP is crap. I was nearly killing my company in the name of blindly insisting Rails was the answer to all questions, timeframes be damned. But when I took a real emotionless non-prejudiced look at it, I realized the language didn’t matter that much. Ruby is prettier. Rails has nice shortcuts. But no big shortcuts I can’t code-up myself in a day if needed. Looked at from a real practical point of view, I could do anything in PHP, and there were many business reasons to do so.

#2 — Our entire company’s stuff was in PHP. Don’t underestimate integration.

By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change. 85 employees re-trained. All customers and clients calling up furious that One Big Day, with questions about the new system. Instead, I was able to slowly gut the ugly PHP and replace it with beautiful PHP. Launch in stages. No big re-training.

#3 — I don’t want what i don’t need

I admire the hell out of the Rails core gang that actually understand every line inside Rails itself. But I don’t. And I’m sure I will never use 90% of it. With my little self-made system, every line is only what’s absolutely necessary. That makes me extremely happy and comfortable.

#4 — It’s small and fast

One little 2U server is serving up a ton of cdbaby.com traffic damn fast with hardly any load.

#5 — It’s built to my tastes

I don’t need to adapt my ways to Rails. I tell PHP exactly what I want to do, the way I want to do it, and it doesn’t complain. I was having to hack-up Rails with all kinds of plugins and mods to get it to be the multi-lingual integration to our existing 95-table database. My new code was made just for me. The most efficient possible code to work with our exact needs.

#6 — I love SQL

Speaking of tastes: one tiny but important thing: I love SQL. I dream in queries. I think in tables. I was always fighting against Rails and its migrations hiding my beloved SQL from me.

#7 — Programming languages are like girlfriends. The new one is better because you are better

Rails was an amazing teacher. I loved it’s “do exactly as I say” paint-by-numbers framework that taught me some great guidelines. I love Ruby for making me really understand OOP. God, Ruby is so beautiful. I love you, Ruby. But the main reason that any programmer learning any new language thinks the new language is so much better than the old one is because he’s a better programmer now! You look back at your old ugly PHP code, compared to your new beautiful Ruby code, and think, “God that PHP is ugly!” But don’t forget you wrote that PHP years ago and are unfairly discriminating against it now. It’s not the language (entirely). It’s you. You’re better now. Give yourself some credit.

Ok. All that being said, I’m looking forward to using Rails some day when I start a brand new project from scratch, with Rails in mind from the beginning.

But I hope that this reaches someone somewhere thinking, “God our old code is ugly. If we only threw it all away and did it all over in Rails, it’d be so much easier!”