Weblog on marginalia.nu

An NSFW filter for Marginalia Search

Mon, 30 Mar 2026 00:00:00 +0000

… optional, that is.

I’ve been working on an NSFW filter for Marginalia Search, as that is something some people have asked for, primarily API consumers.

The search engine has had some domain based filtering for a while, based on the UT1 lists, but that isn’t a very comprehensive approach.

We’ll land on a single hidden layer neural network approach, implemented from scratch, but before landing on that, many other things were tried along the way.

AI makes you boring

Thu, 19 Feb 2026 00:00:00 +0000

This post is an elaboration on a comment I made on Hacker News recently, on a blog post that showed an increase in volume and decline in quality among the “Show HN” submissons.

I don't actually mind AI-aided development, a tool is a tool and should be used if you find it useful, but I think the vibe coded Show HN projects are overall pretty boring. They generally don't have a lot of work put into them, and as a result, the author (pilot?) hasn't generally thought too much about the problem space, and so there isn't really much of a discussion to be had.
The cool part about pre-AI show HN is you got to talk to someone who had thought about a problem for way longer than you had. It was a real opportunity to learn something new, to get an entirely different perspective.

Index Compression, Query Execution Improvements

Fri, 13 Feb 2026 00:00:00 +0000

The Marginalia Search index has recently seen some design tweaks to make it perform better, primarily the introduction of postings list compression.

Last year, the index was partially re-implemented with SSDs in mind. This was largely a success, but left some lingering issues with tail latencies that sometimes weren’t what they needed to be.

To ensure predictable execution times, the query execution is provided a timeout value, after which it will wrap up and return the best results it’s found. Query execution was so flaky that the actual timeout used when terminating the execution used to be something like 50ms lower than the provided value. This is obviously not a fantastic state of affairs.

Trust in Ranking

Sat, 31 Jan 2026 00:00:00 +0000

The Marginalia Search default ranking algorithm recently saw a fairly radical improvement, due to a new domain trust system that drastically reduces the number of content farm results, as long as there are human results it usually finds them across all the usual test queries.

Recently fixing a few bugs that made the search engine work more correctly had the unexpected and undesired side-effect of also making it surface more search engine spam and content farm-type results.

You should probably tell your audience what your blog posts are about as early as possible

Wed, 28 Jan 2026 00:00:00 +0000

Being clear about what your blog posts are about lets people who are interested in what you have to say find your writing more easily. The more paragraphs you spend getting to the point, the bigger the odds they’ll lose patience and click on something else before you’ve presented your thesis.

When publishing articles online, no matter how obscure the subject matter, there is almost always some people who will be into what you have to say. The main thing that keeps them from finding your writing is that they need to discover that it exists. Discovery isn’t made a lick easier by vague titles and long rambling introductions.

New Search Filtering in Web and API

Mon, 08 Dec 2025 00:00:00 +0000

The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI.

This was originally going to be an incredibly chaotic update, both annuncing the new features and doing a technical walkthrough of the changes but that ambition turned out a bit too chaotic, so let’s split them up and focus on the feature announcement bit today.

New Search Filtering GUI

It’s now possible to define a custom filter in the GUI, on the marginalia-search.com version of the website!

Language Support for Marginalia Search

Mon, 06 Oct 2025 00:00:00 +0000

One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish.

These changes are now live for testing, but with an extremely small corpus of documents.

As the search engine has been up to this point built with English in mind, some anglo-centric assumptions made it into its code. A lot of the research on search engines generally seems to embed similar assumptions.

The CoPilot productivity paradox

Sat, 06 Sep 2025 00:00:00 +0000

I’ve been using the CoPilot plugin for IntelliJ on and off for the last few years, and while initially pretty enthusiastic, I’ve come to first disable it and then delete it altogether along with JetBrains’ local AI-completions, and generally felt this has been an improvement in productivity and a reduction of frustration.

CoPilot is pretty good at taking things that are already pretty fast, such as monotonous code transformations like mapping an object to a SQL statement, and then making that even faster.

Snark, Ironic Detachment, Authenticity

Thu, 28 Aug 2025 00:00:00 +0000

How you engage with the world changes how you experience the world, and how the world experiences you.

A snarky and cynical approach, by its default assumption that things are shit, or if they are not yet shit will inevitably turn to; such an approach will give your world a malodorous brownish tint.

Granted, snark gives you plausible deniability, a motte-and-bailey that protects you from direct criticism, encountering backlash you can always backpedal and say it was just a joke that you accidentally took a bit too far.

Faster Index I/O with NVMe SSDs

Sat, 16 Aug 2025 00:00:00 +0000

The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes.

The index is already fairly large, but can sometimes feel smaller than it is, and paradoxically, query performance is a big part of why. If each query has a budget of 100-250ms, a design that finds and ranks results faster in that time period will produce better search results. There are other limitations as well, query understanding is still somewhat limited, where only minor changes to a query can unearth dozens of new related results.

Finding Dead Websites

Tue, 17 Jun 2025 00:00:00 +0000

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change.

This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking.

Profiling Websites

Thu, 29 May 2025 00:00:00 +0000

The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements.

The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code.

It turns out to be somewhat difficult to determine what a website that has non-trivial javascript will look like based its source code alone, as this would require us to among other things solve the halting problem.

A 2030 morning routine

Fri, 23 May 2025 00:00:00 +0000

You wake at 05:30 in the morning, feeling somewhat groggy.

Instead of the alarm clock ringing like it normally does, a cheerful hologram appears: “Hi! I’m Kyle, your new alarm clock assistant!” You get dressed as Kyle explains all of the fantastic things he is capable of.

You head over to the coffee machine. “Hey there! I’m Evan! Are you ready for AI in your coffee? But first - tell me about yourself!”. You ignore Evan’s monologue and close your eyes as the synthetic coffee replacement is brewing. Real coffee costs more than your coffee maker nowadays, so it has to suffice.

PDF to Text, a challenging problem

Tue, 13 May 2025 00:00:00 +0000

The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months.

Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format.

It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”. These glyphs may be rotated, overlap, and appear out of order, with very little semantic information attached to them.

Debugging A Crawler Stall

Tue, 22 Apr 2025 00:00:00 +0000

Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client. This seemed like a good idea at the time, but has led to a fair number of headaches.

Java’s HttpClient has one damning flaw, and that that it doesn’t support socket timeouts.

Its only supported timeout values are time to connect, and time until first byte of the response. This means the client can get stuck on a read call if a server stops responding, potentially for a very long time!

Crawl Order and Disorder

Thu, 27 Mar 2025 00:00:00 +0000

A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains.

This has been actualized recently, since the migration to slop crawl data has dropped memory requirements of the crawler by something like 80%, and as such I’ve been able to increase the number of crawling tasks, which has led to a bizarre case where 99.9% of the crawling is done in 4 days, and the remaining 0.1% takes a week.

Marginalia Search receives second nlnet grant

Tue, 25 Mar 2025 00:00:00 +0000

I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant.

All the details are not yet finalized, but tentatively the grant will go toward addressing most of the items in the project roadmap for 2025.

I’ve already been working full time on the project since summer 2023, and this grant secures additional development time, and extends the runway to a comfortable degree.

Will post more details as they are finalized.

Improved ways to operate a rude crawler

Sat, 22 Mar 2025 00:00:00 +0000

This text is satirical in nature.

Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt. In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup is the one.

Marginalia Search: 4 Years

Mon, 03 Mar 2025 00:00:00 +0000

This update is a few days late, the canonical birth date of the project is Feb 26.

It has been another year of Marginalia Search. The project is still ongoing, still my full time job, although the project is entering a somewhat more mature phase of development, most of the big pieces are in place and do a decent job at what they do.

The roadmap for the project is available on GitHub.

RSS Feeds and Real Time Crawling

Thu, 26 Dec 2024 00:00:00 +0000

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed.

The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any given point in time. To help compensate for this, a new crawler and index partition has been added that focuses on recently updated content.

Notes on binary soup

Tue, 05 Nov 2024 00:00:00 +0000

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before.

This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

Phrase Matching in Marginalia Search

Mon, 30 Sep 2024 00:00:00 +0000

Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query.

This is a write-up about implementing this change. This is going to be a relatively long post, as it represents about 4 months of work.

I’m also happy and grateful to announce that the nlnet people reached out after the run of the grant was over and asked me if I had more work in the pipe, and agreed to fund this change as well!

The sorry state of Java deserialization

Sun, 22 Sep 2024 00:00:00 +0000

I’ve been on a bit of a frustration-driven quest to solve a problem I frequently encounter working on the search engine, that is, reading data from disk.

You’d think this would be a pretty basic thing, but doing this in a way that is half-way performant is surprisingly hard and requires avoiding basically all the high level tools at your disposal.

There’s a common sentiment that modern hardware is fast, so this may not matter, but we aren’t speaking a 30% performance hit, the the question is how many orders of magnitude you’re willing to forego.

Less Coffee, Better Sleep

Wed, 31 Jul 2024 00:00:00 +0000

As an experiment, I’ve reduced my coffee-intake to a single cup a day for about a week now. It’s made an enormous difference in sleep, mood and energy. I get tired at night, fall asleep quickly, and wake up refreshed.

As mentioned previously in the context of morning sunlight exposure—another thing that’s aided my sleeping habits, but is somewhat less practical to sustain as it requires fair weather—I’ve always been slow to get going in the morning, active at night, bad at getting to bed at sane hours. Tired when I should be awake, and awake when I should be tired.

One year of solo dev, wrapping up the grant-funded work

Tue, 18 Jun 2024 00:00:00 +0000

A year ago I walked out of the office for the last time. I handed in my corpo laptop, said some good-byes, and since then I have been my own boss.

This first year has been funded by an NLnet grant, which I’m in the midst of wrapping up. As of now, the work is all done, the final request for payment has been sent.

There’s a similar last-day-of-school levity to both these events.

Feynman's Garden

Sun, 26 May 2024 00:00:00 +0000

The best description of my problem solving process is the Feynman algorithm, which is sometimes presented as a joke where the hidden subtext is “be smart”, but I disagree. The “algorithm” is a surprisingly lucid description of how thinking works in the context of hard problems where the answer can’t simply be looked up or trivially broken down, iterated upon in a bottom-up fashion, or approached with similar methods.

Feynman’s thinking algorithm is described like this:

Experiment in Java native calls

Thu, 16 May 2024 00:00:00 +0000

I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search.

The Marginalia Search engine is mainly written in Java, which is language that is good at many things, but not particularly pleasant to work with when it comes to low level systems programming.

Unfortunately, a part of building an internet search engine involves database-adjacent low level programming.

Using DuckDB to seamlessly query a large parquet file over HTTP

Sun, 05 May 2024 00:00:00 +0000

A neat property of the parquet file format is that it’s designed with block I/O in mind, so that when you are interested in only parts of the contents of a file, it’s possible to some extent to only read that data. Many tools are aware of this property, and DuckDB is one of them. Depending on which circles you run in, a lesser known aspect of HTTP is range requests, where you specify which bytes in a file to be retrieved. It’s possible to combine this trio of properties to read remote parquet files directly in DuckDB.

Query Parsing and Understanding

Wed, 17 Apr 2024 00:00:00 +0000

Been working on improving Marginalia Search query parsing and understanding. This is going to be a pretty long update, as it’s a few months’ work.

Apart from cleaning up the somewhat messy query parsing code, a problem I’m trying to address is that the search engine is currently only good at dealing with fairly focused queries, they don’t need to be short, but if you try to qualify a search that is too broad by adding more terms, it often doesn’t produce anything useful.

Deep Bug

Wed, 10 Apr 2024 00:00:00 +0000

The project has been haunted by a mysterious bug since sometime February. It relates to the code that constructs the index, particularly the code that merges partial indices.

In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement.

You can conceptualize the revese index itself as two files, one with offset pointers into another file, which has sorted numbers. This code runs after each partition finishes crawling and processing its data, and has a run time of about 4 hours.

The Yak Shave

Wed, 28 Feb 2024 00:00:00 +0000

I set out a little over a week ago to add a service registry to Marginalia Search, primarily to reduce its dependence on docker. I would like it to be able to run on bare metal as well, which poses a problem since configuring the application manually is a bit of a headache with dozens of ports that need to be set up. It would also be desirable to be able to run multiple instances of important services in order elliminate downtime during upgrades.

Marginalia: 3 Years

Sun, 25 Feb 2024 00:00:00 +0000

It’s been three years since the inception of Marginalia Search, then a dinky experiment to find where the heck the cool Internet has gone, now my full time job.

While there’s always things that can be improved, it’s fair to say the search engine has never worked as well as it does right now.

A great number of milestones have been reached, perhaps biggest of all the search engine has moved out of my living room and into a proper enterprise server.

Best SEO spam 2024 reddit

Wed, 07 Feb 2024 00:00:00 +0000

One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time.

I’ve been noticing the search engine spam strategy of adding ‘reddit’ to page titles for a few years now, but it feels like it’s been growing a lot recently. I don’t think it’s actually working, but it’s so cute that they are trying.

Contexts, Friction and Distractions

Tue, 30 Jan 2024 00:00:00 +0000

I get significantly more work done when I unplug my computer from the Internet. It’s not that my productive output drops to zero when I’m plugged in, but more like 70%.

Despite many of the tools that I use requiring a connection, and certainly the Internet containing a wealth of information that might expedite my work, these benefits are drastically outweighed by the wealth of distractions also available.

It’s very appealing, when the code is compiling or the docker containers restarting, to sneak open a browser tab with hacker news, or the Χ formerly known as Twitter, youtube, mastodon, a news site, or something similar to pass those minutes.

Charlatans spreading misleading beginner advice are the evolutionary crabs of youtube content creators

Mon, 29 Jan 2024 00:00:00 +0000

You have a hobby you’ve been into for a decade or more. You like talking about your hobby, and your friends and family, after listening to these things for as long as you’ve been into them, maybe aren’t as excited to always hear about it as you are about discussing them, so in an act of compassion you create a youtube channel where you can monologue about your passion instead.

A Hobby Coding Biography

Thu, 28 Dec 2023 00:00:00 +0000

This is a bit of a retrospective of every project I’ve worked on, as far as I remember them. I’ve tried to unearth any artifacts that remain.

Far from everything is flattering and resounding success, but then again, maybe that’s good. There are definitely patterns in the things that didn’t pan out.

Earliest Traces

I was definitely programming stuff, but I don’t think it ever amounted to anything tangible. It was more like playing house, I built GUIs that looked like real applications but didn’t really do anything but look cool to me. Real hackertyper vibes. This was in Java, Delphi and VB script mostly. I also built a bunch of websites on geocities and angelfire. This is definitely as far back as 1999. I did some QBASIC stuff even earlier, like basic text based games and drawing circles and stuff.

A Frivolous Feature

Fri, 22 Dec 2023 00:00:00 +0000

Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS. At a glance this seems like a somewhat frivolous feature, but it has interesting effects.

Autonomous Systems are part of the Internet’s routing infrastructure. If your mental model of an IP number is that they are the phone number of the computer, this is something akin to a postal code. Digging much deeper than that into BGP and autonomous systems is not really in the scope of this article, but Wikipedia has a relatively lucid article on this.

How To Read An Article On The Internet

Wed, 20 Dec 2023 00:00:00 +0000

 A simple guide to reading
 in 9 simple steps

Navigate to the desired article.
Dismiss the GDPR banner
It may seem safe to start reading, but you need to wait about 10 seconds as the various ad auctions resolve and scripts load in
Wait while the article is populated with ads. While the article is in front of you, there is no point to starting to read yet, as the minute’s worth of layout shift will make you lose your place.
Once the layout has settled and not shifted for 10 seconds, move the mouse cursor toward the URL bar to trigger any sort of exit intent scripts you might otherwise accidentally invoke while reading.
Scroll to the end of the article to ensure a paywall doesn’t appear halfway through the article.
Dismiss any calls to subscribe to a newsletter that may have appeared in step 6.
Press Ctrl+F and search for out-of-place paragraphs beginning with “Benefits of”; this is a strong indicator the article is some ChatGPT content farm gibberish.
Press Ctrl+P to bring up the print preview dialog and read the article in print preview

Some also say you should press the back button once to get any back button hijackings out of the way, but your mileage vay vary.

WARC'in the crawler

Wed, 20 Dec 2023 00:00:00 +0000

The Marginalia Crawler has seen improvements! A long term problem with the crawler design is that if for whatever reason the crawler shuts down, then it needs to re-start fetching whatever domains it was currently traversing during the termination from zero.

This isn’t fantastic, since not only does crawling a website take a fair bit of time, it’s a nuisance for the server admins to re-crawl stuff that was already fetched, and a real liability for ending up in robots.txt or some iptables ruleset.

Anchor Tags

Tue, 07 Nov 2023 00:00:00 +0000

I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage.

The problem I’m attempting to address is that many websites don’t really describe themselves particularly well. As Steve Ballmer’s stage performance once illustrated, merely repeating a word doesn’t on its own make what you’re saying relevant to the term.

Another good example of how it falls short is PuTTY’s website, which will be used as a pilot case to improve.

Partitioning The Index

Mon, 30 Oct 2023 00:00:00 +0000

So a bit of an update on what I’ve been working on. This will be adapted into release notes in a while, but I haven’t quite wrapped a bow on the change set yet.

Still, it has certainly been a few weeks. Didn’t quite land how busy I’ve been until I set down to draft this post. Them’s some changes, and I’m skipping a few to keep this meandering post at a sane length.

The List

Thu, 19 Oct 2023 00:00:00 +0000

As a general observation, I tend to be more productive when I know what to do next at any given moment.

There are days when I’ve seemingly gotten a “week” of work done on an afternoon, those are the days when what I needed to do was very clear, and I basically just had a list of items to tick off one by one.

There have admittedly also those ignoble weeks weeks when I’ve gotten an afternoon’s work done, mostly they are weeks when it’s not been at all clear what to do next.

Moving Marginalia to a New Server

Sat, 07 Oct 2023 00:00:00 +0000

So the search engine is moving to a new server soon, thanks to the generous grant mentioned recently.

If you visit search.marginalia.nu now, it may or may not use the old or new server. It’ll be like this for a while, since I need them both for testing and maintenance type work.

I’ll also apologize if this post is a bit chaotic. It is a reflection of a very chaotic couple of weeks that apart from setting up this migration also involved a very short notice invitation for a presentation at ossym23.

A Disk Usage Mystery

Sat, 23 Sep 2023 00:00:00 +0000

I ran into a bit of a puzzling situation yesterday, testing some of the new index construction changes before they’re going live in a few days.

The process crashed with a pretty non-descript stack trace complaining about illegal instructions, so first glance it looked more like it was within the realm of freak JVM bug, cosmic ray, hardware error maybe.

I was doing this on my developer workstation, which also spawned a popup complaining that the hard drive it was working on had nearly run out of space, and inferred that the error probably was due to memory mapping more space onto a disk than what was possible.

Marginalia Search receives FUTO Grant

Fri, 15 Sep 2023 00:00:00 +0000

I’m happy to announce that the generous people at FUTO have granted the project $15,000 with no strings attached to help the search engine out with some more server power.

FUTO is a young Austin, TX-based organization “dedicated to developing, both through in-house engineering and investment, technologies that frustrate centralization and industry consolidation”. It’s one to keep an eye on, I believe their heart is in the right place and they have every possibility of making a real difference.

Absurd Success

Wed, 30 Aug 2023 00:00:00 +0000

So… I’ve had the most unreal week of coding. Zero exaggeration, I’ve halved the RAM requirements of the search engine, removed the need to take the system offline during an upgrade, removed hard limits on how many documents can be indexed, and quadrupled soft limits on how many keywords can be in the corpus.

It’s been a long term goal to keep it possible to run and operate the system on low-powered hardware, and so far improvements have been made, to the point where my 32 Gb RAM developer machine feels spacey rather than cramped, but this set of changes takes it several notches further.

Sleeping at Night

Tue, 22 Aug 2023 00:00:00 +0000

I’ve started going on a long walk each morning immediately as I wake up, and it’s had the unexpected side-effect of fixing my broken circadian rhythm.

For decades, as long as I can remember, I’ve been what you might consider a serious night owl. Regardless of how long I slept or when I woke up, I would get nothing requiring any sort of thought done until sometime after lunch, and it wasn’t really until late at night that my brain really kicked into gear.

Life in 1080p

Sat, 12 Aug 2023 00:00:00 +0000

I recently got a smaller computer screen. It’s actually not that small, it’s 27", but the resolution is modest compared to what is available. And in short, it’s fantastic. It’s not an expensive screen, it’s not a fancy screen; but it’s comparatively a small screen.

For a few years I was using a 34" ultra-wide monitor, which has been causing me nothing but grief. It’s sort of crept up on me that so many small annoyances in my computer-use all originated from using this screen. I want to elaborate on what’s been chafing. I want to be explicit that these are my experiences as someone who primarily uses a computer screen for programming. If you’re using your computer for something else, your experiences may differ.

Message Queues, State Machines, Actors, UI

Sat, 12 Aug 2023 00:00:00 +0000

This is a bit of an what I’ve been working on style of post. It’s also a bit of a complement for the release notes of the upcoming release which should be dropping in a week or so. There’s some spit and polish still missing from these things, but if I don’t write about them now too much will have been ejected from the cache to make a well written post about it.

Full Time

Fri, 16 Jun 2023 00:00:00 +0000

I’m working on Marginalia Search full time.

I left the office for the last time today, and it’s the strangest feeling. I’ve quit jobs, taken time off work, been laid off, but this is different from any of those things. This is deliberate.

There’s a note of relief. I’ve essentially been working two pretty demanding jobs; one for pay and one for passion and the joy of making a difference.

Killing Community

Sun, 11 Jun 2023 00:00:00 +0000

This is a theory that’s previously been stated in log/39-normie-hypothesis.gmi, but I think it’s worth expanding on as it’s become very relevant with the recent Reddit shit-show actualizing just how bad that website has gotten along with social media in general.

I think the model demonstrates how the ’enshittification’ process is an inevitability with any social media that is run on a venture capital model.

An online community can be like a village, where you have familiar faces, collective experiences, shared values and so forth. It can be like a village and be five people, it can be like a village and be a thousand people.

Confessions

Fri, 19 May 2023 00:00:00 +0000

I use print debugging all the time

I know how to use a debugger. I use a debugger sometimes, but most of my debugging is done by print statements that are like

A
B
C
.
.
5
D
.
D
,
{true, 30}
.
,
.
,
10
E

I think Clean Code makes some valid points

I don’t think it should be your bible or treated as infallable, having seen the sort of code that came before it, yeah, Uncle Bob got some things right. He rightfully gets some shit for the stuff that didn’t turn out perfectly well as a product of the time, but at the same time, I think we’ve sort of become blind to how much he got right.

R.I.P. Memex

Mon, 08 May 2023 10:49:28 +0200

I killed the old memex.marginalia.nu site. Not because it wasn’t great, but because I don’t have the time to maintain the software, which was quite janky, and perhaps most of all I wasn’t really feeling it.

The new site looks superficially similar, but it’s actually just a Hugo template that emulates some of the memex’ capabilities. Although some of the coolest stuff is sadly gone as a result.

Thankfully I decided to use an extremely portable markup format when building the original memex software, which meant that porting it over to hugo was literally just a matter of writing a brief python script.

An off-ramp from the digital IKEA maze

Thu, 04 May 2023 00:00:00 +0000

There is an episode of Star Trek where a character is for plot reasons trapped in a shrinking parallel universe. As time passes, people she knows one by one just vanish and she is the only one who seems to notice. Eventually it gets to an absurd point. She asks if it really makes sense if a ship made for a thousand people would have a crew of a few people, and everyone just sort of like shrugs and looks at her like she’s crazy. That’s basically what the last decade of the Internet[1]. It feels like it’s shrinking. Like parts of it are vanishing.

Thoughts on AI and AI-veganism

Fri, 21 Apr 2023 00:00:00 +0000

I’ve come to think LLMs/GPTs/whatever are a threat to conventional search engines because the modern web is an unbelievably annoying dumpster fire.

They don’t really provide better or faster answers, what they provide is an experience that is not a complete pain in the ass.

This frog has been simmering for a long while now and we’re so used to it that seeing literally anything else seems revolutionary.

You visit a website and need to dismiss a cookie policy notification, a request to show popups, a request to know your location, an invitation to subscribe to a newsletter, a sales rep wants to have a chat with you, then you get random layout shifts for several minutes as all the ad auctions finish, and then just as you’re ready to read the content the website crashes and reloads and the circus starts over again. You try to leave, and it hijacks your back button into a redirect loop.

Going Github

Sat, 25 Mar 2023 00:00:00 +0000

I’ve moved Marginalia’s sources to Github. Can’t pick every battle.

The main reason is I’m kind of tired of the amount of spam bots that keep signing up to my Gitea. The juice of self-hosting a public-access git forge, even locked down to prevent arbitrary repo creation, that juice just isn’t worth the squeeze.

This is not without some consideration.

To be blunt, I don’t like Github. Their use of dark patterns leaves a real nasty after-taste. I’m also old enough to remember the Microsoft of the early 2000s very vividly.

Search Result Quality For Multiple Terms

Thu, 23 Mar 2023 00:00:00 +0000

This is a bit of a follow up to the previous post.

The Grand Code Restructuring [ 2023-03-17 ]

Marginalia’s search result quality has, for a long while, been pretty good as long as your search query is a single term, but for multiple search terms it’s been a bit hit-and-miss. Marginalia was never great at this, but the quality of results in this usage pattern has taken a bit of a dive recently due to a re-write of the index last fall.

The Grand Code Restructuring

Fri, 17 Mar 2023 00:00:00 +0000

In general I don’t like to fuss over code, but this is exactly what I’ve been doing in preparation of the NLnet funded work. I’ve spent the last month restructuring Marginalia’s code base. It’s not completely done, but I’ve made great headway.

Things got the way they got because in general for experimental solo-development projects, I think it makes sense to be fairly tolerant of technical debt.

Since refactoring is something that is extremely difficult to break up into parallel tracks or do in small iterations, the cost of refactoring is effectively multiplied by the number of people that could be working on the code.

Marginalia Search: 2 years, big news

Sun, 26 Feb 2023 00:00:00 +0000

No time like the project’s two year anniversary to drop this particular bomb…

Marginalia’s gotten an NLNet grant. This means I’ll be able to work full time on this project at least a year.

https://nlnet.nl/project/Marginalia/

This grant is essentially the best-case scenario for funding this project. It’ll be able to remain independent, open-source, and non-profit.

I won’t start in earnest for a few months as I’ve got loose ends to tie up before I can devote that sort of time. More details to come, but I’ll say as much as the first step is a tidying up of the sources and a move off my self-hosted git instance to an external git host yet to be decided.

A new approach to domain ranking

Mon, 06 Feb 2023 00:00:00 +0000

This is a very brief post announcing a fascinating discovery.

It appears to be possible to use the cosine similarity approach powering explore2.marginalia.nu as a substitute for the link graph in an eigenvector-based ranking algorithm (i.e. PageRank).

The original PageRank algorithm can be conceptualized as a simulation of where a random visitor would end up if they randomly clicked links on websites. With this model in mind, the modification replaces the link-clicking with using explore2 for navigation.

Are you ok?

Fri, 27 Jan 2023 00:00:00 +0000

I don’t know if I’m just imagining it, but has the Internet gone progressively more crazy the last decade or so?

It’s like everyone is so damn angry all the time. If they aren’t angry they’re bitter and resentful. And when they aren’t angry or bitter, they’re so depressed they’re barely able to crawl out of bed. And if they aren’t angry, bitter, or depressed, they have crippling anxiety. Every other week there’s some public blow-out where some person or another just loses their shit.

Memex Design

Fri, 13 Jan 2023 00:00:00 +0000

For clarification, this is discussing no other thing called Memex than memex.marginalia.nu, the website you’re probably visiting right now. That, or you’re reading this over gemini at marginalia.nu, which is serving the same content over a different protocol.

I wanted to build a cross-protocol static site generator designed in a way that is equally understandable by both humans and machines. This groundedness is an appealing property I really admire about the gemini protocol and gemtext format. It’s something I want to explore if it’s possible to extend to software in general.

Faster Index Joins

Tue, 03 Jan 2023 00:00:00 +0000

The most common (and most costly) operation of the marginalia search engine’s index is something like given a set of documents containing one keyword, find each documents containing another keyword.

The naive approach is to just iterate over each document identifier in the first set and do a membership test in the b-tree containing the second. This is an O(m log n)-operation, which on paper is pretty fast.

It turns out it can be made faster.

Creepy Website Similarity

Mon, 26 Dec 2022 00:00:00 +0000

This is a write-up about an experiment from a few months ago, in how to find websites that are similar to each other. Website similarity is useful for many things, including discovering new websites to crawl, as well as suggesting similar websites in the Marginalia Search random exploration mode.

A link to a slapdash interface for exploring the experimental data.

The approach chosen was to use the link graph look for websites that are linked to from the same websites. This turned out to work remarkably well.

On Wizards and Sorcerers

Fri, 23 Dec 2022 00:00:00 +0000

While this post is about programming, it also draws an extended analogy to dungeons and dragons, specifically two of its classes, that correspond to two attitudes toward programming.

In D&D, wizards study magic. They prepare their magic spells ahead of time. While they may learn a large number of magic spells, they need to prepare them ahead of time and can’t just cast them at will.

Wizard programmers prefer up-front design. They apply reason and logic to divide and conquer a large problem, they rely on building blocks like design patterns and algorithms. Wizards rely on explicit knowledge.

The best ideas come AFK

Mon, 07 Nov 2022 00:00:00 +0000

I get my best ideas when I’m not working.

This seems paradoxical, but past a point, the more I work on a project the slower it seems to go. I’ll find changes to do, but lose any sort of vision.

If I’m not programming at all, I rarely get good ideas as well.

There appears to be some magic stoichiometric mixture where I work on a project for a while, then force myself to take a break somewhere far away from any keyword for a day or two, the ideas start to roll in at a pace where I can barely keep up to write them down.

Carbon Dating HTML

Thu, 27 Oct 2022 00:00:00 +0000

One of the more common feature requests I’ve gotten for Marginalia Search is the ability to search by date. I’ve been a bit reluctant because this has the smell of a a surprisingly hard problem. Or rather, a surprisingly large number of easy problems.

The initial hurdle we’ll encounter is that among structured data, pubDate in available in RDFa, OpenGraph, JSON+LD, and Microdata.

A few examples:

<meta property="datePublished" content="2022-08-24" />
<meta itemprop="datePublished" content="2022-08-24" />
<meta property="article:published_time" content="2022-08-24T14:39:14Z" />
<script type="application/ld+json">
{"datePublished":"2022-08-24T14:39:14Z"}
</script>

So far not so that bad. This is at least a case where the web site tells you that here is the pub-date, the exact format of the date may vary, but this is solvable.

Scaling doesn't scale

Tue, 25 Oct 2022 00:00:00 +0000

By which I mean there are deeply problematic assumptions in the very notion of scaling: Scaling changes the rules, and scaling problems exist in both directions. If what you are doing effortlessly scales up, it almost always means it’s egregiously sub-optimal given your present needs.

These assertions are all very abstract. I’ll illustrate with several examples, to try and build an intuition for scaling. You most likely already know what I’m saying is true, but you may need reminding that this is how it works.

Marginalia's Index Reaches 100,000,000 Documents

Fri, 21 Oct 2022 00:00:00 +0000

A very brief note to announce reaching a long term goal and major milestone for marginalia search.

The search engine now indexes 106,857,244 documents!

The previous record was a bit south of seventy million. A hundred million has been a pie-in-the-sky goal for a very long time. It’s seemed borderline impossible to index a that many documents on a PC. Turns out it’s not. It’s more than possible.

Twice this may even be technically doable, but is way past the pain point of sheer logistics. It’s already a real headache to deal with this much data.

The Evolution of Marginalia's crawling

Tue, 23 Aug 2022 00:00:00 +0000

In the primordial days of Marginalia Search, it used a dynamic approach to crawling the Internet.

It ran a number of crawler threads, 32 or 64 or some such, that fetched jobs from a director service, that grabbed them straight out of the URL database, these jobs were batches of 100 or so documents that needed to be crawled.

Crawling was not planned ahead of time, but rather decided through a combination of how much of a website had been visited, and the quality score of that website determined where to go next. It also promoted crawling websites adjacent to high quality websites.

Marginaliacoin, and hidden forums

Thu, 18 Aug 2022 00:00:00 +0000

I discovered someone has made a cryptocurrency called “Memex Marginalia Inu”. It appears to have been created February 23, which is around when the entry “I Have No Capslock And I Must Scream” went absurdly viral to the point where Elon Musk tweeted a link to it.

I Have No Capslock…

Mr Musk’s twitter orbit is exceptionally strange. The tweet was followed by a deluge of bizarre activity, strange emails with calls about stonk canine lunar expeditions, and apparently also a cryptocurrency land-grab of sorts. I can’t claim to understand why, but many of the emails got after the tweet were on the theme “what does this mean?”, almost as though Elon’s tweet was some sort of prophetic omen.

Botspam Apocalypse

Wed, 03 Aug 2022 00:00:00 +0000

Bots are absolutely crippling the Internet ecosystem.

The “future” in the film Terminator 2 is set in the 2020s. If you apply its predictions to the running of a website, it’s honestly very accurate.

Modern bot traffic is virtually indistinguishable from human traffic, and can pummel any self-hosted service into the ground, flood any form with comment spam, and is a chronic headache for almost any small scale web service operator.

On Prescriptive Descriptions

Thu, 14 Jul 2022 00:00:00 +0000

I’d like to discuss a mental somersault that I’ve found has caused me a lot of grief in the past, which is prescriptive descriptions. Let’s break this down a bit:

A descriptive statement is a statement about how something is.
A prescriptive statement is a statement about how something must be, a rule or a law.

If I stay up a bit late, do most of my work in the evenings, wake up tired and just sort of putter about until noon, I might describe myself as a night person because of this.

Fun with Anchor Text Keywords

Thu, 23 Jun 2022 00:00:00 +0000

Anchor texts are a very useful source of keywords for a search engine, and in an older version of the search engine, it used the text of such hyperlinks as a supplemental source for keywords, but due to a few redesigns, this feature has fallen off.

Last few days has been spent working on trying to re-implement it in a new and more powerful fashion. This has largely been enabled by a crawler re-design from a few months ago, which offers the crawled data in a lot more useful fashion and allows a lot more flexible post-processing.

marginalia.nu goes open source

Fri, 27 May 2022 00:00:00 +0000

After a bit of soul searching with regards to the future of the website, I’ve decided to open source the code for marginalia.nu, all of its services, including the search engine, encyclopedia, memex, etc.

A motivating factor is the search engine has sort of grown to a scale where it’s becoming increasingly difficult to productively work on as a personal solo project. It needs more structure. What’s kept me from open sourcing it so far has also been the need for more structure. The needs of the marginalia project, and the needs of an open source project have effectively aligned.

I don't know how to build software

Fri, 06 May 2022 00:00:00 +0000

There are a lot of ways of building software, there are many languages you could choose to build it with, many libraries to rely on, many frameworks to leverage, many architectural approaches, many platforms to choose, many paradigms of daily operations to follow.

It takes years to get in-depth experience with just one permutation of these options.

I’ve been programming for over twenty years, only half the time professionally, but that is how long I’ve been building software. I’ve built about twelve applications in my twenty years of development, of varying size and complexity.

Uncertain Future For Marginalia Search

Thu, 28 Apr 2022 00:00:00 +0000

I found myself effectively without a job on short notice.

I’m not at all worried about finding another one, I have savings, and I have experience, and I have demonstrable skill. What I am concerned about is finding a source of income that’s compatible with putting some time on my personal projects.

Last bunch of years, I’ve been working 32 hour weeks, which is a pretty sweet deal especially combined with the zero hour commute you get working from home during the pandemic. Not every employer is fine with that, and while I do have options, I’m in a worse bargaining position than I have been before.

Lexicon Architectural Rubberducking

Mon, 11 Apr 2022 00:00:00 +0000

I’m going to think out loud for a moment about a problem I’m considering.

RAM is a precious resource on any server. Look at VPS servers, and you’ll be hard pressed to find one with much more than 32 Gb. Look at leasing a dedicated server, and it’s the RAM that really drives up the price. My server has 128 Gb, and it it’s so full it needs to unbutton its pants to sit down comfortably. Anything I can offload to disk is great.

The Bargain Bin B-Tree

Thu, 07 Apr 2022 00:00:00 +0000

I’ve been working lately on a bit of an overhaul of how the search engine does indexing. How it indexes its indices. “Index” is a bit of an overloaded term here, and it’s not the first that will crop up.

Let’s start from the beginning and build up and examine the problem of searching for a number in a list of numbers. You have a long list of numbers, let’s sort them because why not.

Is There A Better Hard Drive Metaphor?

Sun, 03 Apr 2022 00:00:00 +0000

This is mostly a post to complain about something that chafes. I wish there was a programming language (ideally several) that acknowledged that computers have hard drives, not just a processor, RAM and other_devices[].

Something that has struck me when I’ve been working with the search engine is how unfinished the metaphor for accessing physical disks is in most programming languages. It feels like an after-thought, half left to the operating system to figure out, a byzantine relic of the days when computers had tape drives and not SSDs.

Growing Pains

Wed, 23 Mar 2022 00:00:00 +0000

The search engine index has grown quite considerably the last few weeks. It’s actually surpassed 50 million documents, which is quite some milestone. In February it was sitting at 27-28 million or so.

About 80% of this is side-loading all of stackoverflow and stackexchange, and part of it is additional crawling.

The crawler has to date fetched 91 million URLs, but only about a third of what is fetched actually qualifies for indexing for various reasons, some links may be dead, some may be redirects, some may just have too much javascript and cruft to qualify.

The Static File Startup

Fri, 18 Mar 2022 00:00:00 +0000

Note: This is satirical in nature. Slight CW if you are at a point in life where “Office Space” has unveiled itself as a disturbing existential horror movie. This taps into that the same darkness.

A tale of six brave Internet pioneers.

Senior Business Founder / Senior CEO – Zach Senior Tech Lead / Senior Architect / Senior CTO – Kevin Senior Backend dev Senior Frontend dev – Erin Two Senior UX engineers

A meditation on correctness in software

Mon, 14 Mar 2022 00:00:00 +0000

Let’s define a simple mathematical function, the function will perform integer factoring. It will take an integer, and return two integers, the product of which is the first integer.

F(int32 n) = (int32 A, int32 B)

so that

A*B = n

This is fairly straight forward, mathematical, objective. Let’s examine some answers an implementation might give.

F 50 = (5, 10) on ARM F 50 = (10, 5) on Intel

This seems like a bug, so let’s add the requirement that A <= B for deterministic results.

Marginalia Search: 1 year

Fri, 25 Feb 2022 00:00:00 +0000

I’ve caught some bug and don’t have the energy to write more than a brief note.

I want to commemorate the fact that work on the Marginalia search engine started one year ago. The first commit was on February 26th 2021, and contained a sketch for a website crawler and some data models.

In many ways, the paint is barely dry, yet it feels like this project has been around for a long while.

I have no capslock and I must scream

Mon, 21 Feb 2022 00:00:00 +0000

In a near future, a team of desktop computer designers are looking at the latest telemetry and updating the schematics of the hardware-as-a-service self-assembling nanohardware.

Steve: “Hmm, they don’t seem to be using the power button very often.”

Bob: “Compared to the other buttons, it’s only used 0.1% of the time”

Steve: “Remove it?”

Bob: “Remove it!”

Computers now instantly boot up when plugged into the wall, and run until the plug is pulled. No more start-up time, the cases are aesthetically cleaner, and manufacturing cost is down at least a fraction of a dollar.

Drive Failure

Sat, 19 Feb 2022 00:00:00 +0000

Not what I had intended to do this Saturday, but a hard drive failed on the server this morning, or at least so it seemed. MariaDB server went down, dmesg was full of error messages for the nvme drive it’s running off. That’s a pretty important drive.

The drive itself may actually be okay, the working hypothesis is either the drive itself or the bus overheated and reset. After a reboot the system seems fine.

The Anatomy of Search Engine Spam

Mon, 07 Feb 2022 00:00:00 +0000

Black hat SEO is endlessly fascinating phenomenon to study. This post is about some tactics they use to make their sites rank higher.

The goal of blackhat SEO is to boost the search engine ranking of a page nobody particularly wants to see, usually ePharma, escort services, online casinos, shitcoins, hotel bookings; the bermuda pentagon of shady websites.

The theory behind most modern search engines is that if you get links from a high ranking domain, then your domain gets a higher ranking as well, which increases the traffic. The reality is a little more complicated than that, but this is a sufficient mental model to understand the basic how-to.

Can we unfuck internet discoverability?

Fri, 04 Feb 2022 00:00:00 +0000

I’ve been thinking a lot about how difficult it has become to discover quality content on the Internet, not because it isn’t there, but because the signal to noise ratio is really bad, and most venues of discovery don’t seem to be able to handle it.

Recommendation algorithms seem to work almost too well, to the point where it’s all kind of just showing you things you already like, rarely anything new that you might like. It’s an absolute tragedy both for small websites and for their potential audience.

Discovery and Design Considerations

Tue, 18 Jan 2022 00:00:00 +0000

It’s been a productive several weeks. I’ve got the feature pulling updates from RSS working, as mentioned earlier.

I’ve spent the last weeks designing the search engine’s web design, and did the MEMEX too for good measure.

It needed to be done as the blog theme that previously made the foundation for the design off had several problems, including loading a bunch of unnecessary fonts, and not using the screen space of desktop browsers well at all.

Pseudonymous

Sat, 15 Jan 2022 00:00:00 +0000

A person might think I’m illusive, writing and working under a pseudonym. It’s not that I’m hiding, if you send me an email, I’ll respond to you with an email address containing a decent chunk of my real name. It’s not out of shame I wear clothes.

Besides bringing utility, marginalia.nu is an experiment, a bit of an art project, a place to challenge conventions and see what is and isn’t necessary.

Dark

Sun, 02 Jan 2022 00:00:00 +0000

As is often the case these dark winter seasons, I’ve fallen into a bit of a funk. Inspiration it seems is as rare as sunlight, and sunlight is scarce indeed in the winters of the north.

I do know what is missing, novelty. I’ve fallen into consuming “content”. Infinite scroll is the torture rack of the spirit. What is necessary doing new things and seeing new inspiring sights, exposing myself to new inspiring thoughts.

Search Result Relevance

Fri, 10 Dec 2021 00:00:00 +0000

This entry is about a few problems the search engine has been struggling with lately, and how I’ve been attempting to remedy them.

Before the article starts, I wanted to share an amusing new thing in the world of Internet spam.

For a while, people have been adding things like “reddit” to the end of their Google queries to get less blog spam. Well, guess what? The blog spammers are adding “reddit” to the end of their titles now.

Wasted Resources

Sat, 04 Dec 2021 00:00:00 +0000

At a previous job, we had a new and fancy office. The light switches were state of the art. There was an on button, and a separate off button. When you pressed the on button, the lights would fade on. When you pressed the off button, they would fade off. In the cloud somewhere was two functions that presumably looked a bit like this:

fun turnOnLamp() {
 while (!bright()) increaseBrightness();
}
fun turnOffLamp() {
 while (!dark()) decreaseBrightness();
}

I have deduced this from the fact that if you pressed both buttons at the same time, the lights would flicker on and off until someone was contacted to restart something. It is a marvellous time to be alive when you need to reboot your light switches because of a race condition. Modern computers are so fast that we often don’t even recognize when we are doing things inefficiently. We can end messages half way around the world to turn on the lights and it seems like it’s just a wire between the switch and the lamp.

A brief hypothesis about "normies"

Sat, 13 Nov 2021 00:00:00 +0000

The phenomenon of “normies” is an interesting one. The term itself is a bit problematic and not one I’d typically use, but as a phenomenon they are still worth investigating.

Their perhaps biggest distinguishing feature is that they don’t get “it”, whatever it is. It’s tempting to think that these are an especially mindless type of person with no personality and little in terms of thought going on.

I have a theory that normies may not actually exist. That is, you can’t actually show me a person that is a normie. They are a mirage.

Old and New

Fri, 12 Nov 2021 00:00:00 +0000

I’ve been thinking recently about the emphasis put on “new”, specifically for search engines, but the discussion has some merit even in a wider context. I will start wide and narrow down.

It is common to conflate new with good, and most being young sometime between 1950-2000 will indeed have seen marvellous improvements in quality of life and technology with each passing year. In the light of that, it’s at least easy to explain how one might confuse the two.

A Jaunt Through Keyword Extraction

Thu, 11 Nov 2021 00:00:00 +0000

Search results are only as good as the search engine’s ability to figure out what a page is about. Sure a keyword may appear in a page, but is it the topic of the page, or just some off-hand mention?

I didn’t really know anything about data mining or keyword extraction starting out, so I’ve had to learn on the fly. I’m just going to briefly list some of my first naive attempts at keyword extraction, just to give a context.

Localized Programming Languages

Fri, 05 Nov 2021 00:00:00 +0000

This is an reply to a series of posts on anglo-centrism in programming languages that have been floating around in Gemini lately.

gemini://nytpu.com/gemlog/2021-10-31.gmi
gemini://alsd.eu/en/2021-11-04-thoughts-anglocentrism-cs.gmi

Around thirty years ago I was a kid with a computer. I learned to program quite a few years before I learned English. I also used DOS without understanding English. I knew what to type to do things, but I didn’t know what the words meant. I could start programs, I’d play in QBASIC, write small programs and amusements. To me “PRINT” was the word that made text appear on the screen. I learned years later the word meant something in English. To show you what my child eyes saw, I think rot13 does convey the experience quite well;

Keeping Gemini Difficult

Thu, 04 Nov 2021 00:00:00 +0000

This is a response to the post “Making Gemini Easy” over on ~tomasino, and the title is a bit tongue-in-cheek haha-but-no-really.

gemini://tilde.team/~tomasino/journal/20211103-making-gemini-easy.gmi

I think the idea that we need to shield the users from how technology works is a terrible, terrible mistake. It disempowers the users, and concentrates power in the hands of a technological elite, and that divide is only going to grow.

We already have an alarming number of people working with computers, and some may even be programmers, that simply do not understand how computers work. Their only concept of a computer is the user interface on the screen. The rest is unintelligible wizardry. Nobody has told them, it’s been deemed too complicated, nothing for them to worry their little heads about.

A Polemic Against Internet Arguments

Tue, 02 Nov 2021 00:00:00 +0000

I want you to consider for a moment all the human lifetime wasted in ideological stalemates on the Internet, all that energy, all that anger and frustration. Imagine if you take even a fraction of that time, and put it to creating something constructive instead, learning skills, doing anything meaningful.

It boggles the mind, doesn’t it? It must amount to entire human lifetimes every week.

Ideology, or in a wider sense, ethics, is all about what should be done. How we should live. These aren’t statements about the world, but opinions about what the world should look like. They aren’t true or false, and any argument against them always boils down to “I disagree!”. Arguing about ethical systems is some of the least constructive things a human can do. It is more pointless than masturbation, which at least feels good for a moment.

The Parable of A Rude Guest

Thu, 28 Oct 2021 00:00:00 +0000

You are invited to a dinner party. After talking for a while food is served on the table. You pounce. “Haha, suckers!”, you think, and load all the food on your plate and leave nothing but scraps for the hosts. You feel victorious. Serves them right for inviting you into their home. You wolf down the food with ravenous appetite while they look on.

That was tasty, but now you got a piece of meat stuck between your teeth so you go to the bathroom and borrow some floss and use one the hosts’ toothbrushes. You also use the toilet but don’t flush because you don’t think you are going to use it again.

Bot Apologetics

Mon, 25 Oct 2021 00:00:00 +0000

There has been a bit of discussion over on Gemini recently regarding poorly behaved bots. I feel I need to add some perspective from the other side; as a bot operator (even though I don’t operate Gemini bots).

Writing a web spider is pretty easy on paper. You have your standards, and you can test against your own servers to make sure it behaves before you let it loose.

You probably don’t want to pound the server into silicon dust, so you add a crawl delay and parallelize the crawling, and now you have code that’s a lot harder to comprehend. This is likely the cause of some weird bot behavior, including mishandling of redirect loops or repeated visits to the same address. Multi-threaded orchestration based on a rapidly mutating data set is difficult to get right (the working set of the spider by necessity changes as it goes). You can iron a lot of this out locally, but some problems won’t crop up until you really push the limits with real-world scenarios.

Shaking N-gram needles from large haystacks

Fri, 22 Oct 2021 00:00:00 +0000

A recurring problem when searching for text is identifying which parts of the text are in some sense useful. A first order solution is to just extract every word from the text, and match documents against whether they contain those words. This works really well if you don’t have a lot of documents to search through, but as the corpus of documents grows, so does the number of matches.

It’s possible to bucket the words based on where they appear in the document, but this is not something I’m doing at the moment and not something I will implement in the foreseeable future.

Unintuitive Optimization

Wed, 13 Oct 2021 00:00:00 +0000

Optimization is arguably a lot about intuition. You have a hunch, and see if it sticks. Sure you can use profilers and instrumentation, but they are more like hunch generators than anything else.

This one wasn’t as intuitive, at least not to me, but it makes sense when you think about it.

I have an 8 Gb file of dense binary data. This data consists of 4 Kb chunks and is an unsorted list containing first an URL identifier with metadata and then a list of word identifiers. This is a sort of journal that the indexer produces during crawling. Its main benefit is that this can be done quickly with very high fault tolerance. Since it’s only ever added to, if anything does go wrong you can just truncate the bad part at the end and keep going.

The Mystery of the Ceaseless Botnet DDoS

Sun, 10 Oct 2021 00:00:00 +0000

I’ve been dealing with a botnet for the last few days, that’s been sending junk search queries at an increasingly aggressive rate. They were reasonably easy to flag and block but just kept increasing the rate until that stopped working.

Long story short, my patience ran out and put my website behind cloudflare. I didn’t want to have to do this, because it does introduce a literal man in the middle and that kinda undermines the whole point of HTTPS, but I just don’t see any way around it. I just can’t spend every waking hour playing whac-a-mole with thousands of compromised servers flooding me with 50,000 search requests an hour. That’s five-six times more than when I was on the front page of HackerNews, and the attempts only increased.

Web Browsing

Sat, 09 Oct 2021 00:00:00 +0000

An idea I’ve had for a long time with regards to navigating the web is to find a way to browse it.

“Browse” a difficult word to use, because it has a newer connotation of just using a web browser, I mean it in the old pre-Internet sense, browse like when you flip through a magazine, or peruse an antiques shop, not really looking for anything in particular just sort of seeing if anything catches your eye.

Getting with the times

Wed, 06 Oct 2021 00:00:00 +0000

Since my search engine has expanded its scope to include blogs as well as primordial text documents, I’ve done some thinking about how to keep up with newer websites that actually grow and see updates.

Otherwise, as the crawl goes on, it tends to find fewer and fewer interesting web pages, and as the interesting pages are inevitably crawled to exhaustion, accumulate an ever growing amount of junk.

Re-visiting each page and looking for new links in previously visited pages is probably off the table, that’s something I can maybe do once a month.

Experimenting with Personalized PageRank

Sat, 02 Oct 2021 00:00:00 +0000

The last few days I’ve felt like my first attempt at a ranking algorithm for the search engine was pretty good, like it was producing some pretty interesting results. It felt close to what I wanted to accomplish.

The first ranking algorithm was a simple link-counting algorithm that did some weighting to promote pages that look in a certain fashion. It did seem to keep the page quality up, but also seemed to as a strange side-effect promote very “1996”-looking websites. This isn’t quite what I wanted to accomplish, I wanted to promote new sites as well as long as they were rich in content.

Astrolabe - The October Update

Fri, 01 Oct 2021 00:00:00 +0000

https://search.marginalia.nu

The October Update is live. It introduced drastically improved topic identification and an actual ranking algorithm; and the result is interesting to say the least. What’s striking is how much it’s beginning to feel like a search engine. When it fails to find stuff, you can kinda see how.

I’ve played with it for a while now and it does seem to produce relevant results for a lot of topics. A trade down in whimsical results but a big step up if you are looking for something specific, at least within the domain of topics where there are results to find.

Thoughts on Silly Hats

Mon, 27 Sep 2021 00:00:00 +0000

If you look back in history to the turn of the 20th century, you will find a lot of people wearing hats. Women wore some arguably pretty funny over the top hats. A while earlier, over the men were arguably funny top hats. This isn’t the first time men and women wore funny clothes. The aristocrats in 1680s France looked pretty silly too.

The reason we do this, wear silly hats, is because it is fashionable. Compliance with the some perceived fashion trend is one way we compete with fellow human beings, a measuring stick we use to evaluate our standing within society. Oh, you merely wear a modest and peculiar hat? Well mine is bigger and sillier still, therefore I am better!

Re: Software and Branding

Tue, 21 Sep 2021 00:00:00 +0000

gemini://friends.riverside.camp/~clarity/journal/branding.gmi
gemini://idiomdrottning.org/re-branding

Some interesting thoughts going around on the topic of branding in software and websites. I’ve had thoughts like these too, and designed a lot of my website purposefully un-branded. I have no logos, no banners, barely navigational links. I figured I would just see what happens if you subvert this paradigm of web design, since it is stuff you typically just “scroll past” to get to what you care about like you skip the beginning of every youtube video. Since the point has never been to create a brand, I figured I would elevate the message so far above everything else that there simply is nothing else.

Against the Flood

Sun, 19 Sep 2021 00:00:00 +0000

So hacker news apparently discovered my search engine, and really took a liking to the idea. Actually that’s a bit of an understatement, the thread has gotten 3.3k points and lingered on the front page for half a week. And I wasn’t planning for it to go quite that public yet. It has quietly been online for a while, but it was only very recently it started to feel like it was really coming together. It wasn’t perfect, there was still a lot of jankiness and limitations that could have been fixed with more time. The index was half the size it should have been. Someone discovered it and shared it. It took off like a rocket, and I’m still at a loss for words at the reception it’s gotten. I have received so many encouraging comments, emails, offers of collaboration, a few have even joined the patreon. I’ve been working through all the messages and I aim to reply to them all, but it takes time. I’m very grateful for all of this, since I half thought I was alone in this.

New Solutions Creating Old Problems

Tue, 14 Sep 2021 00:00:00 +0000

I’ve spent some time the last week optimizing how the search engine identifies appropriate search results, putting far more consideration into where and how the search terms appear in the page when determining the order they are presented.

Search-result relevance is a pretty difficult problem, but I do think the changes has brought the search engine in a very good direction.

A bit simplified, I’m building tiered indices, ranging from

The Curious Case of the Dot-Com Link Farms

Thu, 09 Sep 2021 00:00:00 +0000

I spent some time today weeding out yet more link-farms from my search engine’s index.

Typically what I would do is just block the subnet assigned to the VPS provider they’re on, and that does seem to work rather well. The cloud providers that don’t police what they host is almost always home to quite a lot of this stuff, so I don’t particularly mind scorching some earth in the name of a clean index.

The Small Website Discoverability Crisis

Wed, 08 Sep 2021 00:00:00 +0000

There are a lot of small websites on the Internet: Interesting websites, beautiful websites, unique websites.

Unfortunately they are incredibly hard to find. You cannot find them on Google or Reddit, and while you can stumble onto them with my search engine, it is not in a very directed fashion.

It is an unfortunate state of affairs. Even if you do not particularly care for becoming the next big thing, it’s still discouraging to put work into a website and get next to no traffic beyond the usual bots.

Soaring High

Thu, 02 Sep 2021 00:00:00 +0000

I’m currently indexing with my search engine. This isn’t an always-on sort of an affair, but rather something I turn on and off as it tends to require at least some degree of babysitting.

I’ve also been knocked out by the side-effects of the vaccine shot I got the other day, so it’s been mostly hands-off “parenting”.

What I’m trying to figure out just how far I can take it. I really don’t know. I took some backups and just let it do its thing relatively unmonitored.

Git Isn't A Web Service

Sat, 28 Aug 2021 00:00:00 +0000

This an expansion on a comment I left on Lettuce’s gemlog post, “Personal Experiences and Opinions on Version Control Software”.

I’ve seen similar questions posed several times recently, in essence people searching for a good git provider.

The thing is you don’t need a git provider. Git is a shell command, and you can host a server yourself with almost no extra work. You can even host it off a system you don’t have administrative access to.

Cursed Motivation

Fri, 27 Aug 2021 00:00:00 +0000

A question I often see asked is one along the lines of

How do I motivate myself do (something)

… where something may be eat healthier, go to the gym, work on some project, study hard, &c.

This idea of motivation is interesting. I think it in part comes from the school system, where teachers and parents often talk about motivating the children to study, perhaps with some sort of reward system. I haven’t been able to pinpoint exactly who introduced the idea, but my hunch is based on never seeing the particular usage of the word in a book printed before the late 20th century.

Stages of Being

Mon, 23 Aug 2021 00:00:00 +0000

@sdfgeoff asked an interesting question on station just a while ago

 How much of your lives do you spend living (or watching) someone elses?

It reminded me of an interesting tool kit for understanding being.

The neoplatonists describe a hierarchy of being. This is sometimes attributed to Renaissance enfant terrible Pico della Mirandola’s oration of the dignity of man, but it’s given the most cursory mention. I’ve attached the quote at the bottom of the post. Plato’s Republic is probably a better source, even if it doesn’t draw up the hierarcy quite in this fashion.

Enter the Circle of Blame

Sun, 15 Aug 2021 00:00:00 +0000

So that IPCC report, huh. It’s provoked interesting behavior in people. I’ll skip over the minority that deny the findings, many people seem to agree that the report isn’t great news. Then they stop to look around and start pointing fingers.

In summary:

The producers blame the consumers for making the wrong purchases.
The consumers blame the producers for producing the wrong things.
The voters blame the politicians for incorrect policies.
The politicians blame the voters for expressing the wrong wants.
The right blames the market for not adapting fast enough.
The left blames capitalism for being shortsighted.
The dogs blame the cats and the cats blame the mice.

We can run around this endless circle of blame until it’s 2121, and we still won’t have gotten any closer to finding a solution. What we’re looking for is some witch to burn, someone to be really mad at for causing us grief; but what we need is to change our behavior. Don’t forget: The producers are people. The consumers are people. The voters are people. The politicians are people. The rich, the poor, the right, the left, the market, capitalism; it’s all people.

Rendered static HTML

Fri, 13 Aug 2021 00:00:00 +0000

The technological choices we make determine the rules we have to abide by.

If every page load incurs hundreds of database calls on the server, and 30 seconds of javascripting on the front-end, then obviously you need to reduce the number of page loads to a minimum. They are frustrating for the user and expensive for the server. This makes the front-end even more slow and stateful, and so the urgency for reducing page loads increases even further.

Bye, Bye, Gmail

Wed, 04 Aug 2021 00:00:00 +0000

I finally got around to moving @marginalia.nu off gmail. I’ve been planning to do so for a while, and vascillated between self-hosting and using a provider. I ended up going for the latter. Even though I do like to self-host my stuff as much as possible, email servers seem like a lot of work. And I got a VPS account included in the price, which is nice. Means I can use it to do some off-site backups without having to use dropbox or similar.

Dying, Every Day (Re: Last times)

Wed, 04 Aug 2021 00:00:00 +0000

Dece’s post “last times” made me associate to one of my favorite thoughts from Roman philosopher Seneca, who was counting his days having fallen out of favor with Emperor Nero.

In the first of his moral epistles to Lucilius, he asks:

 Quem mihi dabis [...] qui intellegat se cotidie mori?

 Who can you show me [...] that understands he is dying every day?

It’s a fascinating and useful reversal of perspective. Our final day is not the day our life is suddenly taken away from us, but the last of our allotted days. Every preceding day has already been marked by our dying. Every day is a day that will never return, every moment is sand irreversibly flowing through the hour glass.

The Astrolabe Part II: The Magic Power of Sampling Bias

Tue, 03 Aug 2021 00:00:00 +0000

As I have mentioned earlier, perhaps the biggest enemy of PageRank is the hegemony of PageRank-style algorithms. Once an algorithm like that becomes not only dominant, but known, it also creates a market for leveraging its design particulars.

Homogenous ecosystems are almost universally bad. It doesn’t really matter if it’s every computer running Windows XP, or every farmer planting genetically identical barley, what you get is extreme susceptibility to exploitation.

The System Upgrade

Fri, 30 Jul 2021 00:00:00 +0000

Early this winter, when I set up the server that would eventually become marginalia.nu, I did so in order to try out some technology I thought looked cool (proxmox, zfs), and stuff I was exposed to at work and didn’t really see the point of so as to see if we could get on better terms with if I had more control (kubernetes).

I based the system on ProxMox, a Linux based virtualization server, which ran a series of virtual machines and containers.

Whatever happened to the Memex?

Wed, 28 Jul 2021 00:00:00 +0000

I stumbled upon the Memex, which is a spiritual predecessor to hypertext technology. It was supposed to be a sort of personal data store that allows the user to link and annotate various documents in order to produce a sort of external memory, a private knowledge bank that associates ideas in a similar way a human brain does. The operator could also save and share associative trails through the information.

Local Backlinks

Mon, 26 Jul 2021 00:00:00 +0000

Maintaining links is difficult. My gemini server doesn’t have a lot of pages, but already maintaining links between relevant pages is growing more tedious by the page. It’s going to become untenable soon.

In part inspired by Antenna, I had the idea of extracting local backlinks, and automatically appending them to the pages that are linked. That way all local links are effectively bidirectional. If new a new post links to an old post, the old post automatically links to the new post. Old pages will thus over time accumulate more links to new pages without manual maintenance.

Index Optimizations

Fri, 23 Jul 2021 00:00:00 +0000

Don’t chase small optimizations

Said some smart person at some particular time, probably. If not, he ought to have; if worse comes to worst, I’m declaring it now. The cost of 2% here and 0.5% there is high, and the benefits are (by definition) low.

I have been optimizing Astrolabe, my search engine. The different kind of Search Engine Optimization. I’ve spent a lot of time recently doing soft optimization, improving the quality and relevance of search results, to great results. I’ll write about that later.

The Mind's A Field

Sun, 18 Jul 2021 00:00:00 +0000

I find if I do not regularly plant interesting thoughts in my mind, it will rarely grow them spontaneously. I need to read interesting books, or have interesting conversations; if I do not, I’ll look back at my ideas and find I haven’t really had any in a very long time. Thoughts will grow whether I take care to plant them or not, but what grows if I have been careless is weeds.

On Link Farms

Wed, 14 Jul 2021 00:00:00 +0000

I’m in the midst of rebuilding the index of my search engine to allow for better search results, and I’ve yet again found need to revisit how I handle link farms. It’s an ongoing arms race between search engines and link farmers to adjust (and circumvent) the detection algorithms. Detection and mitigation of link farms is something I’ve found I need to modify very frequently, as they are constantly evolving to look more like real websites.

Writing for Reading

Mon, 12 Jul 2021 00:00:00 +0000

I’m struck by how easy it is to read things on Gemini. Not just skimming, but actual reading, like you would a book. Not everything written is great, but it’s usually worth reading all the same. Venturing out into the land of contemporary HTML is different, and it’s not a subtle difference.

I’ve written before about the plague of inline links on Wikipedia, and this is largely a continuation of that discourse looking at other design elements.

Re: To unit test or not to unit test, that is the question

Thu, 08 Jul 2021 00:00:00 +0000

gemini://gemini.conman.org/boston/2021/07/07.1

I felt the need to add some thoughts tangentially related to this post by Sean Conner.

Why do we hold unit tests in such high regard?

Enterprise software development (Agile with a TM at the end), and to an increasing degree open source software development has really accepted the Unit Test as personal lord and savior deep within their souls. If it doesn’t have coverage, it’s bad. If it has coverage, it’s good.

The Astrolabe Part I: Lenscraft

Wed, 07 Jul 2021 00:00:00 +0000

Something you probably know, but may not have thought about a lot is that the Internet is large. It is unbelievably vast beyond any human comprehension. What you think of as “The Internet” is a tiny fraction of that vast space with its billions upon billions of websites.

We use various technologies, such as link aggregators and search engines to find our way and make sense of it all. Our choices in navigational aides also shapes the experience we have of the Internet. These convey a warped sense of what the Internet truly is. There is no way of not doing that. Since nothing can communicate the raw reality of the internet to a human mind, concessions need to be made. Some content needs to be promoted, other needs to be de-emphasized. An objective rendering is a pipe dream, even a fair random sample is a noisy incomprehensible mess.

Thoughts on the linkpocalypse

Wed, 30 Jun 2021 00:00:00 +0000

For a long while, I have been puzzled by the strangest problem: My attention span is really bad when I use a computer. I’m an avid reader of esoteric books. I have (recently) read the notoriously dry Confessions of Saint Augustine in print, it was a slog for certain, but it really doesn’t compare with the struggles I have when it comes to bringing myself to reading even a few paragraphs of text on a screen. It surely can’t be the screen itself, can it? That doesn’t seem plausible.