home

Peter Molnar

https://petermolnar.superfeedr.com/
Peter Molnar
https://petermolnar.net/feed/


Content, bloat, privacy, archives

Have you ever reached the point when you started questioning why you’re doing something? I have, but never before with my website.

What is my purpose? The unfortunate, sentient robot Rick created for the sole purpose of passing the butter.

The precursor to petermolnar.net started existing for a very simple reason: I wanted an online home and I wanted to put “interesting” things on it. It was in 1999, before chronological ordering took over the internet.1 Soon it got a blog-ish stream, then a portfolio for my photos, later tech howtos and long journal entries, but one thing was consistent for a very long time: the majority of the content was made by me.

After encountering the indieweb movement2 I started developing the idea of centralising one’s self. I wrote about it not once3 but twice4, but going through with importing bookmarks and favourites had an unexpected outcome: they heavily outweighed my original content.

Do you know what happens when your own website doesn’t have your own content? It starts feeling distant and unfamiliar. When you get here, you either leave the whole thing behind or reboot it somehow. I couldn’t imagine not having a website, so I rebooted.

I kept long journal entries; notes, for replies to other websites and for short entries; photos; and tech articles - the rest needs to continue it’s life either archived privately or forgotten for good.

Outsourcing bookmarks

The indieweb wiki entry on bookmark says5:

Why should you post bookmark posts? Good question. People seem to have reasons for doing so. (please feel free to replace this rhetorical question with actual reasoning)

Since that didn’t help, I stepped back one step further: why do I bookmark?

Usually it’s because I found them interesting and/or useful. What I ended up having was a date of bookmarking, a title, a URL, and some badly applied tags. In this form, bookmarks on my site were completely useless: I didn’t have the content that made them interesting nor a way to search them properly.

To solve the first problem, the missing content, my initial idea was to leave everything in place and pull an extract of the content to have something to search in. It didn’t go well. There’s a plethora of js;dr6 sites these days, which don’t, any more, offer a working, plain HTML output without executing JavaScript. For archival purposes, archive.org introduced an arcane file format, WARC7: it saves everything about the site, but there is no way to simply open it for view. Saving pages with crawlers including media files generated a silly amount of data on my system and soon became unsustainable.

Soon I realised I’m trying to solve a problem others worked on for years, if not decades, so I decided to look into existing bookmark managers. I tried two paid services, Pinboard8 and Pocket9 first. Pocket would be unbeatable, even though it’s not self hosted, if the article extracts they make were available through their API. They are not. Unfortunately Pinboard wasn’t giving me much over my existing crawler solutions.

The winner was Wallabag10: it’s self-hosted, which is great, painful to install and set up, which is not, but it’s completely self-sustaining, runs on SQLite and good enough for me.

There was only one problem: none of these offered archival copies of images, and some of the bookmarks I made were solely for the photos on the sites. I found a format, called MHTML11, also known as .eml, which is perfect for single-file archives of HTML pages: it inlines all images as base64 encoded data.

However, no browser offers a save-as-mhtml in headless mode, so to get your archives, you’ll need to revisit your bookmarks. All of them. I enabled12 save as MHTML in Chrome (Firefox doesn’t know this format), installed the Wayback Machine13 extension and saved GBs of websites. I also added them into Wallabag. It’s an interesting, though very long journey, but you’ll rediscover a lot of things for sure.

When this was done, I dropped thousands of bookmark entries from my site.

If I do want to share a site, I’ll write a note about it, but bookmarks, without context, belong to my archives.

(Some) microblog imports should never have happened

I had iterations of imports, so after bookmarks it seemed reasonable to check what else may simply be noise on my site.

Back in the days people mostly wrote much lengthier entries: journal-like diary pages, thoughts, and it was, nearly always, anonymous. It all happened under pseudonyms.

Parallel to this there were the oldschool instant messengers, like ICQ and MSN Messenger. In many cases, though you all had handles, or numbers, or usernames, you knew exactly who you were talking to. Most of these programs had a feature called status message - looking back at it they may have been precursors to microblogging, but there was a huge difference: they were ephemeral.

With the rise of Twitter and Facebook status message also came (forced?) real identities, and tools letting us post from anywhere, within seconds. The content that earlier landed in status messages - XY is listening to…., Feels like…, etc - suddenly became readable at any time, sometimes to anyone.

I had content like this and I am, as well, guilty of posting short, meaningless, out-of-context entries. Imported burps of private life; useless shares of music pointing to long dead links; one-liner jokes, linking to bash.org; tiny replies and notes that should have been sent privately, either via email or some other mechanism.

Some things are meant to be ephemeral, no matter how loud the librarian is screaming deep inside me. Others belong in logs, and probably not on the public internet.

I deleted most of them and placed a HTTP 410 Gone message for their URLs.

Reposts are messy

For a few months I’ve been silently populating a category that I didn’t promote openly: favorites. At that page, I basically had a lot of reposts: images and galleries, with complete content, but with big fat URLs over them, linking to the original content.

By using a silo you usually give permission to the silo to use your work and there. Due to the effects of votes and likes (see later) you do, in fact, boost the visibility of the artist. Note that usually these permissions are much broader, than you imagine: a lawyer reworded the policy of Instagram to let everyone understand, that by using the service, you allow them to do more or less anything the want to with your work14.

But what is you take content out of a silo? The majority of images and works are not licensed in any special way, meaning you need to assume full copyright protection. Copyright prohibits publishing works without the author’s explicit consensus, so when you repost something that doesn’t indicate it’s OK with it - Creative Commons, Public Domain, etc -, what you do is illegal.

Also: for me, it feels like reposts, without notifying the creator, even though the licence allows it, are somewhat unfair - which is exactly what I was doing with these. Webmentions15 would like to address this by having an option to send notifications and delete requests, but silos are not there yet to send or to receive any of these.

There is a very simple solution: avoid reposting anything without being sure it’s licence allows you. Save it in a private, offline copy, if you really want to. Cweiske had a nice idea about adding source URLs into JPG XMP metadata 16, so you know where it’s from.

Silo reactions only make sense within the silo

When I started writing this entry, I differentiated 3, not-comment reaction types in silos:

A reaction is a social interaction, essentially a templated comment. “Well done”, “I disagree”, “buu”, “acknowledged”, ❤, 👍, ★, and so on. I asked my wife what she thinks about likes, why she uses them, and I got an unexpected answer: because, unlike with regular, text comments, others will not be able react to it - so no trolling or abuse is possible.

A vote has direct effect on ranking: think reddit up- and downvotes. Ideally it’s anonymous: list of voters should not be displayed, not even for the owner of the entry.

A bookmark is solely for one’s self: save this entry because I value it and I want to be able to find it again. They should have no social implications or boosting effect at all.

In many of the silos these are mixed - a Twitter fav used to range from an appreciation to a sarcastic meh17. With a range of reactions available this may get simpler to differentiate, but a like in Facebook still counts as both a vote and a reaction.

I thought a lot about reactions and I came to the conclusion that I should not have them on my site. The first problem is they will be linking into a walled garden, without context, maybe pointing at a private(ish) post, available to a limited audience. If the content is that good, bookmark it as well. If it’s a reaction for the sake of being social, it’s ephemeral.

Conclusions

Don’t let your ideas take over the things you enjoy. Some ideas can be beneficial, others are passing experiments.

There’s a lot of data worth collecting: scrobbles, location data, etc., but these are logs, and most of them, in my opinion, should be private. If I’m getting paranoid about how much services know about me, I shouldn’t publish the same information publicly either.

And finally: keep things simple. I’m finding myself throwing out my filter coffee machine and replacing it with a pot that has a paper filter slot - it makes an even better coffee and I have to care about one less electrical thing. The same should apply for my web presence: the simpler is usually better.


  1. https://stackingthebricks.com/how-blogs-broke-the-web/

  2. https://indieweb.org/

  3. https://petermolnar.net/indieweb-decentralize-web-centralizing-ourselves/

  4. https://petermolnar.net/personal-website-as-archiving-vault/

  5. https://indieweb.org/bookmark

  6. http://tantek.com/2015/069/t1/js-dr-javascript-required-dead

  7. http://www.archiveteam.org/index.php?title=Wget_with_WARC_output

  8. http://pinboard.in/

  9. http://getpocket.com/

  10. https://wallabag.org/en

  11. https://en.wikipedia.org/wiki/MHTML

  12. https://superuser.com/a/445988

  13. https://chrome.google.com/webstore/detail/waybackmachine/gofnhkhaadkoabedkchceagnjjicaihi

  14. https://qz.com/878790/a-lawyer-rewrote-instagrams-terms-of-service-for-kids-now-you-can-understand-all-of-the-private-data-you-and-your-teen-are-giving-up-to-social-media/

  15. https://webmention.net/draft/#sending-webmentions-for-deleted-posts

  16. http://cweiske.de/tagebuch/exif-url.htm

  17. http://time.com/4336/a-simple-guide-to-twitter-favs/

Sat, 28 Oct 2017 15:00:00 +0000


Lizard at La Palma

Camera
PENTAX K-5 II s
Aperture
f/6.3
Shutter speed
1/640 sec
Focal length (as set)
300.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 55-300mm F4-5.8 ED WR

La Palma is full of these little fellows, curious, fast, beautiful. If you give them some pressed fruit bars, they come very close.

Mon, 30 Oct 2017 18:00:00 +0000


Glencoe, Scotland

Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/60 sec
Focal length (as set)
16.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Clencoe in Scotland is a manificent place, but it feels incredibly distant from what I’m usually used to. Vast, flowing clouds, close enough that sometimes they are just above your head, snow capped, steep (although not too large) mountains all around.

Tue, 31 Oct 2017 18:00:00 +0000


Mountains around Glencoe

Camera
PENTAX K-5 II s
Aperture
f/9.0
Shutter speed
1/60 sec
Focal length (as set)
48.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

There is a large parking lot in Glencoe where everybody stoppes to take pictures. While the view on the opposite is indeed magnificent, if you turn around, you can see just as interesting and just as beautiful landscapes as well.

Wed, 01 Nov 2017 18:00:00 +0000


Trees burning in the sunset

Camera
PENTAX K-5 II s
Aperture
f/7.1
Shutter speed
1/60 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 800
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

We passed this tiny island on our way to Kilchurn Castle and back. Going there, they were simple, grey trees, but when the sun was setting, it was lighting up the branches just like it was the colour of the leaves or the trees themselves. You need to look closely to realise, it’s only a trick of the light.

Thu, 02 Nov 2017 18:00:00 +0000


Island on Loch Awe

Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/60 sec
Focal length (as set)
68.0 mm
Sensitivity
ISO 400
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Sunsets have been photographed so many times that when you see one that’s truly outstanding you end up finding yourself looking for excuses why you took the picture. It’s simple: it was, in fact, beautiful.

Fri, 03 Nov 2017 18:00:00 +0000


Sunset at Loch Awe

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/125 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Sunsets can be magnificent all around the UK, but they seem to be astonishing in Scotland - if you’re lucky enough to see the sun. We were, thank fully.

Sat, 04 Nov 2017 18:00:00 +0000


Sunset at Kilchurn Castle

Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/60 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

At the end of a wonderful organised day tour we visited Loch Awe and Kilchurn Castle. We a decent weather all the day - given it was winter in Scotland, I was very glad for it, but the sunset was certainly magnificent.

Sun, 05 Nov 2017 18:00:00 +0000


Powerscourt Waterfall Forest - greens

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/200 sec
Focal length (as set)
45.0 mm
Sensitivity
ISO 200
Lens
K or M Lens

Irish forests are beautiful, and all the greens collide before autumn sits in.

Tue, 07 Nov 2017 18:00:00 +0000


Powerscourt Waterfall Forest - browns

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/20 sec
Focal length (as set)
45.0 mm
Sensitivity
ISO 800
Lens
K or M Lens

Occasionally it’s worth getting off the usual tracks and paths; you end up in abandoned, unmaintained parts of the forest, leading into dark rows of pines all around you.

Wed, 08 Nov 2017 18:00:00 +0000


A piece of Powerscourt Waterfall

Camera
PENTAX K-5 II s
Aperture
f/20.0
Shutter speed
1/6 sec
Focal length (as set)
48.0 mm
Sensitivity
ISO 160
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Only a small portion of the huge Powerscourt Waterfall in Ireland. The place is beautiful, however, it can get very crowded, so taking images of the whole waterfall is very tricky.

Thu, 09 Nov 2017 18:00:00 +0000


Stream with oak leaves

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/80 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 200
Lens
K or M Lens

Sometimes the simple, small things, are just as gorgeous: this was in the stream that as flowing down from a gigantic waterfall in Ireland.

Fri, 10 Nov 2017 18:00:00 +0000


The top of Timanfaya

Camera
PENTAX K-5 II s
Aperture
f/16.0
Shutter speed
1/125 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 160
Lens
smc PENTAX-DA 35mm F2.4 AL

At the top of Timanfaya National Park on Lanzarote, there is a building, designed by César Manrique, called El Diablo: it uses completely natural hot air coming from the depths of the Earth to cook. This is the view in front of it.

Sat, 16 Dec 2017 08:00:00 +0000


Layers of volcanic ash

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/400 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 200
Lens
K or M Lens

Lanzarote has a unique lake: it’s completely green, called El Lago Verde. The colour is due to the different volcanic ash falling into the small lake: all these magnificent slopes around it are contributing to creating that green water.

Sun, 17 Dec 2017 08:00:00 +0000


Panorama of Timanfaya National Park

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
sec
Focal length (as set)
Sensitivity
ISO
Lens

The Timanfaya National Park is a vast wasteland of volcanic eruptions happened mostly around 1730. It is unique, beautiful, looks like an alien planet, and quite dangerous to walk on your own. The road on the left is the route the camels take when going to work and getting home - they are only serving tourists these days though.

My camera doesn’t have a panorama function: this image is glued from 3 handheld shoots, combined perfectly with a program called Hugin.

Mon, 18 Dec 2017 08:00:00 +0000


Wine yard on Lanzarote

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/320 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

On that black slope you can see small semi-circles of stones: those are humiditiy traps with vine in their middle. Wine from Lanzarote is sweet and very nice, and knowing it’s growing on a martinan land makes it even more exciting.

Tue, 19 Dec 2017 08:00:00 +0000


Panorama from Mirador De Morro Velosa

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
sec
Focal length (as set)
Sensitivity
ISO
Lens

When visiting a mirador, one might expect a place to buy coffee next to a magnificent view, but Mirador De Morro Velosa offers a lot more. Besides the panoramic view, the building also functions as a museum about the islands and their origins - and yes, you can still get a Cortado leche v leche.

Since my camera doesn’t have a panorama function, I had to improvise: 4 handheld shoots combined perfectly with Hugin turned out better, than I expected.

Wed, 20 Dec 2017 08:00:00 +0000


A desert of Fuerteventura

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
sec
Focal length (as set)
Sensitivity
ISO
Lens

The image was taken from the top of Mirador De Morro Velosa, on our way to Betancuria.

Fri, 22 Dec 2017 08:00:00 +0000


Peñitas Dam

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/400 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 160
Lens
smc PENTAX-DA 35mm F2.4 AL

During the past centuries there were numerous attempts to make Fuerteventura greener again - one of them was the Peñitas Dam. Unfortunately the erosion of the surrounding mountains is too strong, pushing a lot of soil into the dam, making the water completely disappear in the mud. It still contains a lot of water though, given how happy the trees are in the valley - trees, which are already rare at Fuerteventura.

(Also: a small praise for my Pentax camera, for producing an image like this, ouf of the machine without post-processing or filters.)

Sat, 23 Dec 2017 08:00:00 +0000


Mountains of Fuerteventura

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/500 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 200
Lens
smc PENTAX-DA 35mm F2.4 AL

There is a rather empty road on Fuerteventura, the one that goes through Betancuria, in the mountains, to the south, FV-30. I can certainly recommend taking this route if you can: it’s calm, outerworldly, with ravens visiting you at the miradors, showing all the beautiful mountains on the west side of the island.

Sun, 24 Dec 2017 08:00:00 +0000


Pozo Negro

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/125 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 200
Lens
smc PENTAX-DA 35mm F2.4 AL

Pozo Negro got about 2 sentences in the guide book for Fuerteventura, and most of those 2 lines was that it has 2 pescados, fish restaurants. Well, it did, in fact had nice fish, and if you’re there by sunset, like we were, it also offers beautiful colours.

Mon, 25 Dec 2017 08:00:00 +0000


Road to Malpaís Grande

Camera
PENTAX K-5 II s
Aperture
f/9.0
Shutter speed
1/125 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 200
Lens
smc PENTAX-DA 35mm F2.4 AL

The road to and from Pozo Negro is quiet, but that doesn’t mean there is nothing to look at: it comes all the way along the black fields of lava which started at Malpaís Grande and ended up at the sea at Pozo Negro.

Tue, 26 Dec 2017 08:00:00 +0000


Puertito de los Molinos

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/125 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 200
Lens
K or M Lens

Puertito de los Molinos has huge, steep, beautiful walls of rock surrounding it; unfortunately it’s more or less impossible to climb down on them. We planned to get here by sunset, but it was pitch black when we actually arrived, so instead before getting our flight home, we got up early and came back for sunrise. It was worth it: when sunrise comes with the high tide the waves are magnificent, even with a calm weather.

Wed, 27 Dec 2017 08:00:00 +0000


Living with a rugged, cheap Chinese Android phone, the Nomu S10

Disclaimer: I didn’t get paid for this entry: I simply liked this phone enough to share the whys and the fixes for it’s problems. It also serves as notes for myself.

Update 2018-04-18: the phone’s rubber exteriors started to deteriorate on the bottom right corner, where it contacts my hand most of the time; some of this is visible on the first image already, but it gets much worse surprisingly fast. I’ve found a glue - namely Loctite Power Flex Super Glue - which holds it, but it’s quite ugly after the fix. A single year out of a device like this is not very good, but considering the price, it’s not a big surprise. Unfortunately there is no repair kit available in any form, and I seriously dislike generating a phone per year amount of waste. However, I still like most of it’s aspect very much.

How the Nomu S10 found me

Nomu S10 after a few months of regular use in pockets and bags
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/40 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 3200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

About half a year ago we (my wife and me) decided we need to replace our phones - again. The Galaxy S4 was a sad choice after a Nexus 4 and a Galaxy Nexus, but those had to be replaced: they were literally falling apart and none of them supported European LTE bands - 3, 8, 20.

The Samsung Galaxy S4 is a weak, fragile creature, with weird bugs, a bloated, locked system, and with a slippery, rounded-everywhere shape that will certainly land on the tarmac a few times due to this. It has an impressive range of sensors, but that’s it, and I wanted a phone which works reliably and which is a bit more rugged.

I don’t believe in the current trend of bezel-free design and I’m very tired of all-glass, expensive, overheating, bloated, gargantuan “phones”, so I started looking into less-known brands. Initially into names like Oneplus, Xiaomi and some similar, sort of established manufacturers, but soon I learnt getting the actually good models in Europe is quite hard and due to this, nearly none of them covers LTE band 20 - the one giffgaff1, my service provider has access to.

The danger of less known brands is that they can be cheap for very logical reason: many of them are built cheap. On the bright side, the usually bring a surprising but good feature: most of the time these phones feature a near-raw AOSP system, without any bloat, and only minor software modifications.

Not long into my search I found a gem: the Nomu S102 and the Nomu brand in general. Apart from the S10, there is now the S10 Pro3, the S204 and S305 model. All of them are IP68 - waterproof enough to survive being fully submerged under water for a little while, rugged, with huge batteries.

The S10 is a relatively basic phone. It lacks NFC, 5GHz WiFi, and any new, shiny tech - but it’s loud, has a very capable 5000mAh battery, and covers more or less anything that falls under regular use. Yes, it’s heavy. It also has a metal frame inside - some way it resembles a ThinkPad from the old days. It packs 2GB RAM, 16GB storage, and either expandable with microSD, or it accepts 2 SIMs. The screen is a HD IPS screen, which I prefer to the OLED screens personally. The 2.4GHz Wifi is fast and stable and has a better reception than my previous, high-end phones did.

It also comes with a protective foil by default - actually it comes with 2 layers; after removing the first, a second is still protecting the display. Also has some version of Qualcomm Quick Charge which is also supported by the European charger it came with, which is certainly needed for the monster battery it has.

Though the stock firmware accepted SD cards up to 32GB, for my surprise, one of the system updates made 64GB possible as well - I don’t have any bigger ones, but I wouldn’t be surprised if the upper limit was 256GB.

Overall it has all the features you’d want from a phone that was built to last and be a companion for rough weather or outdoor activities.

I got a very good phone for the price and it worked well enough that I even used it without rooting.

Same Nomu S10, again after a few months of use
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/40 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 3200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Troubles arise: the Triada malware fiasco

One day my wife came home, telling me that her phone started to act weird: it randomly shows full screen popups, installs random software, and generally acting weird.

Knowing the she is quite careful and not installing basically anything on her phone after a few initial, well established apps, it looked very much like either a hack through other networks she connects to or something completely out of our control.

Unfortunately it turned out to be the later: there has been reports that a good amount of similarly cheap Chinese phones got infected with malware - from the manufacturer6! Interestingly the malware did not manifest on my phone, but the possibility that it might got me itchy.

Having a little experience in flashing custom ROMs the solution was obvious: I need to find a malware-free ROM for the Nomu and re-flash them.

The last time I had to dwell deep into Android and flashing was with my beloved HTC Desire - in 2011 - apparently a lot has changed since, and the malware, Triada, managed to sit into a part of the system, called Zygote, which is deep enough that even with root privileges you couldn’t get rid of it.

So here is how to get rid of the Triada malware on a Nomu S10 by flashing a malware free ROM.

The search for a malware-free ROM - the downside of niche phones

When you have a widely bought and known phone there are usually plenty of custom ROMs floating around for it. This used to be the case for the Nexus 4, the Galaxy S4, even for the HTC Desire.

This is not the case with the Nomu S10: it’s rare, so nobody made a full-on custom ROM for it. I started looking into forums and threads on the topic and thankfully I came across a French forum7 mentioning something called Archos 50 Saphir ROM. I was hoping to find some Cyanogenmod LineageOS based ROM, but for my surprise, Archos is a retailer: they sell Archos branded phones. One of these is what they call 50 Saphir8 - which is a re-branded Nomu S10.

Apparently the S10 had been sold to various “brands” who buy cheap phones, put a logo and a ROM without malware on them, and sell them for double the price.

This is the Archos ROM I ended up using:

https://www.dropbox.com/s/yk2t2mlv1uch0y7/Archos-50-Saphir-14-OTA.zip?dl=0

You can also get it from needrom.com9. It’s 1.1GB, so if you’re tight on bandwidth, don’t grab it immediately.

Another option is to get one of the newer, official Nomu ROMs, from their own site10 - the version 1.1.4, uploaded 2017-11-11 seems to be running fine and malware free for me for the past month:

https://drive.google.com/file/d/1ZXlnlic2z10CSFGoCf7nHygfTWsnFsAq/

I went with this solution, although it’s reasonable to be cautious and stick to the Archos version.

OEM Unlocking - the upside of OEM phones

Note: OEM unlocking can trigger a complete factory reset, deleting anything on your phone, excluding the microSD card. I’d still suggest removing even the microSD card and saving everything before starting the process.

Since the S10 has been sold for other brands to be used with their labels, Nomu allowed these resellers to flash their own operating system. This is good news for us: it means we can do the same, and rather simply.

There is a process in newer Androids that protects your phone from being overwritten by a mere USB cable and a laptop which requires a working operating system. If you don’t have one, you won’t be able to unlock flashing on your phone, so first to a factory reset and then follow these steps.

In your running Android:

  1. enable the Developer mode:
    1. go to Settings
    2. enter About phone
    3. tap Build number approximately 6 times
  2. once you have Developer options under Settings, enter it
  3. enable OEM unlocking

For the next stage, you’ll need a tool called fastboot. I’m using Debian, which is a linux distribution, and I have fastboot available from apt (the Debian app store, one could say) . I have no idea how to install it on Windows, but there are many tutorial available on the internet, and the commands should work once you have it. The commands below need to be executed as root user in linux.

Once you have fastboot:

  1. power off your phone
  2. press and hold volume up and the power buttons
  3. you’ll be greeted with a little menu, looking like:
Select Boot Mode:
[VOLUME_UP to select, VOLUME_DOW is OK.]

[Recovery   Mode]       <<==
[Fastboot   Mode]
[Normal     Mode]

Select Fastboot mode by pressing volume up and press volume down. The screen will now print a new line:

=> FASTBOOT mode...

On your laptop, connect the USB cable and test if you see the device in your laptop:

fastboot devices
EEMRTK5PBEJ78DWS        fastboot

If you don’t see the device, there is something wrong; try to repeat the process from the beginning of OEM unlocking.

Now enter:

fastboot flashing unlock

Note: it used to be fastboot oem unlock. Now it’s fastboot flashing unlock. If it doesn’t work, try the oem command version.

It will ask for confirmation and you’ll have to press the corresponding volume button - read the instructions on the phone, but volume up should enable to unlock.

Congratulations! You can now install any ROM made for your phone.

Things that might go wrong with this:

Flashing the stock ROM

Using SP Flash Tool (UPDATE 2018-04-20)

I turns out I was unaware of a tool - namely the SP Flash Tool - which is the official way of flashing ROMs on MediaTek based systems.

When the phone is turned off and connected to a computer via USB, it shows up as modem (!) device, as ttyACM. The SP Flash tool uses this to flash the ROM, but in order to do that - even if the flash tool is run by root - needs some tweaking on the linux side.

In order to get this supported on Debian, some udev rules need to be added:

Run (as root):

cat > etc/udev/rules.d/20-mediatek-blacklist.rules << EOF
ATTRS{idVendor}=="0e8d", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="6000", ENV{ID_MM_DEVICE_IGNORE}="1"
EOF

cat > etc/udev/rules.d/80-mediatek-usb.rules << EOF
SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="*"
EOF

systemctl restart udev.service

Once done add your user to the dialout and uucp groups as:

usermod -a -G dialout,uucp YOUR_USERNAME

Sp Flash tool needs an old version of libpng12, so get that from the Debian packages, or from the jessie (oldstable) repository:

wget http://ftp.uk.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb
dpkg -i libpng12-0_1.2.50-2+deb8u3_amd64.deb
rm libpng12-0_1.2.50-2+deb8u3_amd64.deb

This should make it possible to flash, using the SP flash tool, which can be downloaded from spflashtool.com11.

Credit due to Miss Montage on needrom.com12 for finding these out.

Using fastboot

Originally I tried to flash the stock ROM via the stock recovery, but every single time I got an error, telling me the zip file is corrupted. After a while I decided to take another approach.

Once you have the stock ZIP, extract it:

unzip NOMU_S10_COMMON_V1.1.4_2017_11_01_FQ5C62WTE1D.XWP.NOMU.M0.HB.H.SSXSSYDAKLS23.1101.V3.14.zip -d nomu_s10_1.1.4

It will create the following files in the nomu_s10_1.1.4 directory:

APDB_MT6735_S01_alps-mp-m0.mp1_W16.47
boot.img
boot-verified.img
BPLGUInfoCustomAppSrcP_MT6735_S00_MOLY_LR9_W1444_MD_LWTG_MP_V88_P92_1_lwg_n
cache.img
Checksum.ini
lk.bin
lk-verified.bin
logo.bin
logo-verified.bin
md1arm7.img
md1dsp.img
md1rom.img
md3rom.img
MT6737T_Android_scatter.txt
preloader_fq5c62wt_xwp_nomu.bin
preloader.img
recovery.img
recovery-verified.img
secro.img
secro-verified.img
system.img
trustzone.bin
userdata.img
V18S NOMU 软件配置说明.txt

The following will render your phone temporarily useless,; you will be without recovery, left only with a bootloader, for a short period. I seriously recommend only doing this with a fully charged phone and avoiding any accidental reboots during the process.

First, wipe the relevant partitions:

fastboot erase system
fastboot erase boot
fastboot erase recovery
fastboot erase cache

Now flash them:

cd nomu_s10_1.1.4
fastboot flash boot boot.img
fastboot flash recovery recovery.img
fastboot flash cache cache.img
fastboot flash system system.img

It will take a while, be patient.

Flashing the Archos ROM

Just use the same method as above, but instead of extracting the Nomu zip, extract the Archos zip.

However, unlike the stock ROM, the Archos ROM can be installed via adb sideload or simply selecting the zip in the dirty booted TWRP recovery described below. In case you’re familiar with custom recovery zip installing then there’s no need for the extract magic, but that method works just fine as well.

[Optional] How to root the Nomu S10 - dirty booting TWRP recovery

Dirty boot means we don’t flash the recovery partition, only load it on the fly from the laptop and use it temporarily - think about it as a live linux distribution. I found a TWRP custom recovery which worked very well for me; unfortunately I don’t remember the source, so I’ve uploaded it to Dropbox:

TWRP 3.0.2 recovery for Nomu S10:

https://www.dropbox.com/s/yebkei44w9oq6kh/nomu-s10_twrp-3.0.2.img?dl=0

Once you’re in the recovery, you can install whatever you want, including SuperSu:

https://www.dropbox.com/s/semoj6evcio3nyw/SuperSU-v2.82-201705271822.zip?dl=0

SuperSu is a “systemless root” method, which means instead of touching the system partition, which triggers an alarm with many root detectors, it puts the su binary into /sbin - which is part of the boot image. It can be simply reverted, should that be needed.

I didn’t have luck installing Magisk. While it worked flawlessly on a LineageOS install on a Nexus 10, no matter what I tried, it never worked here, but every single attempt triggered a factory reset. Just use SuperSu. If you do need features Magisk provides, maybe take a look at the Xposed framework instead, that worked well:

https://www.dropbox.com/s/xuqmah3vwh3nsre/xposed-v88.2-sdk23-arm.zip?dl=0

The simplest for these to install is to put them on the microSD card and select them as zip to be installed from TWRP. I will not cover the process this here, there are very good howto about TWRP and installing zips.

Waterproof - as long as you are careful

Not too long ago we spent a few days abroad, next to the sea, and found a tiny pool, left with water during low tide. The pool had a surprising amount if small fish and a small amount of coral in it, and, since the S10 is marketed as IP68, we decided to take a few underwater pictures with it. It’s important to say that one of the first things we did when we got the phones was submerge it in a lake to test the waterproofness, and there were not issues, but it was just a few seconds.

It’s a terrible underwater camera, so don’t use it as one. The touchscreen gets mad and recognizes the water as constant touch, so pushing the camera shutter button is hard and tricky.

But after finishing taking the shoots it seemed like my wife’s phone got a leak somewhere. It could have been a tiny bit of looseness in the rubber USB cover, or something completely unrelated, but after a few minutes out of the water it developed weird errors. The screen showed a lot of vertical lines, fading or altering the colour “behind” them, the resolution looked like it fell back to 320x240, and charging worked only sporadically. Apart from this, it was still working, receiving calls, responsive to touch.

After getting home I returned it to Amazon, who, due to the lack of replacement units with the original seller, refunded the value (the phone was less, the 6 months old). In my opinion, and according to the marketing, we did nothing out of ordinary use. We bought another one.

Touchscreen quality issues? (it may have been the only problematic one we got)

After buying the replacement for the defective one the next we received seemed fine - until you poked the touchscreen around the ‘a’ character on the keyboard. From that point, it started acting like the screen was being touched at multiple locations - would have made a nice stock video for ghost movies.

This was an obvious, immediate return, and we are now with yet another one without glitches - I hope it stays like that.

And you’re still recommending it?!

Oh, yes.

It’s cheap, with very nice features: small, compared to the battery it packs; runs for days on a single charge even with reasonable use; waterproof enough to survive being out in the rain for a long while, even submerged (when the flaps a strictly closed); the features it has all work; it’s “hackable” by default (the good way, unlike the Samsung or HTC phones).

The malware-by-manufacturer is not a unique problem and not strictly a problem of no-name manufacturers. There were reports that similar issues hit well known brands, such as Samsung, LG, Asus - at a point, the problem even included some Nexus 5 phones by Google13.

My sole remaining moan with the phone is that it’s glued. To take it apart, you’d need a special machine or a rather precise heat-gun and a steady hand. However, all the ones with screws, like the Blackview BV6000, are much larger and heavier.

So yes: regardless of the potential malware issue, I’m still recommending this phone, but do make sure you have a virus-free OS once before you start using it as your main phone.


  1. https://www.giffgaff.com/orders/affiliate/petermolnar2

  2. http://amzn.to/2AioYRp

  3. http://amzn.to/2CCxb9e

  4. http://amzn.to/2CNBf3a

  5. http://amzn.to/2E3E7IK

  6. https://news.drweb.com/news/?i=11390&lng=en

  7. https://forums.malwarebytes.com/topic/200072-trojantriada

  8. http://www.archos.com/gb-en/products/smartphones/saphir/archos_50saphir/index.html

  9. https://www.getdroidtips.com/stock-rom-archos-50-saphir/

  10. http://www.nomu.hk/s10-rom-download/

  11. https://spflashtool.com/download/SP_Flash_Tool_v5.1744_Linux.zip

  12. https://www.needrom.com/download/how-to-setup-sp-flash-tool-linux-mtk

  13. https://arstechnica.com/information-technology/2017/03/preinstalled-malware-targets-android-users-of-two-companies/

Sun, 07 Jan 2018 11:00:00 +0000


LinkedIn is ignoring user settings

A few days ago, on the #indieweb Freenode channel1 one of the users asked if we knew an indieweb-friendly way of getting data out of LinkedIn. I wasn’t paying attention to any recent news related to LinkedIn, though I’ve heard a few things, such as they are struggling to prevent data scraping: the note mentioned that they believe it’s a problem that employers keep an eye on changes in LinkedIn profiles via 3rd party. This, indeed, can be an issue, but there are ways to manage this within LinkedIn: your public profile settings2.

In my case, this was set to visible to everyone for years, and by the time I had to set it up (again: years), it was working as intended. But a few days ago, for my surprise, visiting my profile while logged out resulted in this:

LinkedIn showing a paywall-like ‘authwall’ for profiles set explicitly to public for everyone

and this:

$ wget -O- https://www.linkedin.com/in/petermolnareu
--2018-01-14 10:26:12--  https://www.linkedin.com/in/petermolnareu
Resolving www.linkedin.com (www.linkedin.com)... 91.225.248.129, 2620:109:c00c:104::b93f:9001
Connecting to www.linkedin.com (www.linkedin.com)|91.225.248.129|:443... connected.
HTTP request sent, awaiting response... 999 Request denied
2018-01-14 10:26:12 ERROR 999: Request denied.

or this:

$ curl https://www.linkedin.com/in/petermolnareu
<html><head>
<script type="text/javascript">
window.onload = function() {
  // Parse the tracking code from cookies.
  var trk = "bf";
  var trkInfo = "bf";
  var cookies = document.cookie.split("; ");
  for (var i = 0; i < cookies.length; ++i) {
    if ((cookies[i].indexOf("trkCode=") == 0) && (cookies[i].length > 8)) {
      trk = cookies[i].substring(8);
    }
    else if ((cookies[i].indexOf("trkInfo=") == 0) && (cookies[i].length > 8)) {
      trkInfo = cookies[i].substring(8);
    }
  }

  if (window.location.protocol == "http:") {
    // If "sl" cookie is set, redirect to https.
    for (var i = 0; i < cookies.length; ++i) {
      if ((cookies[i].indexOf("sl=") == 0) && (cookies[i].length > 3)) {
        window.location.href = "https:" + window.location.href.substring(window.location.protocol.length);
        return;
      }
    }
  }

  // Get the new domain. For international domains such as
  // fr.linkedin.com, we convert it to www.linkedin.com
  var domain = "www.linkedin.com";
  if (domain != location.host) {
    var subdomainIndex = location.host.indexOf(".linkedin");
    if (subdomainIndex != -1) {
      domain = "www" + location.host.substring(subdomainIndex);
    }
  }

  window.location.href = "https://" + domain + "/authwall?trk=" + trk + "&trkInfo=" + trkInfo +
      "&originalReferer=" + document.referrer.substr(0, 200) +
      "&sessionRedirect=" + encodeURIComponent(window.location.href);
}
</script>
</head></html>
So I started digging. According to the LinkedIn FAQ3 there is a page where you can set your profile’s public visibility. Those settings, for me, were still set to:
LinkedIn public profile settings

Despite the settings, there is no public profile for logged out users.

I’d like to understand what it going on, because so far, this looks like a fat lie from LinkedIn. Hopefully just a bug.

UPDATE

I tried setting referrers and user agents, used different IP addresses, still nothing. I can’t type today and managed to mistype https://google.com - the referrer ended up as https:/google.com. So, following the notes on HN, setting a referrer to Google sometimes works. After a few failures it will lock you out again, referrer or not. This is even uglier if it was a proper authwall for everyone.

curl 'https://www.linkedin.com/in/petermolnareu' \
-e 'https://google.com/' \
-H 'accept-encoding: text' -H \
'accept-language: en-US,en;q=0.9,' \
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
<!DOCTYPE html>...

  1. https://chat.indieweb.org

  2. https://www.linkedin.com/public-profile/settings

  3. https://www.linkedin.com/help/linkedin/answer/83?query=public

Sun, 14 Jan 2018 12:00:00 +0000


Hol vasmű állott, most kőhalom 2

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/60 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 1600
Lens
K or M Lens

Empty windows, nature slowly taking over, rubble everywhere - a few decades of decay.

Wed, 14 Feb 2018 10:00:00 +0000


Hol vasmű állott, most kőhalom

Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/100 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 3200
Lens
smc PENTAX-DA 35mm F2.4 AL

Just another abandoned, decaying factory in Budapest.

Thu, 15 Feb 2018 10:00:00 +0000


How to install microG an odexed stock android ROM

Why would anyone want an android without Google?

About 1.5 years ago I attended to a wedding. It took place outside of the city, at a restaurant with a very nice garden, where I’ve never before. In about 2 hours into the happening, my phone buzzed. I took it out, expecting a text message, or similar, but no: it was Google Maps, asking me if I’m at the place where I am, and since I’m there, could I upload some pictures of the place?

Since then this had become regular, until the point it became obstructive and annoying. I’m not alone: Brad Frost’s1 entry talks about the same problem. I’ve tried everything to turn Google location tracking off. I went to Location History2 and disabled it. I went through the Google Maps application and removed every notification. This latter one cleared the madness for a while - until a new version of Maps came up, which introduced some extra notification setting, which then showed yet another popup out of the blue.

Google fails to respect user settings lately, turning into a desperately annoying data hoarding monster. They’ve been doing nasty things, like silently collecting cell tower information, even with location settings disabled for years3, but with the coming of GDPR4 they need to get consent - hence the silly amount of notifications they are bombarding you with.

Once I set a setting in a service, I expect it to stay they way I set it. I expect backwards compatibility, backfilled data, if needed. Google and Facebook are both failing at this; Facebook always had, Google only recently started. New app, we renamed all the settings, let’s reset them to the default level of annoyance!

The whole problem on android can be traced back to one omnipotent application: Google Services Framework. This silent, invisible beast upgrades itself and Play Store whenever, wherever it wants it to. It does this all in the background, not even letting you know. If you happen to run an ancient phone, like my HTC Desire5, it will fill up that generous 150MB you have for user apps without a blink of an eye, and let you wonder why your phone can’t boot any more.

The extremely sad part is that everyone started depending on GMS - Google Mobile Services - for convenience: it provides push services, you don’t have to run you own. It all leads to the point that android, while in theory is Open Source, it will never be Free from Google, in it’s current form.

Enter microG6: a few enthusiasts with the same feelings as me, but with actual results. microG is a service level replacement for Google; it’s Free and Open Source, it’s transparent. There’s only one problem: it’s very tricky to install it on nieche phones, with odexed ROMs.

So I made a guide. While this guide was made using a Nomu S107, but given it’s an AOSP based ROM with tiny modifications, I’m fairly certain it can be applied to any similar, no-name less known brands and phones.

Important notes

The methods below might void your warranty. It might brick your phone. It will take while. It can cause an unexpected amount of hair pulled out.

Never do it on your only phone, or a phone you value high. I take no responsibility if anything goes wrong.

It was done on a Nomu S10, with Android 6.0 Marshmallow. It will most probably need to be altered for other versions.

The heavy lifting, the work, are all done by magnificent people out there; this article is merely summary of existing knowledge.

The only thing I can assure is that it worked for me, but it took a weekend to get to the bottom of it.

Prerequisites

Operating system, adb, fastboot

The guide was made for Debian based linux, including Ubuntu and Mint.

I assume you have a general understanding and familiarity with fastboot and adb - these are both needed during the process.

It’s doable on Windows as well, with very similar steps, but I don’t have a Windows, so I can’t make that guide.

SP Flash Tool (flashing stock ROMs on MediaTek devices)

The stock ZIPs Nomu provide can’t be flashed via the regular recoveries, like TWRP. As a workaround, I used to extract it and flash the pieces by fastboot - that is because I wasn’t aware of a tool, called SP Flash Tool and the MediaTek download mode.

When the phone is turned off and connected to a computer via USB, it shows up as modem (!) device, as ttyACM. The SP Flash tool uses this to flash the ROM, but in order to do that - even if the flash tool is run by root - needs some tweaking on the linux side.

In order to get this supported on Debian, some udev rules need to be added:

Run (as root):

cat > etc/udev/rules.d/20-mediatek-blacklist.rules << EOF
ATTRS{idVendor}=="0e8d", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="6000", ENV{ID_MM_DEVICE_IGNORE}="1"
EOF

cat > etc/udev/rules.d/80-mediatek-usb.rules << EOF
SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="*"
EOF

systemctl restart udev.service

Once done add your user to the dialout and uucp groups as:

usermod -a -G dialout,uucp YOUR_USERNAME

Sp Flash tool needs an old version of libpng12, so get that from the Debian packages, or from the jessie (oldstable) repository:

wget http://ftp.uk.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb
dpkg -i libpng12-0_1.2.50-2+deb8u3_amd64.deb
rm libpng12-0_1.2.50-2+deb8u3_amd64.deb

This should make it possible to flash, using the SP flash tool, which can be downloaded from spflashtool.com8.

Credit due to Miss Montage on needrom.com9 for finding these out.

TWRP recovery for Nomu S10

Do not flash TWRP recovery on the Nomu S10. There is some kind of safety check, which wants to trigger a factory reset, so what happens in short is the following:

At this point it’s stuck with a boot logo on screen (not even a bootloop); it can’t even be turned off, it’s without a working system, no recovery, and a locked fastboot. If you reach this point, use SP Flash Tool described above.

Instead of flashing, a method called dirty boot will be used: via fastboot, the TWRP image will be booted from the PC, not from the phone, because TWRP is still neede in order to flash custom ZIPs. Jemmini10 made one for the Nomu S10; it in a zip11 ; extract it to get a .img file.

SuperSu flashable ZIP

Many things have changed since the early days of rooting android (2.3 and 4.x). Nowadays fiddling with the system /bin might result in mayhem, so clever people came up with the idea to put su, the actual binary needed for rooting into the boot image, ending up in /sbin, without triggering any “security”. One of these systemless rooting apps is SuperSu12, which you’ll need in a flashable zip13 format.

Xposed framework ZIP and apk

A vast amount of Android’s real potential is lock behind bars - the reason is “security”. I’m putting this in quotes, because it’s smoke and mirrors: malware can use vulnerabilities to install itself deep inside the system that it’s impossible to even detect it, yet you’re not allowed to have full access on your phone. Not even the security suits and the malware scanners ask for root, and without that, they are not much more than a paid, bad joke.

Anyway: the XPosed Framework14 is here to help. It’s a utility which lets you install modules which modules tweak low level android behaviour. For example, Volumesteps+ will let you change the granularity of volume steps, which is very useful for someone who’d find a volume level between the factory 8 and 9 the best. For us, the important module will be FakeGApps, which allows signature spoofing15, which is a hard requirement for microG to work.

For reasons I’m yet to understand, I had to both flash the zip16 and install the apk17 version to get XPosed on the phone. In theory, only one should have been enough, but for now, get both.

NanoDroid microG ZIP

NanoDroid18 is originally a Magisk module (it’s another systemless rooting and framework, but I could never get it running on the Nomu), but it’s also available as flashable ZIPs.

For our needs, only the NanoDroid microG zip19 is required.

The actual un-googling

Summary

  1. root the phone and install xposed
  2. enable signature spoofing via xposed module fakegapps
  3. removed all google related apps, libs, entries
  4. install microg

Now: in details.

1. Rooting the Nomu S10 with SuperSu and Xposed

Important: OEM unlocking will trigger a factory reset, it will wipe every user and app setting from the phone.

  1. Start android
  2. Enable Developer Options:
  3. Enable OEM unlocking
  4. Enable USB Debugging (ADB)
  5. reboot into fastboot:
  6. in fastboot, issue the oem unlock command:
  7. reboot the phone: fastboot reboot
  8. let the factory reset flush through
  9. re-enable Developer Options (see 2.)
  10. verify that OEM unlocking is on - is should be. If not, go back to 1. and start again.
  11. re-enable USB debugging (see 3.)
  12. boot into fastboot (see 5.)
  13. “dirty” boot TWRP recovery:
  14. install SuperSu via TWRP
  15. install Xposed via TWRP (see 14.)
  16. reboot the phone
  17. install the Xposed apk

2. Enable signature spoofing with FakeGapps

Once the Xposed Installer is up and running it will look like this:

Running and enabled Xposed

Under Menu, Download search for FakeGApps20, click Download and enable it:

FakeGApps in Xposed

For FakeGApps to take effect, the phone has to be rebooted, but it can be done together with the next step.

3. Remove all google apps and libraries

These commands should be run either via an adb shell, or from a Terminal Emulator on the phone.

adb shell

# become root - SuperSu will prompt for verification
su -
# remount / and /system for read-write
mount -o remount,rw /
mount -o remount,rw /system
# create a file with the list it items to delete
cat > /sdcard/ungoogle.sh << EOF
rm -rf /system/app/CarHomeGoogle*
rm -rf /system/app/ChromeBookmarksSyncAdapter*
rm -rf /system/app/ConfigUpdater*
rm -rf /system/app/FaceLock*
rm -rf /system/app/GenieWidget*
rm -rf /system/app/Gmail*
rm -rf /system/app/Gmail2
rm -rf /system/app/GmsCore*
rm -rf /system/app/Google*
rm -rf /system/app/LatinImeTutorial*
rm -rf /system/app/LatinImeDictionaryPack*
rm -rf /system/app/MarketUpdater*
rm -rf /system/app/MediaUploader*
rm -rf /system/app/NetworkLocation*
rm -rf /system/app/OneTimeInitializer*
rm -rf /system/app/Phonesky*
rm -rf /system/app/PlayStore*
rm -rf /system/app/SetupWizard*
rm -rf /system/app/Talk*
rm -rf /system/app/Talkback*
rm -rf /system/app/Vending*
rm -rf /system/app/VoiceSearch*
rm -rf /system/app/VoiceSearchStub*
rm -rf /system/etc/permissions/com.google.android.maps.xml
rm -rf /system/etc/permissions/com.google.android.media.effects.xml
rm -rf /system/etc/permissions/com.google.widevine.software.drm.xml
rm -rf /system/etc/permissions/features.xml
rm -rf /system/etc/preferred-apps/google.xml
rm -rf /system/etc/g.prop
rm -rf /system/addon.d/70-gapps.sh
rm -rf /system/framework/com.google.android.maps.jar
rm -rf /system/framework/com.google.android.media.effects.jar
rm -rf /system/framework/com.google.widevine.software.drm.jar
rm -rf /system/lib/libfilterpack_facedetect.so
rm -rf /system/lib/libfrsdk.so
rm -rf /system/lib/libgcomm_jni.so
rm -rf /system/lib/libgoogle_recognizer_jni.so
rm -rf /system/lib/libgoogle_recognizer_jni_l.so
rm -rf /system/lib/libfacelock_jni.so
rm -rf /system/lib/libfacelock_jni.so
rm -rf /system/lib/libgtalk_jni.so
rm -rf /system/lib/libgtalk_stabilize.so
rm -rf /system/lib/libjni_latinimegoogle.so
rm -rf /system/lib/libflint_engine_jni_api.so
rm -rf /system/lib/libpatts_engine_jni_api.so
rm -rf /system/lib/libspeexwrapper.so
rm -rf /system/lib/libvideochat_stabilize.so
rm -rf /system/lib/libvoicesearch.so
rm -rf /system/lib/libvorbisencoder.so
rm -rf /system/lib/libpicowrapper.so
rm -rf /system/priv-app/CarHomeGoogle*
rm -rf /system/priv-app/ChromeBookmarksSyncAdapter*
rm -rf /system/priv-app/ConfigUpdater*
rm -rf /system/priv-app/FaceLock*
rm -rf /system/priv-app/GenieWidget*
rm -rf /system/priv-app/Gmail*
rm -rf /system/priv-app/GmsCore*
rm -rf /system/priv-app/Google*
rm -rf /system/priv-app/LatinImeTutorial*
rm -rf /system/priv-app/LatinImeDictionaryPack*
rm -rf /system/priv-app/MarketUpdater*
rm -rf /system/priv-app/MediaUploader*
rm -rf /system/priv-app/NetworkLocation*
rm -rf /system/priv-app/OneTimeInitializer*
rm -rf /system/priv-app/Phonesky*
rm -rf /system/priv-app/PlayStore*
rm -rf /system/priv-app/SetupWizard*
rm -rf /system/priv-app/Talk*
rm -rf /system/priv-app/Talkback*
rm -rf /system/priv-app/Vending*
rm -rf /system/priv-app/VoiceSearch*
rm -rf /system/priv-app/VoiceSearchStub*
EOF
# execut the created list
sh /sdcard/ungoogle.sh

3. Install NanoDroid microG

Once Google is cleaned up and the FakeGapps module is ready, reboot into recovery (see 12. and 13.) and install the NanoDroid zip via TWRP.

If you done everything right, there will be no Google Services or apps left, if not - as I did - a few leftovers will need to be manually cleaned up.

If the microG flashing was successful, an app, called microG settings will show up:

FakeGApps in Xposed

Done!


  1. http://bradfrost.com/blog/post/google-you-creepy-sonofabitch/

  2. https://myaccount.google.com/activitycontrols/location

  3. https://qz.com/1131515/google-collects-android-users-locations-even-when-location-services-are-disabled/

  4. https://www.eugdpr.org/

  5. https://en.wikipedia.org/wiki/HTC_Bravo

  6. https://microg.org/

  7. http://www.nomu.hk/pro/s10_product_show/

  8. https://spflashtool.com/download/SP_Flash_Tool_v5.1744_Linux.zip

  9. https://www.needrom.com/download/how-to-setup-sp-flash-tool-linux-mtk

  10. https://forum.xda-developers.com/showthread.php?t=3482755

  11. https://forum.xda-developers.com/attachment.php?attachmentid=3947063

  12. http://www.supersu.com/

  13. https://s3-us-west-2.amazonaws.com/supersu/download/zip/SuperSU-v2.82-201705271822.zip

  14. https://forum.xda-developers.com/showthread.php?t=3034811

  15. https://github.com/microg/android_packages_apps_GmsCore/wiki/Signature-Spoofing

  16. https://dl-xda.xposed.info/framework/sdk23/arm/xposed-v89-sdk23-arm.zip

  17. https://forum.xda-developers.com/attachment.php?attachmentid=4393082

  18. http://nanolx.org/nanolx/nanodroid

  19. https://downloads.nanolx.org/NanoDroid/Stable/NanoDroid-microG-16.1.20180209.zip

  20. http://repo.xposed.info/module/com.thermatk.android.xf.fakegapps

Sat, 24 Feb 2018 10:00:00 +0000


We are living in instant messenger hell

Note: I have updated some parts of this entry. This is due to the fact that I wrote about XMP without spending enough time exploring what it’s really capable of, for which I’m sorry. I made changes to my article according to these finds.

Me vs. IM

Before the dawn of the always online era (pre 2007) the world of instant messengers was completely different. For me, it all started with various IRC1 rooms, using mIRC2, later extended with ICQ3 in 1998.

I loved ICQ. I loved it’s notifications sound - I have it as notification sound on my smartphone and it usually results in very confused expressions from people who haven’t heard the little ‘ah-oooh’ for a decade -, it’s capability of sending and receiving files, the way you could search for people based on location, interest tags, etc.

The sixth protocol version appeared in ICQ 2000b and faced a complete rework. Encryption was significantly improved. Thanks to the new protocol, ICQ learned how to call phones, and send SMS and pager messages. Users also got the option of sending contact requests to other users.4

Around this time, Windows included an instant messenger in their operating systems: MSN Messenger5, later renamed to Windows Live Messenger. It was inferior, but because it was built in to Windows, it took all the ICQ users away. It’s completely dead now.

The multiplication of messengers had one useful effect though: people who got fed up running multiple clients for the same purpose - to message people - came up with the idea if multi-protocol applications. I used Trillian6 for many years, followed by Pidgin7 once I switched to linux.

With the help of these multi-protocol miracles it wasn’t an issues when newcomers like Facebook or Google released their messaging functionality: both were built in top of XMPP8, an open standard for instant messaging, and they were both supported out of the box in those programs.

Around this time came Skype and it solved all the video call problems with ease. It was fast, p2p, encrypted, ran on every platform, supported more or less everything people needed, including multiple instances for multiple accounts. Skype was on a good way to eliminate everything else. Unfortunately none of the multi-protocol messengers ever had a native support to it: it only worked if a local instance of Skype was running.

A few years later iPhone appeared and it ate the consumer world; not long before that, BlackBerry did the same to the business. Smartphones came with their own, new issues: synchronization, and resource (battery and bandwidth) limitations. ICQ existed for Symbian S60, Windows CE, and a bunch of other, ancient platforms, but by the time iPhones and BlackBerries roamed the mobile land, it was in a neglected state in AOL and missed a marvellous opportunity.

Both of those problems were known and addressed in the XMPP specification. The protocol was low on resources by design, it supported device priority, and XEP-0280: Message Carbons9 took care of delivering messages to multiple clients. There was a catch though: none of the well known XMPP providers supported any of these additions, so you ended up using either your mobile device or your computer exclusively at the same time. Most of the big system - AOL, Yahoo!, MSN, Skype, etc - didn’t even have a client for iOS, let alone for Android that time.

This lead to a new type of messenger generation: mobile only apps. WhatsApp10, BlackBerry Messenger11, Viber, etc - none of them offered any real way to be used from the desktop, and they all required - they still do - a working mobile phone number even to register.

For reasons I’m yet to comprehend, both Google and Facebook abandoned XMPP instead of extending fully implementing it. Google went completely proprietary and replaced gtalk12 with Hangouts13; Facebook started using MQTT14 for their messenger applications. Both of them were simple enough to be reverse engineered and added to libpurple, but they both tried to reinvent something that already existed.

For Skype, this was a turning point: it was bought by Microsoft, and they slowly moved it from p2p to a completely centralised webapp. The official reasoning included something about power hungry p2p connections… Soon, Skype lost all of it’s appeal from it’s previous iterations: video and voice was lagging, it was consuming silly amount of resources, it was impossible to stop it on Android, etc. Today, it resembles nothing from the original, incredible, p2p, secure, decentralised, resource-aware application it used to be.

I had to install WhatsApp yesterday - I resisted it as long as I could. It completely mangled competition in the UK and the Netherlands: nobody is willing to use anything else, not even regular text (SMS) or email. It did all this despite it’s lack of multi-device support, and the fact that it’s now owned by one of the nastiest, people-ignorant businesses around the globe15.

So, all together, in February 2018, for work and personal communication, I need to be able to connect to:

* I still have some contacts on ICQ, though it’s a wasteland, and I can’t even remember the last time I actually talked to anyone on it. This sort of applies to Hangouts: those who used to use it are now mostly on Facebook.

** WeChat is, so far, only a thing if you have Chinese contacts or if you live in/visit China. It’s dominating China so far that other networks, like QQ, can be more or less ignored, but WeChat is essential.

If I install all those things on my phone, I’ll run out of power in a matter of hours and the Nomu has an internal 5000mAh brick. They will consume any RAM I throw at them, and I don’t even want to think about the privacy implications: out of curiosity I checked the ICQ app, but the policy pushed into my face on the first launch is rather scary. As for Facebook: I refuse to run Facebook in any form on my phone apart from ‘mbasic’, the no javascript web interface.

Typing on a touchscreen inefficient, and I’m very far from being a keyboard nerd; my logs will be application specific and probably not in any readable/parsable format.

On top of all this, a few days ago Google announced Google Hangouts Chat17. Right now, Google has the following applications to cover text, voice, and video chat:

That’s 5 applications. 5. Only from Google.

Words for the future

I really, really want one, single thing, which allows me:

I sort of liked is Telegram18: cross device support, surprisingly fast and low on resources, but it gets attacked because they dared to roll their own crypto, and, in the end, it’s still a centralised service, ending up as just another account to connect to, and just another app to run. Since I wrote this entry, a few has tried to point out, that Telegram is not better, than WhatsApp or Signal, but I have to disagree. Yes, WhatsApp is encrypted by default - this also means I need to run my phone as a gateway all the time. No phone = no desktop user. The desktop “app” is a power and resource eater Electron app.

Others asked about Signal. It’s doing encryption the paranoid, aggressive way, but the same time, it depends on Google Services Framework on Android, Twilio, AWS, requires a smartphone app, eliminates 3rd party client options, and will only run the “desktop” Electron app if you pair it with a phone app - in which case it’s very similar to WhatsApp. Like it or not, it’s also a silo, with centralised services, even though you could, in theory, be able to install a complicated server of your own, that relies on the services listed above. It might be better, then WhatsApp, definitely not better from a usability point of view, than Telegram. Privacy wise… unless I can run my own server, without those kind of dependencies, no, thanks - it’s just another silo.

I also believe OTR-like encryption is overrated, or at least not as important as many presses. Most of the messages will tell you less, than their metadata, so what’s the point? Most of the encryption protocols are exclusive per connected client, meaning you can’t have multiple devices with the same account exchanging the same messages - hence the need for the phone apps as gateways. XMPP with OMEMO19 is tacking this - if that’s on by default, that could work. Note: TLS, infrastructure level encryption is a must, that is without question.

While Matrix20 looks promising, it’s an everything-on-HTTP solution, which I still find odd and sad. HTTP is not a simple protocol - yes, it’s omnipresent, but that doesn’t make it the best for a particular purpose. There’s another problem with it: no big player bought in which could bring the critical mass of users, and without that, it’s practically impossible to get people to use it.

Video and voices calls are, in general, in a horrible shape: nearly everything is doing WebRTC, which, while usually works, is a terrible performer, insanely heavy on CPU, and, most of the time, always tries to go for the highest quality, consuming bandwidth like there is no tomorrow.

All this leaves me with XMPP and SIP.

XMPP is and could be able to cover everything, and, on top of it, it’s federated, like email: anyone can run their own instance. I’m still a fan of email (yes, you read that right), and a signficant part of it is due to the options you can choose from: providers, clients, even being your own email service.

Unlike with most solutions and silos, the encryption problem (namely that if encryption is on, only one of the devices can get the messages, our you need to use a router device, like WhatsApp does) is covered and done with the XMPP extension OMEMO21. It’s a multi-client encryption protocoll, that allows simultaneous devices to connect and encrypt at once.

In case of XMPP, voice and video could be handled by a P2P protocol, Jingle22, but, unfortunately, it’s rarely supported. On Android, I found Astrachat23 which can do it, but it lacks many features when it comes to text based communications, unlike Conversations24. On desktop, I’m having serious problems getting Pidgin use video, so not everything is working yet.

This is where SIP comes in: an old, battle tested, proven VOIP protocol, which, so far, worked for me without any glitch in 2018. A few years ago many mobile providers were blocking SIP (among other VOIP protocols), but it’s getting much better. Unfortunately I have not started running my own VOIP exchange yet, and ended up using Linphone25 as software and provider - for now. The unfortunate part of SIP is that Pidgin doesn’t support it in any form.

There is one, very significant problem left: conformist people. I understand WhatsApp is simple and convenient, but it’s a Facebook owned, phone only system.

I’d welcome thoughts and recommendations on how to make your friends use something that’s not owned by Facebook.

Until then, I’ll keep using Pidgin, with a swarm of plugins that need constant updating.

Adding networks to Pidgin (technical details)

Pidgin, which I mentioned before, is a multi protocol client. Out of the box, it’s in a pretty bad shape: AIM, MSN, and Google Talk are dead as doornail, most of the systems it supports are arcane (eg. Zephyr) or sort of forgotten (ICQ). The version 3 of pidgin, and it’s library, libpurple, has been in the making for a decade and it’s still far ahead; the current 2.x line is barely supported.

There is hope however: people keep adding support for new systems, even to ones without proper or documented API.

For those who want to stick to strictly text interfaces, Bitlbee has a way to be compiled with libpurple support, but it’s a bit weird to use when you have the same contact or same names present on multiple networks.

The guides below are made for Debian and it’s derivatives, like Ubuntu and Mint. In order to build any of the plugins below, some common build tools are needed, apart from the per plugin specific ones:

sudo apt install libprotobuf-dev protobuf-compiler build-essential
sudo apt-get build-dep pidgin

How to conect to Skype with Pidgin (or libpurple)

The current iteration of the Skype plugin uses the web interface to connect to the system. It doesn’t offer voice and video calls, but it supports individual and group chats alike.

If you have 2FA on, you’ll need to use your app password as password and tick the Use alternative login method on the Advanced tab when adding the account.

git clone https://github.com/EionRobb/Skype4pidgin
cd Skype4pidgin/Skypeweb
cmake .
make
sudo make install

How to connect to Google Hangouts with Pidgin (or libpurple)

I’ve taken the instructions from the author’s bitbucket site26:

sudo apt install -y libpurple-dev libjson-glib-dev libglib2.0-dev libprotobuf-c-dev protobuf-c-compiler mercurial make
hg clone https://bitbucket.org/EionRobb/purple-hangouts/
cd purple-hangouts
make
sudo make install

How to connec to Facebook and/or Workplace by Facebook with Pidgin (or libpurple)

The Workplace support is not yet merged into the main code: it’s in the wip-work-chat branch. More information in the support ticket27.

Workplace and it’s ‘buddy’ list is sort of a mystery at this point in time, so don’t expect everything to run completely smooth, but it’s much better, than nothing.

In order to log in to a Workplace account, tick Login as Workplace account on the Advanced tab.

git clone https://github.com/dequis/purple-facebook
cd purple-facebook
git checkout wip-work-chat
./autogen.sh
./configure
make
sudo make install

How to conect to Telegram with Pidgin (or libpurple)

The Telegram plugin works nicely, including inline images and and to end encrypted messages. Voice supports seems to be lacking unfortunately.

sudo apt install libgcrypt20-dev libwebp-dev
git clone https://github.com/majn/telegram-purple
cd telegram-purple
git submodule update --init --recursive
./configure
make
sudo make install

How to connect to WhatsApp with Pidgin (or libpurple)

Did I mention I hate this network? First of all a note: WhatsApp doesn’t allow 3rd party applications at all. They might ban the phone number you use for life. This ban may be extended to Facebook with the same phone number but this has never been officially confirmed.

Apart from that it needs a lot of hacking around: the plugin is not enough, because WhatsApp doesn’t tell you your password. In order to get your password, you need to fake a ‘registration’ from the computer.

Even if you do this, only one device will work: the other instances will get logged out, so there is no way to use WhatsApp from your phone and from your laptop. It’s 2007 again, except it’s mobile only instead of desktop only.

Please stop using WhatsApp and use something with a tad more openness in it; XMPP, Telegram, SIP, ICQ… basically anything.

If you’re stuck with needing to communicate with stubborn and lazy people, like I am, continue reading, and install the plugin for pidgin:

sudo apt install libprotobuf-dev protobuf-compiler
git clone https://github.com/jakibaki/whatsapp-purple/
cd whatsapp-purple
make
sudo make install

However, this is not enough: the next step is yowsup, a command line python utility that allows you to ‘register’ to WhatsApp and reveals that so well hidden password.

sudo pip3 install yowsup

Once done, you need to first request an SMS, meaning you’ll need a number that’s able to receive SMS. Replace the COUNTRYCODE and PHONENUMBER string with your country code and phone number without prefixes, so for United Kingdom, that would be:

No 00, or + before the full international phone number.

$ yowsup-cli registration --requestcode sms -p PHONENUMBER --cc COUNTRYCODE --env android

    yowsup-cli  v2.0.15
    yowsup      v2.5.7

    Copyright (c) 2012-2016 Tarek Galal
    http://www.openwhatsapp.org

    This software is provided free of charge. Copying and redistribution is
    encouraged.

    If you appreciate this software and you would like to support future
    development please consider donating:
    http://openwhatsapp.org/yowsup/donate


    INFO:yowsup.common.http.warequest:b'{"login":"PHONENUMBER","status":"sent","length":6,"method":"sms","retry_after":78,"sms_wait":78,"voice_wait":65}\n'
    status: b'sent'
    length: 6
    method: b'sms'
    retry_after: 78
    login: b'PHONENUMBER'

Once you got the SMS, use the secret code:

$ yowsup-cli registration --register SECRET-CODE -p PHONENUMBER --cc COUNTRYCODE --env android

    yowsup-cli  v2.0.15
    yowsup      v2.5.7

    Copyright (c) 2012-2016 Tarek Galal
    http://www.openwhatsapp.org

    This software is provided free of charge. Copying and redistribution is
    encouraged.

    If you appreciate this software and you would like to support future
    development please consider donating:
    http://openwhatsapp.org/yowsup/donate

    INFO:yowsup.common.http.warequest:b'{"status":"ok","login":"PHONENUMBER","type":"existing","edge_routing_info":"CAA=","chat_dns_domain":"sl","pw":"[YOUR WHATSAPP PASSWORD YOU NEED TO COPY]=","expiration":4444444444.0,"kind":"free","price":"$0.99","cost":"0.99","currency":"USD","price_expiration":1520591114}\n'
    status: b'ok'
    login: b'PHONENUMBER'
    pw: b'YOUR WHATSAPP PASSWORD YOU NEED TO COPY'
    type: b'existing'
    expiration: 4444444444.0
    kind: b'free'
    price: b'$0.99'
    cost: b'0.99'
    currency: b'USD'
    price_expiration: 1520591114

That YOUR WHATSAPP PASSWORD YOU NEED TO COPY is the password you need to put in the password field of the account; the username is your PHONENUMBER.

How to connect to WeChat with Pidgin (or libpurple)

If there is something worse, than WhatsApp, it’s WeChat: app only and rather agressive when it comes to accessing private data on the phone. If you want to use it, but avoid actually serving data to it, I recommend getting the Xposed Framework28 with XPrivacyLua29 on your phone before WeChat and restricting WeChat with it as much as possible.

sudo apt install cargo clang
git clone https://github.com/sbwtw/pidgin-wechat
cd pidgin-wechat
cargo build
sudo cp target/debug/libwechat.so /usr/lib/purple-2/

Pidgin will only ask for a username - fill that in with you WeChat username and connect. Pidgin will soon pop up a window with a QR code - scan it with the WeChat app and follow the process on screen.

Other networks

Pidgin has a list of third party plugins30, but it’s outdated. I’ve been searching for forks and networks missing from the list on Github.

Extra Plugins for Pidgin

Purple Plugin Pack

There are a few useful plugins for Pidgin that can make life simpler; the Purple Plugin Pack31 contains most of the ones in my list:

XMPP Message Carbons

XEP-0280 Message Carbons32 is an extension that allows multiple devices to receive all messages.

sudo apt install libpurple-dev libglib2.0-dev libxml2-dev
git clone https://github.com/gkdr/carbons
cd carbons
make
sudo make install

Once installed, open a chat or conversation that happens on the relevant server and type:

/carbons on

This will not be delivered as message but executed on the server as command. Unfortunately not all of the XMPP servers support this.

OMEMO

OMEMO33 is a multi-legged encryption protocol that allows encrypted messages across multiple devices. It’s built-in into Conversations34, one of the best XMPP clients for Android - Pidgin doesn’t have it by default.

sudo apt install git cmake libpurple-dev libmxml-dev libxml2-dev libsqlite3-dev libgcrypt20-dev
git clone https://github.com/gkdr/lurch/
cd lurch
git submodule update --init --recursive
make
sudo make install

Message Delivery Receipts35

Yet another missing by default XMPP extension, which is quite useful.

git clone https://git.assembla.com/pidgin-xmpp-receipts.git 
cd pidgin-xmpp-receipts/
make
sudo cp xmpp-receipts.so /usr/lib/purple-2/

Porting old logs to Pidgin

I wrote a Python script which can port some old logs into Pidgin. It can deal with unmodifies logs from:

As for ZNC and Facebook, a lot of handywork is needed - see the comments in the script.

Requirements:

pip3 install arrow bs4

And the script:

import os
import sqlite3
import logging
import re
import glob
import sys
import hashlib
import arrow
import argparse
from bs4 import BeautifulSoup
import csv


def logfilename(dt, nulltime=False):
    if nulltime:
        t = '000000'
    else:
        t = dt.format('HHmmss')

    return "%s.%s%s%s.txt" % (
        dt.format("YYYY-MM-DD"),
        t,
        dt.datetime.strftime("%z"),
        dt.datetime.strftime("%Z")
    )


def logappend(fpath,dt,sender,msg):
    logging.debug('appending log: %s' % (fpath))
    with open(fpath, 'at') as f:
        f.write("(%s) %s: %s\n" % (
        dt.format('YYYY-MM-DD HH:mm:ss'),
        sender,
        msg
    ))
    os.utime(fpath, (dt.timestamp, dt.timestamp))
    os.utime(os.path.dirname(fpath), (dt.timestamp, dt.timestamp))


def logcreate(fpath,contact, dt,account,plugin):
    logging.debug('creating converted log: %s' % (fpath))
    if not os.path.exists(fpath):
        with open(fpath, 'wt') as f:
            f.write("Conversation with %s at %s on %s (%s)\n" % (
                contact,
                dt.format('ddd dd MMM YYYY hh:mm:ss A ZZZ'),
                account,
                plugin
            ))


def do_facebook(account, logpathbase):
    plugin = 'facebook'

    # the source for message data is from a facebook export
    #
    # for the buddy loookup: the  pidgin buddy list xml (blist.xml) has it, but
    # only after the alias was set for every facebook user by hand
    # the file contains lines constructed:
    # UID\tDisplay Nice Name
    #
    lookupf = os.path.expanduser('~/tmp/facebook_lookup.csv')
    lookup = {}
    with open(lookupf, newline='') as csvfile:
        reader = csv.reader(csvfile, delimiter='\t')
        for row in reader:
            lookup.update({row[1]: row[0]})

    # the csv file for the messages is from the Facebook Data export
    # converted with https://pypi.python.org/pypi/fbchat_archive_parser
    # as: fbcap messages.htm -f csv > ~/tmp/facebook-messages.csv
    dataf = os.path.expanduser('~/tmp/facebook-messages.csv')
    reader = csv.DictReader(open(dataf),skipinitialspace=True)
    for row in reader:
        # skip conversations for now because I don't have any way of getting
        # the conversation id
        if ', ' in row['thread']:
            continue

        # the seconds are sometimes missing from the timestamps
        try:
            dt = arrow.get(row.get('date'), 'YYYY-MM-DDTHH:mmZZ')
        except:
            try:
                dt = arrow.get(row.get('date'), 'YYYY-MM-DDTHH:mm:ssZZ')
            except:
                logging.error('failed to parse entry: %s', row)

        dt = dt.to('UTC')
        contact = lookup.get(row.get('thread'))
        if not contact:
            continue
        msg = row.get('message')
        sender = row.get('sender')

        fpath = os.path.join(
            logpathbase,
            plugin,
            account,
            contact,
            logfilename(dt, nulltime=True)
        )

        if not os.path.isdir(os.path.dirname(fpath)):
            os.makedirs(os.path.dirname(fpath))
        logcreate(fpath, contact, dt, account, plugin)
        logappend(fpath, dt, sender, msg)


def do_zncfixed(znclogs, logpathbase, znctz):
    # I manually organised the ZNC logs into pidgin-like
    # plugin/account/contact/logfiles.log
    # structure before parsing them
    LINESPLIT = re.compile(
        r'^\[(?P<hour>[0-9]+):(?P<minute>[0-9]+):(?P<second>[0-9]+)\]\s+'
        r'<(?P<sender>.*?)>\s+(?P<msg>.*)$'
    )
    searchin = os.path.join(
        znclogs,
        '**',
        '*.log'
    )
    logs = glob.glob(searchin, recursive=True)
    for log in logs:
        contact = os.path.basename(os.path.dirname(log))
        account = os.path.basename(os.path.dirname(os.path.dirname(log)))
        plugin = os.path.basename(os.path.dirname(os.path.dirname(os.path.dirname(log))))
        logging.info('converting log file: %s' % (log))
        dt = arrow.get(os.path.basename(log).replace('.log', ''), 'YYYY-MM-DD')
        dt = dt.replace(tzinfo=znctz)


        if contact.startswith("#"):
            fname = "%s.chat" % (contact)
        else:
            fname = contact

        fpath = os.path.join(
            logpathbase,
            plugin,
            account,
            fname,
            logfilename(dt)
        )

        if not os.path.isdir(os.path.dirname(fpath)):
            os.makedirs(os.path.dirname(fpath))

        with open(log, 'rb') as f:
            for line in f:
                line = line.decode('utf8', 'ignore')
                match = LINESPLIT.match(line)
                if not match:
                    continue
                dt = dt.replace(
                    hour=int(match.group('hour')),
                    minute=int(match.group('minute')),
                    second=int(match.group('second'))
                )
                logcreate(fpath, contact, dt, account, plugin)
                logappend(fpath, dt, match.group('sender'), match.group('msg'))


def do_msnplus(msgpluslogs, logpathbase, msgplustz):
    NOPAR = re.compile(r'\((.*)\)')
    NOCOLON = re.compile(r'(.*):?')

    searchin = os.path.join(
        msgpluslogs,
        '**',
        '*.html'
    )
    logs = glob.glob(searchin, recursive=True)
    plugin = 'msn'
    for log in logs:
        logging.info('converting log file: %s' % (log))
        contact = os.path.basename(os.path.dirname(log))

        with open(log, 'rt', encoding='UTF-16') as f:
            html = BeautifulSoup(f.read(), "html.parser")
            account = html.find_all('li', attrs={'class':'in'}, limit=1)[0]
            account = NOPAR.sub('\g<1>', account.span.string)
            for session in html.findAll(attrs={'class': 'mplsession'}):
                dt = arrow.get(
                    session.get('id').replace('Session_', ''),
                    'YYYY-MM-DDTHH-mm-ss'
                )
                dt = dt.replace(tzinfo=msgplustz)
                seconds = int(dt.format('s'))

                fpath = os.path.join(
                    logpathbase,
                    plugin,
                    account,
                    contact,
                    logfilename(dt)
                )

                if not os.path.isdir(os.path.dirname(fpath)):
                    os.makedirs(os.path.dirname(fpath))

                for line in session.findAll('tr'):
                    if seconds == 59:
                        seconds = 0
                    else:
                        seconds = seconds + 1

                    tspan = line.find(attrs={'class': 'time'}).extract()
                    time = tspan.string.replace('(', '').replace(')','').strip().split(':')

                    sender = line.find('th').string
                    if not sender:
                        continue

                    sender = sender.strip().split(':')[0]
                    msg = line.find('td').get_text()

                    mindt = dt.replace(
                        hour=int(time[0]),
                        minute=int(time[1]),
                        second=int(seconds)
                    )

                    logcreate(fpath, contact, dt, account, plugin)
                    logappend(fpath, mindt, sender, msg)


def do_trillian(trillianlogs, logpathbase, trilliantz):
    SPLIT_SESSIONS = re.compile(
        r'^Session Start\s+\((?P<participants>.*)?\):\s+(?P<timestamp>[^\n]+)'
        r'\n(?P<session>(?:.|\n)*?)(?=Session)',
        re.MULTILINE
    )

    SPLIT_MESSAGES = re.compile(
        r'\[(?P<time>[^\]]+)\]\s+(?P<sender>.*?):\s+'
        r'(?P<msg>(?:.|\n)*?)(?=\n\[|$)'
    )

    searchin = os.path.join(
        trillianlogs,
        '**',
        '*.log'
    )

    logs = glob.glob(searchin, recursive=True)
    for log in logs:
        if 'Channel' in log:
            logging.warn(
                "Group conversations are not supported yet, skipping %s" % log
            )
            continue

        logging.info('converting log file: %s' % (log))
        contact = os.path.basename(log).replace('.log', '')
        plugin = os.path.basename(os.path.dirname(os.path.dirname(log))).lower()

        with open(log, 'rb') as f:
            c = f.read().decode('utf8', 'ignore')

            for session in SPLIT_SESSIONS.findall(c):
                participants, timestamp, session = session
                logging.debug('converting session starting at: %s' % (timestamp))
                participants = participants.split(':')
                account = participants[0]
                dt = arrow.get(timestamp, 'ddd MMM DD HH:mm:ss YYYY')
                dt = dt.replace(tzinfo=trilliantz)
                fpath = os.path.join(
                    logpathbase,
                    plugin,
                    participants[0],
                    contact,
                    logfilename(dt)
                )

                if not os.path.isdir(os.path.dirname(fpath)):
                    os.makedirs(os.path.dirname(fpath))

                seconds = int(dt.format('s'))
                curr_mindt = dt
                for line in SPLIT_MESSAGES.findall(session):
                    # this is a fix for ancient trillian logs where seconds
                    # were missing
                    if seconds == 59:
                        seconds = 0
                    else:
                        seconds = seconds + 1

                    time, sender, msg = line
                    try:
                        mindt = arrow.get(time,
                        'YYYY.MM.DD HH:mm:ss')
                    except:
                        time = time.split(':')
                        mindt = dt.replace(
                            hour=int(time[0]),
                            minute=int(time[1]),
                            second=int(seconds)
                        )

                    # creating the filw with the header has to be here to
                    # avoid empty or status-messages only files
                    logcreate(fpath, participants[1], dt, account, plugin)
                    logappend(fpath, mindt, sender, msg)

            if params.get('cleanup'):
                print('deleting old log: %s' % (log))
                os.unlink(log)


def do_skype(skypedbpath, logpathbase):
    db = sqlite3.connect(skypedbpath)

    cursor = db.cursor()
    cursor.execute('''SELECT `skypename` from Accounts''')
    accounts = cursor.fetchall()
    for account in accounts:
        account = account[0]
        cursor.execute('''
        SELECT
            `timestamp`,
            `dialog_partner`,
            `author`,
            `from_dispname`,
            `body_xml`
        FROM
            `Messages`
        WHERE
            `chatname` LIKE ?
        ORDER BY
            `timestamp` ASC
        ''', ('%' + account + '%',))

        messages = cursor.fetchall()
        for r in messages:
            dt = arrow.get(r[0])
            dt = dt.replace(tzinfo='UTC')
            fpath = os.path.join(
                logpathbase,
                account,
                r[1],
                logfilename(dt, nulltime=True)
            )

            if not os.path.isdir(os.path.dirname(fpath)):
                os.makedirs(os.path.dirname(fpath))

            logcreate(fpath, r[1], dt, account, 'skype')
            logappend(fpath, dt, r[3], r[4])


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Parameters for Skype v2 logs to Pidgin logs converter')

    parser.add_argument(
        '--skype_db',
        default=os.path.expanduser('~/.skype/main.db'),
        help='absolute path to skype main.db'
    )

    parser.add_argument(
        '--pidgin_logs',
        default=os.path.expanduser('~/.purple/logs/skype'),
        help='absolute path to Pidgin skype logs'
    )

    parser.add_argument(
        '--facebook_account',
        default='',
        help='facebook account name'
    )

    parser.add_argument(
        '--loglevel',
        default='warning',
        help='change loglevel'
    )

    for allowed in ['skype', 'trillian', 'msnplus', 'znc', 'facebook']:
        parser.add_argument(
            '--%s' % allowed,
            action='store_true',
            default=False,
            help='convert %s logs' % allowed
        )

        if allowed != 'skype' or allowed != 'facebook':
            parser.add_argument(
                '--%s_logs' % allowed,
                default=os.path.expanduser('~/.%s/logs' % allowed),
                help='absolute path to %s logs' % allowed
            )

            parser.add_argument(
                '--%s_timezone' % allowed,
                default='UTC',
                help='timezone name for %s logs (eg. US/Pacific)' % allowed
            )

    params = vars(parser.parse_args())

    # remove the rest of the potential loggers
    while len(logging.root.handlers) > 0:
        logging.root.removeHandler(logging.root.handlers[-1])

    LLEVEL = {
        'critical': 50,
        'error': 40,
        'warning': 30,
        'info': 20,
        'debug': 10
    }

    logging.basicConfig(
        level=LLEVEL[params.get('loglevel')],
        format='%(asctime)s - %(levelname)s - %(message)s'
    )

    if params.get('facebook'):
        logging.info('facebook enabled')
        do_facebook(
            params.get('facebook_account'),
            params.get('pidgin_logs')
        )


    if params.get('skype'):
        logging.info('Skype enabled; parsing skype logs')
        do_skype(
            params.get('skype_db'),
            params.get('pidgin_logs')
        )

    if params.get('trillian'):
        logging.info('Trillian enabled; parsing trillian logs')
        do_trillian(
            params.get('trillian_logs'),
            params.get('pidgin_logs'),
            params.get('trillian_timezone'),
        )

    if params.get('msnplus'):
        logging.info('MSN Plus! enabled; parsing logs')
        do_msnplus(
            params.get('msnplus_logs'),
            params.get('pidgin_logs'),
            params.get('msnplus_timezone'),
        )

    if params.get('znc'):
        logging.info('ZNC enabled; parsing znc logs')
        do_zncfixed(
            params.get('znc_logs'),
            params.get('pidgin_logs'),
            params.get('znc_timezone'),
        )

  1. http://www.irc.org/

  2. https://www.mirc.com/

  3. https://icq.com/

  4. https://medium.com/@Dimitryophoto/icq-20-years-is-no-limit-8734e1eea8ea

  5. https://en.wikipedia.org/wiki/Windows_Live_Messenger

  6. https://www.trillian.im/

  7. http://pidgin.im/

  8. https://xmpp.org/

  9. https://xmpp.org/extensions/xep-0280.html

  10. https://en.wikipedia.org/wiki/Whatsapp

  11. https://en.wikipedia.org/wiki/BlackBerry_Messenger

  12. https://en.wikipedia.org/wiki/Google_talk

  13. https://en.wikipedia.org/wiki/Google_Hangouts

  14. https://en.wikipedia.org/wiki/MQTT

  15. http://www.salimvirani.com/facebook/

  16. https://www.facebook.com/workplace

  17. https://www.blog.google/products/g-suite/move-projects-forward-one-placehangouts-chat-now-available/

  18. https://telegram.org/

  19. https://xmpp.org/extensions/xep-0384.html

  20. https://matrix.org/

  21. https://xmpp.org/extensions/xep-0384.html

  22. https://xmpp.org/extensions/xep-0166.html

  23. https://play.google.com/store/apps/details?id=com.mailsite.astrachat

  24. https://f-droid.org/packages/eu.siacs.conversations/

  25. https://www.linphone.org/

  26. https://bitbucket.org/EionRobb/purple-hangouts/src#markdown-header-compiling

  27. https://github.com/dequis/purple-facebook/issues/371

  28. http://repo.xposed.info/

  29. https://lua.xprivacy.eu/

  30. https://developer.pidgin.im/wiki/ThirdPartyPlugins

  31. https://bitbucket.org/rekkanoryo/purple-plugin-pack/

  32. https://xmpp.org/extensions/xep-0280.html

  33. https://xmpp.org/extensions/xep-0384.html

  34. https://f-droid.org/packages/eu.siacs.conversations/

  35. https://xmpp.org/extensions/xep-0184.html

Sat, 03 Mar 2018 15:00:00 +0000


The internet that took over the Internet

There is a video out there, titled The Fall of The Simpsons: How it Happened1. It starts by introducing a mediocre show that airs every night, called “The Simpsons”, and compares it to a genius cartoon, that used to air in the early 90s, called “The Simpsons”. Watch the video, because it’s good, and I’m about to use it’s conclusion.

It reckons that the tremendous difference is due to shrinking layers in jokes, and, more importantly, in the characters after season 7. I believe something similar happened online, which made the Internet become the internet.

Many moons ago, while still living in London, the pedal of our flatmate’s sewing machine broke down, and I started digging for replacement parts for her. I stumbled upon a detailed website about ancient capacitors2. It resembled other, gorgeous sources of knowledge: one of my all time favourite is leofoo’s site on historical Nikon equipment3. All decades old sites, containing specialist level knowledge on topics only used to be found in books in dusty corners of forgotten libraries.

There’s an interesting article about how chronological ordering destroyed the original way of curating content4 during the early online era, and I think the article got many things right. Try to imagine a slow web: slow connection, slow updates, slow everything. Take away social networks - no Twitter, no Facebook. Forget news aggregators: no more Hacker News or Reddit, not even Technorati. Grab your laptop and put in down on a desk, preferably in a corner - you’re not allowed to move it. Use the HTML version of DuckDuckGo5 to search, and navigate with links from one site to another. That’s how it was like; surfing on the information highway, and if you really want to experience it, UbuWeb6 will allow you to do so.

Most of the content was hand crafted, arranged to be readable, not searchable; it was human first, not machine first. Nearly everything online had a lot of effort put into it, even if the result was eye-blowing red text on blue background7; somebody worked a lot on it. If you wanted it out there you learnt HTML, how to use FTP, how to link, how to format your page.

We used to have homepages. Homes on the Internet. Not profiles, no; profile is something the authorities make about you in dossier.

6 years ago Anil Dash released a video, “The web we lost”8 and lamented the web 2.0 - I despise this phrase; a horrible buzzword everyone used to label anything with; if you put ‘cloud’ and ‘blockchain’ together, you’ll get the level of buzz that was ‘web 2.0’ -, that fall short to social media, but make no mistake: the Internet, the carefully laboured web 1.0, had already went underground when tools made it simple for anyone to publish with just a few clicks.

The social web lost against social media, because it didn’t (couldn’t?) keep up with making things even simpler. Always on, always instant, always present. It served the purpose of a disposable web perfectly, where the most common goal is to seek fame, attention, to follow trends, to gain followers.

There are people who never gave up, and are still tirelessly building tools, protocols, ideas, to lead people out of social media. The IndieWeb9’s goals are simple: own your data, have an online home, and connect with others through this. And so it’s completely reasonable to hear:

I want blogging to be as easy as tweeting.10

But… what will this really achieve? This may sound rude and elitist, but the more I think about it the more I believe: the true way out of the swamp of social media is for things to require a little effort.

To make people think about what they produce, to make them connect to their online content. It’s like IKEA11: once you put time, and a minor amount of sweat - or swearing - into it, it’ll feel more yours, than something comfortably delivered.

The Internet is still present, but it’s shrinking. Content people really care about, customised looking homepages, carefully curated photo galleries are all diminishing. It would be fantastic to return to a world of personal websites, but that needs the love and work that used to be put into them, just like 20 years ago.

At this point in time, most people don’t seem to relate to their online content. It’s expendable. We need to make them care about it, and simpler tooling, on it’s own, will not help with the lack of emotional connection.


  1. https://www.youtube.com/watch?v=KqFNbCcyFkk

  2. http://www.vintage-radio.com/repair-restore-information/valve_capacitors.html

  3. http://www.mir.com.my/rb/photography/

  4. https://stackingthebricks.com/how-blogs-broke-the-web/

  5. https://duckduckgo.com/html/

  6. http://www.slate.com/articles/technology/future_tense/2016/12/ubuweb_the_20_year_old_website_that_collects_the_forgotten_and_the_unfamiliar.html

  7. http://code.divshot.com/geo-bootstrap/

  8. http://anildash.com/2012/12/the-web-we-lost.html

  9. https://indieweb.org

  10. http://www.manton.org/2018/03/indieweb-generation-4-and-hosted-domains.html

  11. https://en.wikipedia.org/wiki/IKEA_effect

Sun, 25 Mar 2018 22:20:00 +0100


Guide on how to make your website printable with CSS

Printing?! It’s 2018!

“Printing” doesn’t always mean putting it on paper. When people print a web article, sometimes it ends up as a PDF, because the HTML save it not usable. The reasons for this differ: JavaScript rendered content, unsaved scripts in the end result, the lack of MHTML support in browsers, etc. What’s important is that providing a print-friendly format for your site makes it possible for people to save it in a usable way.

Printing might still be relevant, because that’s the only method that gives you a physical object. I have long entries about journeys, visits of foreign places. At a certain point in time I was tempted to put together a photobook from the images there, but the truth is: it’s a lot of work, especially if you’ve more or less done it already once by writing your entry.

There’s also the completely valid case of archiving: hard copies have a life of decades if not centuries, when stored properly, unlike any electronic media we currently have as an option.

That little extra CSS

Before jumping into the various hacks that helps printers it’s important to mention, how to add printer-only instructions to your CSS. There are two ways, either using:

@media print {

}

inside an existing CSS, or by adding another CSS file specifically for print media into the HTML <head> section:

<head>
    <link rel="stylesheet" type="text/css" href="print.css" media="print">
</head>

White background, black(ish) text

Most printers operate with plain, white paper, so unless there’s a very, very good reason for printing background color, just get rid of it.

It also applies to the font: a bit lighter from black, so saves tint.

* {
    background-color: #fff !important;
    color: #222;
}

Use printer and PDF safe fonts

If you take a look at the history of printers vs fonts there used to be many problems around this topic - even so they might still require a font cartridge to be able to properly print fonts out of the basic options.1

To avoid rendering problems, aliasing issues, generally speaking: unreadable fonts, stick to one of the base 14 fonts:

which are, by definition, part of the PDF standard2. So for example:

* {
    font-size: 11pt !important;
    font-family: Helvetica, sans-serif !important;
}

If you do insist on special fonts, eg. you have icons in fonts, you might want to consider using SVG instead of fonts for icons - otherwise printing them properly will become a problem.

Besides the potential printing issues one more reason to go with a standard, base font is that if for any reason the text needs to go through character recognition for scanning it back - say it’s an archival hard copy and the only one left after a data loss indicent - the simpler and wider known the font, the better your chances for getting the characters properly recognized.

Pages and page breaks

It’s very annoying to find a heading at the bottom of a printed page, or a paragraph broke into separate pages, although this latter depends on paragraph length. I generally recommend disallowing page breaks at these locations.

Apart from this it’s a good idea to have a margin around the edges so you have an area where you can handle the page, not covering any of the text, or where it can be glued together as pages in a book.

@page {
    margin: 0.5in;
}

h1, h2, h3, h4, h5, h6 {
    page-break-after: avoid !important;
}

p, li, blockquote, figure, img {
    page-break-inside: avoid !important;
}

Images

Printing images is tricky: most of the images are sized for the web and those sizing are too small by resolution, too large by percentage of space taken for printing. The alt-text and the image headline, which is usually in alt and title are also something to consider printing, but unfortunately the href trick doesn’t work with them: that is because you can’t add ::before or ::after to self-closing tags, such as images.

Lately, instead of using simple img tags, I switched to using figure, along with figcaption - this way the headline became possible to print.

Apart from this I’ve limited the size of the images by view-width and view-height, so they never become too large and occupy complete pages.

figure {
    margin: 1rem 0;
}

figcaption {
    text-align: left;
    margin: 0 auto;
    padding: 0.6rem;
    font-size: 0.9rem;
}

figure img {
    display: block;
    max-height: 35vh;
    max-width: 90vw;
    outline: none;
    width: auto;
    height: auto;
    margin: 0 auto;
    padding: 0;
}

This is how images inside figure (should) look in print with the styling above:

This is how images can look like when some width/height limitations are applied in printing

Source codes

If you have code blocks in your page it’s useful to have them coloured, but still dark-on-light.

I’m using Pandoc’s built-in syntax highlighting3 and the following styling for printing:

code, pre {
    max-width: 96%;
    border: none;
    color: #222;
    word-break: break-all;
    word-wrap: break-word;
    white-space: pre-wrap;
    overflow:initial;
    page-break-inside: enabled;
    font-family: "Courier", "Courier New", monospace !important;
}

pre {
    border: 1pt dotted #666;
    padding: 0.6em;
}

/* code within pre - this is to avoid double borders */
pre code {
    border: none;
}

code.sourceCode span    { color: black; }
code.sourceCode span.al { color: black; }
code.sourceCode span.at { color: black; }
code.sourceCode span.bn { color: black; }
code.sourceCode span.bu { color: black; }
code.sourceCode span.cf { color: black; }
code.sourceCode span.ch { color: black; }
code.sourceCode span.co { color: darkgray; }
code.sourceCode span.dt { color: black; }
code.sourceCode span.dv { color: black; }
code.sourceCode span.er { color: black; }
code.sourceCode span.ex { color: darkorange; }
code.sourceCode span.fl { color: black; }
code.sourceCode span.fu { color: darkorange; }
code.sourceCode span.im { color: black; }
code.sourceCode span.kw { color: darkcyan; }
code.sourceCode span.op { color: black; }
code.sourceCode span.ot { color: black; }
code.sourceCode span.pp { color: black; }
code.sourceCode span.sc { color: black; }
code.sourceCode span.ss { color: black; }
code.sourceCode span.st { color: magenta; }
code.sourceCode span.va { color: darkturquoise; }

It should result in something similar:

Color printing source code

The basic CSS solution

Links are the single most important things on the internet; they are the internet. However, when they get printed, the end result usually looks something like this:

Before showing URLs - example showing Wikipedia entry “Mozilla software rebranded by Debian”

In order to avoid this problem, the URLs behind the links need to be shown as if they were part of the text. There is a rather simple way to do it:

a::after {
    content: " (" attr(href) ") ";
    font-size: 90%;
}

but unfortunately it makes the text rather ugly and very hard to read:

After showing URLs

Aaron Gustafson’s solution4

There is a very nice, minimalistic Javascript solution5 that collects all links on the page and converts them into footnotes on the fly, when it detects a print request.

This solution is way nicer, so I certainly recommend using this as well (it’s a supplement for the CSS solution above) even if it requres Javascript: (this is a copy-paste solution, just put it in your header)

<script type="text/javascript">
    // <![CDATA[
    /*------------------------------------------------------------------------------
    Function:       footnoteLinks()
    Author:         Aaron Gustafson (aaron at easy-designs dot net)
    Creation Date:  8 May 2005
    Version:        1.3
    Homepage:       http://www.easy-designs.net/code/footnoteLinks/
    License:        Creative Commons Attribution-ShareAlike 2.0 License
                    http://creativecommons.org/licenses/by-sa/2.0/
    Note:           This version has reduced functionality as it is a demo of
                    the script's development
    ------------------------------------------------------------------------------*/
    function footnoteLinks(containerID,targetID) {
      if (!document.getElementById ||
          !document.getElementsByTagName ||
          !document.createElement) return false;
      if (!document.getElementById(containerID) ||
          !document.getElementById(targetID)) return false;
      var container = document.getElementById(containerID);
      var target    = document.getElementById(targetID);
      var h2        = document.createElement('h2');
      addClass.apply(h2,['printOnly']);
      var h2_txt    = document.createTextNode('Links');
      h2.appendChild(h2_txt);
      var coll = container.getElementsByTagName('*');
      var ol   = document.createElement('ol');
      addClass.apply(ol,['printOnly']);
      var myArr = [];
      var thisLink;
      var num = 1;
      for (var i=0; i<coll.length; i++) {
        var thisClass = coll[i].className;
        if ( coll[i].getAttribute('href') ||
             coll[i].getAttribute('cite') ) {
          thisLink = coll[i].getAttribute('href') ? coll[i].href : coll[i].cite;
          var note = document.createElement('sup');
          addClass.apply(note,['printOnly']);
          var note_txt;
          var j = inArray.apply(myArr,[thisLink]);
          if ( j || j===0 ) {
            note_txt = document.createTextNode(j+1);
          } else {
            var li     = document.createElement('li');
            var li_txt = document.createTextNode(thisLink);
            li.appendChild(li_txt);
            ol.appendChild(li);
            myArr.push(thisLink);
            note_txt = document.createTextNode(num);
            num++;
          }
          note.appendChild(note_txt);
          if (coll[i].tagName.toLowerCase() == 'blockquote') {
            var lastChild = lastChildContainingText.apply(coll[i]);
            lastChild.appendChild(note);
          } else {
            coll[i].parentNode.insertBefore(note, coll[i].nextSibling);
          }
        }
      }
      target.appendChild(h2);
      target.appendChild(ol);
      addClass.apply(document.getElementsByTagName('html')[0],['noted']);
      return true;
    }
    window.onload = function() {
      footnoteLinks('content','content');
    }
    // ]]>
  </script>
  <script type="text/javascript">
    // <![CDATA[
    /*------------------------------------------------------------------------------
    Excerpts from the jsUtilities Library
    Version:        2.1
    Homepage:       http://www.easy-designs.net/code/jsUtilities/
    License:        Creative Commons Attribution-ShareAlike 2.0 License
                    http://creativecommons.org/licenses/by-sa/2.0/
    Note:           If you change or improve on this script, please let us know.
    ------------------------------------------------------------------------------*/
    if(Array.prototype.push == null) {
      Array.prototype.push = function(item) {
        this[this.length] = item;
        return this.length;
      };
    };
    // ---------------------------------------------------------------------
    //                  function.apply (if unsupported)
    //           Courtesy of Aaron Boodman - http://youngpup.net
    // ---------------------------------------------------------------------
    if (!Function.prototype.apply) {
      Function.prototype.apply = function(oScope, args) {
        var sarg = [];
        var rtrn, call;
        if (!oScope) oScope = window;
        if (!args) args = [];
        for (var i = 0; i < args.length; i++) {
          sarg[i] = "args["+i+"]";
        };
        call = "oScope.__applyTemp__(" + sarg.join(",") + ");";
        oScope.__applyTemp__ = this;
        rtrn = eval(call);
        oScope.__applyTemp__ = null;
        return rtrn;
      };
    };
    function inArray(needle) {
      for (var i=0; i < this.length; i++) {
        if (this[i] === needle) {
          return i;
        }
      }
      return false;
    }
    function addClass(theClass) {
      if (this.className != '') {
        this.className += ' ' + theClass;
      } else {
        this.className = theClass;
      }
    }
    function lastChildContainingText() {
      var testChild = this.lastChild;
      var contentCntnr = ['p','li','dd'];
      while (testChild.nodeType != 1) {
        testChild = testChild.previousSibling;
      }
      var tag = testChild.tagName.toLowerCase();
      var tagInArr = inArray.apply(contentCntnr, [tag]);
      if (!tagInArr && tagInArr!==0) {
        testChild = lastChildContainingText.apply(testChild);
      }
      return testChild;
    }
    // ]]>
  </script>
  <style type="text/css" media="screen">
    .printOnly {
      display: none;
    }
  </style>
  <style type="text/css" media="print">
    a:link:after,
    a:visited:after {
      content: " (" attr(href) ") ";
      font-size: 90%;
    }
    html.noted a:link:after,
    html.noted a:visited:after {
      content: '';
    }
  </style>

Alternative approach: always using footnotes for URLs

I little while ago I made a decision to put all links into footnotes by default - no in-text-links which will bring you to another site. This is a design decision and doesn’t apply to most of the already existing sites, but if you, just as me, think, there is value in it, consider it as an option. It also makes the two hacks above obsolete, however, it has it’s own problems, such as reading the site entries via RSS.

Avoid

opacity and transparency: it can get blurry

A simple and sort of lazy solution to, instead of figuring out the proper color code, just apply opacity to a text to make it slightly different from the rest. Unfortunately some of these opacity settings can result in blurry or unusable text:

CSS opacity resulting in blurry text

Therefore I suggest to avoid opacity and transparency on all elements for your printing styles.

Happy printing!


  1. https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/print_c_fonts.mspx

  2. https://en.wikipedia.org/wiki/Portable_Document_Format#Standard_Type_1_Fonts_.28Standard_14_Fonts.29

  3. http://pandoc.org/MANUAL.html#syntax-highlighting

  4. https://alistapart.com/article/improvingprint

  5. https://alistapart.com/d/improvingprintfinal.html

Tue, 17 Apr 2018 19:00:00 +0100


Dawn at Dojo Stara Wieś

Camera
PENTAX K-5 II s
Aperture
f/11.0
Shutter speed
1/80 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 100
Lens
smc PENTAX-DA 35mm F2.4 AL

I had the chance from Pakua UK1 to spend a weekend at Dojo Stara Wieś2 in Poland. Unexpected as it is, the Dojo is a small village, built for Japanese martial arts, in Japanese architectural style.

While it’s not the complete fairytale Japan one might expect, in the end, the only thing one could wish for is a small forest of giant bamboo, because everything else is tranquility here. The ponds were full of huge frogs and lovely newts, the air was filled with loud and happy birds - it’s a lovely place.

I took the picture not that early, sometimes just after sunrise.


  1. https://www.pakuauk.com/

  2. http://www.dojostarawies.com/en.html

Mon, 23 Apr 2018 18:00:00 +0000


Engawa of the dojo building at Dojo Stara Wieś

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
1/60 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 400
Lens
K or M Lens

This is the ourdoor veranda, the engawa of the dojo building itself at Dojo Stara Wieś1. The building hosts three beautiful areas to practice martial arts in a place, which resembles their origin quite well.


  1. http://www.dojostarawies.com/en.html

Tue, 24 Apr 2018 18:00:00 +0000


La Caldera de Taburiente panorama

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
sec
Focal length (as set)
Sensitivity
ISO
Lens

A panorama from Roque de los Muchachos on La Palma at 2426m.

Fri, 25 May 2018 08:00:00 +0000


Stargazers

Camera
PENTAX K-5 II s
Aperture
f/
Shutter speed
sec
Focal length (as set)
Sensitivity
ISO
Lens

The Roque de los Muchachos host a significant amount and rather important astronomy telescopes. Unfortunately visitors are not allowed up here during the night because even that tiny ligth pollution could distort measurements, but it’s certainly a unique view, even during daytime.

Sat, 26 May 2018 10:00:00 +0000


La Caldera de Taburiente

Camera
PENTAX K-5 II s
Aperture
f/5.0
Shutter speed
1/500 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 80
Lens
smc PENTAX-DA 35mm F2.4 AL

The very top of La Palma is the Roque de los Muchachos with it’s 2426m height. On cloudy days, this is the view - that pointy thing in the distance is the Teide on Tenerife, way above the clouds, with it’s 3718m.

Sun, 27 May 2018 10:00:00 +0000


Stream of Cascada de Los Colores

Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/60 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 400
Lens
smc PENTAX-DA 35mm F2.4 AL

On the Canary Island La Palma, unexpectedly, there is a lot of water. Some of this water ends up in the Cascada de Los Colores, a small waterfall of red, yellow, and green streams. Soon the stream becomes mostly red, stays like that for a while, slowly turns into yellow, as other, clear water connects to it, and in the end, it fades into ordinary water.

Mon, 28 May 2018 10:00:00 +0000


Using I²C sensors on a linux via a USB and IIO

Notes: no warranties. This is hardware, so it can cause trouble with your system, especially if you short-circuit something or - as I did once, many moons ago - solder on the fly why the thing is still connected to the USB port. Don’t do that.

Proto-assembly of Digispark ATTiny85, Adafruit BME280, and Adafruit SI1145
Camera
PENTAX K-5 II s
Aperture
f/5.6
Shutter speed
1/60 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 800
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

USB I²C adapter

A few months ago I wrote about using a Raspberry Pi with some I²C sensors to collect data for Collectd1. While it worked well, it made me realise that having the RPi running a full fledged operating system means I need to apply security patches to yet another machine, and that is not something I want to deal with. I also have a former laptop, running as a ZFS based NAS, so why not use that?

After venturing into a fruitless dig to use the I²C port in the VGA connector2 I verified that indeed, as concluded in the tutorial, it doesn’t work with embedded Intel graphics on linux.

Alternative I started looking at USB I²C adapter, but they are expensive. There is one project though, which looked very promising, and it didn’t require a full-fledged Arduino either: Till Harbaum’s I²C-Tiny-USB3.

It uses an ATtiny85 board - as the name suggests, it’s tiny, and turned out to be a perfectly fine USB to I²C adapter. You can buy one here: https://amzn.to/2ubPs6I

Note: there’s an Adafruit FT232H, which, in theory, is capable of the same thing. I haven’t tested it.

I2C-Tiny-USB firmware

The git repository already contains a built hex file, but in case there are any modifications needed to be done, this is how it’s done:

sudo -i
apt install gcc-avr avr-libc
cd /usr/src
git clone https://github.com/harbaum/I2C-Tiny-USB
cd I2C-Tiny-USB/digispark
make hex

Make sure the I2C_IS_AN_OPEN_COLLECTOR_BUS is uncommented; I’ve tried with real pull-up resistors, and, for my surprise, the sensors stopped showing up.

micronucleus flash utility

To flash the hex file, you’ll need micronucleus, a tiny flasher utility.

sudo -i
apt install libusb-dev
cd /usr/src
git clone https://github.com/micronucleus/micronucleus
cd micronucleus/commandline
make CONFIG=t85_default
make install

Run:

micronucleus --run --dump-progress --type intel-hex main.hex

then connect the device through a USB port, and wait for the end of the flash process.

I²C on linux

Surprisingly enough, Debian did not show I²C hubs in /dev - apparently the kernel module for this is not loaded, so load it, and make that load permanent:

sudo -i
modprobe i2c-dev
echo "i2c-dev" >> /etc/modules

Connect the Attiny85

Normally a PC already has a serious amount of I²C adapters. As a result, the new device will show up with an extra device number, which number is rather important. The kernel log can help identify that:

dmesg | grep i2c-tiny-usb
[    3.721200] usb 5-2: Product: i2c-tiny-usb
[    3.725693] i2c-tiny-usb 5-2:1.0: version 2.01 found at bus 005 address 003
[    3.736109] i2c i2c-1: connected i2c-tiny-usb device
[    3.736584] usbcore: registered new interface driver i2c-tiny-usb

To read just the device number:

i2cdev=$(dmesg | grep 'connected i2c-tiny-usb device' | head -n1 | sed -r 's/.*\s+i2c-([0-9]+).*/\1/')

Note: the device number might change after a reboot. For me, it was 10 when simply plugged in, and 1 if it was connected during a reboot.

Detecting I2C devices

i2cdetect is a program that dumps all the devices responding on an I²C adapter. The Adafruit website has a collection for their sensors4. That 1 after the i2cdetect -y is the device number identified in the previous step, and it says I have 2 devices:

sudo -i
i2cdev=$(dmesg | grep 'connected i2c-tiny-usb device' | head -n1 | sed -r 's/.*\s+i2c-([0-9]+).*/\1/')
i2cdetect -y ${i2cdev}
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- 77   

I²C 0x77: BME280 temperature, pressure, humidity sensor5

This is where things got interesting. Normally, when a BME280 sensors comes into play, every tutorial starts pulling out Python for the task, given that most of the Adafruit libraries are in Python.

Don’t get me wrong, those are great libs, and the Python solutions are decent, but doing a pip3 search bme280 resulted in this:

bme280 (0.5)                           - Python Driver for the BME280 Temperature/Pressure/Humidity Sensor from Bosch.
Adafruit-BME280 (1.0.1)                - Python code to use the BME280 temperature/humidity/pressure sensor with a Raspberry Pi or BeagleBone black.
adafruit-circuitpython-bme280 (2.0.2)  - CircuitPython library for the Bosch BME280 temperature/humidity/pressure sensor.
bme280_exporter (0.1.0)                - Prometheus exporter for the Bosh BME280 sensor
RPi.bme280 (0.2.2)                     - A library to drive a Bosch BME280 temperature, humidity, pressure sensor over I2C

Which one to use? Then there are the dependencies, and the code quality varies from one to another.

So I started digging into the internet, github, and other sources, and somehow I realised there’s a kernel module, named bmp280. The BMP280 is a sibling of the BME280 - it’s without the humidity sensor. So the questions was: what in the world is drivers/iio/pressure/bmp280-i2c.c and how can I use it?

It turned out, that apart from hwmon, there’s another sensor library layer in the linux kernel, called Industrial I/O - iio. It was added with this name somewhere in 2012, around 3.156, and it’s purpose is to offer a subsystem fast speed sensors7. While fast speed is not a thing for me this time, but I do trust the kernel code quality.

For my greatest surprise, the BMP280 module is even included in the Debian Sid kernel as a module, and adding it was a mere:

sudo -i
modprobe bmp280
echo "bmp280" >> /etc/modules
modprobe bmp280-i2c
echo "bmp280-i2c" >> /etc/modules

To actually enable the device, the i2c bus has to be told of the sensor’s existence:

sudo -i
i2cdev=$(dmesg | grep 'connected i2c-tiny-usb device' | head -n1 | sed -r 's/.*\s+i2c-([0-9]+).*/\1/')
echo "bme280 0x77" > /sys/bus/i2c/devices/i2c-${i2cdev}/new_device

The kernel log should show something like this:

kernel: bmp280 1-0077: 1-0077 supply vddd not found, using dummy regulator
kernel: bmp280 1-0077: 1-0077 supply vdda not found, using dummy regulator
kernel: i2c i2c-1: new_device: Instantiated device bme280 at 0x77

Verify the device is working:

tree /sys/bus/iio/devices/iio\:device0
/sys/bus/iio/devices/iio:device0
├── dev
├── in_humidityrelative_input
├── in_humidityrelative_oversampling_ratio
├── in_pressure_input
├── in_pressure_oversampling_ratio
├── in_pressure_oversampling_ratio_available
├── in_temp_input
├── in_temp_oversampling_ratio
├── in_temp_oversampling_ratio_available
├── name
├── power
│   ├── async
│   ├── autosuspend_delay_ms
│   ├── control
│   ├── runtime_active_kids
│   ├── runtime_active_time
│   ├── runtime_enabled
│   ├── runtime_status
│   ├── runtime_suspended_time
│   └── runtime_usage
├── subsystem -> ../../../../../../../../../bus/iio
└── uevent

2 directories, 20 files

And that’s it. The BME280 is ready to be used:

for f in  in_pressure_input in_temp_input in_humidityrelative_input; do echo "$f: $(cat /sys/bus/iio/devices/iio\:device0/$f)"; done
in_pressure_input: 102.112671875
in_temp_input: 26050
in_humidityrelative_input: 49.611328125

According to the BME280 datasheet8, under recommended modes of operation (3.5.1 Weather monitoring), the oversampling for each sensor should be 1, so:

sudo -i
echo 1 > /sys/bus/iio/devices/iio\:device0/in_pressure_oversampling_ratio
echo 1 > /sys/bus/iio/devices/iio\:device0/in_temp_oversampling_ratio
echo 1 > /sys/bus/iio/devices/iio\:device0/in_humidityrelative_oversampling_ratio

I²C 0x60: SI1145 UV index, light, IR sensor9

Unlike the BME280, the SI1145 doesn’t have a built-in kernel module in Debian Sid - but it does exist as a kernel module, it’s simply not included in the Debian Kernel. I’ve also learnt that this sensor is a heavyweight player, and that I should have bought something way simpler for mere light measurements; something that’s already included the out-of-the-box kernel modules, like a TSL256110.

But I wasn’t willing to give up the SI1145, being an expensie sensor, so in order to have it in the kernel, I had to compile the kernel module. Before getting started make sure:

Once those two are true, identify the kernel version:

uname -a
Linux system-hostname 4.17.0-1-amd64 #1 SMP Debian 4.17.3-1 (2018-07-02) x86_64 GNU/Linux

The output contains 4.17.3-1 - that is the actual kernel version, not the 4.17.0-1-amd64 which is the Debian name.

Get the kernel; extract it; add the SI1145 to the config; compile the drivers/iio/light modules; add that to the local modules.

sudo -i
cd /usr/src/
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.3.tar.gz
tar xf linux-4.17.3.tar.gz
cd linux-4.17.3
cp /boot/config-4.17.0-1-amd64 .config
cp ../linux-headers-4.17.0-1-amd64/Module.symvers .
echo "CONFIG_SI1145=m" >> .config
make menuconfig
# save it
# exit
make prepare
make modules_prepare
make SUBDIRS=scripts/mod
make M=drivers/iio/light SUBDIRS=drivers/iio/light modules
cp drivers/iio/light/si1145.ko /lib/modules/$(uname -r)/kernel/drivers/iio/light/
depmod
modprobe si1145
echo "si1145" >> /etc/modules

Once that is done, and there are no error messages, enable the device:

sudo -i
i2cdev=$(dmesg | grep 'connected i2c-tiny-usb device' | head -n1 | sed -r 's/.*\s+i2c-([0-9]+).*/\1/')
echo "si1145 0x60" > /sys/bus/i2c/devices/i2c-${i2cdev}/new_device

The kernel log shoud show something like this:

kernel: si1145 1-0060: device ID part 0x45 rev 0x0 seq 0x8
kernel: si1145 1-0060: no irq, using polling
kernel: i2c i2c-1: new_device: Instantiated device si1145 at 0x60

Verify the device is working:

tree /sys/bus/iio/devices/iio\:device1
/sys/bus/iio/devices/iio:device1                                     
├── buffer
│   ├── data_available
│   ├── enable
│   ├── length
│   └── watermark
├── current_timestamp_clock
├── dev
├── in_intensity_ir_offset
├── in_intensity_ir_raw
├── in_intensity_ir_scale
├── in_intensity_ir_scale_available
├── in_intensity_offset
├── in_intensity_raw
├── in_intensity_scale
├── in_intensity_scale_available
├── in_proximity0_raw
├── in_proximity_offset
├── in_proximity_scale
├── in_proximity_scale_available
├── in_temp_offset
├── in_temp_raw
├── in_temp_scale
├── in_uvindex_raw
├── in_uvindex_scale
├── in_voltage_raw
├── name
├── out_current0_raw
├── power
│   ├── async
│   ├── autosuspend_delay_ms
│   ├── control
│   ├── runtime_active_kids
│   ├── runtime_active_time
│   ├── runtime_enabled
│   ├── runtime_status
│   ├── runtime_suspended_time
│   └── runtime_usage
├── sampling_frequency
├── scan_elements
│   ├── in_intensity_en
│   ├── in_intensity_index
│   ├── in_intensity_ir_en
│   ├── in_intensity_ir_index
│   ├── in_intensity_ir_type
│   ├── in_intensity_type
│   ├── in_proximity0_en
│   ├── in_proximity0_index
│   ├── in_proximity0_type
│   ├── in_temp_en
│   ├── in_temp_index
│   ├── in_temp_type
│   ├── in_timestamp_en
│   ├── in_timestamp_index
│   ├── in_timestamp_type
│   ├── in_uvindex_en
│   ├── in_uvindex_index
│   ├── in_uvindex_type
│   ├── in_voltage_en
│   ├── in_voltage_index
│   └── in_voltage_type
├── subsystem -> ../../../../../../../../../bus/iio
├── trigger
│   └── current_trigger
└── uevent

5 directories, 59 files

Note: I tried, others tried, but even though in theory, there’s a temperature sensor on the SI1145, it doesn’t work. It seems like it reads the value on startup, and that’s it.

CLI script

In order to have a quick view, without collectd, or other dependencies, a script like this is more, than sufficient:

#!/usr/bin/env bash

d="$(date)"
temperature=$(echo "scale=2;$(cat /sys/bus/iio/devices/iio\:device0/in_temp_input)/1000" | bc)
pressure=$(echo "scale=2;$(cat /sys/bus/iio/devices/iio\:device0/in_pressure_input)*10/1" | bc) 
humidity=$(echo "scale=2;$(cat /sys/bus/iio/devices/iio\:device0/in_humidityrelative_input)/1" | bc) 
light_vis=$(cat /sys/bus/iio/devices/iio\:device1/in_intensity_raw) 
light_ir=$(cat /sys/bus/iio/devices/iio\:device1/in_intensity_ir_raw) 
light_uv=$(cat /sys/bus/iio/devices/iio\:device1/in_uvindex_raw) 

echo "$(hostname -f) $d

Temperature: $temperature °C
Pressure: $pressure mBar
Humidity: $humidity %
Visible light: $light_vis lm
IR light: $light_ir lm
UV light: $light_uv lm"

The output:

your.hostname Thu Jul 12 08:48:40 BST 2018

Temperature: 25.59 °C
Pressure: 1021.65 mBar
Humidity: 49.28 %
Visible light: 287 lm
IR light: 334 lm
UV light: 12 lm

Note: I’m not completely certain that the light unit is actually in lumens; the documentation is a bit fuzzy about that, so I assumed it is.

Collectd

The next step is to actually collect the sensor readouts from the sensors. I’m still using collectd11, a small, ancient, yet stable and very good little metrics collection system, because it’s enough. It writes ordinary rrd files, which can be plotted into graphs with tools like Collectd Graph Panel12

Unfortunately there’s not yet an iio plugin for collectd (or I couldn’t find it yet, and if you did, please let me know), so I had to add an extremely simple shell script as an exec plugin to collectd.

/usr/local/lib/collectd/iio.sh

#!/usr/bin/env bash

HOSTNAME="${COLLECTD_HOSTNAME:-$(hostname -f)}"
INTERVAL="${COLLECTD_INTERVAL:-60}"

# this will run only on collectd load, and once it's loaded, 
# even though it throws and error, additional runs don't make any
# problems
i2cdev=$(dmesg | grep 'connected i2c-tiny-usb device' | head -n1 | sed -r 's/.*\s+i2c-([0-9]+).*/\1/')
echo "bme280 0x77" > /sys/bus/i2c/devices/i2c-${i2cdev}/new_device
echo "si1145 0x60" > /sys/bus/i2c/devices/i2c-${i2cdev}/new_device


while true; do
    for sensor in /sys/bus/iio/devices/iio\:device*; do 
        name=$(cat "${sensor}/name")
        if [ "$name" == "bme280" ]; then

            # unit: °C
            temp=$(echo "scale=2;$(cat ${sensor}/in_temp_input)/1000" | bc )
            echo "PUTVAL $HOSTNAME/sensors-$name/temperature-temperature interval=$INTERVAL N:${temp}"

            # unit: mBar
            pressure=$(echo "scale=2;$(cat ${sensor}/in_pressure_input)*10/1" | bc)
            echo "PUTVAL $HOSTNAME/sensors-$name/pressure-pressure interval=$INTERVAL N:${pressure}"

            # unit: %
            humidity=$(echo "scale=2;$(cat ${sensor}/in_humidityrelative_input)/1" | bc)
            echo "PUTVAL $HOSTNAME/sensors-$name/percent-humidity interval=$INTERVAL N:${humidity}"

        elif [ "$name" == "si1145" ]; then

            # unit: lumen?
            ir=$(cat ${sensor}/in_intensity_ir_raw)
            echo "PUTVAL $HOSTNAME/sensors-$name/gauge-ir interval=$INTERVAL N:${ir}"

            light=$(cat ${sensor}/in_intensity_raw)
            echo "PUTVAL $HOSTNAME/sensors-$name/gauge-light interval=$INTERVAL N:${light}"

            uv=$(cat ${sensor}/in_uvindex_raw)
            echo "PUTVAL $HOSTNAME/sensors-$name/gauge-uv interval=$INTERVAL N:${uv}"

        fi
    done
    sleep "$INTERVAL"
done

/etc/collectd/collectd.conf

[...]
LoadPlugin "exec"
<Plugin exec>
  Exec "nobody" "/usr/local/lib/collectd/iio.sh"
</Plugin>
[...]

The results are:

BME280 temperature graph in Collectd Graph Panel
SI1145 raw light measurement in Collectd Graph Panel

Conclusions

The Industrial I/O layer is something I’ve heard for the first time, but it’s extremely promising: the code is clean, it already has support for a lot of sensors, and it seems to be possible to extend at a relative easy.

Unfortunately it’s documentation it brief and I’m yet to find any metrics collector that supports it out of the box, but that doesn’t mean there won’t be any very soon.

Currently I’m very happy with my budget I2C USB solution - not having to run a Raspberry Pi for simple metrics collection is certainly in win, and utilising the sensors directly from the kernel also looks very decent.


  1. https://petermolnar.net/raspberry-pi-bme280-si1145-collectd-mosquitto/

  2. https://web.archive.org/web/20160506154718/http://www.paintyourdragon.com/?p=43

  3. https://github.com/harbaum/I2C-Tiny-USB/tree/master/digispark

  4. https://learn.adafruit.com/i2c-addresses

  5. https://www.adafruit.com/product/2652

  6. https://github.com/torvalds/linux/tree/a980e046098b0a40eaff5e4e7fcde6cf035b7c06

  7. https://wiki.analog.com/software/linux/docs/iio/iio

  8. https://cdn-shop.adafruit.com/datasheets/BST-BME280_DS001-10.pdf

  9. https://www.adafruit.com/product/1777

  10. https://www.adafruit.com/product/439

  11. http://collectd.org/

  12. https://github.com/pommi/CGP

Fri, 13 Jul 2018 21:00:00 +0000


Do websites want to force us to use Reader Mode?

Excuse me, sir, but where’s the content?

A couple of days ago I blindly clicked on a link1 on Hacker News2 - it was poiting at a custom domain hosted on Medium. Out of curiosity, I changed the browser size to external 1280x720 - viewport 1280 × 646 -, turned off uBlock Origin3 and noscript4 so I’d mimic a common laptop setup, only to be presented with this:

Screenshot of blog.hiri.com/a-year-on-our-experience-launching-a-paid-proprietary-product-on-linux-db4f9116be08 when the window size is 1280x720

I don’t even know where to start listing the problems.

Screenshot of javascript requests made by blog.hiri.com/a-year-on-our-experience-launching-a-paid-proprietary-product-on-linux-db4f9116be08

So, foolishly, I started a now flagged thread5, begging publishers to go and start a static blog, or just publish this as a plain, HTML document. It would even be better is it was a Word 97 HTML export.

I decided to keep the browser like that, same resolution, no adblockers, and visited 2 more sites: bbc.co.uk, and theguardian.com.

Screenshot of www.bbc.co.uk/news/uk-44933429
Screenshot of javascript requests made by www.bbc.co.uk/news/uk-44933429
Screenshot of www.theguardian.com/world/2018/jul/23/greeks-urged-to-leave-homes-as-wildfires-spread-near-athens
Screenshot of javascript requests made by www.theguardian.com/world/2018/jul/23/greeks-urged-to-leave-homes-as-wildfires-spread-near-athens

Well… at least the BBC doesn’t have sticky headers and/or footers.

How did we get here?

Good examples

Let’s take a look at something, which is actually readable - a random entry from Wikipedia:

Screenshot of a random article from wikipedia

Note the differences:

Or another readable thing:

Screenshot of textfiles.com/magazines/LOD/lod-1 - Legion of Doom technical journal, volume 1, 1987

A 31 years old text file - still perfectly readable.

Or loading the first mentioned article in Firefox Reader Mode6:

Screenshot of a Medium article in Firefox Reader Mode

Developers gonna developer

So back to that thread. While most of the reactions were positive, there were opposing ones as well; here are a few of those.

I barely see the problem. Sure, the header and footer aren’t perfect, but stupidly large? I also don’t feel any “cpu melting javascripts” and my PC is barely usable when I compile anything.For me, Medium provides a very readable experience that is much better than the average static blog. And I don’t have to fear a malware ridden page like an old Wordpress installation. https://news.ycombinator.com/item?id=17592735

WordPress comes with it’s own can of worms, but it did introduce automatic security updates in version 3.77 - that was in 2013 October. Any WordPress installation since have been receiving security patches, and WordPress backports security patches respectfully well.

As for being malware ridden… it doesn’t even make it to the news pages any more when an ad network start spreading malware, but that’s still a thing.8

Why is it that I only ever hear those complaints on HN and never elsewhere… Are you all still using Pentium 3 PCs and 56k modems?

https://news.ycombinator.com/item?id=17592801

A couple of years ago Facebook intruduced 2G Tuesdays9 and that should still be a thing for everyone out there. Rural Scotland? There isn’t any phone signal, let alone 3 or 4G. Rural Germany? 6Mbps/1Mbps wired connections. And that is in Europe. Those who travel enough know this problem very well, and yes, 1.8MB - I initially stated 121kB in my original thread, that was a mistake, and due to uBlock not being completely off - of JavaScript is way too much. It was too much when jquery was served from a single CDN at may even actually got cached in the browser, but compiled, React apps won’t be cached for long.

[…] people nowadays demand rich media content […]

https://news.ycombinator.com/item?id=17592943

I remember when I first saw parallax scroll - of course it made me go “wow”. It was a product commercial, I think, but soon everybody was doing parallax scroll, even for textual content. I was horrible. Slow, extremely hard to read due to all the moving parts.

There were times when I thought mouse trailing bouncing circles10 were cool. It turned out readable, small, fast text is cooler.

Nobody is “demanding” rich media content; people demand content. For free, but that is for another day. With some images, maybe even videos - and for that, we have <img>, <figure>, <video>, with all their glory.

> 121KB javascript is not heavy

Part of the problem is that HTML and CSS alone are horribly outdated in terms of being able to provide a modern-looking UI outside the box.

Want a slider? Unfortunately the gods at W3C/Google/etc. don’t believe in a <input type="slider"> tag. Want a toggle switch? No <input type="toggle">. Want a tabbed interface? No <tabs><tab></tab></tabs> infrastructure. Want a login button that doesn’t look like it came out of an 80’s discotheque? You’re probably going to need Angular, Polymer, MDL or one of those frameworks, and then jQuery to deal with the framework itself. You’re already looking at 70-80kb for most of this stuff alone.

Want your website to be mobile-friendly? Swipe gestures? Pull to refresh? Add another 30-40kb.

Commenting? 20kb.

Commenting with “reactive design” just to make your users feel like their comments went through before they actually went through? 50kb.

Want to gather basic statistics about your users? Add another 10kb of analytics code.

https://news.ycombinator.com/item?id=17594649

This comment is certainly right, when it comes to UI. However… this is an article. Why would an article need swipe gestures, pull-to-refresh? Analytics is an interesting territory, but basics are well covered by analyzing server logs1112.

Mobile friendly design doesn’t need anything at all; it actually needs less: HTML, by design, flows text to the available width, so any text will fill the available container.

For web UI, you need those, yes. To display an article, you really don’t.

Medium vs blogs

I’ve been told that people/companies most usually post to Medium for the following reaons:

As for discoverability, I believe pushing the article link to Reddit, HN, etc. is a significant booster, but merely putting it on medium doesn’t mean anything. I’ve had this problem a long while ago, with personal blogs, as is why is dicoverability never addressed in re-decentralize topics, but the truth is: there is no real need for it. Search engines are wonderful, and if your topic is good enough, people will find it by searching.

The looks more serious problem is funny, given the article I linked is on their own domain - if I wasn’t aware of the generic issues with Medium layouts, I wouldn’t know, it’s a Medium article. One could make any blog look and feel the same. One could export an article from Typora13 and still look professional.

I’ve heard stories of moving to medium brought a lot more “reads” and hits on channels, but I’m sceptical. Eons ago I read an article, when PageRank was still a thing, where a certain site went to be #1 on Google for certain phrases without even contaning that phrase - only the links linking to the site did. The lesson there is that everything can be playes, and I find it hard to believe that purely posting to Medium would boost visibility that much. I could be wrong though.

Proposals - how do we fix this?

Always make the content the priority

There’s an article to read, so let people read it. The rest is secondary for any visitor of yours.

Don’t do sticky headers/footers

But if you really, really have to, make it certain it’s the opposite of the display layout: for horizontal windows, the menu should be on the side; for vertical, it should be on the top.

You don’t even need JS for it, since it’s surprisingly simple to tell horizontal apart from vertical, even in pure CSS, with media queries:

 @media screen and (orientation:portrait) { … }
 @media screen and (orientation:landscape) { … }

Rich media != overdosed JavaScript

Embrace srcset14 and serve different, statically pre-generated images. Seriously consider if you need a framework at all15. BTW, React is the past, from before progessive enhancements, an it came back to haunt us for the rest of the eternity.

Use one good analytics system. There really is no need for multiple ones, just make is sure that one is well configured.

Don’t install yet another commenting system - nobody cares. Learn from the bigger players and think it through if you actually need a commenting system or not16.

Some JS is useful, a lot of JS is completely unneeded for displaying articles. If your text is 8000 characters, there is simply no reasonable excuse to serve 225x more additional code to “enhance” that.

Closure

HTML was invented to easily share text documents. Even if it has images, videos, etc. in them, you’re still sharing text. Never forget that the main purpose is to make that text readable.

There are many people out there with capped, terrible data connection, even in developed countries, and this is not changing in the close future. Every kB counts, let alone MBs.

MBs of Javascript has to be evaluated in the browser, which needs power. Power these days comes from batteries. More code = more drain.

Keep it simple, stupid.


  1. https://blog.hiri.com/a-year-on-our-experience-launching-a-paid-proprietary-product-on-linux-db4f9116be08

  2. https://news.ycombinator.com/

  3. https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/

  4. https://noscript.net/

  5. https://news.ycombinator.com/item?id=17592600

  6. https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free-web-pages

  7. https://codex.wordpress.org/Configuring_Automatic_Background_Updates

  8. https://www.theguardian.com/technology/2016/mar/16/major-sites-new-york-times-bbc-ransomware-malvertising

  9. https://www.theverge.com/2015/10/28/9625062/facebook-2g-tuesdays-slow-internet-developing-world

  10. http://dynamicdrive.com/dynamicindex13/trailer.htm

  11. https://www.awstats.org/

  12. https://matomo.org/log-analytics/

  13. https://typora.io/

  14. https://www.sitepoint.com/how-to-build-responsive-images-with-srcset/

  15. http://youmightnotneedjquery.com/

  16. https://motherboard.vice.com/en_us/article/jp5yx8/im-on-twitter-too

Wed, 25 Jul 2018 10:30:00 +0100


Lessons of running a (semi) static, Indieweb-friendly site for 2 years

In 2016, I decided to leave WordPress behind. Some of their philosophy, mostly the “decisions, not options” started to leave the trail I thought to be the right one, but on it’s own, that wouldn’t have been enough: I had a painful experience with media handling hooks, which were respected on the frontend, and not on the backend, at which point, after staring at the backend code for days, I made up my mind: let’s write a static generator.

This was strictly scratching my own itches1: I wanted to learn Python, but keep using tools, like exiftool and Pandoc, so instead of getting an off the shelf solution, I did actually write my own “static generator” - in the end, it’s a glorified script.

Since the initial idea, I rewrote that script nearly 4 times, mainly to try out language features, async workers for processing, etc, and I’ve learnt a few things in the process. It is called NASG - short for ‘not another static generator’, and it lives on Github2, if anyone wants to see it.

Here are my learnings.

Learning to embrace “buying in”

webmentions

I made a small Python daemon to handle certain requests; one of these routings was to handle incoming webmentions3. It merely put the requests in a queue - apart from some initial sanity checks on the POST request itself -, but it still needed a dynamic part.

This approach also required parsing the source websites on build. After countless iterations - changing parsing libraries, first within Python, then using XRay4 - I had a completely unrelated talk with a fellow sysadmin on how bad we are when in comes to “buying into” a solution. Basically if you feel like you can do it yourself it’s rather hard for us to pay someone - instead we tend to learn it and just do it, let it be piping in the house of sensor automation.

None of these - webmentions, syndication, websub - are vital for my site. Do I really need to handle all of them myself? If I make it sure I can replace them, if the service goes out of business, why not use them?

With that in mind, I decided to use webmention.io5 as my incoming webmention (it even gave pingback support back) handler. I ask the service for any new comments on build, save them as YAML + Markdown, so the next time I only need to parse the new ones.

To send webmentions, Telegraph6 is nice, simple service, that offers an API access, so you don’t have to deal with webmention endpoint discovery. I put down a text file, with slugified names of the source and target URLs to prevent sending the mention any time.

websub

In case of websub7 superfeedr8 does the job quite well.

syndication

For syndication, I decided to go with IFTTT9 brid.gy publish10. IFTTT reads my RSS feed(s) and either creates link-only posts on WordPress11 and Tumblr12, or sends webmentions to brid.gy to publish to links Twitter13 and complete photos to Flickr14

I ended up outsourcing my newsletter as well. Years ago I sent a mail around to friends to ask them if they want updates from my site in mail; a few of them did. Unfortunately Google started putting these in either Spam or Promitions, so it never reached people; the very same happened with Blogtrottr15 mails. To overcome this, I set up a Google Group, where only my Gmail account can post, but anyone can subscribe, and another IFTTT hook16 that sends mails to that group with the contents of anything new in my RSS feed.

Search: keep it server side

I spent days looking for a way to integrate JavaScript based search (lunr.js or elasticlunr.js) in my site. I went as far as embedding JS in Python to pre-populate a search index - but to my horror, that index was 7.8MB at it’s smallest size.

It turns out that the simplest solution is what I already had: SQLite, but it needed some alterations.

The initial solution required a small Python daemon to run in the background and spit extremely simple results back for a query. Besides the trouble of running another daemon, it needed the copy of the nasg git tree for the templates, a virtualenv for sanic (the HTTP server engine I used), and Jinja2 (templating), and a few other bits.

However, there is a simpler, yet uglier solution. Nearly every webserver out in the wild has PHP support these days, including mine, because I’m still running WordPress for friends and family.

To overcome the problem, I made a Jinja2 template, that creates a PHP file, which read-only reads the SQLite file I pre-populate with the search corpus during build. Unfortunately it’s PHP 7.0, so instead of the FTS5 engine, I had to step back and use the FTS4 - still good enough. Apart from a plain, dead simple PHP engine that has SQLite support, there is no need for anything else, and because the SQLite file is read-only, there’s no lock-collision issue either.

About those markup languages…

YAML can get messy

I went with the most common post format for static sites: YAML metadata + Markdown. Soon I started seeing weird errors with ’ and " characters, so I dug into the YAML specification - don’t do it, it’s a hell dimension. There is a subset of YAML, title StrictYAML17 to address some of these problems, but the short summary is: YAML or not, try to use as simple markup as possible, and be consistent.

title: post title
summary: single-line long summary
published: 2018-08-07T10:00:00+00:00
tags:
- indieweb
syndicate:
- https://something.com/xyz

If one decides to use lists by newline and -, stick to that. No inline [] lists, no spaced - prefix; be consistent.

Same applies for dates and times. While I thought the “correct” date format is ISO 8601, that turned out to be a subset of it, named RFC 333918. Unfortunately I started using +0000 format instead of +00:00 from the beginning, so I’ll stick to that.

Markdown can also get messy

There are valid arguments against Markdown19, so before choosing that as my main format, I tested as many as I could20 - in the end, I decided to stick to an extended version of Markdown, because that is still the closest-to-plain-text for my eyes. I also found Typora, which is a very nice Markdown WYSIWYG editor21. Yes, unfortunately, it’s electron based. I’ll swallow this frog for now.

The “extensions” I use with Markdown:

I’ve tried using the Python Markdown module; the end result was utterly broken HTML when I had code blocks with regexes that collided with the regexes Python Markdown was using. I tried the Python markdown2 module - worked better, didn’t support language tag for code blocks.

In the end, I went back to where I started: Pandoc22. The regeneration of the whole site is ~60 seconds instead of ~20s with markdown2, but it doesn’t really matter - it’s still fast.

pandoc --to=html5 --quiet --no-highlight --from=markdown+footnotes+pipe_tables+strikeout+raw_html+definition_lists+backtick_code_blocks+fenced_code_attributes+lists_without_preceding_blankline+autolink_bare_uris

The take away is the same with YAML: do your own ruleset and stick to it; don’t mix other flavours in.

Syntax highlighting is really messy

Pandoc has a built-in syntax highlighting method; so does the Python Markdown module (via Codehilite).

I have some entries that can break both, and break them bad.

Besides broken, Codehilite is VERBOSE. At a certain point, it managed to add 60KB of HTML markup to my text.

A long while ago I tried to completely eliminate JavaScript from my site, because I’m tired of the current trends. However, JS has it’s place, especially as a progessive enhancement23.

That in mind, I went back to the solution that worked the best so far: prism.js24 The difference this time I that I only add it when there is a code block with language property, and I inline the whole JS block in the code - the ‘developer’ version, supporting a lot of languages, weighs around 58KB, which is a lot, but it works very nice, and it very fast.

No JS only means no syntax highlight, but at least my HTML code is readable, unlike with CodeHilite.

Summary

Static sites come with compromises when it comes to interactions, let that be webmentions, search, pubsub. They need either external services, or some simple, dynamic parts.

If you do go with dynamic, try to keep it as simple as possible. If the webserver has PHP support avoid adding a Python daemon and use that PHP instead.

There are very good, completely free services out there, run by mad scientists enthusiasts, like webmention.io and brid.gy. It’s perfectly fine to use them.

Keep your markup consistent and don’t deviate from the feature set you really need.

JavaScript has it’s place, and prism.js is potentially the nicest syntax highlighter currently available for the web.


  1. https://indieweb.org/scratch_your_own_itch

  2. https://github.com/petermolnar/nasg/

  3. http://indieweb.org/webmention

  4. https://github.com/aaronpk/xray

  5. https://webmention.io/

  6. http://telegraph.p3k.io/

  7. https://indieweb.org/websub

  8. https://superfeedr.com/

  9. http://ifttt.com/

  10. https://brid.gy/about#publishing

  11. https://ifttt.com/applets/83096071d-syndicate-to-wordpress-com

  12. https://ifttt.com/applets/83095945d-syndicate-to-tumblr

  13. https://ifttt.com/applets/83095698d-syndicate-to-brid-gy-twitter-publish

  14. https://ifttt.com/applets/83095735d-syndicate-to-brid-gy-publish-flickr

  15. https://blogtrottr.com/

  16. https://ifttt.com/applets/83095496d-syndicate-to-petermolnarnet-googlegroups-com

  17. http://hitchdev.com/strictyaml/features-removed/

  18. https://en.wikipedia.org/wiki/RFC_3339

  19. https://indieweb.org/markdown#Criticism

  20. https://en.wikipedia.org/wiki/List_of_lightweight_markup_languages

  21. http://typora.io/

  22. http://pandoc.org/MANUAL.html#pandocs-markdown

  23. https://en.wikipedia.org/wiki/Progressive_enhancement

  24. https://prismjs.com/

Tue, 07 Aug 2018 18:33:00 +0100


The three Facebooks

I recently wanted to check the upcoming gigs of a music venue. I tried to pull up their website1, but I couldn’t find their agenda there - turned out it’s sort of an abandoned site, because the hosting company is refusing to respond to any requests.

As a result their gigs are listed on Facebook - at least it can be access without logging in. My current browser setup is a bit complex, but the bottom line is I’m routing my Firefox through my home broadband. I’m used to very fast, unlimited desktop connections these days, both at work and at home, but the throttling I introduced by going through a few loops made some problem visible. When I loaded the Facebook page itself, it took quite a long while, even with noscript and ublock origin, and it made me curious, why.

So I made a fresh Firefox profile and loaded all three versions of Facebook I’m aware of.

Desktop

Visiting the main Facebook site from a regular desktop client gives you the whole, full-blown, unfiltered experience - and the raw madness behind it.

The page executed 26.13 MB Javascript. That is 315x the size of the complete jquery framework, 193x of Bootstrap + Popper + Jquery together.

Facebook in full glory mode
Facebook and it’s Javascript

m.facebook.com

.m is for mobile devices only; without faking my resolution and user agent in Firefox dev tools, I couldn’t get there.

It’s better, but it still had 1.28 MB Javascript in the end. On mobile, that is a serious amount of code to be executed.

Facebook in mobile mode - strictly for mobile only though

mbasic.facebook.com

mbasic is a fascinating thing: it doesn’t have JS at all. It’s like the glorious, old days: ugly, very hard to find anything, but incredibly fast and light.

Facebook in good ol’ days mode

Summary

desktop2 m.3 mbasic.4
Uncompressed everything 36.83 MB 2.22 MB 96.91 KB
Total used bandwidth 9.33 MB 1.01 MB 57.98 KB
JS code to execute 26.13 MB 1.28 MB n/a
JS bandwidth 4.22 MB 364.39 KB n/a
JS compression ratio 6.19x 3.59x 1.67x
CSS to parse 1.34 MB 232.81 KB inline
CSS bandwidth 279.73 KB 53.61 KB inline
CSS compression ratio 4.90x 4.34x -
HTML to parse 2.78 MB 172.06 KB 70.20 KB
HTML bandwith 199.73 KB 37.73 KB 14.20 KB
HTML compression ratio 14.25x 4.56x 4.94x

Conclusions

React is evil. It splits code up into small chunks, and on their own, they seem reasonably sized. However, when there’s a myriad of these, they add up.

The compressed vs uncompressed ratio in desktop JS and HTML indicates extreme amount of repetition.

Most resources are unique, hashed names, and I’m guessing many of them are tied to A/B testing or something similar, so caching won’t solve the issue either.

There’s always a balanced way to do things. A couple of years ago, during the times of backbone.js an underscore.js, that balance was found, and everyone should learn from it.

Many moons ago, in 2012 (when Facebook still had an API), an article was published: The Making of Fastbook: An HTML5 Love Story5. It was a demonstration that the already bloated Facebook app could be replaced with a responsive, small, service worker powered HTML5 website.

Facebook won’t change: it will keep being a monster on every level.

Don’t follow their example.


  1. http://yuk.hu/

  2. https://facebook.com/yukbudapest

  3. https://m.facebook.com/yukbudapest

  4. https://mbasic.facebook.com/yukbudapest

  5. https://www.sencha.com/blog/the-making-of-fastbook-an-html5-love-story/

Thu, 23 Aug 2018 10:45:00 +0100


GPS tracking without a server

Nearly all self-hosted location tracking Android applications are based on server-client architecture: the one on the phone collects only a small points, if not only one, and sends it to a configured server. Traccar1, Owntracks2, etc.

While this setup is useful, it doesn’t fit in my static, unless it hurts3 approach, and it needs data connectivity, which can be tricky during abroad trips. The rare occasions in rural Scotland and Wales tought me data connectivity is not omnipresent at all.

There used to be a magnificent little location tracker, which, besides the server-client approach, could store the location data in CSV and KML files locally: Backitude4. The program is gone from Play store, I have no idea, why, but I have a copy of the last APK of it5.

My flow is the following:

Backitude configuration

These are the modified setting properties:

I have an exported preferences file available7.

Syncthing

The syncthing configuration is optional; it could be simple done by manual transfers from the phone. It’s also not the most simple thing to do, so I’ll let the Syncting Documentation8 take care of describing the how-tos.

Python script

Before jumping to the script, there are 3 Python modules it needs:

pip3 install --user arrow gpxpy requests

And the script itself - please replace the INBASE, OUTBASE, and BINGKEY properties. To get a Bing key, visit Bing9.

import os
import sqlite3
import csv
import glob
import arrow
import re
import gpxpy.gpx
import requests

INBASE="/path/to/your/syncthing/gps/files"
OUTBASE="/path/for/sqlite/and/gpx/output"
BINGKEY="get a bing maps key and insert it here"

def parse(row):
    DATE = re.compile(
        r'^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})T'
        r'(?P<time>[0-9]{2}:[0-9]{2}:[0-9]{2})\.(?P<subsec>[0-9]{3})Z$'
    )

    lat = row[0]
    lon = row[1]
    acc = row[2]
    alt = row[3]
    match = DATE.match(row[4])
    # in theory, arrow should have been able to parse the date, but I couldn't get
    # it working
    epoch = arrow.get("%s-%s-%s %s %s" % (
        match.group('year'),
        match.group('month'),
        match.group('day'),
        match.group('time'),
        match.group('subsec')
    ), 'YYYY-MM-DD hh:mm:ss SSS').timestamp
    return(epoch,lat,lon,alt,acc)

def exists(db, epoch, lat, lon):
    return db.execute('''
        SELECT
            *
        FROM
            data
        WHERE
            epoch = ?
        AND
            latitude = ?
        AND
            longitude = ?
    ''', (epoch, lat, lon)).fetchone()

def ins(db, epoch,lat,lon,alt,acc):
    if exists(db, epoch, lat, lon):
        return
    print('inserting data point with epoch %d' % (epoch))
    db.execute('''INSERT INTO data (epoch, latitude, longitude, altitude, accuracy) VALUES (?,?,?,?,?);''', (
        epoch,
        lat,
        lon,
        alt,
        acc
    ))


if __name__ == '__main__':
    db = sqlite3.connect(os.path.join(OUTBASE, 'location-log.sqlite'))
    db.execute('PRAGMA auto_vacuum = INCREMENTAL;')
    db.execute('PRAGMA journal_mode = MEMORY;')
    db.execute('PRAGMA temp_store = MEMORY;')
    db.execute('PRAGMA locking_mode = NORMAL;')
    db.execute('PRAGMA synchronous = FULL;')
    db.execute('PRAGMA encoding = "UTF-8";')

    files = glob.glob(os.path.join(INBASE, '*.csv'))
    for logfile in files:
        with open(logfile) as csvfile:
            try:
                reader = csv.reader(csvfile)
            except Exception as e:
                print('failed to open CSV reader for file: %s; %s' % (logfile, e))
                continue
            # skip the first row, that's headers
            headers = next(reader, None)
            for row in reader:
                epoch,lat,lon,alt,acc = parse(row)
                ins(db,epoch,lat,lon,alt,acc)
        # there's no need to commit per line, per file should be safe enough
        db.commit()

    db.execute('PRAGMA auto_vacuum;')

    results = db.execute('''
        SELECT
            *
        FROM
            data
        ORDER BY epoch ASC''').fetchall()
    prevdate = None
    gpx = gpxpy.gpx.GPX()

    for epoch, lat, lon, alt, acc in results:
        # in case you know your altitude might actually be valid with negative
        # values you may want to remove the -10
        if alt == 'NULL' or alt < -10:
            url = "http://dev.virtualearth.net/REST/v1/Elevation/List?points=%s,%s&key=%s" % (
                lat,
                lon,
                BINGKEY
            )
            bing = requests.get(url).json()
            # gotta love enterprise API endpoints
            if not bing or \
                'resourceSets' not in bing or \
                not len(bing['resourceSets']) or \
                'resources' not in bing['resourceSets'][0] or \
                not len(bing['resourceSets'][0]) or \
                'elevations' not in bing['resourceSets'][0]['resources'][0] or \
                not bing['resourceSets'][0]['resources'][0]['elevations']:
                alt = 0
            else:
                alt = float(bing['resourceSets'][0]['resources'][0]['elevations'][0])
                print('got altitude from bing: %s for %s,%s' % (alt,lat,lon))
                db.execute('''
                    UPDATE
                        data
                    SET
                        altitude = ?
                    WHERE
                        epoch = ?
                    AND
                        latitude = ?
                    AND
                        longitude = ?
                    LIMIT 1
                ''',(alt, epoch, lat, lon))
                db.commit()
            del(bing)
            del(url)
        date = arrow.get(epoch).format('YYYY-MM-DD')
        if not prevdate or prevdate != date:
            # write previous out
            gpxfile = os.path.join(OUTBASE, "%s.gpx" % (date))
            with open(gpxfile, 'wt') as f:
                f.write(gpx.to_xml())
                print('created file: %s' % gpxfile)

            # create new
            gpx = gpxpy.gpx.GPX()
            prevdate = date

            # Create first track in our GPX:
            gpx_track = gpxpy.gpx.GPXTrack()
            gpx.tracks.append(gpx_track)

            # Create first segment in our GPX track:
            gpx_segment = gpxpy.gpx.GPXTrackSegment()
            gpx_track.segments.append(gpx_segment)

        # Create points:
        gpx_segment.points.append(
            gpxpy.gpx.GPXTrackPoint(
                lat,
                lon,
                elevation=alt,
                time=arrow.get(epoch).datetime
            )
        )

    db.close()

Once this is done, the OUTBASE directory will be populated by .gpx files, one per day.

GpsPrune

GpsPrune is a desktop, QT based GPX track visualizer. It needs data connectivity to have nice maps in the background, but it can do a lot of funky things, including editing GPX tracks.

sudo apt install gpsprune

Keep it in mind that the export script overwrites the GPX files, so the data needs to be fixed in the SQLite database.

This is an example screenshot of GpsPrune, about our 2 day walk down from Mount Emei and it’s endless stairs:

emei

Happy tracking!


  1. https://www.traccar.org/

  2. https://owntracks.org/

  3. https://indieweb.org/manual_until_it_hurts

  4. http://www.gpsies.com/backitude.do

  5. gaugler.backitude.apk

  6. https://syncthing.net/

  7. backitude.prefs

  8. https://docs.syncthing.net/intro/getting-started.html

  9. https://msdn.microsoft.com/en-us/library/ff428642

Thu, 27 Sep 2018 11:05:00 +0100


Page created: Wed, Oct 17, 2018 - 09:00 AM GMT