Scrapy is a scrappy little web scraper you can use to scrape web content using CSS or XPath selectors and even Regex. Using CSS selectors is probably the easiest for people who are familiar with the web, but the hardcore user can even mix-n-match.

We can open a shell, a familiar technique to Python programmers, in order to test our scraping before unleashing it on the world.
$ scrapy shell https://mikeward.net
This opens a shell and the HTTP request and response are now available to us, as you can see below.

Read More

Health Information

Installing apps that “help” you with medical conditions can lead to privacy disasters. Your personal information gets shared with the world, and suddenly advertisements for your health situation follow you around the Internet. According to research firm Twinword, an estimated 83% of users searched online for health or medical information. If the average Internet user were to forget their medical issues, they could probably remember by spending an hour on the web watching advertisements.

Installing apps to help you with a pregnancy launches an avalanche of data sharing and advertising that almost defies belief. In the US there are laws that govern the use, sharing and storage of healthcare data, but it is a complex, nuanced issue.

Read More

Getting Started

Install pass – use the package manager for your distro.

$ sudo apt install pass
$ sudo yum install pass
$ sudo pacman -S pass

Let’s try it out.

$ pass
Error: password store is empty. Try "pass init".

$ pass init
Usage: pass init [--path=subfolder,-p subfolder] gpg-id...

We need a GPG key. Your distro should come with GNU Privacy Guard (GPG) installed already; if not install it now.

Read More
cybercorn contest

I made a really difficult contest in the Bitcorns idle farming game. Actually I made three, and the one designed for hackers was recently solved, giving the winner the access to the private key. What this means in practical terms is that they could take the Bitcoin and the twenty five nice Counterparty assets at that address that I’m calling a farm. Cornfused? I’ll explain how this works, and in the process hopefully impress upon you just how insanely difficult this challenge really was.

The Cybercorn card itself is a token on the Bitcoin blockchain, only 32 were issued, it has clues hidden inside it, and it’s a card in the Bitcorns farming game. Not the reduced size one shown here mind you, the real one (shown below on this page) whose SHA256 hash was recorded upon issuance and used as part of one of the clues. That card image is an animated GIF, and when the corns fall down from the tree a clue is revealed on the empty branches.

Read More

When people ask me about metadata and they’re not asking about a phone app, I default to saying exiftool because it’s fabulous and it’s the tool that I like. But it’s rarely a good answer for most people who want simpler things. So I looked at MAT2, the successor to Metadata Anonymisation Toolkit (MAT).

It is a python script with only a couple of options, but supports a LOT of file formats, including most office document formats, image formats, and many audio and video formats.

Read More

Paper wallets are the safest way to store cryptocurrency.

That’s a generalization, and of course the “safest” way to store tokens varies depending both upon your circumstances and what you consider safe. But generally peaking, depite numerous debates that will undoubtedly continue, the fact remains this is the most secure method for storing cryptocurrencies. That’s because you minimize the electronic attack surface, reducing the defense to the physical realm.

There are dangers and pitfalls with paper wallets to be fair, but they are entirely avoidable. That’s what this post is all about – debunking the popular notion that paper wallets are a good choice in theory, but not in practice! Let’s see how it goes, and please – to the makers of hardware wallets, I use your products and love them, it’s not personal.

Read More

Bitcorn is a game based on the Counterparty platform, where Bitcorn farmers hold CROPS that get harvested seasonally. Yes it’s a game, involves cryptocurrency, requires patience, and best of all is totally useless. Trust me, just read a little more before dismissing it out of hand – you may like it.

TL;DR – Hodl CROPS tokens and get Bitcorns airdrops four harvests a year. Like I said, this is for patient farmers. The game runs at least until 2022, when some final prizes will be awarded to farms and coops. Read on to make sense out of all this bitcorn farming and how to get started with your very own Bitcorn farm.

Read More

The Simple Mail Transfer Protocol (SMTP) has survived to this day due to widespread use, but remains a sore spot for privacy advocates and a nightmare for security professionals. Email was the first real application on the Internet, and remained the most popular app in the 1990s, even after the web became popular.

In the late 1970s when the Internet was still being developed, people read and sent email by logging in to a central computer on a console and used a text-based email program. This basic technique of sending text messages and reading them using simple email programs continued to be the primary method of communicating online through the 1990s. But now, with yearly email messages sent somewhere on the order of one hundred trillion, it has grown into an unrecognizable beast.

Read More

You’ve probably heard of two factor authentication, commonly referred to as two factor auth, or simply 2FA. It literally means using two different things to authenticate. Authentication is proving that you are who you claim to be, and providing two distinct forms of credentials is highly recommended. Not all 2FA varieties are equal however, so we’re going to examine this in a bit more detail.

The bottom line is that when 2FA is available, you should try to use it because it provides much stronger security. You might not realize it, but you are already familiar with common forms of two factor authentication. Perhaps you work in a secure area and use both a badge and a passcode to gain access. The badge is one factor, and the passcode is a second factor. This illustrates the main point of 2FA – it is not sufficient to simply have the physical badge, which might be stolen – an attacker would need to know the passcode you carry in your head as well.

Read More

We focus almost exclusively on software that runs locally on linux because of all the great software that is available. But in this post we’ll take a look at a couple of web based reources for doing some pretty effective image forensics.

We test drove these sites using JPEG and PNG image files with content hidden inside. Our PNG test image had another image hidden inside using LSB steganography – using the least significant bits. We used two JPEG images with the same fairly large image hidden inside, one using the Matroschka program and the other using Steghide.

Read More

Did Google do something amazing, again? Should we even care? Is this the beginning of the end? Do we need to worry about quantum computers breaking our encryption? For the answers to these and a down-to-earth explanation of what it all means, please read on.

First of all we don’t know if Google did indeed achieve quantum supremacy. We won’t know for sure for some time to come. TL;DR – if they did, it really doesn’t change anything that will affect online commerce, cryptocurrency, encrypted communications or anything people are worried about. This is all about proving whether or not there are things can be computed with these machines once we get better at building them that cannot be computed by quantum simulators – a.k.a. conventional computers. That’s the short answer, a slightly longer one follows.

Read More

Let’s now take a look at using cryptographic hashing functions in Python programs. We’ll mainly focus on the hashlib module, that provides all the common hashing functions we’re likely to need. To cover almost every use case you’ll only actually need a handful of these despite your interpreter probably supporting more.

To see the hashing functions guaranteed to be supported on all platforms:
>>> hashlib.algorithms_guaranteed
{'sha512', 'md5', 'blake2s', 'sha256', 'sha224', 'sha3_256', 'shake_256', 'sha1', 'shake_128', 'sha3_512', 'sha3_224', 'sha3_384', 'sha384', 'blake2b'}

Read More