Entries found: 140

An LLM TDD loop

Published: Sun, 06 Oct 2024 14:48:50 +0100
Updated: Sun, 06 Oct 2024 14:48:50 +0100
UTC: 2024-10-06 13:48:50+00:00
URL: https://codeinthehole.com/tips/llm-tdd-loop-script/

This post details a Bash script that runs a TDD loop where an LLM is iteratively used to write Python application code to make a fixed set of tests pass. Similar things have been done before1; this script merely demonstrates that such things can be done simply by stitching together open-source LLM tools. Tooling The script uses two commands from Simon Willison: llm — a command line interface for interacting with Large Language Models (LLMs).

Content Preview

This post details a Bash script that runs a TDD loop where an LLM is iteratively used to write Python application code to make a fixed set of tests pass. Similar things have been done before1; this script merely demonstrates that such things can be done simply by stitching together open-source LLM tools. Tooling The script uses two commands from Simon Willison: llm — a command line interface for interacting with Large Language Models (LLMs).

Writing Markdown in Vim

Published: Tue, 06 Jun 2023 00:00:00 +0000
Updated: Tue, 06 Jun 2023 00:00:00 +0000
UTC: 2023-06-06 00:00:00+00:00
URL: https://codeinthehole.com/tips/writing-markdown-in-vim/

I write a lot of Markdown in Vim and have spent considerable energy configuring it to my liking. This post details how I configure Vim1 for writing markdown. It is merely a reference that I can refer others to. File-type settings In ~/.vim/after/ftplugin/markdown.vim (or equivalent), set some buffer-local settings: " Use Vim's spell checker setlocal spell " No line numbers setlocal nonumber Configure K to look up the word under the cursor in a dictionary:

Content Preview

I write a lot of Markdown in Vim and have spent considerable energy configuring it to my liking. This post details how I configure Vim1 for writing markdown. It is merely a reference that I can refer others to. File-type settings In ~/.vim/after/ftplugin/markdown.vim (or equivalent), set some buffer-local settings: " Use Vim's spell checker setlocal spell " No line numbers setlocal nonumber Configure K to look up the word under the cursor in a dictionary:

Tips for using Github Copilot in Vim

Published: Fri, 06 Jan 2023 12:07:03 +0000
Updated: Fri, 06 Jan 2023 12:07:03 +0000
UTC: 2023-01-06 12:07:03+00:00
URL: https://codeinthehole.com/tips/vim-and-github-copilot/

Based on a few weeks of using vim-copilot, I recommend the following: Enable Copilot for the gitcommit, markdown and yaml filetypes: let g:copilot_filetypes = { \ 'gitcommit': v:true, \ 'markdown': v:true, \ 'yaml': v:true \ } By default, these and few others are disabled but I’ve found them to be useful. It’s often amusing to see Copilot’s attempts to complete your commit messages. Disable Copilot for large files as it can be slow and impair the editing experience:

Content Preview

Based on a few weeks of using vim-copilot, I recommend the following: Enable Copilot for the gitcommit, markdown and yaml filetypes: let g:copilot_filetypes = { \ 'gitcommit': v:true, \ 'markdown': v:true, \ 'yaml': v:true \ } By default, these and few others are disabled but I’ve found them to be useful. It’s often amusing to see Copilot’s attempts to complete your commit messages. Disable Copilot for large files as it can be slow and impair the editing experience:

Creating pull requests with GPT3 and random artwork

Published: Thu, 22 Dec 2022 21:42:35 +0000
Updated: Thu, 22 Dec 2022 21:42:35 +0000
UTC: 2022-12-22 21:42:35+00:00
URL: https://codeinthehole.com/projects/pull-requests-with-gpt3-and-random-artwork/

A friend of mine has been retweeting great paintings from Twitter accounts like @HenryRothwell, which I’ve greatly enjoyed. E.g. Good morning - I hope you slept like a caddis-fly larvae-stuffed miller's thumb. I'm starting with 'Winter Landscape', Valerius de Saedeleer, oil on canvas, 1930s. pic.twitter.com/hk6RUFh794 — Henry Rothwell (@HenryRothwell) December 18, 2022 In a similar vein, I’ve started embedding painting images in the description of every pull request I open.

Content Preview

A friend of mine has been retweeting great paintings from Twitter accounts like @HenryRothwell, which I’ve greatly enjoyed. E.g. Good morning - I hope you slept like a caddis-fly larvae-stuffed miller's thumb. I'm starting with 'Winter Landscape', Valerius de Saedeleer, oil on canvas, 1930s. pic.twitter.com/hk6RUFh794 — Henry Rothwell (@HenryRothwell) December 18, 2022 In a similar vein, I’ve started embedding painting images in the description of every pull request I open.

New project: Food price scraper

Published: Sat, 12 Nov 2022 17:51:38 +0000
Updated: Sat, 12 Nov 2022 17:51:38 +0000
UTC: 2022-11-12 17:51:38+00:00
URL: https://codeinthehole.com/projects/food-scraper/

I’ve created a deeply middle-class Git scraper project which tracks the prices of a basket of goods sold by the British online supermarket, Ocado. For example, Lurpak butter: I’ve been looked for an excuse to use Git scraping for ages, and this idea came up as my wife and I were commiserating over how much food prices are increasing at the moment. The project is the codeinthehole/food-scraper repo — see, in particular, the product overview file.

Content Preview

I’ve created a deeply middle-class Git scraper project which tracks the prices of a basket of goods sold by the British online supermarket, Ocado. For example, Lurpak butter: I’ve been looked for an excuse to use Git scraping for ages, and this idea came up as my wife and I were commiserating over how much food prices are increasing at the moment. The project is the codeinthehole/food-scraper repo — see, in particular, the product overview file.

OpenAI Git poetry

Published: Fri, 11 Nov 2022 22:07:17 +0000
Updated: Fri, 11 Nov 2022 22:07:17 +0000
UTC: 2022-11-11 22:07:17+00:00
URL: https://codeinthehole.com/tidbits/openai-git-poetry/

OpenAI provides a REST API where you can generate prompt completions. Here’s a minimal example where a JSON payload is piped into httpie: $ export OPENAI_API_KEY="..." # fill in your API key here $ echo '{"model": "text-davinci-002", "prompt": "Write a poem about cheese"}' \ | http https://api.openai.com/v1/completions Authorization:"Bearer $OPENAI_API_KEY" \ | jq -r '.choices[0].text There's nothing quite like cheese It's rich, creamy, and delicious. Quite. We can extend this by dynamically building the prompt string.

Content Preview

OpenAI provides a REST API where you can generate prompt completions. Here’s a minimal example where a JSON payload is piped into httpie: $ export OPENAI_API_KEY="..." # fill in your API key here $ echo '{"model": "text-davinci-002", "prompt": "Write a poem about cheese"}' \ | http https://api.openai.com/v1/completions Authorization:"Bearer $OPENAI_API_KEY" \ | jq -r '.choices[0].text There's nothing quite like cheese It's rich, creamy, and delicious. Quite. We can extend this by dynamically building the prompt string.

"It was the system. We can't change it."

Published: Sun, 30 Oct 2022 15:31:23 +0000
Updated: Sun, 30 Oct 2022 15:31:23 +0000
UTC: 2022-10-30 15:31:23+00:00
URL: https://codeinthehole.com/tidbits/the-bone-clocks/

From the final novella of David Mitchell’s The Bone Clocks, set in Ireland in 2043: ‘Number one is to survive’, answers Hood, watching the men on the roof. ‘They’re all dead, like my parents. They had a better life than I did, mind. So did you. Your power stations, your cars, your creature comforts. Well, you lived too long. The bill’s due. Today,’ up on the roof the bold is cut on the first panel, ‘you start to pay.

Content Preview

From the final novella of David Mitchell’s The Bone Clocks, set in Ireland in 2043: ‘Number one is to survive’, answers Hood, watching the men on the roof. ‘They’re all dead, like my parents. They had a better life than I did, mind. So did you. Your power stations, your cars, your creature comforts. Well, you lived too long. The bill’s due. Today,’ up on the roof the bold is cut on the first panel, ‘you start to pay.

Auditing 1Password activity

Published: Wed, 29 Jun 2022 21:00:01 +0000
Updated: Wed, 29 Jun 2022 21:00:01 +0000
UTC: 2022-06-29 21:00:01+00:00
URL: https://codeinthehole.com/tips/auditing-1password-activity/

As a 1Password admin, there are common audit questions you need to answer around who created various resources. The web dashboard is excellent but some questions are still fiddly to answer. Questions like: Who created this group? Who created this vault? Who created this item? Who has access to this item? This post is a note-to-self on how to find these answers1. Who created this group? If the group hasn’t had much activity, you can see the creator in the activity timeline section of the group detail page in the web dashboard.

Content Preview

As a 1Password admin, there are common audit questions you need to answer around who created various resources. The web dashboard is excellent but some questions are still fiddly to answer. Questions like: Who created this group? Who created this vault? Who created this item? Who has access to this item? This post is a note-to-self on how to find these answers1. Who created this group? If the group hasn’t had much activity, you can see the creator in the activity timeline section of the group detail page in the web dashboard.

Flakey tests in a large Python/Django codebase

Published: Wed, 08 Jun 2022 09:45:59 +0100
Updated: Wed, 08 Jun 2022 09:45:59 +0100
UTC: 2022-06-08 08:45:59+00:00
URL: https://codeinthehole.com/news/oe-tech-flakey-tests/

For the record, I wrote up some patterns that cause flakey tests on the Kraken Technologies tech blog.

Content Preview

For the record, I wrote up some patterns that cause flakey tests on the Kraken Technologies tech blog.

Explaining

Published: Thu, 14 Oct 2021 16:29:42 +0100
Updated: Thu, 14 Oct 2021 16:29:42 +0100
UTC: 2021-10-14 15:29:42+00:00
URL: https://codeinthehole.com/tips/explaining/

Someone has asked a question (in Slack or Github) and you’re about to write an explanation. But before you start typing, ask yourself this: Is this the best place to answer this question? Because the place where the question is asked is generally not the best place to write a detailed answer. “Why did you do it that way?” Imagine you’ve requested a review on a Github pull request and one of the reviewers has questioned why something is done the way it is.

Content Preview

Someone has asked a question (in Slack or Github) and you’re about to write an explanation. But before you start typing, ask yourself this: Is this the best place to answer this question? Because the place where the question is asked is generally not the best place to write a detailed answer. “Why did you do it that way?” Imagine you’ve requested a review on a Github pull request and one of the reviewers has questioned why something is done the way it is.

Vim's useful lists

Published: Mon, 03 May 2021 09:27:17 +0100
Updated: Mon, 03 May 2021 09:27:17 +0100
UTC: 2021-05-03 08:27:17+00:00
URL: https://codeinthehole.com/tips/vim-lists/

A good understanding of Vim’s various lists is a massive productivity boost — it’s taken me many years of Vim use to truly appreciate this

Content Preview

A good understanding of Vim’s various lists is a massive productivity boost — it’s taken me many years of Vim use to truly appreciate this. This post summarises some of Vim’s lists, detailing their purpose and how to make the most of them. Contents: Quickfix list Tip: Use mappings for faster browsing Tip: Define a mapping to :grep for the word under the cursor Tip: Custom grepprg programs Tip: Custom makeprg programs Tip: Operating on the error list Tip: Populate the quickfix list in a subshell Location list Jump list Tip: Use jump commands to navigate Change list Tip: Jump to the previous location where insert mode was used Buffer list Tip: Select buffers with FZF Argument list Tag match list Tip: Use Universal Ctags Tip: Use :tjump as your default “jump to tag” command Tip: FZF also has a tag searching functionality Include file list and definition lists Summary I need to jump somewhere I need to apply the same transformation in lots of places Quickfix list A global list of locations populated via one of these commands:

A Vim mapping for opening virtualenv files

Published: Mon, 16 Nov 2020 15:45:30 +0000
Updated: Mon, 16 Nov 2020 15:45:30 +0000
UTC: 2020-11-16 15:45:30+00:00
URL: https://codeinthehole.com/tips/a-vim-mapping-for-opening-virtualenv-files/

Here’s a useful command-mode mapping for Python development: " ~/.vim/ftplugin/python.vim function! VirtualEnvSitePackagesFolder() " Try a few candidate Pythons to see which this virtualenv uses. for python in ["python3.7", "python3.8", "python3.9"] let candidate = $VIRTUAL_ENV . "/lib/" . python if isdirectory(candidate) return candidate . "/site-packages/" endif endfor return "" endfunction cnoremap %v <C-R>=VirtualEnvSitePackagesFolder()<cr> In command-mode, %v will expand to the path of your virtualenv’s site-packages folder in, which makes it easy to quickly open a third-party module.

Content Preview

Here’s a useful command-mode mapping for Python development: " ~/.vim/ftplugin/python.vim function! VirtualEnvSitePackagesFolder() " Try a few candidate Pythons to see which this virtualenv uses. for python in ["python3.7", "python3.8", "python3.9"] let candidate = $VIRTUAL_ENV . "/lib/" . python if isdirectory(candidate) return candidate . "/site-packages/" endif endfor return "" endfunction cnoremap %v <C-R>=VirtualEnvSitePackagesFolder()<cr> In command-mode, %v will expand to the path of your virtualenv’s site-packages folder in, which makes it easy to quickly open a third-party module.

Setting up a 2020 MacBook Pro for Python development

Published: Sun, 19 Jul 2020 00:00:00 +0100
Updated: Sun, 19 Jul 2020 00:00:00 +0100
UTC: 2020-07-18 23:00:00+00:00
URL: https://codeinthehole.com/guides/settings-up-a-2020-macbook-for-python-development/

This is a note-to-self on setting up a 2020 13-inch MacBook Pro, largely for Python development. I imagine it will all be out-of-date by the time I set-up my next laptop but it’s possible it might be useful to someone in the meantime. Applications I don’t tend to use Homebrew Cask and suffer the indignity of either installing them from the App Store or downloading .dmg files and dragging them into /Applications.

Content Preview

This is a note-to-self on setting up a 2020 13-inch MacBook Pro, largely for Python development. I imagine it will all be out-of-date by the time I set-up my next laptop but it’s possible it might be useful to someone in the meantime. Applications I don’t tend to use Homebrew Cask and suffer the indignity of either installing them from the App Store or downloading .dmg files and dragging them into /Applications.

Conditional nested blocks in Terraform

Published: Tue, 14 Jul 2020 22:05:23 +0100
Updated: Tue, 14 Jul 2020 22:05:23 +0100
UTC: 2020-07-14 21:05:23+00:00
URL: https://codeinthehole.com/tips/conditional-nested-blocks-in-terraform/

Here’s a useful technique for using Terraform’s dynamic blocks to create conditional nested blocks. Maintenance mode As an example, let’s create a “maintenance mode” for a service which allows a “under maintenance” holding page to be served when a Terraform variable is set. This is useful if you need to stop all traffic to a RDS database server so it can be upgraded. Define a boolean maintenance_mode variable: # variables.

Content Preview

Here’s a useful technique for using Terraform’s dynamic blocks to create conditional nested blocks. Maintenance mode As an example, let’s create a “maintenance mode” for a service which allows a “under maintenance” holding page to be served when a Terraform variable is set. This is useful if you need to stop all traffic to a RDS database server so it can be upgraded. Define a boolean maintenance_mode variable: # variables.

Maintainable Terraform CIDR lists

Published: Wed, 08 Jul 2020 00:00:00 +0000
Updated: Wed, 08 Jul 2020 00:00:00 +0000
UTC: 2020-07-08 00:00:00+00:00
URL: https://codeinthehole.com/tips/terraform-cidrs/

Problem Your Terraform config requires managing many CIDRs that control firewall ingress rules. You’ve been storing these in a CSV string: variable "client_cidrs" { default="50.1.1.1/32,44.2.1.0/32", } which is fed to a aws_security_group somewhere in your configuration. The CIDRs change frequently and maintaining this variable is difficult as it’s hard to track where each individual CIDR came from. Solution Use a HCL list variable which allows each entry to have an associated comment explaining what the CIDR corresponds to:

Content Preview

Problem Your Terraform config requires managing many CIDRs that control firewall ingress rules. You’ve been storing these in a CSV string: variable "client_cidrs" { default="50.1.1.1/32,44.2.1.0/32", } which is fed to a aws_security_group somewhere in your configuration. The CIDRs change frequently and maintaining this variable is difficult as it’s hard to track where each individual CIDR came from. Solution Use a HCL list variable which allows each entry to have an associated comment explaining what the CIDR corresponds to:

Easy Github URLs from Vim

Published: Tue, 23 Jun 2020 15:49:58 +0100
Updated: Tue, 23 Jun 2020 15:49:58 +0100
UTC: 2020-06-23 14:49:58+00:00
URL: https://codeinthehole.com/tips/easy-github-urls-from-vim/

URLs are great aren’t they? You include them in your Slack messages and your co-workers can see exactly what you’re talking about in a single click. I wish people would use them more (and design apps that support them properly). Anyway, a super-useful Vim mapping I use is: vnoremap <leader>gb :GBrowse! master:%<cr> which, after visually selecting a block of code, grabs its Github URL from the HEAD of master and copies it to the clipboard.

Content Preview

URLs are great aren’t they? You include them in your Slack messages and your co-workers can see exactly what you’re talking about in a single click. I wish people would use them more (and design apps that support them properly). Anyway, a super-useful Vim mapping I use is: vnoremap <leader>gb :GBrowse! master:%<cr> which, after visually selecting a block of code, grabs its Github URL from the HEAD of master and copies it to the clipboard.

Software development tips – part 2

Published: Fri, 24 Apr 2020 10:04:16 +0000
Updated: Fri, 24 Apr 2020 10:04:16 +0000
UTC: 2020-04-24 10:04:16+00:00
URL: https://codeinthehole.com/tips/software-development-tips-part2/

On code smells: If your codebase is tightly coupled to data in your database (ie the codebase has a data value hard-coded), it is a sign you should extract that data from your database into a code-layer model. Never use numbered variable names (eg account1 = ...) – there’s always a better way. On good things I always tell team members about: The writing and conference talks by James Mickens are great - start with “Computers are a sadness, I am the cure” from Monitorama 2014.

Content Preview

On code smells: If your codebase is tightly coupled to data in your database (ie the codebase has a data value hard-coded), it is a sign you should extract that data from your database into a code-layer model. Never use numbered variable names (eg account1 = ...) – there’s always a better way. On good things I always tell team members about: The writing and conference talks by James Mickens are great - start with “Computers are a sadness, I am the cure” from Monitorama 2014.

Resolving conflicts during a Git rebase

Published: Thu, 05 Mar 2020 00:00:00 +0000
Updated: Thu, 05 Mar 2020 00:00:00 +0000
UTC: 2020-03-05 00:00:00+00:00
URL: https://codeinthehole.com/guides/resolving-conflicts-during-a-git-rebase/

Resolving conflicts from a Git rebase can be tricky. But don’t worry – here’s a comprehensive guide to how to resolve them. There’s three phases: Which commit of mine is conflicting? What changes were made in the target branch that conflict with my commit? Resolve conflicts safely These are accurate as of Git v2.23 and are for resolving conflicts using the command line. Let’s walk through the process. A conflict happens On your working branch, you run:

Content Preview

Resolving conflicts from a Git rebase can be tricky. But don’t worry – here’s a comprehensive guide to how to resolve them. There’s three phases: Which commit of mine is conflicting? What changes were made in the target branch that conflict with my commit? Resolve conflicts safely These are accurate as of Git v2.23 and are for resolving conflicts using the command line. Let’s walk through the process. A conflict happens On your working branch, you run:

Software development tips – part 1

Published: Wed, 26 Feb 2020 10:04:16 +0000
Updated: Wed, 26 Feb 2020 10:04:16 +0000
UTC: 2020-02-26 10:04:16+00:00
URL: https://codeinthehole.com/tips/software-development-tips-part1/

On software development: Everything you create that has a name lives in a namespace. Remember this. Ensure the names you pick are unique and unambiguous within their namespace. If a 500 Internal Server Error HTTP response can be induced from your web app through a carefully crafted request, it needs fixing. Don’t assume anything about the incoming request. On tools: Using Alfred with a well-stocked cupboard of Chrome bookmarks is a massive productivity win.

Content Preview

On software development: Everything you create that has a name lives in a namespace. Remember this. Ensure the names you pick are unique and unambiguous within their namespace. If a 500 Internal Server Error HTTP response can be induced from your web app through a carefully crafted request, it needs fixing. Don’t assume anything about the incoming request. On tools: Using Alfred with a well-stocked cupboard of Chrome bookmarks is a massive productivity win.

Vim text-objects for Python development

Published: Thu, 13 Jun 2019 00:00:00 +0000
Updated: Thu, 13 Jun 2019 00:00:00 +0000
UTC: 2019-06-13 00:00:00+00:00
URL: https://codeinthehole.com/tips/vim-text-objects/

Text objects, as in the iw from ciw (“change inner word”), form an important part of your Vim mentalese1. This post details those that I find most useful for Python and Django development. For brevity, the leading a (mnemonic: “a"n) or i (mnemonic: “inner”), that you combine with the following commands to form the full text-object, are omitted. From core Vim: w → a word W → a WORD t → a HTML/XML tag s → a sentence p → a paragraph From the (highly recommended) wellle/targets.

Content Preview

Text objects, as in the iw from ciw (“change inner word”), form an important part of your Vim mentalese1. This post details those that I find most useful for Python and Django development. For brevity, the leading a (mnemonic: “a"n) or i (mnemonic: “inner”), that you combine with the following commands to form the full text-object, are omitted. From core Vim: w → a word W → a WORD t → a HTML/XML tag s → a sentence p → a paragraph From the (highly recommended) wellle/targets.

Debugging Vim by example

Published: Thu, 28 Mar 2019 00:00:00 +0000
Updated: Thu, 28 Mar 2019 00:00:00 +0000
UTC: 2019-03-28 00:00:00+00:00
URL: https://codeinthehole.com/tips/debugging-vim-by-example/

There’s only so far you can get by cargo-culting other people’s ~/.vim folders. An important next step is understanding how to debug Vim; knowing what to do when it’s slow or misbehaving. Learn how to scratch things that itch. This post illustrates a range of debugging and profiling approaches for Vim by walking through real issues I’ve recently investigated, diagnosed and resolved. There’s very little to copy into your ~/.

Content Preview

There’s only so far you can get by cargo-culting other people’s ~/.vim folders. An important next step is understanding how to debug Vim; knowing what to do when it’s slow or misbehaving. Learn how to scratch things that itch. This post illustrates a range of debugging and profiling approaches for Vim by walking through real issues I’ve recently investigated, diagnosed and resolved. There’s very little to copy into your ~/.

Beware of changing the 'related name' of a Django model field

Published: Mon, 18 Mar 2019 17:05:53 +0000
Updated: Mon, 18 Mar 2019 17:05:53 +0000
UTC: 2019-03-18 17:05:53+00:00
URL: https://codeinthehole.com/news/oe-related-name-outage/

I wrote up a lesson on the dangers of renaming the related_name of a Django model field over on the Octopus Energy tech blog.

Content Preview

I wrote up a lesson on the dangers of renaming the related_name of a Django model field over on the Octopus Energy tech blog.

Using black and isort with Vim

Published: Fri, 08 Mar 2019 09:22:04 +0000
Updated: Fri, 08 Mar 2019 09:22:04 +0000
UTC: 2019-03-08 09:22:04+00:00
URL: https://codeinthehole.com/tips/using-black-and-isort-with-vim/

FYI, the easiest way to get Vim to automatically run black and isort over a Python buffer when saving is to use Ale’s fixer functionality. " In ~/.vim/after/ftplugin/python.vim (or somewhere like that) let b:ale_fixers = ['black', 'isort'] let b:ale_fix_on_save = 1 If you’re only using black/isort in a subset of your projects, you can enable the b:ale_fix_on_save setting conditionally: let b:ale_fix_on_save = 0 let filepath = expand('%:p:h') if match(filepath, 'some-project-name') !

Content Preview

FYI, the easiest way to get Vim to automatically run black and isort over a Python buffer when saving is to use Ale’s fixer functionality. " In ~/.vim/after/ftplugin/python.vim (or somewhere like that) let b:ale_fixers = ['black', 'isort'] let b:ale_fix_on_save = 1 If you’re only using black/isort in a subset of your projects, you can enable the b:ale_fix_on_save setting conditionally: let b:ale_fix_on_save = 0 let filepath = expand('%:p:h') if match(filepath, 'some-project-name') !

Avoiding package lock-out when provisioning Ubuntu 18.04 machines

Published: Wed, 06 Mar 2019 15:49:53 +0000
Updated: Wed, 06 Mar 2019 15:49:53 +0000
UTC: 2019-03-06 15:49:53+00:00
URL: https://codeinthehole.com/tips/avoiding-package-lockout-in-ubuntu-1804/

When provisioning a virtual machine running Ubuntu 16.04 or later, a common problem if being unable to install packages since another process is holding a lock (eg on /var/lib/dpkg/lock-frontend). This happens as Ubuntu VMs typically start several package-management programs unattended-upgrades and its associated apt.daily service — on boot, and these will block your provisioning scripts. You’ll see an error like this: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Content Preview

When provisioning a virtual machine running Ubuntu 16.04 or later, a common problem if being unable to install packages since another process is holding a lock (eg on /var/lib/dpkg/lock-frontend). This happens as Ubuntu VMs typically start several package-management programs unattended-upgrades and its associated apt.daily service — on boot, and these will block your provisioning scripts. You’ll see an error like this: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Installing the latest RabbitMQ on Ubuntu 18.04

Published: Thu, 21 Feb 2019 12:51:47 +0000
Updated: Thu, 21 Feb 2019 12:51:47 +0000
UTC: 2019-02-21 12:51:47+00:00
URL: https://codeinthehole.com/tips/install-latest-rabbitmq-on-ubuntu-1804/

I wasted a morning trying to install RabbitMQ v3.7.12 (the latest version as of Feb 2019) on an Ubuntu 18.04 machine using Puppet. This as tricky as: Only RabbitMQ version 3.6.10 is available from the default repositories; Getting Puppet to install packages from custom locations can be painful. Solution Use these Puppet modules in your Puppetfile: mod 'computology-packagecloud', '0.3.2' mod 'garethr-erlang', '0.3.0' mod 'puppet-rabbitmq', "9.0.0" and something like this in your manifest code:

Content Preview

I wasted a morning trying to install RabbitMQ v3.7.12 (the latest version as of Feb 2019) on an Ubuntu 18.04 machine using Puppet. This as tricky as: Only RabbitMQ version 3.6.10 is available from the default repositories; Getting Puppet to install packages from custom locations can be painful. Solution Use these Puppet modules in your Puppetfile: mod 'computology-packagecloud', '0.3.2' mod 'garethr-erlang', '0.3.0' mod 'puppet-rabbitmq', "9.0.0" and something like this in your manifest code:

Easy to change

Published: Mon, 24 Dec 2018 11:51:36 +0000
Updated: Mon, 24 Dec 2018 11:51:36 +0000
UTC: 2018-12-24 11:51:36+00:00
URL: https://codeinthehole.com/tips/easy-to-change/

I care about writing maintainable software1: code that is a pleasure to work with in the long-term; where new requirements can be accommodated smoothly and swiftly – above all, software that is easy to change. Memorise that phrase. I feel we lose sight of this overarching guiding principle and harm our codebases by dogmatically pursuing well-intentioned but proximate goals. The main thing the future maintainer of your code (i.e. you) cares about is whether the code is easy to change2.

Content Preview

I care about writing maintainable software1: code that is a pleasure to work with in the long-term; where new requirements can be accommodated smoothly and swiftly – above all, software that is easy to change. Memorise that phrase. I feel we lose sight of this overarching guiding principle and harm our codebases by dogmatically pursuing well-intentioned but proximate goals. The main thing the future maintainer of your code (i.e. you) cares about is whether the code is easy to change2.

Listing groups in G-Suite

Published: Sat, 27 Oct 2018 12:53:24 +0100
Updated: Sat, 27 Oct 2018 12:53:24 +0100
UTC: 2018-10-27 11:53:24+00:00
URL: https://codeinthehole.com/tips/listing-google-groups/

Oddly, you can’t pull a report of all groups from G-Suite like you can for users. The only option is to use the API. Here’s how. Follow steps 1 and 2 from the quickstart guide but instead of the sample Python script, use this: from httplib2 import Http from googleapiclient.discovery import build from oauth2client import file, client, tools def main(): _print_all_groups() def _print_all_groups(): for group in _all_groups(): print(group['email']) def _all_groups(): """ Return a list of all group dicts.

Content Preview

Oddly, you can’t pull a report of all groups from G-Suite like you can for users. The only option is to use the API. Here’s how. Follow steps 1 and 2 from the quickstart guide but instead of the sample Python script, use this: from httplib2 import Http from googleapiclient.discovery import build from oauth2client import file, client, tools def main(): _print_all_groups() def _print_all_groups(): for group in _all_groups(): print(group['email']) def _all_groups(): """ Return a list of all group dicts.

Advanced pull-request crafting

Published: Sun, 03 Jun 2018 12:30:05 +0100
Updated: Sun, 03 Jun 2018 12:30:05 +0100
UTC: 2018-06-03 11:30:05+00:00
URL: https://codeinthehole.com/tips/advanced-pull-request-crafting/

I spend most of my day reviewing pull requests in Github. These are my working notes on what makes a good PR. Purpose? It should be clear to the reviewer what change is being made and, crucially, why. So ensure your title and description convey the purpose of the PR. Consider including: Screenshots — such as snaps or gifs of a new UI, or graphs of the devastating performance improvement your PR delivers.

Content Preview

I spend most of my day reviewing pull requests in Github. These are my working notes on what makes a good PR. Purpose? It should be clear to the reviewer what change is being made and, crucially, why. So ensure your title and description convey the purpose of the PR. Consider including: Screenshots — such as snaps or gifs of a new UI, or graphs of the devastating performance improvement your PR delivers.

Shortcuts of the old and minimalist

Published: Mon, 27 Nov 2017 12:27:45 +0100
Updated: Mon, 27 Nov 2017 12:27:45 +0100
UTC: 2017-11-27 11:27:45+00:00
URL: https://codeinthehole.com/tips/shortcuts-of-the-old-and-minimalist/

As I get older and grumpier, I increasingly value clean, uncluttered working environments. I’m sure I’m not the only one, so here’s a few useful practices and shortcuts that help me avoid using the mouse and satisfy my need for productivity micro-optimisation. They are mainly for macOS users. Hide everything Adjust your system preferences to automatically hide the Dock and menu bar (there’s a checkbox in “General”). With these two changes, a maximised window is effectively full-screen.

Content Preview

As I get older and grumpier, I increasingly value clean, uncluttered working environments. I’m sure I’m not the only one, so here’s a few useful practices and shortcuts that help me avoid using the mouse and satisfy my need for productivity micro-optimisation. They are mainly for macOS users. Hide everything Adjust your system preferences to automatically hide the Dock and menu bar (there’s a checkbox in “General”). With these two changes, a maximised window is effectively full-screen.

Joining between date and timestamp fields in Postgres

Published: Thu, 16 Nov 2017 15:50:04 +0000
Updated: Thu, 16 Nov 2017 15:50:04 +0000
UTC: 2017-11-16 15:50:04+00:00
URL: https://codeinthehole.com/tips/joining-between-date-and-datetime-fields-in-postgres/

Joining tables on date and timestamp with timezone fields in Postgres1 needs careful handling because of time zones and daylight-saving time. To illustrate, assume we have two tables: t1 which has a field of type date and a foreign-key t2_id to t2 which has a field of timestamp with timezone. We want to build SQL queries that join between these two tables with additional date constraints on the join. Assume the date values in t1 correspond to the Europe/London time zone.

Content Preview

Joining tables on date and timestamp with timezone fields in Postgres1 needs careful handling because of time zones and daylight-saving time. To illustrate, assume we have two tables: t1 which has a field of type date and a foreign-key t2_id to t2 which has a field of timestamp with timezone. We want to build SQL queries that join between these two tables with additional date constraints on the join. Assume the date values in t1 correspond to the Europe/London time zone.

Little-known words relevant to software development

Published: Tue, 07 Nov 2017 22:41:21 +0100
Updated: Tue, 07 Nov 2017 22:41:21 +0100
UTC: 2017-11-07 21:41:21+00:00
URL: https://codeinthehole.com/lists/little-known-words-relevant-to-software-development/

A curated1 collection of words-of-the-day from @qikipedia: Word of the day: ARSLE - to move backwards. — Quite Interesting (@qikipedia) May 14, 2017 Word of the day: VERSCHLIMMBESSERN - (German) - to make something worse while attempting to make it better — Quite Interesting (@qikipedia) July 3, 2017 Word of the day: PAREIDOLIA - the imagined perception of a pattern or image where it doesn't exist. — Quite Interesting (@qikipedia) October 21, 2017 Word of the day : FANKLE (Scottish) - to tangle or entangle something.

Content Preview

A curated1 collection of words-of-the-day from @qikipedia: Word of the day: ARSLE - to move backwards. — Quite Interesting (@qikipedia) May 14, 2017 Word of the day: VERSCHLIMMBESSERN - (German) - to make something worse while attempting to make it better — Quite Interesting (@qikipedia) July 3, 2017 Word of the day: PAREIDOLIA - the imagined perception of a pattern or image where it doesn't exist. — Quite Interesting (@qikipedia) October 21, 2017 Word of the day : FANKLE (Scottish) - to tangle or entangle something.

Using a custom Sentry client

Published: Mon, 16 Oct 2017 16:33:45 +0100
Updated: Mon, 16 Oct 2017 16:33:45 +0100
UTC: 2017-10-16 15:33:45+00:00
URL: https://codeinthehole.com/news/oe-tech-custom-sentry-client/

I wrote up a short tip on using a custom Sentry client.

Content Preview

I wrote up a short tip on using a custom Sentry client.

Why your Django models are fat

Published: Wed, 04 Oct 2017 10:30:02 +0000
Updated: Wed, 04 Oct 2017 10:30:02 +0000
UTC: 2017-10-04 10:30:02+00:00
URL: https://codeinthehole.com/lists/why-your-models-are-fat/

Because: You moved everything out of the views.py modules when you heard fat controllers were bad. And you’ve read that fat models are a good idea1. You need some extra functionality in a template and this is the easiest way to shoehorn it in. Eg: <p> The current balance is {{ account.get_balance_via_three_network_calls_lol }} </p> You can quickly solve your current problem by adding n more lines to that method (repeat ad infinitum)2.

Content Preview

Because: You moved everything out of the views.py modules when you heard fat controllers were bad. And you’ve read that fat models are a good idea1. You need some extra functionality in a template and this is the easiest way to shoehorn it in. Eg: <p> The current balance is {{ account.get_balance_via_three_network_calls_lol }} </p> You can quickly solve your current problem by adding n more lines to that method (repeat ad infinitum)2.

Bash error reporting

Published: Sat, 30 Sep 2017 22:44:03 +0100
Updated: Sat, 30 Sep 2017 22:44:03 +0100
UTC: 2017-09-30 21:44:03+00:00
URL: https://codeinthehole.com/tips/bash-error-reporting/

Two tips: Fail fast You probably already know that you can force Bash scripts to exit immediately if there’s an error (that is, if any command exits with a non-zero exit code) using: #!/usr/bin/env bash set -e but it’s even better to use: set -eu -o pipefail so that the script: exits on an error (-e, equivalent to -o errexit); exits on an undefined variable (-u, equivalent to -o nounset); exits on an error in piped-together commands (-o pipefail)1.

Content Preview

Two tips: Fail fast You probably already know that you can force Bash scripts to exit immediately if there’s an error (that is, if any command exits with a non-zero exit code) using: #!/usr/bin/env bash set -e but it’s even better to use: set -eu -o pipefail so that the script: exits on an error (-e, equivalent to -o errexit); exits on an undefined variable (-u, equivalent to -o nounset); exits on an error in piped-together commands (-o pipefail)1.

Using pgbadger with AWS RDS

Published: Tue, 29 Aug 2017 14:08:08 +0100
Updated: Tue, 29 Aug 2017 14:08:08 +0100
UTC: 2017-08-29 13:08:08+00:00
URL: https://codeinthehole.com/tips/using-pgbadger-with-aws-rds/

There’s two non-obvious things to know when starting to use pgbadger with AWS RDS. First, set: log_statement = None Don’t set this to all as the AWS docs suggest. Further, don’t waste your time trying to add a DB parameter to set log_line_prefix to pgbadger’s recommended value: it’s not possible1. Instead tell pgbadger about the log format that RDS insists on: pgbadger -f stderr -p '%t:%r:%u@%d:[%p]:' postgres.log Hope that saves you some time.

Content Preview

There’s two non-obvious things to know when starting to use pgbadger with AWS RDS. First, set: log_statement = None Don’t set this to all as the AWS docs suggest. Further, don’t waste your time trying to add a DB parameter to set log_line_prefix to pgbadger’s recommended value: it’s not possible1. Instead tell pgbadger about the log format that RDS insists on: pgbadger -f stderr -p '%t:%r:%u@%d:[%p]:' postgres.log Hope that saves you some time.

A mnemonic for mock decorators

Published: Sat, 26 Aug 2017 00:00:00 +0000
Updated: Sat, 26 Aug 2017 00:00:00 +0000
UTC: 2017-08-26 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-mnemonic-for-mocks/

Using both @mock.patch decorators and py.test fixtures can be confusing as it’s not always clear what order arguments are being injected. For instance, which of these is right? This: @mock.patch.object(module, 'collaborator_1') @mock.patch.object(module, 'collaborator_2') def test_something_in_module(collaborator_1, collaborator_2, some_pytest_fixture): pass Or this? @mock.patch.object(module, 'collaborator_2') @mock.patch.object(module, 'collaborator_1') def test_something_in_module(collaborator_1, collaborator_2, some_pytest_fixture): pass Or this? @mock.patch.object(module, 'collaborator_2') @mock.patch.object(module, 'collaborator_1') def test_something_in_module(some_pytest_fixture, collaborator_1, collaborator_2): pass I’ve wasted considerable time using PDB to work out what is being injected where.

Content Preview

Using both @mock.patch decorators and py.test fixtures can be confusing as it’s not always clear what order arguments are being injected. For instance, which of these is right? This: @mock.patch.object(module, 'collaborator_1') @mock.patch.object(module, 'collaborator_2') def test_something_in_module(collaborator_1, collaborator_2, some_pytest_fixture): pass Or this? @mock.patch.object(module, 'collaborator_2') @mock.patch.object(module, 'collaborator_1') def test_something_in_module(collaborator_1, collaborator_2, some_pytest_fixture): pass Or this? @mock.patch.object(module, 'collaborator_2') @mock.patch.object(module, 'collaborator_1') def test_something_in_module(some_pytest_fixture, collaborator_1, collaborator_2): pass I’ve wasted considerable time using PDB to work out what is being injected where.

Your codebase is your house

Published: Fri, 30 Jun 2017 11:26:39 +0100
Updated: Fri, 30 Jun 2017 11:26:39 +0100
UTC: 2017-06-30 10:26:39+00:00
URL: https://codeinthehole.com/tips/your-codebase-is-your-/

“Remember, code is your house, and you have to live in it.” - Michael Feathers 🏠 — Programming Wisdom (@CodeWisdom) April 19, 2017 This is the best metaphor I know for promoting or defending software quality. We do live in our codebases: shoddy software engineering has direct, easily-visualised analogues from construction and house maintenance. For instance, I’m sure you’ve seen these pull requests before1: Furthermore, the episode “Hurrican Neddy” from season 8 of The Simpsons is a parable of a failed software project, where feature delivery is prioritised at the expense of maintainability.

Content Preview

“Remember, code is your house, and you have to live in it.” - Michael Feathers 🏠 — Programming Wisdom (@CodeWisdom) April 19, 2017 This is the best metaphor I know for promoting or defending software quality. We do live in our codebases: shoddy software engineering has direct, easily-visualised analogues from construction and house maintenance. For instance, I’m sure you’ve seen these pull requests before1: Furthermore, the episode “Hurrican Neddy” from season 8 of The Simpsons is a parable of a failed software project, where feature delivery is prioritised at the expense of maintainability.

Git tips for working with pull requests

Published: Tue, 06 Jun 2017 21:30:36 +0100
Updated: Tue, 06 Jun 2017 21:30:36 +0100
UTC: 2017-06-06 20:30:36+00:00
URL: https://codeinthehole.com/tips/open-github-pull-request-command/

I spend at least 50% of each day reviewing, amended (and occasionally merging) pull requests, adding both commits and comments. As such I often want to quickly jump from a terminal window to the pull request detail page to check previous comments or add new. Even with the excellent hub git wrapper, there’s no easy way to do this. I can jump to the pull request list page with: git pulls where pulls is aliased in ~/.

Content Preview

I spend at least 50% of each day reviewing, amended (and occasionally merging) pull requests, adding both commits and comments. As such I often want to quickly jump from a terminal window to the pull request detail page to check previous comments or add new. Even with the excellent hub git wrapper, there’s no easy way to do this. I can jump to the pull request list page with: git pulls where pulls is aliased in ~/.

Converting JSON into CSV data for Google Sheets

Published: Tue, 16 May 2017 21:52:21 +0100
Updated: Tue, 16 May 2017 21:52:21 +0100
UTC: 2017-05-16 20:52:21+00:00
URL: https://codeinthehole.com/tips/json-to-google-sheets/

Like many people, I use Google Sheets to quickly create and share tabular data. As well as creating spreadsheets by pasting results generated in psql, I often create reports from JSON files using JQ. This post is a note-to-self on how to do this. Here’s a command to create a tab-separated report from a JSON events file exported from Loggly: $ cat loggly_events.json | \ jq -r '.events[].event.json | select(.params | has("payment_day")) | [.

Content Preview

Like many people, I use Google Sheets to quickly create and share tabular data. As well as creating spreadsheets by pasting results generated in psql, I often create reports from JSON files using JQ. This post is a note-to-self on how to do this. Here’s a command to create a tab-separated report from a JSON events file exported from Loggly: $ cat loggly_events.json | \ jq -r '.events[].event.json | select(.params | has("payment_day")) | [.

Podcast.__init__ on Oscar

Published: Mon, 10 Apr 2017 23:00:20 +0100
Updated: Mon, 10 Apr 2017 23:00:20 +0100
UTC: 2017-04-10 22:00:20+00:00
URL: https://codeinthehole.com/news/podcast-init/

I was interviewed, along with Michael van Tellingen, on the “Podcast.__init__” Python podcast to talk about Oscar.

Content Preview

I was interviewed, along with Michael van Tellingen, on the “Podcast.__init__” Python podcast to talk about Oscar.

Reorganising a Consul key-value store

Published: Mon, 27 Mar 2017 15:25:05 +0100
Updated: Mon, 27 Mar 2017 15:25:05 +0100
UTC: 2017-03-27 14:25:05+00:00
URL: https://codeinthehole.com/tips/migrate-consul-key-value-store/

If your Consul key-value store is structured as: / A/ X = 1 Z = 2 Y = 3 C D but you now realise you should have namespaced everything within WEBSERVER/ (or something like that): / WEBSERVER/ A/ X = 1 Z = 2 Y = 3 C D then this Bash script will help you migrate: #!/bin/bash set -e # Exit on error # Emit "key value" lines for all keys in Consul's KV store keys_and_values() { # Recursively fetch all values from Consul but exclude: # (a) those that end in / (as these are folders) # (b) those that start with WEBSERVER/ (as that's where we are migrating # to).

Content Preview

If your Consul key-value store is structured as: / A/ X = 1 Z = 2 Y = 3 C D but you now realise you should have namespaced everything within WEBSERVER/ (or something like that): / WEBSERVER/ A/ X = 1 Z = 2 Y = 3 C D then this Bash script will help you migrate: #!/bin/bash set -e # Exit on error # Emit "key value" lines for all keys in Consul's KV store keys_and_values() { # Recursively fetch all values from Consul but exclude: # (a) those that end in / (as these are folders) # (b) those that start with WEBSERVER/ (as that's where we are migrating # to).

Lies you're told on a software project

Published: Tue, 21 Mar 2017 11:54:01 +0000
Updated: Tue, 21 Mar 2017 11:54:01 +0000
UTC: 2017-03-21 11:54:01+00:00
URL: https://codeinthehole.com/lists/lies/

In approximate chronological order: “Sorry I missed stand-up this morning” “I didn’t see that email” “I’ll get back to you shortly” “Sounds easy - should only take a couple of days” “It’s nearly finished” “I didn’t have time to write tests” “We’ll clean this up later” 1 “Lessons have been learnt” Remember: "Later equals never." - Le Blanc's law — Programming Wisdom (@CodeWisdom) March 20, 2017  ↩︎

Content Preview

In approximate chronological order: “Sorry I missed stand-up this morning” “I didn’t see that email” “I’ll get back to you shortly” “Sounds easy - should only take a couple of days” “It’s nearly finished” “I didn’t have time to write tests” “We’ll clean this up later” 1 “Lessons have been learnt” Remember: "Later equals never." - Le Blanc's law — Programming Wisdom (@CodeWisdom) March 20, 2017  ↩︎

Hugo

Published: Thu, 16 Mar 2017 10:30:02 +0000
Updated: Thu, 16 Mar 2017 10:30:02 +0000
UTC: 2017-03-16 10:30:02+00:00
URL: https://codeinthehole.com/news/hugo/

I’ve migrated this site to Hugo so I can host it on Github pages1. Hugo is a fast and well thought-out static site generator, written in Golang. It’s easy to learn and has some neat features2 - the trickiest part is understanding the difference between various ways pages are organised: “sections”, “types”, “taxonomies” etc. The Vim plugin vim-markdown provides good support for authoring Hugo posts since it supports syntax-highlighting for:

Content Preview

I’ve migrated this site to Hugo so I can host it on Github pages1. Hugo is a fast and well thought-out static site generator, written in Golang. It’s easy to learn and has some neat features2 - the trickiest part is understanding the difference between various ways pages are organised: “sections”, “types”, “taxonomies” etc. The Vim plugin vim-markdown provides good support for authoring Hugo posts since it supports syntax-highlighting for:

Django, ELB health checks and continuous delivery

Published: Thu, 05 May 2016 00:00:00 +0000
Updated: Thu, 05 May 2016 00:00:00 +0000
UTC: 2016-05-05 00:00:00+00:00
URL: https://codeinthehole.com/news/oe-tech-elb-health-checks/

I wrote an article on Django, ELB health checks and continuous delivery.

Content Preview

I wrote an article on Django, ELB health checks and continuous delivery.

Testing for missing migrations in Django

Published: Thu, 21 Jan 2016 00:00:00 +0000
Updated: Thu, 21 Jan 2016 00:00:00 +0000
UTC: 2016-01-21 00:00:00+00:00
URL: https://codeinthehole.com/news/oe-tech-missing-migrations/

Octopus Energy has a tech blog. I wrote an article on Testing for missing migrations in Django.

Content Preview

Octopus Energy has a tech blog. I wrote an article on Testing for missing migrations in Django.

Octopus Energy

Published: Mon, 23 Nov 2015 00:00:00 +0000
Updated: Mon, 23 Nov 2015 00:00:00 +0000
UTC: 2015-11-23 00:00:00+00:00
URL: https://codeinthehole.com/news/octopus-energy/

Last Wednesday was my last day at Yoyo Wallet. Thursday marked my first day at Octopus Energy. I’m deeply excited about Octopus. Through innovative use of technology, especially automation and cloud computing, Octopus Energy is going to radically change how people consume and pay for energy in the UK. We’re going to be both cheaper and massively better at customer service than the much-maligned incumbents. Right now, we’re looking to build a superb tech team.

Content Preview

Last Wednesday was my last day at Yoyo Wallet. Thursday marked my first day at Octopus Energy. I’m deeply excited about Octopus. Through innovative use of technology, especially automation and cloud computing, Octopus Energy is going to radically change how people consume and pay for energy in the UK. We’re going to be both cheaper and massively better at customer service than the much-maligned incumbents. Right now, we’re looking to build a superb tech team.

A useful template for commit messages

Published: Fri, 02 Oct 2015 00:00:00 +0000
Updated: Fri, 02 Oct 2015 00:00:00 +0000
UTC: 2015-10-02 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-useful-template-for-commit-messages/

Here’s a useful heuristic for writing better commit messages. Set your commit message template to: # If applied, this commit will... # Why is this change needed? Prior to this change, # How does it address the issue? This change # Provide links to any relevant tickets, articles or other resources and you’ll be guided into writing concise commit subjects in the imperative mood — a good practice. See rule 5 of Chris Beam’s “How to write a commit message” for the inspiration of this tip and more reasoning on the use of the imperative mood.

Content Preview

Here’s a useful heuristic for writing better commit messages. Set your commit message template to: # If applied, this commit will... # Why is this change needed? Prior to this change, # How does it address the issue? This change # Provide links to any relevant tickets, articles or other resources and you’ll be guided into writing concise commit subjects in the imperative mood — a good practice. See rule 5 of Chris Beam’s “How to write a commit message” for the inspiration of this tip and more reasoning on the use of the imperative mood.

Copying Postgres output into a spreadsheet

Published: Wed, 02 Sep 2015 00:00:00 +0000
Updated: Wed, 02 Sep 2015 00:00:00 +0000
UTC: 2015-09-02 00:00:00+00:00
URL: https://codeinthehole.com/tips/copying-postgres-output-into-a-spreadsheet/

I often need to grab information from a Postgres database and paste it into a spreadsheet for sharing with others. Google Sheets needs the pasted data to be tab-separated in order to be correctly split into columns. This isn’t the default behaviour for psql but here’s how to configure psql’s output to get it. At a psql prompt, switch to unaligned output => \a set the field separator to a tab character:

Content Preview

I often need to grab information from a Postgres database and paste it into a spreadsheet for sharing with others. Google Sheets needs the pasted data to be tab-separated in order to be correctly split into columns. This isn’t the default behaviour for psql but here’s how to configure psql’s output to get it. At a psql prompt, switch to unaligned output => \a set the field separator to a tab character:

An SSH tip for modern AWS patrons

Published: Sat, 02 May 2015 00:00:00 +0000
Updated: Sat, 02 May 2015 00:00:00 +0000
UTC: 2015-05-02 00:00:00+00:00
URL: https://codeinthehole.com/tips/an-ssh-tip-for-modern-aws-patrons/

Cloud computing and immutable infrastructure deployments have changed the way I use SSH. I miss the days when I could run: ssh app1-prod to jump onto a machine and investigate an issue. This would work as, back in the days of yore, your web servers didn’t change IP address several times a week so I could create a helpful alias in ~/.ssh/config: Host app1-prod User example_user HostName 74.207.251.29 This circumvented the labour-intensive act of typing in the remote username and IP address when SSHing around town.

Content Preview

Cloud computing and immutable infrastructure deployments have changed the way I use SSH. I miss the days when I could run: ssh app1-prod to jump onto a machine and investigate an issue. This would work as, back in the days of yore, your web servers didn’t change IP address several times a week so I could create a helpful alias in ~/.ssh/config: Host app1-prod User example_user HostName 74.207.251.29 This circumvented the labour-intensive act of typing in the remote username and IP address when SSHing around town.

commandlinefu.com is in new hands

Published: Sat, 04 Apr 2015 00:00:00 +0000
Updated: Sat, 04 Apr 2015 00:00:00 +0000
UTC: 2015-04-04 00:00:00+00:00
URL: https://codeinthehole.com/news/commandlinefucom-is-in-new-hands/

For the record, I no longer maintain commandlinefu.com. I’ve handed over the baton to notable Bay Area celebrity, Jon Hendren (@fart). This is a good thing for the site and its users as I have been unable to do much maintenance in recent times. I look forward to seeing how the site evolves.

Content Preview

For the record, I no longer maintain commandlinefu.com. I’ve handed over the baton to notable Bay Area celebrity, Jon Hendren (@fart). This is a good thing for the site and its users as I have been unable to do much maintenance in recent times. I look forward to seeing how the site evolves.

Backing up Postgres database rows before deleting them

Published: Thu, 19 Mar 2015 00:00:00 +0000
Updated: Thu, 19 Mar 2015 00:00:00 +0000
UTC: 2015-03-19 00:00:00+00:00
URL: https://codeinthehole.com/tips/backing-up-database-rows-in-postgres-before-deleting-them/

Hmmm, this delete statement is taking longer than I thought it would… If you ever have to manually run a SQL delete statement in psql, you can back-up the rows you’re about to delete: \copy ( select * from $table where id in ( ... ) ) to '/tmp/backup.csv' and, if you’ve got your filter clause wrong (we’ve all done it), restore them with: \copy $table from '/tmp/backup.csv' Moderately useful.

Content Preview

Hmmm, this delete statement is taking longer than I thought it would… If you ever have to manually run a SQL delete statement in psql, you can back-up the rows you’re about to delete: \copy ( select * from $table where id in ( ... ) ) to '/tmp/backup.csv' and, if you’ve got your filter clause wrong (we’ve all done it), restore them with: \copy $table from '/tmp/backup.csv' Moderately useful.

Avoiding clashing Django migrations

Published: Sat, 31 Jan 2015 00:00:00 +0000
Updated: Sat, 31 Jan 2015 00:00:00 +0000
UTC: 2015-01-31 00:00:00+00:00
URL: https://codeinthehole.com/tips/avoiding-clashing-django-migrations/

Managing South migrations on a multi-developer Django project can be painful. Developers working on separate branches will often create migrations for the same app with the same migration number1. When merged into master, these clashing migrations can cause deployment hiccups as South will complain if migrations are applied out of order. There are various techniques available for dealing with this2, but what we do at Yoyo is test for such clashes as part of our Travis continuous integration.

Content Preview

Managing South migrations on a multi-developer Django project can be painful. Developers working on separate branches will often create migrations for the same app with the same migration number1. When merged into master, these clashing migrations can cause deployment hiccups as South will complain if migrations are applied out of order. There are various techniques available for dealing with this2, but what we do at Yoyo is test for such clashes as part of our Travis continuous integration.

Bootstrapped virtualenvs

Published: Fri, 24 Oct 2014 00:00:00 +0000
Updated: Fri, 24 Oct 2014 00:00:00 +0000
UTC: 2014-10-24 00:00:00+00:00
URL: https://codeinthehole.com/tips/bootstrapped-virtualenvs/

The excellent virtualenvwrapper supports a postmkvirtualenv script to bootstrap your virtual environments. Here’s a useful implementation: #!/usr/bin/env bash # Grab project name from virtualenv name NAME=$(basename $VIRTUAL_ENV) # Set terminal title on postactivate echo "title $NAME" > $VIRTUAL_ENV/bin/postactivate # Change directory to root of project on postactivate. We assume the # mkvirtualenv is being run from the root of the project. This line # will need to edited later if not.

Content Preview

The excellent virtualenvwrapper supports a postmkvirtualenv script to bootstrap your virtual environments. Here’s a useful implementation: #!/usr/bin/env bash # Grab project name from virtualenv name NAME=$(basename $VIRTUAL_ENV) # Set terminal title on postactivate echo "title $NAME" > $VIRTUAL_ENV/bin/postactivate # Change directory to root of project on postactivate. We assume the # mkvirtualenv is being run from the root of the project. This line # will need to edited later if not.

Integrating Django application metrics into Zabbix

Published: Wed, 22 Oct 2014 00:00:00 +0000
Updated: Wed, 22 Oct 2014 00:00:00 +0000
UTC: 2014-10-22 00:00:00+00:00
URL: https://codeinthehole.com/tips/integrating-django-application-metrics-into-zabbix/

At Tangent, we use Zabbix for monitoring and alerting. This is a note-to-self on how to configure application monitoring. Management command You need a script that prints out a value to STDOUT. A simple management command suffices: from django.core.management.base import BaseCommand, CommandError class Command(BaseCommand): def add_arguments(self, parser): parser.add_argument('args', nargs='*') def handle(self, *args, **options): if not args: print self.usage() return method_name = 'fetch__%s' % args[0] if not hasattr(self, method_name): raise CommandError("No method found with name '%s'" % method_name) print getattr(self, method_name)(*args[1:]) def usage(self): fetchers = [m for m in dir(self) if m.

Content Preview

At Tangent, we use Zabbix for monitoring and alerting. This is a note-to-self on how to configure application monitoring. Management command You need a script that prints out a value to STDOUT. A simple management command suffices: from django.core.management.base import BaseCommand, CommandError class Command(BaseCommand): def add_arguments(self, parser): parser.add_argument('args', nargs='*') def handle(self, *args, **options): if not args: print self.usage() return method_name = 'fetch__%s' % args[0] if not hasattr(self, method_name): raise CommandError("No method found with name '%s'" % method_name) print getattr(self, method_name)(*args[1:]) def usage(self): fetchers = [m for m in dir(self) if m.

Linking to Github

Published: Thu, 17 Jul 2014 00:00:00 +0000
Updated: Thu, 17 Jul 2014 00:00:00 +0000
UTC: 2014-07-17 00:00:00+00:00
URL: https://codeinthehole.com/tips/linking-to-github/

It was rightly pointed out yesterday that it’s dangerous to link to lines or blocks of code on Github without explicitly specifying the commit hash in the URL. On this theme, consider this git command: $ git browse -u -- commit/$(git rev-parse HEAD) https://github.com/tangentlabs/django-oscar/commit/17851d4b66922f8d7e203e2b469040690c84a0db This emits the Github URL to the HEAD commit on the current branch, specifying the commit hash in the URL. Note that the browse subcommand is provided by the excellent hub library.

Content Preview

It was rightly pointed out yesterday that it’s dangerous to link to lines or blocks of code on Github without explicitly specifying the commit hash in the URL. On this theme, consider this git command: $ git browse -u -- commit/$(git rev-parse HEAD) https://github.com/tangentlabs/django-oscar/commit/17851d4b66922f8d7e203e2b469040690c84a0db This emits the Github URL to the HEAD commit on the current branch, specifying the commit hash in the URL. Note that the browse subcommand is provided by the excellent hub library.

Continuously rebuild your project

Published: Wed, 18 Jun 2014 00:00:00 +0000
Updated: Wed, 18 Jun 2014 00:00:00 +0000
UTC: 2014-06-18 00:00:00+00:00
URL: https://codeinthehole.com/tips/continuously-re-build-your-project/

New developers joining a project will often find that the project won’t build cleanly on their machine, and hours of time will be sunk into setting up the project so work can start. This is sad and expensive for all concerned. This is a particular menace in agencies (or anywhere with lots of small projects) where a large team of developers need to jump between projects. Tools like Vagrant and Docker can help but aren’t the panacea they first seem to be1.

Content Preview

New developers joining a project will often find that the project won’t build cleanly on their machine, and hours of time will be sunk into setting up the project so work can start. This is sad and expensive for all concerned. This is a particular menace in agencies (or anywhere with lots of small projects) where a large team of developers need to jump between projects. Tools like Vagrant and Docker can help but aren’t the panacea they first seem to be1.

Using the silver searcher with Vim

Published: Tue, 17 Jun 2014 00:00:00 +0000
Updated: Tue, 17 Jun 2014 00:00:00 +0000
UTC: 2014-06-17 00:00:00+00:00
URL: https://codeinthehole.com/tips/using-the-silver-searcher-with-vim/

If you’re not doing this already, then you should use the Silver Searcher within Vim for rapid, convenient file searching. In a nutshell, ag offers similar functionality to ack but with much better performance. It’s easily installed - on OSX, run: brew install the_silver_searcher Urge Vim to use it for :grep commands by adding the following to ~/.vimrc: if executable('ag') " Note we extract the column as well as the file and line number set grepprg=ag\ --nogroup\ --nocolor\ --column set grepformat=%f:%l:%c%m endif :grep searches are now lightning-fast and respectful of your ~/.

Content Preview

If you’re not doing this already, then you should use the Silver Searcher within Vim for rapid, convenient file searching. In a nutshell, ag offers similar functionality to ack but with much better performance. It’s easily installed - on OSX, run: brew install the_silver_searcher Urge Vim to use it for :grep commands by adding the following to ~/.vimrc: if executable('ag') " Note we extract the column as well as the file and line number set grepprg=ag\ --nogroup\ --nocolor\ --column set grepformat=%f:%l:%c%m endif :grep searches are now lightning-fast and respectful of your ~/.

The road to Oscar 1.0

Published: Thu, 01 May 2014 00:00:00 +0000
Updated: Thu, 01 May 2014 00:00:00 +0000
UTC: 2014-05-01 00:00:00+00:00
URL: https://codeinthehole.com/projects/the-road-to-oscar-10/

Oscar 0.7 was released this week, a comparatively minor house-keeping release that allowed us to reduce our pull-request and issue backlogs. We have a plan for Oscar through 2014 and this article outlines the roadmap. v1.0 We’re aiming to be v1.0 before the end of the year1. Oscar has been beta for nearly two years and its APIs are stabilising. We’re ready to commit to the backward compatibility responsibilities associated with coming out of beta.

Content Preview

Oscar 0.7 was released this week, a comparatively minor house-keeping release that allowed us to reduce our pull-request and issue backlogs. We have a plan for Oscar through 2014 and this article outlines the roadmap. v1.0 We’re aiming to be v1.0 before the end of the year1. Oscar has been beta for nearly two years and its APIs are stabilising. We’re ready to commit to the backward compatibility responsibilities associated with coming out of beta.

Command-line tips for effective release announcements

Published: Thu, 16 Jan 2014 00:00:00 +0000
Updated: Thu, 16 Jan 2014 00:00:00 +0000
UTC: 2014-01-16 00:00:00+00:00
URL: https://codeinthehole.com/tips/command-line-tips-for-effective-release-announcements/

We finally released Oscar 0.6 last week. The process brought home the importance of writing your release notes as you go rather than at the end. It’s a real pain to extract the key changes from 1200 commits spread over the last 8 months. Lesson learnt. This article is largely a note-to-self in case I have to repeat the process. However, if you do find yourself in a similar position, here are a few command-line tricks for analysing your git history.

Content Preview

We finally released Oscar 0.6 last week. The process brought home the importance of writing your release notes as you go rather than at the end. It’s a real pain to extract the key changes from 1200 commits spread over the last 8 months. Lesson learnt. This article is largely a note-to-self in case I have to repeat the process. However, if you do find yourself in a similar position, here are a few command-line tricks for analysing your git history.

How to install PostGIS and GeoDjango on Ubuntu

Published: Fri, 04 Oct 2013 00:00:00 +0000
Updated: Fri, 04 Oct 2013 00:00:00 +0000
UTC: 2013-10-04 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-install-postgis-and-geodjango-on-ubuntu/

Despite its extensive documentation, getting GeoDjango installed and configured can be a pain. Here are my notes for future reference: Installation on Ubuntu 12.04 First, ensure your system locale is UTF8 as PostgreSQL uses it to determine its default encoding during installation: export LANGUAGE="en_US.UTF-8" export LANG="en_US.UTF-8" export LC_ALL="en_US.UTF-8" Now install dependencies: sudo apt-get update sudo apt-get install postgresql-server-dev-9.1 postgresql-9.1-postgis PostgreSQL should now be installed and running with UTF8 encodings. Verify this with:

Content Preview

Despite its extensive documentation, getting GeoDjango installed and configured can be a pain. Here are my notes for future reference: Installation on Ubuntu 12.04 First, ensure your system locale is UTF8 as PostgreSQL uses it to determine its default encoding during installation: export LANGUAGE="en_US.UTF-8" export LANG="en_US.UTF-8" export LC_ALL="en_US.UTF-8" Now install dependencies: sudo apt-get update sudo apt-get install postgresql-server-dev-9.1 postgresql-9.1-postgis PostgreSQL should now be installed and running with UTF8 encodings. Verify this with:

Enhancing your Git commit editor

Published: Thu, 08 Aug 2013 00:00:00 +0000
Updated: Thu, 08 Aug 2013 00:00:00 +0000
UTC: 2013-08-08 00:00:00+00:00
URL: https://codeinthehole.com/tips/enhancing-your-git-commit-editor/

Confession: I am a pedant, especially around commit messages. I often find myself writing very similar commit messages (like “Bump version to 0.4.3”) and want to ensure I use the same wording each time. Thanks to @LuRsT, I learnt how to employ Git’s prepare-commit-msg hook to display the last 5 commit messages when I’m editing a commit message. Use the following .git/hooks/prepare-commit-msg hook: #!/bin/sh NUM_COMMITS=5 FORMAT="# %h %s [%an]" COMMITS="$(git log --pretty="${FORMAT}" -${NUM_COMMITS})" HEADER="# # Last ${NUM_COMMITS} commits # ----------------------" recent_commits() { echo "${HEADER}" echo "${COMMITS}" } COMMIT_FILE=$1 SOURCE=$2 SHA=$3 case "$SOURCE" in merge|squash|message) ;; ""|commit|template) if [ -z "$SHA" ]; then recent_commits >> $COMMIT_FILE fi ;; *) echo "Unexpected type '$SOURCE' in prepare-commit-msg hook" >&2 exit 1 esac then your default commit template looks like this:

Content Preview

Confession: I am a pedant, especially around commit messages. I often find myself writing very similar commit messages (like “Bump version to 0.4.3”) and want to ensure I use the same wording each time. Thanks to @LuRsT, I learnt how to employ Git’s prepare-commit-msg hook to display the last 5 commit messages when I’m editing a commit message. Use the following .git/hooks/prepare-commit-msg hook: #!/bin/sh NUM_COMMITS=5 FORMAT="# %h %s [%an]" COMMITS="$(git log --pretty="${FORMAT}" -${NUM_COMMITS})" HEADER="# # Last ${NUM_COMMITS} commits # ----------------------" recent_commits() { echo "${HEADER}" echo "${COMMITS}" } COMMIT_FILE=$1 SOURCE=$2 SHA=$3 case "$SOURCE" in merge|squash|message) ;; ""|commit|template) if [ -z "$SHA" ]; then recent_commits >> $COMMIT_FILE fi ;; *) echo "Unexpected type '$SOURCE' in prepare-commit-msg hook" >&2 exit 1 esac then your default commit template looks like this:

Dumping and restoring a PostGIS database

Published: Sun, 28 Jul 2013 00:00:00 +0000
Updated: Sun, 28 Jul 2013 00:00:00 +0000
UTC: 2013-07-28 00:00:00+00:00
URL: https://codeinthehole.com/tips/dumping-and-restoring-a-postgis-database/

I wasted at least twenty minutes getting this to work. These are my notes. Problem You are using a PostGIS database and want to take a backup copy from production and restore it in a different environment. One complication is that each environment connects to its database with its own user. This is a common scenario if you are using GeoDjango. Solution Suppose your production database is called “myproject_prod” which you connect to with user “myproject_prod_role” and you want to replace your existing stage database “myproject_stage” that you connect to with user “myproject_stage_role”.

Content Preview

I wasted at least twenty minutes getting this to work. These are my notes. Problem You are using a PostGIS database and want to take a backup copy from production and restore it in a different environment. One complication is that each environment connects to its database with its own user. This is a common scenario if you are using GeoDjango. Solution Suppose your production database is called “myproject_prod” which you connect to with user “myproject_prod_role” and you want to replace your existing stage database “myproject_stage” that you connect to with user “myproject_stage_role”.

purl, URI templates and generated tests

Published: Tue, 09 Jul 2013 00:00:00 +0000
Updated: Tue, 09 Jul 2013 00:00:00 +0000
UTC: 2013-07-09 00:00:00+00:00
URL: https://codeinthehole.com/projects/purl-uri-templates-and-generated-tests/

TLDR: Parameterised tests are a useful testing technique. Both Nose and py.test support them. URI templates in purl The newly released purl 0.8 (a URL library of mine) supports URI templates as per RFC 6570. These allow new URL instances to be created by passing bindings to a template instance. >>> import purl >>> tpl = purl.Template('http://www.google.com{path}') >>> tpl.expand({'path': ['a', 'b', 'c']}).as_string() 'http://www.google.com/a/b/c' Alternatively, you can expand template strings directly:

Content Preview

TLDR: Parameterised tests are a useful testing technique. Both Nose and py.test support them. URI templates in purl The newly released purl 0.8 (a URL library of mine) supports URI templates as per RFC 6570. These allow new URL instances to be created by passing bindings to a template instance. >>> import purl >>> tpl = purl.Template('http://www.google.com{path}') >>> tpl.expand({'path': ['a', 'b', 'c']}).as_string() 'http://www.google.com/a/b/c' Alternatively, you can expand template strings directly:

A deferred logging file handler for Django

Published: Wed, 12 Jun 2013 00:00:00 +0000
Updated: Wed, 12 Jun 2013 00:00:00 +0000
UTC: 2013-06-12 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-deferred-logging-file-handler-for-django/

At Tangent we handle environment-specific configuration of Django projects using the method outlined by David Cramer. This involves distinguishing between core settings (which we keep in core/default.py) and environment specific settings (eg core/stage.py, core/test.py). The standard settings.py module imports all defaults and then uses a environmental shell variable to determine which environment settings module to import. A problem One tricky issue with this arrangement is logging to file. Ideally, we want to define a single LOGGING dict in the default settings but have file logging use an environment-specific folder.

Content Preview

At Tangent we handle environment-specific configuration of Django projects using the method outlined by David Cramer. This involves distinguishing between core settings (which we keep in core/default.py) and environment specific settings (eg core/stage.py, core/test.py). The standard settings.py module imports all defaults and then uses a environmental shell variable to determine which environment settings module to import. A problem One tricky issue with this arrangement is logging to file. Ideally, we want to define a single LOGGING dict in the default settings but have file logging use an environment-specific folder.

Conditional logic in Django forms

Published: Sat, 01 Jun 2013 00:00:00 +0000
Updated: Sat, 01 Jun 2013 00:00:00 +0000
UTC: 2013-06-01 00:00:00+00:00
URL: https://codeinthehole.com/tips/conditional-logic-in-django-forms/

The problem It’s common for UX professionals to design forms like the following: where radio buttons are employed to split the form into sections, each of which can have its own fields which are only mandatory if the parent radio button is checked. Thus the validation logic is conditional on the submitted form data. Such requirements are slightly tricky to capture in Django as they tread slightly outside the normal path of form validation.

Content Preview

The problem It’s common for UX professionals to design forms like the following: where radio buttons are employed to split the form into sections, each of which can have its own fields which are only mandatory if the parent radio button is checked. Thus the validation logic is conditional on the submitted form data. Such requirements are slightly tricky to capture in Django as they tread slightly outside the normal path of form validation.

PyPI README badges

Published: Fri, 03 May 2013 00:00:00 +0000
Updated: Fri, 03 May 2013 00:00:00 +0000
UTC: 2013-05-03 00:00:00+00:00
URL: https://codeinthehole.com/tips/pypi-readme-badges/

Thanks to @kuramanga, it’s now possible to add shiny PyPi badges to your Python project READMEs that indicate the latest released version on PyPI and the total number of downloads. This screenshot is taken from django-oscar’s README. Embed these badges in your own repo as Restructured text: .. image:: https://pypip.in/v/$REPO/badge.png :target: https://crate.io/packages/$REPO/ :alt: Latest PyPI version .. image:: https://pypip.in/d/$REPO/badge.png :target: https://crate.io/packages/$REPO/ :alt: Number of PyPI downloads or Markdown: [](https://crate.io/packages/$REPO/) [!

Content Preview

Thanks to @kuramanga, it’s now possible to add shiny PyPi badges to your Python project READMEs that indicate the latest released version on PyPI and the total number of downloads. This screenshot is taken from django-oscar’s README. Embed these badges in your own repo as Restructured text: .. image:: https://pypip.in/v/$REPO/badge.png :target: https://crate.io/packages/$REPO/ :alt: Latest PyPI version .. image:: https://pypip.in/d/$REPO/badge.png :target: https://crate.io/packages/$REPO/ :alt: Number of PyPI downloads or Markdown: [](https://crate.io/packages/$REPO/) [!

A useful Git post-checkout hook for Python repos

Published: Tue, 23 Apr 2013 00:00:00 +0000
Updated: Tue, 23 Apr 2013 00:00:00 +0000
UTC: 2013-04-23 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-useful-git-post-checkout-hook-for-python-repos/

Every now and again, an innocent python developer checks out a new git branch then proceeds to bang their head against a bug caused by an orphaned .pyc file from the previous branch. Since *.pyc files are typically in the repo’s .gitignore file, they are not removed when switching branches and can cause issues if the corresponding .py is removed. This can be neatly addressed through a ‘post checkout’ hook which deletes all such files.

Content Preview

Every now and again, an innocent python developer checks out a new git branch then proceeds to bang their head against a bug caused by an orphaned .pyc file from the previous branch. Since *.pyc files are typically in the repo’s .gitignore file, they are not removed when switching branches and can cause issues if the corresponding .py is removed. This can be neatly addressed through a ‘post checkout’ hook which deletes all such files.

Disable database access when writing unit tests in Django

Published: Mon, 22 Apr 2013 00:00:00 +0000
Updated: Mon, 22 Apr 2013 00:00:00 +0000
UTC: 2013-04-22 00:00:00+00:00
URL: https://codeinthehole.com/tips/disable-database-access-when-writing-unit-tests-in-django/

Consider this curio: import mock from django.utils.functional import curry no_database = curry( mock.patch, 'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError("Using the database is not permitted"))) This snippet creates a decorator that can wrap a test case or method and raises an exception if the database is accessed. This can be useful if you’re a puritan about true unit tests. Use by wrapping a TestCase subclass: from django.test import TestCase @no_database() class UnitTestCase(TestCase): ... or method:

Content Preview

Consider this curio: import mock from django.utils.functional import curry no_database = curry( mock.patch, 'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError("Using the database is not permitted"))) This snippet creates a decorator that can wrap a test case or method and raises an exception if the database is accessed. This can be useful if you’re a puritan about true unit tests. Use by wrapping a TestCase subclass: from django.test import TestCase @no_database() class UnitTestCase(TestCase): ... or method:

How to install PIL on 64-bit Ubuntu 12.04

Published: Thu, 18 Apr 2013 00:00:00 +0000
Updated: Thu, 18 Apr 2013 00:00:00 +0000
UTC: 2013-04-18 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-install-pil-on-64-bit-ubuntu-1204/

Problem You want to install PIL on 64-bit Ubuntu 12.04 (Precise Pangolin). Solution With pip already installed, install the required development packages: sudo apt-get install python-dev libjpeg-dev libfreetype6-dev zlib1g-dev and symlink the three image libraries into /usr/lib: sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/ PIL should now install with support for JPEGs, PNGs and FreeType, as indicated by the compilation output:

Content Preview

Problem You want to install PIL on 64-bit Ubuntu 12.04 (Precise Pangolin). Solution With pip already installed, install the required development packages: sudo apt-get install python-dev libjpeg-dev libfreetype6-dev zlib1g-dev and symlink the three image libraries into /usr/lib: sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/ PIL should now install with support for JPEGs, PNGs and FreeType, as indicated by the compilation output:

Converting Github issues into pull requests

Published: Mon, 04 Mar 2013 00:00:00 +0000
Updated: Mon, 04 Mar 2013 00:00:00 +0000
UTC: 2013-03-04 00:00:00+00:00
URL: https://codeinthehole.com/tips/converting-github-issues-into-pull-requests/

Using the Hub library, it’s possible to convert Github issues into pull requests. This gives rise to a useful Github workflow which this article describes. This is nothing new; it’s been written about before. However, this is something I do all the time whilst developing Oscar and I’m fed up with explaining it. This article is a reference I can point people at. Workflow Discuss Discuss an idea for a new feature on the project mailing list.

Content Preview

Using the Hub library, it’s possible to convert Github issues into pull requests. This gives rise to a useful Github workflow which this article describes. This is nothing new; it’s been written about before. However, this is something I do all the time whilst developing Oscar and I’m fed up with explaining it. This article is a reference I can point people at. Workflow Discuss Discuss an idea for a new feature on the project mailing list.

Altering Postgres table columns with South

Published: Tue, 19 Feb 2013 00:00:00 +0000
Updated: Tue, 19 Feb 2013 00:00:00 +0000
UTC: 2013-02-19 00:00:00+00:00
URL: https://codeinthehole.com/tips/altering-postgres-table-columns-with-south/

Problem You’re using Postgres with Django. You change a field type of one of your models, generate an --auto South migration and attempt to run it. However, South chokes on the new migration complaining that the data in the column cannot be cast to the new type. For instance, I recently changed a CharField to a TimeField but the corresponding migration lead to: Running migrations for stores: - Migrating forwards to 0009_auto__chg_field_openingperiod_start__chg_field_openingperiod_end.

Content Preview

Problem You’re using Postgres with Django. You change a field type of one of your models, generate an --auto South migration and attempt to run it. However, South chokes on the new migration complaining that the data in the column cannot be cast to the new type. For instance, I recently changed a CharField to a TimeField but the corresponding migration lead to: Running migrations for stores: - Migrating forwards to 0009_auto__chg_field_openingperiod_start__chg_field_openingperiod_end.

Configuring logging for Postgres.app

Published: Mon, 14 Jan 2013 00:00:00 +0000
Updated: Mon, 14 Jan 2013 00:00:00 +0000
UTC: 2013-01-14 00:00:00+00:00
URL: https://codeinthehole.com/tips/configuring-logging-for-postgresapp/

Problem You’re using Postgres.app on a Mac for local development but are getting SQL errors from your application. You’re seeing an error message: ERROR: current transaction is aborted, commands ignored until end of transaction block This isn’t useful: you want to know which query is generating the error. Solution Turn on Postgres logging and watch the log files when the error is generated. This is done by editing the postgresql.conf config file, whose location can be found from the “Server Settings” option in the Postgres.

Content Preview

Problem You’re using Postgres.app on a Mac for local development but are getting SQL errors from your application. You’re seeing an error message: ERROR: current transaction is aborted, commands ignored until end of transaction block This isn’t useful: you want to know which query is generating the error. Solution Turn on Postgres logging and watch the log files when the error is generated. This is done by editing the postgresql.conf config file, whose location can be found from the “Server Settings” option in the Postgres.

Mathematics and engineering

Published: Fri, 14 Dec 2012 00:00:00 +0000
Updated: Fri, 14 Dec 2012 00:00:00 +0000
UTC: 2012-12-14 00:00:00+00:00
URL: https://codeinthehole.com/tidbits/mathematics-and-engineering/

Structure and Interpretation of Computer Programs is full of interesting footnotes. Here’s a good one about the Fermat test for prime numbers: Numbers that fool the Fermat test are called Carmichael numbers, and little is known about them other than that they are extremely rare. There 255 Carmichael below 100,000,000. The smallest few are 561, 1105, 1729, 2465, 2821 and 6601. In testing primality of very large numbers chosen at random, the chance of stumbling upon a value that fools the Fermat test is less than the chance that cosmic radiation will cause the computer to make an error in carrying out a correct algorithm.

Content Preview

Structure and Interpretation of Computer Programs is full of interesting footnotes. Here’s a good one about the Fermat test for prime numbers: Numbers that fool the Fermat test are called Carmichael numbers, and little is known about them other than that they are extremely rare. There 255 Carmichael below 100,000,000. The smallest few are 561, 1105, 1729, 2465, 2821 and 6601. In testing primality of very large numbers chosen at random, the chance of stumbling upon a value that fools the Fermat test is less than the chance that cosmic radiation will cause the computer to make an error in carrying out a correct algorithm.

Effective pull requests and other good practices for teams using github

Published: Sat, 20 Oct 2012 00:00:00 +0000
Updated: Sat, 20 Oct 2012 00:00:00 +0000
UTC: 2012-10-20 00:00:00+00:00
URL: https://codeinthehole.com/tips/pull-requests-and-other-good-practices-for-teams-using-github/

I work at an agency where we pay $200 a month to Github for their platinum plan. This article is a summary of an internal talk I gave on making the most of our subscription. There’s nothing original here: it’s just a collection of tips that I’ve harvested over the last few years. I’m publishing this article mainly so I have something to refer future employees to. Use pull requests Pull requests are an excellent tool for fostering code review.

Content Preview

I work at an agency where we pay $200 a month to Github for their platinum plan. This article is a summary of an internal talk I gave on making the most of our subscription. There’s nothing original here: it’s just a collection of tips that I’ve harvested over the last few years. I’m publishing this article mainly so I have something to refer future employees to. Use pull requests Pull requests are an excellent tool for fostering code review.

How to chroot a user in Ubuntu 12.04

Published: Tue, 16 Oct 2012 00:00:00 +0000
Updated: Tue, 16 Oct 2012 00:00:00 +0000
UTC: 2012-10-16 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-chroot-a-user-in-ubuntu-1204/

External parties often need to upload data to your application. Sadly, most ask for a FTP server. Push back against this and suggest they use sFTP. This article explains how to set-up a chroot-ed user in Ubuntu 12.04 so that an external party can upload data to your application securely. This is mainly for my own reference. User set-up Create user with a dummy shell: adduser --shell=/bin/false barry and alter the ownership and permissions of their home folder:

Content Preview

External parties often need to upload data to your application. Sadly, most ask for a FTP server. Push back against this and suggest they use sFTP. This article explains how to set-up a chroot-ed user in Ubuntu 12.04 so that an external party can upload data to your application securely. This is mainly for my own reference. User set-up Create user with a dummy shell: adduser --shell=/bin/false barry and alter the ownership and permissions of their home folder:

Prefer WebTest to Django's test client for functional tests

Published: Sun, 09 Sep 2012 00:00:00 +0000
Updated: Sun, 09 Sep 2012 00:00:00 +0000
UTC: 2012-09-09 00:00:00+00:00
URL: https://codeinthehole.com/tips/prefer-webtest-to-djangos-test-client-for-functional-tests/

Since watching Carl Meyer’s superb ‘Testing and Django’ talk, I’ve been using Ian Bicking’s WebTest library for functional tests, via django-webtest. I’ve been really impressed and I’d like to stress one of Carl’s points - that using WebTest for functional tests is superior to using the Django client. Why? Several reasons - here’s a few: WebTest allows you to model a user’s experience much more closely as it is smart about mark-up.

Content Preview

Since watching Carl Meyer’s superb ‘Testing and Django’ talk, I’ve been using Ian Bicking’s WebTest library for functional tests, via django-webtest. I’ve been really impressed and I’d like to stress one of Carl’s points - that using WebTest for functional tests is superior to using the Django client. Why? Several reasons - here’s a few: WebTest allows you to model a user’s experience much more closely as it is smart about mark-up.

Cacheback - asynchronous cache refreshing for Django

Published: Sun, 02 Sep 2012 00:00:00 +0000
Updated: Sun, 02 Sep 2012 00:00:00 +0000
UTC: 2012-09-02 00:00:00+00:00
URL: https://codeinthehole.com/projects/cacheback-asynchronous-cache-refreshing-for-django/

Inspired by Jacob Kaplan-Moss’s excellent talk “Django doesn’t scale” at this year’s OSCon, I’ve put together a Django package for re-populating caches asynchronously. It provides a simple API for wrapping expensive read operations that caches results and uses Celery to repopulate items when they become stale. It can be used as a decorator for simple cases but provides an extensible class for more fine-grained control. It also provides helper classes for working with querysets.

Content Preview

Inspired by Jacob Kaplan-Moss’s excellent talk “Django doesn’t scale” at this year’s OSCon, I’ve put together a Django package for re-populating caches asynchronously. It provides a simple API for wrapping expensive read operations that caches results and uses Celery to repopulate items when they become stale. It can be used as a decorator for simple cases but provides an extensible class for more fine-grained control. It also provides helper classes for working with querysets.

Use models for uploads

Published: Thu, 19 Jul 2012 00:00:00 +0000
Updated: Thu, 19 Jul 2012 00:00:00 +0000
UTC: 2012-07-19 00:00:00+00:00
URL: https://codeinthehole.com/tips/use-models-for-uploads/

All Django developers will deal with file uploads at some point. I contend that it’s a good practice to use models to capture the upload metadata and to track processing status. This article explains how and why. An e-commerce example Suppose your e-commerce application allows admins to upload CSV files to update product stock levels (a common requirement). A typical file may comprise a SKU and a stock level: 9781231231999,0 9781231231999,4 9781231231999,2 .

Content Preview

All Django developers will deal with file uploads at some point. I contend that it’s a good practice to use models to capture the upload metadata and to track processing status. This article explains how and why. An e-commerce example Suppose your e-commerce application allows admins to upload CSV files to update product stock levels (a common requirement). A typical file may comprise a SKU and a stock level: 9781231231999,0 9781231231999,4 9781231231999,2 .

Vim macros for adding i18n support to Django templates

Published: Fri, 06 Jul 2012 00:00:00 +0000
Updated: Fri, 06 Jul 2012 00:00:00 +0000
UTC: 2012-07-06 00:00:00+00:00
URL: https://codeinthehole.com/tips/vim-macros-for-adding-i18n-support-to-django-templates/

Problem You want to add i18n support to an existing project. One part of this is modifying all templates to use the {% trans "..." %} block around all hard-coded strings. When you have a lot of templates, this gets pretty tedious. Solution Use Vim macros! Macro 1 - Convert tag text To convert <h1>Welcome to my site</h1> to <h1>{% trans "Welcome to my site" %}</h1> use the macro vitc{% trans "" %}<ESC>4hp which breaks down as:

Content Preview

Problem You want to add i18n support to an existing project. One part of this is modifying all templates to use the {% trans "..." %} block around all hard-coded strings. When you have a lot of templates, this gets pretty tedious. Solution Use Vim macros! Macro 1 - Convert tag text To convert <h1>Welcome to my site</h1> to <h1>{% trans "Welcome to my site" %}</h1> use the macro vitc{% trans "" %}<ESC>4hp which breaks down as:

A data migration for every Django project

Published: Sat, 16 Jun 2012 00:00:00 +0000
Updated: Sat, 16 Jun 2012 00:00:00 +0000
UTC: 2012-06-16 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-data-migration-for-every-django-project/

How to use a South data migration to avoid accidentally sending emails from example.com. Problem Consider the following snippet from Django’s docs1 for sending a confirmation email: from django.contrib.sites.models import Site from django.core.mail import send_mail def register_for_newsletter(request): current_site = Site.objects.get_current() send_mail( 'Thanks for subscribing to %s alerts' % current_site.name, 'Thanks for your subscription. We appreciate it.\n\n-The %s team.' % current_site.name, 'editor@%s' % current_site.domain, [user.email] ) Here the domain for the email sender is taken from the ‘current site’ instance, which is controlled by Django’s ‘Sites’ framework and accessible by a custom method on the manager of the Site model.

Content Preview

How to use a South data migration to avoid accidentally sending emails from example.com. Problem Consider the following snippet from Django’s docs1 for sending a confirmation email: from django.contrib.sites.models import Site from django.core.mail import send_mail def register_for_newsletter(request): current_site = Site.objects.get_current() send_mail( 'Thanks for subscribing to %s alerts' % current_site.name, 'Thanks for your subscription. We appreciate it.\n\n-The %s team.' % current_site.name, 'editor@%s' % current_site.domain, [user.email] ) Here the domain for the email sender is taken from the ‘current site’ instance, which is controlled by Django’s ‘Sites’ framework and accessible by a custom method on the manager of the Site model.

Django, Nginx, WSGI and encoded slashes

Published: Sat, 05 May 2012 00:00:00 +0000
Updated: Sat, 05 May 2012 00:00:00 +0000
UTC: 2012-05-05 00:00:00+00:00
URL: https://codeinthehole.com/tips/django-nginx-wsgi-and-encoded-slashes/

Problem You are serving a Django application using Nginx to proxy to an Apache server running mod_wsgi and you want to allow slashes in your URL keywords. For example, you may want to edit some attribute of the page at URL /; hence, you want to use a URL regex of the form: url(r'/edit/page/(?P<page_url>.*)/$', ...) and use the URL /edit/page/%2F/ to edit this page, where the third path segment is URL-encoded.

Content Preview

Problem You are serving a Django application using Nginx to proxy to an Apache server running mod_wsgi and you want to allow slashes in your URL keywords. For example, you may want to edit some attribute of the page at URL /; hence, you want to use a URL regex of the form: url(r'/edit/page/(?P<page_url>.*)/$', ...) and use the URL /edit/page/%2F/ to edit this page, where the third path segment is URL-encoded.

purl - immutable URL objects for Python

Published: Thu, 19 Apr 2012 00:00:00 +0000
Updated: Thu, 19 Apr 2012 00:00:00 +0000
UTC: 2012-04-19 00:00:00+00:00
URL: https://codeinthehole.com/projects/purl-immutable-url-objects-for-python/

Working with URLs in Python feels clunky when it should be pleasant. In urlparse and urllib, the standard library has all the functionality you need, but the code you have to write is often cumbersome and unclear. For instance, here’s a typical test method that makes an assertion about a query parameter: import urlparse def test_url_has_correct_query_parameter(self): url = get_url_from_somewhere() parse_result = urlparse.parseurl(url) query_params = urlparse.parse_qs(parse_result.query) self.assertEqual('testing', query_params['q'][0]) Not terrible, but could be more concise.

Content Preview

Working with URLs in Python feels clunky when it should be pleasant. In urlparse and urllib, the standard library has all the functionality you need, but the code you have to write is often cumbersome and unclear. For instance, here’s a typical test method that makes an assertion about a query parameter: import urlparse def test_url_has_correct_query_parameter(self): url = get_url_from_somewhere() parse_result = urlparse.parseurl(url) query_params = urlparse.parse_qs(parse_result.query) self.assertEqual('testing', query_params['q'][0]) Not terrible, but could be more concise.

Embedding HTML in Django messages

Published: Thu, 12 Apr 2012 00:00:00 +0000
Updated: Thu, 12 Apr 2012 00:00:00 +0000
UTC: 2012-04-12 00:00:00+00:00
URL: https://codeinthehole.com/tips/embedding-html-in-django-messages/

Problem You want to embed HTML within a message using Django’s messages framework. This is a reasonably common requirement - for instance, it’s common to want to include a link within the message, perhaps pointing the user towards a sign-in or registration page. This problem exists as of Django 1.4 but may be solved within the framework in later versions. Solution Use the extra_tags keyword argument to pass a flag indicating that the message is safe for rendering without escaping.

Content Preview

Problem You want to embed HTML within a message using Django’s messages framework. This is a reasonably common requirement - for instance, it’s common to want to include a link within the message, perhaps pointing the user towards a sign-in or registration page. This problem exists as of Django 1.4 but may be solved within the framework in later versions. Solution Use the extra_tags keyword argument to pass a flag indicating that the message is safe for rendering without escaping.

csvfilter - a Python command-line tool for manipulating CSV data

Published: Sun, 01 Apr 2012 00:00:00 +0000
Updated: Sun, 01 Apr 2012 00:00:00 +0000
UTC: 2012-04-01 00:00:00+00:00
URL: https://codeinthehole.com/projects/csvfilter-a-python-command-line-tool-for-manipulating-csv-data/

Problem You want a unix-like tool for manipulating CSV data from the command-line. The standard tools cut and awk aren’t always suitable as they don’t handle quoting and escaping which are common in CSVs. Solution Use the CSV manipulation function csvfilter, a simple Python library I’ve put together. Install with: pip install csvfilter Sample usage: # Pluck columns 2, 5 and 6 cat in.csv | csvfilter -f 2,5,6 > out.csv # Pluck all columns except 4 cat in.

Content Preview

Problem You want a unix-like tool for manipulating CSV data from the command-line. The standard tools cut and awk aren’t always suitable as they don’t handle quoting and escaping which are common in CSVs. Solution Use the CSV manipulation function csvfilter, a simple Python library I’ve put together. Install with: pip install csvfilter Sample usage: # Pluck columns 2, 5 and 6 cat in.csv | csvfilter -f 2,5,6 > out.csv # Pluck all columns except 4 cat in.

How to reload Django's URL config

Published: Wed, 21 Mar 2012 00:00:00 +0000
Updated: Wed, 21 Mar 2012 00:00:00 +0000
UTC: 2012-03-21 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-reload-djangos-url-config/

Problem For some reason, you need to reload your Django URL config. Normally, the root URL config will be imported and stored in memory when your server process starts up. Occasionally though, you may want to reload it. This can be the case if your URL configuration changes depending on certain parameters. Solution You can reload the URL config using the following snippet: import sys from django.conf import settings def reload_urlconf(urlconf=None): if urlconf is None: urlconf = settings.

Content Preview

Problem For some reason, you need to reload your Django URL config. Normally, the root URL config will be imported and stored in memory when your server process starts up. Occasionally though, you may want to reload it. This can be the case if your URL configuration changes depending on certain parameters. Solution You can reload the URL config using the following snippet: import sys from django.conf import settings def reload_urlconf(urlconf=None): if urlconf is None: urlconf = settings.

Validating international postcodes in Django

Published: Tue, 13 Mar 2012 00:00:00 +0000
Updated: Tue, 13 Mar 2012 00:00:00 +0000
UTC: 2012-03-13 00:00:00+00:00
URL: https://codeinthehole.com/tips/validating-international-postcodes-in-django/

Problem You want to validate a post/zip-code when you only know the country at runtime. Solution Use this snippet to dynamically fetch a validation method from Django’s suite of “localflavor” form fields using the ISO 3166-1 two character country code. def get_postcode_validator(country_code): # Django 1.3 uses 'UK' instead of GB - this changes in 1.4 if country_code == 'GB': country_code = 'UK' module_path = 'django.contrib.localflavor.%s' % country_code.lower() try: module = __import__(module_path, fromlist=['forms']) except ImportError: # No forms module for this country return lambda x: x fieldname_variants = ['%sPostcodeField', '%sPostCodeField', '%sPostalCodeField', '%sZipCodeField',] for variant in fieldname_variants: fieldname = variant % country_code.

Content Preview

Problem You want to validate a post/zip-code when you only know the country at runtime. Solution Use this snippet to dynamically fetch a validation method from Django’s suite of “localflavor” form fields using the ISO 3166-1 two character country code. def get_postcode_validator(country_code): # Django 1.3 uses 'UK' instead of GB - this changes in 1.4 if country_code == 'GB': country_code = 'UK' module_path = 'django.contrib.localflavor.%s' % country_code.lower() try: module = __import__(module_path, fromlist=['forms']) except ImportError: # No forms module for this country return lambda x: x fieldname_variants = ['%sPostcodeField', '%sPostCodeField', '%sPostalCodeField', '%sZipCodeField',] for variant in fieldname_variants: fieldname = variant % country_code.

How to sync PyCon videos to your iPhone

Published: Sun, 11 Mar 2012 00:00:00 +0000
Updated: Sun, 11 Mar 2012 00:00:00 +0000
UTC: 2012-03-11 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-sync-pycon-videos-to-your-iphone/

Problem You want to all the PyCon videos on your iPhone for offline viewing. Solution Use the following Python script to fetch pycon videos from YouTube and convert them to M4V format so they can be imported to iTunes. This gist is stale now - the code for downloading videos has been expanding into a [Github repository](https://github.com/codeinthehole/pyvideo2quicktime). To run the script: Ensure you have requests and BeautifulSoup installed in your Python environment; Ensure you have ffmpeg available on your path; Download youtube-dl to the same directory as this script Run this script using:

Content Preview

Problem You want to all the PyCon videos on your iPhone for offline viewing. Solution Use the following Python script to fetch pycon videos from YouTube and convert them to M4V format so they can be imported to iTunes. This gist is stale now - the code for downloading videos has been expanding into a [Github repository](https://github.com/codeinthehole/pyvideo2quicktime). To run the script: Ensure you have requests and BeautifulSoup installed in your Python environment; Ensure you have ffmpeg available on your path; Download youtube-dl to the same directory as this script Run this script using:

Tips for using a git pre-commit hook

Published: Mon, 05 Mar 2012 00:00:00 +0000
Updated: Mon, 05 Mar 2012 00:00:00 +0000
UTC: 2012-03-05 00:00:00+00:00
URL: https://codeinthehole.com/tips/tips-for-using-a-git-pre-commit-hook/

Here’s a few tips for using a Git pre-commit hook. Keep your hook script in source control Commit your hook script (say pre-commit.sh) at the root of your project and include the installation instructions in your README/documentation to encourage all developers use it. Installation is nothing more than: ln -s ../../pre-commit.sh .git/hooks/pre-commit Then everyone benefits from running the same set of tests before committing and updates are picked up automatically.

Content Preview

Here’s a few tips for using a Git pre-commit hook. Keep your hook script in source control Commit your hook script (say pre-commit.sh) at the root of your project and include the installation instructions in your README/documentation to encourage all developers use it. Installation is nothing more than: ln -s ../../pre-commit.sh .git/hooks/pre-commit Then everyone benefits from running the same set of tests before committing and updates are picked up automatically.

Confoo 2012 presentations

Published: Sat, 03 Mar 2012 00:00:00 +0000
Updated: Sat, 03 Mar 2012 00:00:00 +0000
UTC: 2012-03-03 00:00:00+00:00
URL: https://codeinthehole.com/talks/confoo-2012-presentations/

I gave two presentations at Confoo 2012 - below are the abstracts and slides. Migrating from PHP to Python and Django Abstract: Many programmers cut their teeth with PHP before getting interested in Python. However, while the languages seem similar at first, there are many new concepts to grasp and a rich ecosystem of tools to master. This isn’t easy, but with the right advice, a strong PHP developer can become an excellent Python developer very quickly.

Content Preview

I gave two presentations at Confoo 2012 - below are the abstracts and slides. Migrating from PHP to Python and Django Abstract: Many programmers cut their teeth with PHP before getting interested in Python. However, while the languages seem similar at first, there are many new concepts to grasp and a rich ecosystem of tools to master. This isn’t easy, but with the right advice, a strong PHP developer can become an excellent Python developer very quickly.

Testing HTTPS handling in Django

Published: Thu, 01 Mar 2012 00:00:00 +0000
Updated: Thu, 01 Mar 2012 00:00:00 +0000
UTC: 2012-03-01 00:00:00+00:00
URL: https://codeinthehole.com/tips/testing-https-handling-in-django/

Problem You want to test how your application handles HTTPS requests. Solution Use the following to simulate a HTTPS request using the Django test client: from django.test.client import Client client = Client() response = client.get(url, **{'wsgi.url_scheme': 'https'}) Discussion The standard way to test for a HTTPS request is using the is_secure method of the django.http.HttpRequest class1 and its subclasses. As of Django 1.3, the implementation of this method checks whether an environmental variable HTTPS is equal to “on”:

Content Preview

Problem You want to test how your application handles HTTPS requests. Solution Use the following to simulate a HTTPS request using the Django test client: from django.test.client import Client client = Client() response = client.get(url, **{'wsgi.url_scheme': 'https'}) Discussion The standard way to test for a HTTPS request is using the is_secure method of the django.http.HttpRequest class1 and its subclasses. As of Django 1.3, the implementation of this method checks whether an environmental variable HTTPS is equal to “on”:

Prefer data migrations to initial data

Published: Sat, 25 Feb 2012 00:00:00 +0000
Updated: Sat, 25 Feb 2012 00:00:00 +0000
UTC: 2012-02-25 00:00:00+00:00
URL: https://codeinthehole.com/tips/prefer-data-migrations-to-initial-data/

Django provides several mechanisms for loading initial data for models, such as leveraging JSON fixtures or files of raw SQL - James Bennett offers a good overview. Each documented method involves initialising data as part of the syncdb event, either by loading a fixture file or by hooking into the syncdb signal. However, there is a serious pitfall with these techniques, as described in the Django docs: This is extremely convenient, but be careful: remember that the data will be refreshed every time you run syncdb.

Content Preview

Django provides several mechanisms for loading initial data for models, such as leveraging JSON fixtures or files of raw SQL - James Bennett offers a good overview. Each documented method involves initialising data as part of the syncdb event, either by loading a fixture file or by hooking into the syncdb signal. However, there is a serious pitfall with these techniques, as described in the Django docs: This is extremely convenient, but be careful: remember that the data will be refreshed every time you run syncdb.

A Fabric function for git tagging

Published: Thu, 09 Feb 2012 00:00:00 +0000
Updated: Thu, 09 Feb 2012 00:00:00 +0000
UTC: 2012-02-09 00:00:00+00:00
URL: https://codeinthehole.com/tips/a-fabric-function-for-git-tagging/

Listed below is a Fabric function for determining the appropriate git reference to deploy during a deployment. It works well with projects run using the git-flow development model. Set-up Assume there is a test environment where: the QA team to assess release candidates developers to run integration tests developers can deploy ‘debug’ builds from a specific (untagged) commit There will also be stage and production environments. Fabric function The following function can be used as part of Fabric build script.

Content Preview

Listed below is a Fabric function for determining the appropriate git reference to deploy during a deployment. It works well with projects run using the git-flow development model. Set-up Assume there is a test environment where: the QA team to assess release candidates developers to run integration tests developers can deploy ‘debug’ builds from a specific (untagged) commit There will also be stage and production environments. Fabric function The following function can be used as part of Fabric build script.

Solving MySQL connection problems caused by a dead name server

Published: Thu, 02 Feb 2012 00:00:00 +0000
Updated: Thu, 02 Feb 2012 00:00:00 +0000
UTC: 2012-02-02 00:00:00+00:00
URL: https://codeinthehole.com/tips/solving-mysql-connection-problems-caused-by-a-dead-name-server/

A client site went down today. This is what happened and how it was fixed. Symptoms The immediate symptom is that your application servers can’t connect to your database servers. Attempted connections get an error message: Can’t connect to MySQL server on ‘10.10.110.11’ (111) The relevant MySQL process list reveals a long list of attempted connections in state login: root@server-db1:~ $ mysqladmin processlist +-----+----------------------+--------------------+----+---------+------+-------+ | Id | User | Host | db | Command | Time | State | +-----+----------------------+--------------------+----+---------+------+-------+ .

Content Preview

A client site went down today. This is what happened and how it was fixed. Symptoms The immediate symptom is that your application servers can’t connect to your database servers. Attempted connections get an error message: Can’t connect to MySQL server on ‘10.10.110.11’ (111) The relevant MySQL process list reveals a long list of attempted connections in state login: root@server-db1:~ $ mysqladmin processlist +-----+----------------------+--------------------+----+---------+------+-------+ | Id | User | Host | db | Command | Time | State | +-----+----------------------+--------------------+----+---------+------+-------+ .

Auto-setting terminal titles for python virtual environments

Published: Mon, 23 Jan 2012 00:00:00 +0000
Updated: Mon, 23 Jan 2012 00:00:00 +0000
UTC: 2012-01-23 00:00:00+00:00
URL: https://codeinthehole.com/tips/auto-setting-terminal-titles-for-python-virtual-environments/

Problem You’re a python hacker using virtualenv and virtualenvwrapper on a range of projects. After a few hours in the office and much context switching, your terminal emulator is bursting with open tabs with the unhelpful title ‘bash’ and it’s difficult to remember which tab is for which project. This is making you unhappy. Solution Use your postactivate1 script to set the terminal title when you activate a virtual environment. Add something similar to the following to your postactivate script:

Content Preview

Problem You’re a python hacker using virtualenv and virtualenvwrapper on a range of projects. After a few hours in the office and much context switching, your terminal emulator is bursting with open tabs with the unhelpful title ‘bash’ and it’s difficult to remember which tab is for which project. This is making you unhappy. Solution Use your postactivate1 script to set the terminal title when you activate a virtual environment. Add something similar to the following to your postactivate script:

Introducing unittest-xml: testing XML in Python

Published: Fri, 13 Jan 2012 00:00:00 +0000
Updated: Fri, 13 Jan 2012 00:00:00 +0000
UTC: 2012-01-13 00:00:00+00:00
URL: https://codeinthehole.com/projects/introducing-unittest-xml-testing-xml-in-python/

For some reason, I keep finding myself writing unit tests that need to make assertions about an XML document. To keep things DRY, I’ve packaged up my custom assertion methods as a PyPi module: unittest-xml. There’s a small chance it may be useful to others. Sample Usage Enable the additional assert methods using a mixin: import unittest from xmltest import XMLAssertions class SampleTestCase(unittest.TestCase, XMLAssertions): ... Now suppose that the expected XML from some SUT1 is:

Content Preview

For some reason, I keep finding myself writing unit tests that need to make assertions about an XML document. To keep things DRY, I’ve packaged up my custom assertion methods as a PyPi module: unittest-xml. There’s a small chance it may be useful to others. Sample Usage Enable the additional assert methods using a mixin: import unittest from xmltest import XMLAssertions class SampleTestCase(unittest.TestCase, XMLAssertions): ... Now suppose that the expected XML from some SUT1 is:

Rewriting codeinthehole.com

Published: Fri, 06 Jan 2012 00:00:00 +0000
Updated: Fri, 06 Jan 2012 00:00:00 +0000
UTC: 2012-01-06 00:00:00+00:00
URL: https://codeinthehole.com/news/rewriting-codeintheholecom/

I rewrote this blog recently with the following aims: to make it as easy as possible to create a new article, using reStructuredText (RST); to clean up and simplify the design, focussing on the readability of articles that include code snippets. I knew that there were various static blogs out there, with many supporting RST, but I still fancied the challenge of crafting something specific to my needs. There’s nothing wrong with wheel re-invention if you want to learn about wheels.

Content Preview

I rewrote this blog recently with the following aims: to make it as easy as possible to create a new article, using reStructuredText (RST); to clean up and simplify the design, focussing on the readability of articles that include code snippets. I knew that there were various static blogs out there, with many supporting RST, but I still fancied the challenge of crafting something specific to my needs. There’s nothing wrong with wheel re-invention if you want to learn about wheels.

How to set-up MySQL for Python on Ubuntu

Published: Thu, 05 Jan 2012 00:00:00 +0000
Updated: Thu, 05 Jan 2012 00:00:00 +0000
UTC: 2012-01-05 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-set-up-mysql-for-python-on-ubuntu/

This is just for my own reference as I always forget the dependencies for setting up MySQL on a new machine. Installation Starting with a vanilla Lucid install1, install pip and upgrade to the latest version: apt-get install python-pip pip install -U pip Next, install the required development packages: apt-get install python-dev libmysqlclient-dev then pip install MySQL-python should complete successfully. Symptoms of missing headers Without libmysqlclient-dev, you’ll see something like this:

Content Preview

This is just for my own reference as I always forget the dependencies for setting up MySQL on a new machine. Installation Starting with a vanilla Lucid install1, install pip and upgrade to the latest version: apt-get install python-pip pip install -U pip Next, install the required development packages: apt-get install python-dev libmysqlclient-dev then pip install MySQL-python should complete successfully. Symptoms of missing headers Without libmysqlclient-dev, you’ll see something like this:

Using pip and requirements.txt to install from the HEAD of a Github branch

Published: Sat, 20 Aug 2011 00:00:00 +0000
Updated: Sat, 20 Aug 2011 00:00:00 +0000
UTC: 2011-08-20 00:00:00+00:00
URL: https://codeinthehole.com/tips/using-pip-and-requirementstxt-to-install-from-the-head-of-a-github-branch/

Problem The python package installer pip can be used to install directly from Github, like so: pip install git+git://github.com/tangentlabs/django-oscar.git#egg=django-oscar This will install from the HEAD of the master branch. However, when you use pip freeze to export your dependencies (usually to a requirements.txt file), pip will fix the reference to a specific commit by including its ID within the URL: $ pip freeze | grep oscar -e git://github.com/tangentlabs/django-oscar.git@d636b803d98cd1d3edd01821d4fb2a01ce215ee4#egg=django_oscar-dev Hence running pip install -r requirements.

Content Preview

Problem The python package installer pip can be used to install directly from Github, like so: pip install git+git://github.com/tangentlabs/django-oscar.git#egg=django-oscar This will install from the HEAD of the master branch. However, when you use pip freeze to export your dependencies (usually to a requirements.txt file), pip will fix the reference to a specific commit by including its ID within the URL: $ pip freeze | grep oscar -e git://github.com/tangentlabs/django-oscar.git@d636b803d98cd1d3edd01821d4fb2a01ce215ee4#egg=django_oscar-dev Hence running pip install -r requirements.

Console logging to STDOUT in Django

Published: Tue, 16 Aug 2011 00:00:00 +0000
Updated: Tue, 16 Aug 2011 00:00:00 +0000
UTC: 2011-08-16 00:00:00+00:00
URL: https://codeinthehole.com/tips/console-logging-to-stdout-in-django/

Problem By default in Python (and Django), the documented console handler emits to STDERR, but you want it to use STDOUT instead. This is often desired for management commands that run as cronjobs. Solution For Python 2.6, use the following LOGGING config in your settings to specify a different output stream: import sys LOGGING = { 'handlers': { 'console':{ 'level':'INFO', 'class':'logging.StreamHandler', 'strm': sys.stdout }, ... } } For Python 2.7+, the keyword argument to the constructor of logging.

Content Preview

Problem By default in Python (and Django), the documented console handler emits to STDERR, but you want it to use STDOUT instead. This is often desired for management commands that run as cronjobs. Solution For Python 2.6, use the following LOGGING config in your settings to specify a different output stream: import sys LOGGING = { 'handlers': { 'console':{ 'level':'INFO', 'class':'logging.StreamHandler', 'strm': sys.stdout }, ... } } For Python 2.7+, the keyword argument to the constructor of logging.

Running django cronjobs within a virtualenv

Published: Thu, 11 Aug 2011 00:00:00 +0000
Updated: Thu, 11 Aug 2011 00:00:00 +0000
UTC: 2011-08-11 00:00:00+00:00
URL: https://codeinthehole.com/tips/running-django-cronjobs-within-a-virtualenv/

If you use virtual environments on your django servers, then getting manage.py commands to run from cron is a little tricky. You need to activate the virtualenv before running the command and so you might think that the following would work: */10 * * * * root source /var/www/mysite/virtualenvs/dev/bin/activate && /var/www/mysite/build/dev/manage.py some_custom_command > /dev/null It doesn’t, although it’s tricky to spot why as /var/log/syslog doesn’t give much away (Debian-specific of course).

Content Preview

If you use virtual environments on your django servers, then getting manage.py commands to run from cron is a little tricky. You need to activate the virtualenv before running the command and so you might think that the following would work: */10 * * * * root source /var/www/mysite/virtualenvs/dev/bin/activate && /var/www/mysite/build/dev/manage.py some_custom_command > /dev/null It doesn’t, although it’s tricky to spot why as /var/log/syslog doesn’t give much away (Debian-specific of course).

Multi-scp: copying a file to all hosts in your SSH config

Published: Sun, 22 May 2011 00:00:00 +0000
Updated: Sun, 22 May 2011 00:00:00 +0000
UTC: 2011-05-22 00:00:00+00:00
URL: https://codeinthehole.com/projects/multi-scp-copying-a-file-to-all-hosts-in-your-ssh-config/

I am totally reliant on my bash aliases and readline configuration. When working on a new server, the first thing I do is copy over my local Bash (~/.bashrc) and Readline (~/.inputrc) configuration files. One mildly annoying issue is when you update a config file, it’s a pain to copy it onto all your remote hosts. To scratch this itch, I wrote a simple node.js executable for copying a file to all the defined hosts in your ~/.

Content Preview

I am totally reliant on my bash aliases and readline configuration. When working on a new server, the first thing I do is copy over my local Bash (~/.bashrc) and Readline (~/.inputrc) configuration files. One mildly annoying issue is when you update a config file, it’s a pain to copy it onto all your remote hosts. To scratch this itch, I wrote a simple node.js executable for copying a file to all the defined hosts in your ~/.

Coloured output while doing TDD with Django and Fabric

Published: Wed, 20 Apr 2011 00:00:00 +0000
Updated: Wed, 20 Apr 2011 00:00:00 +0000
UTC: 2011-04-20 00:00:00+00:00
URL: https://codeinthehole.com/tips/coloured-output-while-doing-tdd-with-django-and-fabric/

I’m a big fan of using PHPUnit with console colours turned on (using the --colors option). Eg: It helps gets into the natural “red, green, refactor” rhythm. I’m currently totally immersed in Django, and greatly miss the lack of colour support within the “test” management command. A simple workaround for this is to use Fabric with a few modified color commands. Your fabric file should include the following: from fabric.colors import _wrap_with green_bg = _wrap_with('42') red_bg = _wrap_with('41') # Set the list of apps to test env.

Content Preview

I’m a big fan of using PHPUnit with console colours turned on (using the --colors option). Eg: It helps gets into the natural “red, green, refactor” rhythm. I’m currently totally immersed in Django, and greatly miss the lack of colour support within the “test” management command. A simple workaround for this is to use Fabric with a few modified color commands. Your fabric file should include the following: from fabric.colors import _wrap_with green_bg = _wrap_with('42') red_bg = _wrap_with('41') # Set the list of apps to test env.

Domain-model-mapper - A PHP Data Mapper implementation

Published: Sun, 13 Mar 2011 00:00:00 +0000
Updated: Sun, 13 Mar 2011 00:00:00 +0000
UTC: 2011-03-13 00:00:00+00:00
URL: https://codeinthehole.com/projects/domain-model-mapper-a-php-data-mapper-implementation/

At various PHP conferences and meetups over the last few weeks, I’ve seen attention drawn to the Data Mapper design pattern. This is an elegant pattern that splits the responsibilities of business logic and persistence. In the words of pattern supremo Martin Fowler: The Data Mapper is a layer of software that separates the in-memory objects from the database. Its responsibility is to transfer data between the two and also to isolate them from each other.

Content Preview

At various PHP conferences and meetups over the last few weeks, I’ve seen attention drawn to the Data Mapper design pattern. This is an elegant pattern that splits the responsibilities of business logic and persistence. In the words of pattern supremo Martin Fowler: The Data Mapper is a layer of software that separates the in-memory objects from the database. Its responsibility is to transfer data between the two and also to isolate them from each other.

Commandlinefu presentation at Confoo 2011

Published: Sat, 12 Mar 2011 00:00:00 +0000
Updated: Sat, 12 Mar 2011 00:00:00 +0000
UTC: 2011-03-12 00:00:00+00:00
URL: https://codeinthehole.com/talks/commandlinefu-presentation-at-confoo-2011/

The presentation I gave at Confoo 2011 is now available online, there is feedback on joind.in. The slides are written in Eric Meyer’s S5, which is a XHTML, CSS and Javascript based slide show format. I enjoyed using this as it was easy to port my styles across from the website CSS file, it also means the presentation is viewable as a normal webpage (no flash required). However, the downside is that it is not easy to submit the slides to Slideshare which is the common thing to do.

Content Preview

The presentation I gave at Confoo 2011 is now available online, there is feedback on joind.in. The slides are written in Eric Meyer’s S5, which is a XHTML, CSS and Javascript based slide show format. I enjoyed using this as it was easy to port my styles across from the website CSS file, it also means the presentation is viewable as a normal webpage (no flash required). However, the downside is that it is not easy to submit the slides to Slideshare which is the common thing to do.

The British way of dealing with foreign APIs.

Published: Fri, 04 Feb 2011 00:00:00 +0000
Updated: Fri, 04 Feb 2011 00:00:00 +0000
UTC: 2011-02-04 00:00:00+00:00
URL: https://codeinthehole.com/tidbits/the-british-way-of-dealing-with-foreign-apis/

A bad joke told in Python: def call_foreign_api(str): try: foreign_api(str) except NotUnderstoodError: foreign_api(str.upper())

Content Preview

A bad joke told in Python: def call_foreign_api(str): try: foreign_api(str) except NotUnderstoodError: foreign_api(str.upper())

nurl - an immutable URL object for node.js

Published: Wed, 10 Nov 2010 00:00:00 +0000
Updated: Wed, 10 Nov 2010 00:00:00 +0000
UTC: 2010-11-10 00:00:00+00:00
URL: https://codeinthehole.com/projects/nurl-an-immutable-url-object-for-nodejs/

I’m a big fan of Value Objects - some of the most useful classes I’ve ever written have been lightweight wrappers around strings. For my first contribution to the node.js ecosystem, I’ve published a simple, immutable URL object to the Node package manager (npm) directory. It provides a light-weight, immutable URL object that encapsulates the functionality of the existing ‘url’ and ‘querystring’ modules in a single object as well as offering various extension methods that make working with URLs easy.

Content Preview

I’m a big fan of Value Objects - some of the most useful classes I’ve ever written have been lightweight wrappers around strings. For my first contribution to the node.js ecosystem, I’ve published a simple, immutable URL object to the Node package manager (npm) directory. It provides a light-weight, immutable URL object that encapsulates the functionality of the existing ‘url’ and ‘querystring’ modules in a single object as well as offering various extension methods that make working with URLs easy.

mysqldump with wildcard table matching

Published: Fri, 05 Nov 2010 00:00:00 +0000
Updated: Fri, 05 Nov 2010 00:00:00 +0000
UTC: 2010-11-05 00:00:00+00:00
URL: https://codeinthehole.com/tips/mysqldump-with-wildcard-table-matching/

Ever wanted to use mysqldump to dump tables that match a wildcard pattern? I have. It’s not currently supported as an option but can be achieved with a little bash magic. Here’s how: #!/bin/bash if [ $# -lt 2 ] then echo "Usage: `basename $0` database wildcardpattern" echo "Eg: `basename $0` mydatabase App_%" exit 1 fi database=$1 pattern=$2 mysqldump $database `mysql -ND $database -e "SHOW TABLES LIKE '$pattern'" | awk '{printf $1" "}'` This uses a simple SQL query to extract all the table names that match the pattern and concatenate them in the format that mysqldump expects.

Content Preview

Ever wanted to use mysqldump to dump tables that match a wildcard pattern? I have. It’s not currently supported as an option but can be achieved with a little bash magic. Here’s how: #!/bin/bash if [ $# -lt 2 ] then echo "Usage: `basename $0` database wildcardpattern" echo "Eg: `basename $0` mydatabase App_%" exit 1 fi database=$1 pattern=$2 mysqldump $database `mysql -ND $database -e "SHOW TABLES LIKE '$pattern'" | awk '{printf $1" "}'` This uses a simple SQL query to extract all the table names that match the pattern and concatenate them in the format that mysqldump expects.

How to sync a MySQL table between two remote databases.

Published: Fri, 03 Sep 2010 00:00:00 +0000
Updated: Fri, 03 Sep 2010 00:00:00 +0000
UTC: 2010-09-03 00:00:00+00:00
URL: https://codeinthehole.com/tips/how-to-sync-a-mysql-table-between-two-remote-databases/

Definitely tricker than you might think. Seems like it should be trivial using SELECT ... INTO OUTFILE and LOAD DATA INFILE ... to make the transfer via dumping the table into a temporary file. However, SELECT ... INTO OUTFILE creates a file on the remote server rather than locally. This prevents the use of LOAD DATA INFILE for the second step as the file being loaded has to be local or on the destination server.

Content Preview

Definitely tricker than you might think. Seems like it should be trivial using SELECT ... INTO OUTFILE and LOAD DATA INFILE ... to make the transfer via dumping the table into a temporary file. However, SELECT ... INTO OUTFILE creates a file on the remote server rather than locally. This prevents the use of LOAD DATA INFILE for the second step as the file being loaded has to be local or on the destination server.

Phing trick for avoiding deploying debug code

Published: Sun, 22 Aug 2010 00:00:00 +0000
Updated: Sun, 22 Aug 2010 00:00:00 +0000
UTC: 2010-08-22 00:00:00+00:00
URL: https://codeinthehole.com/tips/phing-trick-for-avoiding-deploying-debug-code/

As the saying goes: Fool me once, shame on you; fool me twice, shame on me Ensuring mistakes aren’t repeated is a commonplace activity for any development team. This can manifest itself in many ways such as writing regression tests, stepping up your code reviews, adding stories to a testing plan or humiliating the developer in question through use of an unusual (dunce’s) hat. We had an issue recently where some debugging code got committed and wasn’t picked up during testing.

Content Preview

As the saying goes: Fool me once, shame on you; fool me twice, shame on me Ensuring mistakes aren’t repeated is a commonplace activity for any development team. This can manifest itself in many ways such as writing regression tests, stepping up your code reviews, adding stories to a testing plan or humiliating the developer in question through use of an unusual (dunce’s) hat. We had an issue recently where some debugging code got committed and wasn’t picked up during testing.

Return false with prudence

Published: Thu, 28 Jan 2010 00:00:00 +0000
Updated: Thu, 28 Jan 2010 00:00:00 +0000
UTC: 2010-01-28 00:00:00+00:00
URL: https://codeinthehole.com/tips/return-false-with-prudence/

From “Javascript: the good parts”: It is rarely possible for standards committees to remove imperfections from a language because doing so would cause the breakdage of all of the bad programs that depend on those bad parts. They are usually powerless to do anything except heap more features on top of the existing pile of imperfections. Douglas Crockford’s terse yet lucid javascript primer makes some excellent points on writing in a language with more than its fair of share of shortcomings.

Content Preview

From “Javascript: the good parts”: It is rarely possible for standards committees to remove imperfections from a language because doing so would cause the breakdage of all of the bad programs that depend on those bad parts. They are usually powerless to do anything except heap more features on top of the existing pile of imperfections. Douglas Crockford’s terse yet lucid javascript primer makes some excellent points on writing in a language with more than its fair of share of shortcomings.

Javascript refactoring for customising shared libraries

Published: Tue, 13 Oct 2009 00:00:00 +0000
Updated: Tue, 13 Oct 2009 00:00:00 +0000
UTC: 2009-10-13 00:00:00+00:00
URL: https://codeinthehole.com/tips/javascript-refactoring-for-customising-shared-libraries/

One difficulty working with a shared in-house framework is it is difficult to maintain a common javascript file that is valid across multiple applications. This is currently an issue we face at Tangent, where we run a generic e-commerce platform which we customise to the needs of each client. Most of these e-commerce applications have a javascript-rich checkout page whose functionality differs in small yet significant ways such as the required and optional fields within a delivery address, or the range of delivery options available.

Content Preview

One difficulty working with a shared in-house framework is it is difficult to maintain a common javascript file that is valid across multiple applications. This is currently an issue we face at Tangent, where we run a generic e-commerce platform which we customise to the needs of each client. Most of these e-commerce applications have a javascript-rich checkout page whose functionality differs in small yet significant ways such as the required and optional fields within a delivery address, or the range of delivery options available.

A pseudo-code job advert and its discontents

Published: Thu, 09 Jul 2009 00:00:00 +0000
Updated: Thu, 09 Jul 2009 00:00:00 +0000
UTC: 2009-07-09 00:00:00+00:00
URL: https://codeinthehole.com/tidbits/a-pseudo-code-job-advert-and-its-discontents/

Based on the success of a highly tongue-in-cheek ad for a project manager, we recently experimented with a similar approach for finding developers: a job ad written in PHP. Now I appreciate this is deeply lame, but the results of the campaign were quite surprising - more of which in a minute. First, hold your nose and parse the following: <?php class TangentLabs extends HoxtonWebCompany implements InnovativeWebsites, WorldBeatingApplications, IngeniousECommerce { const vacancyForBrilliantDevelopers = true; public $benefits = array( 'Smartest web agency in London', 'Working on inventive web apps, using cutting-edge technology', 'Super-friendly work environment, working within genuinely brilliant dev team', ); public $drawbacks = null; public $sampleProjects = array( 'http://www.

Content Preview

Based on the success of a highly tongue-in-cheek ad for a project manager, we recently experimented with a similar approach for finding developers: a job ad written in PHP. Now I appreciate this is deeply lame, but the results of the campaign were quite surprising - more of which in a minute. First, hold your nose and parse the following: <?php class TangentLabs extends HoxtonWebCompany implements InnovativeWebsites, WorldBeatingApplications, IngeniousECommerce { const vacancyForBrilliantDevelopers = true; public $benefits = array( 'Smartest web agency in London', 'Working on inventive web apps, using cutting-edge technology', 'Super-friendly work environment, working within genuinely brilliant dev team', ); public $drawbacks = null; public $sampleProjects = array( 'http://www.

Deploying cron jobs using Phing

Published: Sun, 31 May 2009 00:00:00 +0000
Updated: Sun, 31 May 2009 00:00:00 +0000
UTC: 2009-05-31 00:00:00+00:00
URL: https://codeinthehole.com/tips/deploying-cron-jobs-using-phing/

Deploying applications that depend on cron-jobs can be a pain. However, Phing can be used to make such deployments easy - here’s how… Consider an application folder structure as follows: /builds /development /test /stage /src /cron.d appname-__BUILD__-order-processing /scripts /order-processing handle-ready-to-ship-orders.php handle-cancellations.php ... /public /classes ... All development work takes place within the /src folder while the /builds/* folders are used as targets in deployment. This system allows multiple builds to happily co-exist on the same server and the whole application infrastructure to be moved between servers easily as the structure in source control mirrors that of the server.

Content Preview

Deploying applications that depend on cron-jobs can be a pain. However, Phing can be used to make such deployments easy - here’s how… Consider an application folder structure as follows: /builds /development /test /stage /src /cron.d appname-__BUILD__-order-processing /scripts /order-processing handle-ready-to-ship-orders.php handle-cancellations.php ... /public /classes ... All development work takes place within the /src folder while the /builds/* folders are used as targets in deployment. This system allows multiple builds to happily co-exist on the same server and the whole application infrastructure to be moved between servers easily as the structure in source control mirrors that of the server.

Auto-generating an FAQ with Prototype

Published: Mon, 25 May 2009 00:00:00 +0000
Updated: Mon, 25 May 2009 00:00:00 +0000
UTC: 2009-05-25 00:00:00+00:00
URL: https://codeinthehole.com/tips/auto-generating-an-faq-with-prototype/

Have just been writing an FAQ page for commandlinefu.com. Documenting is always tiresome, but FAQs particularly so when hand-coding the HTML links between each question and the summary table at the top of the page. Javascript to the rescue: I cobbled together a quick Prototype script which automatically generates the FAQ summary links by parsing the DOM for the appropriate links: document.observe("dom:loaded", function () { $$("a.question").each(function (ele) { var id = ele.

Content Preview

Have just been writing an FAQ page for commandlinefu.com. Documenting is always tiresome, but FAQs particularly so when hand-coding the HTML links between each question and the summary table at the top of the page. Javascript to the rescue: I cobbled together a quick Prototype script which automatically generates the FAQ summary links by parsing the DOM for the appropriate links: document.observe("dom:loaded", function () { $$("a.question").each(function (ele) { var id = ele.

Inspecting Javascript objects

Published: Sun, 24 May 2009 00:00:00 +0000
Updated: Sun, 24 May 2009 00:00:00 +0000
UTC: 2009-05-24 00:00:00+00:00
URL: https://codeinthehole.com/tips/inspecting-javascript-objects/

Learning Ruby or Python from the command-line prompt is greatly enhanced by the built-in inspection methods these languages provide. These allow the methods and properties of an object to be interrogated via a simple method call which returns an array of all property or method names. For instance, in IRB (the interactive Ruby command-line) we can interrogate the integer object: irb(main):001:0>my_int = 1 irb(main):002:0>my_int.methods This returns an array of all method names for integer objects:

Content Preview

Learning Ruby or Python from the command-line prompt is greatly enhanced by the built-in inspection methods these languages provide. These allow the methods and properties of an object to be interrogated via a simple method call which returns an array of all property or method names. For instance, in IRB (the interactive Ruby command-line) we can interrogate the integer object: irb(main):001:0>my_int = 1 irb(main):002:0>my_int.methods This returns an array of all method names for integer objects:

Phing, Xinc and Nabaztags

Published: Wed, 06 May 2009 00:00:00 +0000
Updated: Wed, 06 May 2009 00:00:00 +0000
UTC: 2009-05-06 00:00:00+00:00
URL: https://codeinthehole.com/tips/phing-xinc-and-nabaztags/

Finally got around to setting up continuous integration for some of the projects that comprise the day-job. We’re using the PEAR package Xinc, which has proved to be excellent thus far - especially as it integrates so well with my deployment tool of choice: Phing. Part of the fun in setting it up was looking for suitable feedback mechanisms or devices. Email notifications are a given but there are a range of more interesting feedback mechanisms available such as toolbar notifications, remote-controlled lava lamps, or plain humiliation tactics (such as making the person who broke the build wear the dunce’s hat till it is fixed).

Content Preview

Finally got around to setting up continuous integration for some of the projects that comprise the day-job. We’re using the PEAR package Xinc, which has proved to be excellent thus far - especially as it integrates so well with my deployment tool of choice: Phing. Part of the fun in setting it up was looking for suitable feedback mechanisms or devices. Email notifications are a given but there are a range of more interesting feedback mechanisms available such as toolbar notifications, remote-controlled lava lamps, or plain humiliation tactics (such as making the person who broke the build wear the dunce’s hat till it is fixed).

Ingenious use of an anonymous function

Published: Wed, 15 Apr 2009 00:00:00 +0000
Updated: Wed, 15 Apr 2009 00:00:00 +0000
UTC: 2009-04-15 00:00:00+00:00
URL: https://codeinthehole.com/tidbits/ingenious-use-of-an-anonymous-function/

Just stumbled across a gem of a question whilst idly browsing Stack Overflow: Design a function f, such that: f(f(n)) == -n where n is a 32-bit signed integer; you can’t use complex numbers arithmetic. Interesting in its own right, what makes this particularly intriguing is that the question doesn’t specify a language to use - indeed, the choice of language has a major say in the range of solutions available.

Content Preview

Just stumbled across a gem of a question whilst idly browsing Stack Overflow: Design a function f, such that: f(f(n)) == -n where n is a 32-bit signed integer; you can’t use complex numbers arithmetic. Interesting in its own right, what makes this particularly intriguing is that the question doesn’t specify a language to use - indeed, the choice of language has a major say in the range of solutions available.

Using a Phing filter to flush browser caches

Published: Sun, 15 Mar 2009 00:00:00 +0000
Updated: Sun, 15 Mar 2009 00:00:00 +0000
UTC: 2009-03-15 00:00:00+00:00
URL: https://codeinthehole.com/tips/using-a-phing-filter-to-flush-browser-caches/

A quick Phing tip that’s made my life easier when deploying new versions of commandlinefu.com. One of the key performance recommendations from Steve Souders’ excellent “High Performance Websites” is to use Expires HTTP headers to set far-future expiration dates for your site components (such as images, Javascript files and CSS stylesheets). This way, browsers can cache the files between requests giving a performance boost to your site. Assuming you’re using Apache for serving, the following settings can be used to set these headers for all Javascript and CSS files (there are a few alternative ways of achieving the same result):

Content Preview

A quick Phing tip that’s made my life easier when deploying new versions of commandlinefu.com. One of the key performance recommendations from Steve Souders’ excellent “High Performance Websites” is to use Expires HTTP headers to set far-future expiration dates for your site components (such as images, Javascript files and CSS stylesheets). This way, browsers can cache the files between requests giving a performance boost to your site. Assuming you’re using Apache for serving, the following settings can be used to set these headers for all Javascript and CSS files (there are a few alternative ways of achieving the same result):

The most important command-line tip - incremental history searching with .inputrc

Published: Tue, 03 Feb 2009 00:00:00 +0000
Updated: Tue, 03 Feb 2009 00:00:00 +0000
UTC: 2009-02-03 00:00:00+00:00
URL: https://codeinthehole.com/tips/the-most-important-command-line-tip-incremental-history-searching-with-inputrc/

Getting <www.commandlinefu.com> off the ground has renewed my interest in Bash, UNIX and all things command-line. Powerful one-liners are things of beauty and are worth collecting; however what I consider to be the most influential command-line tip I know covers four: "\e[A": history-search-backward "\e[B": history-search-forward "\e[C": forward-char "\e[D": backward-char These lines need to be placed in your ~/.inputrc file, the start-up script for the Readline utility used by Bash (as well as several other applications) and others).

Content Preview

Getting <www.commandlinefu.com> off the ground has renewed my interest in Bash, UNIX and all things command-line. Powerful one-liners are things of beauty and are worth collecting; however what I consider to be the most influential command-line tip I know covers four: "\e[A": history-search-backward "\e[B": history-search-forward "\e[C": forward-char "\e[D": backward-char These lines need to be placed in your ~/.inputrc file, the start-up script for the Readline utility used by Bash (as well as several other applications) and others).

Current pet project: Command-Line-Fu

Published: Thu, 22 Jan 2009 00:00:00 +0000
Updated: Thu, 22 Jan 2009 00:00:00 +0000
UTC: 2009-01-22 00:00:00+00:00
URL: https://codeinthehole.com/projects/current-pet-project-command-line-fu/

If you’re anything like me, you spend a lot of time at the UNIX command-line manipulating the filesystem, configuring Linux, playing with services and so forth. As any UNIX user knows, tremendous power can be wielded through judicious function selection, piping and output redirection. It’s often quite staggering what can be achieved in a single line given a rudimentary knowledge of sed, grep, awk, cut… Indeed, when I stumble upon a line of particular elegance or usefulness, I generally log them to a Tomboy note (fired up in a flash using Gnome-do).

Content Preview

If you’re anything like me, you spend a lot of time at the UNIX command-line manipulating the filesystem, configuring Linux, playing with services and so forth. As any UNIX user knows, tremendous power can be wielded through judicious function selection, piping and output redirection. It’s often quite staggering what can be achieved in a single line given a rudimentary knowledge of sed, grep, awk, cut… Indeed, when I stumble upon a line of particular elegance or usefulness, I generally log them to a Tomboy note (fired up in a flash using Gnome-do).

Phing task to create an Unfuddle message

Published: Sun, 11 Jan 2009 00:00:00 +0000
Updated: Sun, 11 Jan 2009 00:00:00 +0000
UTC: 2009-01-11 00:00:00+00:00
URL: https://codeinthehole.com/tips/phing-task-to-create-an-unfuddle-message/

Another day, another new Phing task; again integrating with project management software - this time the excellent Unfuddle. I’ve been playing with Unfuddle for a few days now and it’s very impressive. You get SVN and git hosting as well as superb issue tracking. It also supports simple project messages (which are displayed on the project dashboard) and so-called notebooks which are essentially project wikis that can be used to house documentation and manuals.

Content Preview

Another day, another new Phing task; again integrating with project management software - this time the excellent Unfuddle. I’ve been playing with Unfuddle for a few days now and it’s very impressive. You get SVN and git hosting as well as superb issue tracking. It also supports simple project messages (which are displayed on the project dashboard) and so-called notebooks which are essentially project wikis that can be used to house documentation and manuals.

Phing task to update Twitter status

Published: Sat, 10 Jan 2009 00:00:00 +0000
Updated: Sat, 10 Jan 2009 00:00:00 +0000
UTC: 2009-01-10 00:00:00+00:00
URL: https://codeinthehole.com/tips/phing-task-to-update-twitter-status/

At Tangent Labs, we’re currently experimenting with integrating Twitter into our project workflow to provide a latest activity feed in a easily digestible format (for both developers and non-technical people). For a pilot project, we’ve created a Twitter account and added an SVN post-commit hook script that updates Twitter with the latest commit information (commit message, affected files, author). We’re going to integrate our bug-tracking software shortly too but that’s not the subject of this post.

Content Preview

At Tangent Labs, we’re currently experimenting with integrating Twitter into our project workflow to provide a latest activity feed in a easily digestible format (for both developers and non-technical people). For a pilot project, we’ve created a Twitter account and added an SVN post-commit hook script that updates Twitter with the latest commit information (commit message, affected files, author). We’re going to integrate our bug-tracking software shortly too but that’s not the subject of this post.

Deploying to a shared hosting environment using Phing

Published: Sun, 04 Jan 2009 00:00:00 +0000
Updated: Sun, 04 Jan 2009 00:00:00 +0000
UTC: 2009-01-04 00:00:00+00:00
URL: https://codeinthehole.com/tips/deploying-to-a-shared-hosting-environment-using-phing/

Deploying applications to shared hosting environments can be a pain when no SSH access is provided. Consequently, it’s hard to avoid using FTP to deploy files from your development environment to a production server. In such trying circumstances, it’s easy to form self-destructive habits like using drag-and-drop FTP deployment - a very bad thing. Much better is to write an automated deployment script so that you can build to production in one clean step, a key tenet of The Joel Test for writing better code (highly recommended).

Content Preview

Deploying applications to shared hosting environments can be a pain when no SSH access is provided. Consequently, it’s hard to avoid using FTP to deploy files from your development environment to a production server. In such trying circumstances, it’s easy to form self-destructive habits like using drag-and-drop FTP deployment - a very bad thing. Much better is to write an automated deployment script so that you can build to production in one clean step, a key tenet of The Joel Test for writing better code (highly recommended).

Monitoring MySQL with Ganglia and gmetric

Published: Sun, 14 Dec 2008 00:00:00 +0000
Updated: Sun, 14 Dec 2008 00:00:00 +0000
UTC: 2008-12-14 00:00:00+00:00
URL: https://codeinthehole.com/tips/monitoring-mysql-with-ganglia-and-gmetric/

Following some server monitoring advice from the excellent ‘Building Scalable Web Sites’ by Cal Henderson, I’ve recently been experimenting with Ganglia on a cluster of servers at Tangent Labs. It has proved to be deeply impressive and has given us a great tool for keeping an eye on how our servers are performing, as well as providing an invaluable diagnostic tool should things go wrong. In essence, Ganglia is a distributed monitoring application that allows statistics on a cluster of servers to be aggregated in a single place.

Content Preview

Following some server monitoring advice from the excellent ‘Building Scalable Web Sites’ by Cal Henderson, I’ve recently been experimenting with Ganglia on a cluster of servers at Tangent Labs. It has proved to be deeply impressive and has given us a great tool for keeping an eye on how our servers are performing, as well as providing an invaluable diagnostic tool should things go wrong. In essence, Ganglia is a distributed monitoring application that allows statistics on a cluster of servers to be aggregated in a single place.

High Performance Web Sites by Steve Souders

Published: Sat, 06 Dec 2008 00:00:00 +0000
Updated: Sat, 06 Dec 2008 00:00:00 +0000
UTC: 2008-12-06 00:00:00+00:00
URL: https://codeinthehole.com/books/high-performance-web-sites-by-steve-souders/

High Performance Web Sites (HPWS) is essentially a book-length exposition of the YSlow extension to Firebug augmented with case studies of popular US websites. YSlow itself links to some explanation paragraphs regarding the various guidelines that are used to grade the performance of a website. However, even for those familiar with this documentation, HPWS is still an excellent resource on how the performance of a web app can be tuned.

Content Preview

High Performance Web Sites (HPWS) is essentially a book-length exposition of the YSlow extension to Firebug augmented with case studies of popular US websites. YSlow itself links to some explanation paragraphs regarding the various guidelines that are used to grade the performance of a website. However, even for those familiar with this documentation, HPWS is still an excellent resource on how the performance of a web app can be tuned.

Checking all MySQL tables

Published: Sun, 23 Nov 2008 00:00:00 +0000
Updated: Sun, 23 Nov 2008 00:00:00 +0000
UTC: 2008-11-23 00:00:00+00:00
URL: https://codeinthehole.com/tips/checking-all-mysql-tables/

It’s well known that MyISAM tables are prone to corruption and need to be regularly checked and repaired. Moreover, in a production environment, it can be beneficial to run a daily check of all tables and mail news of any errors to an appropriate developer/DBA. There are two options for checking MySQL tables. The most effective method is to run the myisamchk utility directly on the index files (.MYI) of the tables in question (some simple shell expansion makes this easy):

Content Preview

It’s well known that MyISAM tables are prone to corruption and need to be regularly checked and repaired. Moreover, in a production environment, it can be beneficial to run a daily check of all tables and mail news of any errors to an appropriate developer/DBA. There are two options for checking MySQL tables. The most effective method is to run the myisamchk utility directly on the index files (.MYI) of the tables in question (some simple shell expansion makes this easy):

Javascript cookie objects using Prototype and JSON

Published: Sat, 08 Nov 2008 00:00:00 +0000
Updated: Sat, 08 Nov 2008 00:00:00 +0000
UTC: 2008-11-08 00:00:00+00:00
URL: https://codeinthehole.com/tips/javascript-cookie-objects-using-prototype-and-json/

It’s sometime useful to interact with cookies directly on the client-side using javascript. This can be useful for a variety of situations, such as persisting display settings between page requests without storing anything on the server. I’ve also used them to display a simple welcome message to new visitors. It can make your controller code simpler if this kind of simple display logic is contained entirely on the client side.

Content Preview

It’s sometime useful to interact with cookies directly on the client-side using javascript. This can be useful for a variety of situations, such as persisting display settings between page requests without storing anything on the server. I’ve also used them to display a simple welcome message to new visitors. It can make your controller code simpler if this kind of simple display logic is contained entirely on the client side.

Date conditional redirects with mod_rewrite

Published: Fri, 07 Nov 2008 00:00:00 +0000
Updated: Fri, 07 Nov 2008 00:00:00 +0000
UTC: 2008-11-07 00:00:00+00:00
URL: https://codeinthehole.com/tips/date-conditional-redirects-with-mod_rewrite/

The Apache module mod_rewrite is capable of some very cool stuff. One neat trick is to use the time and date variables to control redirects and URL rewriting. This is useful if you have a URL that you don’t want to be exposed to the world until a certain date has passed - this could be the case with special offers and competitions which have a one-off static page that isn’t to be revealed until a specified date.

Content Preview

The Apache module mod_rewrite is capable of some very cool stuff. One neat trick is to use the time and date variables to control redirects and URL rewriting. This is useful if you have a URL that you don’t want to be exposed to the world until a certain date has passed - this could be the case with special offers and competitions which have a one-off static page that isn’t to be revealed until a specified date.

Creating large XML files with PHP

Published: Fri, 31 Oct 2008 00:00:00 +0000
Updated: Fri, 31 Oct 2008 00:00:00 +0000
UTC: 2008-10-31 00:00:00+00:00
URL: https://codeinthehole.com/tips/creating-large-xml-files-with-php/

When creating large XML files with PHP, there are some important considerations to bear in mind with regards to scalability. There are several libraries available for writing XML files of small to intermediate size (such as DOMDocument), but when dealing with very large files (eg. > 500Mb, or several million elements), these libraries are no longer useful as the size of the file then can create is memory-bound. For example, DOMDocument stores the XML tree in memory while it is being built - you then flush it out to file after all elements have been created:

Content Preview

When creating large XML files with PHP, there are some important considerations to bear in mind with regards to scalability. There are several libraries available for writing XML files of small to intermediate size (such as DOMDocument), but when dealing with very large files (eg. > 500Mb, or several million elements), these libraries are no longer useful as the size of the file then can create is memory-bound. For example, DOMDocument stores the XML tree in memory while it is being built - you then flush it out to file after all elements have been created:

Monitoring MySQL

Published: Sun, 26 Oct 2008 00:00:00 +0000
Updated: Sun, 26 Oct 2008 00:00:00 +0000
UTC: 2008-10-26 00:00:00+00:00
URL: https://codeinthehole.com/tips/monitoring-mysql/

Just a quick tip on monitoring the queries that mysql is handling on a production site. You can use the mysqladmin tool to return a list of the processes currently being handled. Combining this with the UNIX watch command allows a real-time monitoring of what’s going on. watch -n 1 mysqladmin processlist The -n 1 specifies that mysqladmin executes every second. Depending on your set-up, you may need to specify a mysql user and password:

Content Preview

Just a quick tip on monitoring the queries that mysql is handling on a production site. You can use the mysqladmin tool to return a list of the processes currently being handled. Combining this with the UNIX watch command allows a real-time monitoring of what’s going on. watch -n 1 mysqladmin processlist The -n 1 specifies that mysqladmin executes every second. Depending on your set-up, you may need to specify a mysql user and password:

Following log files with tail -f

Published: Wed, 22 Oct 2008 00:00:00 +0000
Updated: Wed, 22 Oct 2008 00:00:00 +0000
UTC: 2008-10-22 00:00:00+00:00
URL: https://codeinthehole.com/tips/following-log-files-with-tail-f/

UNIX is a majestic onion of discovery. Every day a new layer of understanding can be peeled away to give some new pungent goodness. Today’s was the ‘follow’ option of the tail command. It’s commonplace to use tail for viewing the recent entries to a log file: tail /var/log/apache2/error.log Much more useful is set the ‘follow’ option so that, rather than echoing to STDOUT and returning control to the prompt - tail continues to watch the file in question and echos additional lines to the terminal.

Content Preview

UNIX is a majestic onion of discovery. Every day a new layer of understanding can be peeled away to give some new pungent goodness. Today’s was the ‘follow’ option of the tail command. It’s commonplace to use tail for viewing the recent entries to a log file: tail /var/log/apache2/error.log Much more useful is set the ‘follow’ option so that, rather than echoing to STDOUT and returning control to the prompt - tail continues to watch the file in question and echos additional lines to the terminal.

Writing a Thesis in LaTeX

Published: Thu, 10 Nov 2005 00:00:00 +0000
Updated: Thu, 10 Nov 2005 00:00:00 +0000
UTC: 2005-11-10 00:00:00+00:00
URL: https://codeinthehole.com/guides/writing-a-thesis-in-latex/

Overview This article is a guide to constructing a decent parent file for a thesis or dissertation compiled in Latex. The specific details implemented here, and included in the example files, are those set out by the guidelines for submission to the University of Nottingham, but can be easily amended to suit any sensible requirements. Considerable attention has been paid to presenting the final document as a PDF file, which keeps the file size manageable (compared to postscript) and allows groovy add-ons such as hyperlinks and back-referencing.

Content Preview

Overview This article is a guide to constructing a decent parent file for a thesis or dissertation compiled in Latex. The specific details implemented here, and included in the example files, are those set out by the guidelines for submission to the University of Nottingham, but can be easily amended to suit any sensible requirements. Considerable attention has been paid to presenting the final document as a PDF file, which keeps the file size manageable (compared to postscript) and allows groovy add-ons such as hyperlinks and back-referencing.

Tutorial: the complex Ginzburg-Landau equation

Published: Tue, 01 Feb 2005 00:00:00 +0000
Updated: Tue, 01 Feb 2005 00:00:00 +0000
UTC: 2005-02-01 00:00:00+00:00
URL: https://codeinthehole.com/projects/ginzburg-landau-tutorial/

I wrote a tutorial on the complex Ginzburg-Landau equation as a set of simple HTML pages.

Content Preview

I wrote a tutorial on the complex Ginzburg-Landau equation as a set of simple HTML pages.

About

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/about/

Hello, I’m David, a software engineer. Employment I’m Head of Software Engineering at Kraken Technologies, the technology part of Octopus Energy Group. I help run a department of engineers building the “Kraken” software platform that underpins the business. Octopus Energy became the 15th UK tech “unicorn” in May 2020, after investment from Origin Energy. Previously I’ve worked as: Director of Engineering at Yoyo Wallet, a payments and loyalty start-up; a Technical Director (and various other roles) at Tangent Snowball, a digital agency; an Operational Researcher at HM Revenue and Customs.

Content Preview

Hello, I’m David, a software engineer. Employment I’m Head of Software Engineering at Kraken Technologies, the technology part of Octopus Energy Group. I help run a department of engineers building the “Kraken” software platform that underpins the business. Octopus Energy became the 15th UK tech “unicorn” in May 2020, after investment from Origin Energy. Previously I’ve worked as: Director of Engineering at Yoyo Wallet, a payments and loyalty start-up; a Technical Director (and various other roles) at Tangent Snowball, a digital agency; an Operational Researcher at HM Revenue and Customs.

Code

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/code/

I am a software engineer. Open-source I occasionally write open-source software. A selection of projects are listed below but see my Github profile for a more exhaustive list. Oscar Oscar is an open-source e-commerce framework for Django. I was the original author and a maintainer for several years. Sadly, I no longer have the time and am no longer involved in the project. django-oscar - domain-driven e-commerce for Django django-oscar-accounts - managed accounts using double-entry bookkeeping django-oscar-datacash - a Django app for integrating with the DataCash payment gateway django-oscar-paypal - Oscar integration with PayPal django-oscar-gocardless - Oscar integration with GoCardless.

Content Preview

I am a software engineer. Open-source I occasionally write open-source software. A selection of projects are listed below but see my Github profile for a more exhaustive list. Oscar Oscar is an open-source e-commerce framework for Django. I was the original author and a maintainer for several years. Sadly, I no longer have the time and am no longer involved in the project. django-oscar - domain-driven e-commerce for Django django-oscar-accounts - managed accounts using double-entry bookkeeping django-oscar-datacash - a Django app for integrating with the DataCash payment gateway django-oscar-paypal - Oscar integration with PayPal django-oscar-gocardless - Oscar integration with GoCardless.

Music

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/music/

Some curated content can be found at: youtube.com/user/codeinthehole soundcloud.com/codeinthehole mixcloud.com/codeinthehole Some curated Other good sources for stuff to code to: WeFunk radio rain.today (also try Piano jazz and rain) Spotify playlists Lewis Ofman, L’Impératrice — that kind of thing. Haunting waltzes. Classical/jazz fusion — better than you might think!

Content Preview

Some curated content can be found at: youtube.com/user/codeinthehole soundcloud.com/codeinthehole mixcloud.com/codeinthehole Some curated Other good sources for stuff to code to: WeFunk radio rain.today (also try Piano jazz and rain) Spotify playlists Lewis Ofman, L’Impératrice — that kind of thing. Haunting waltzes. Classical/jazz fusion — better than you might think!

Now

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/now/

June 2024 Culture: Books I’ve enjoyed since November: The Mating Season by P.G. Wodehouse Clash of Kings by George R. R. Martin Storm of Swords by George R. R. Martin His Bloody Project by Graeme Macrae Burnet Demon Copperhead by Barbara Kingsolver The Finckler Question by Howard Jacobson Dune by Frank Herbert The Sense of an Ending by Julian Barnes The Narrow Road to the Deep North by Richard Flanagan The Blind Assassin by Margaret Atwood The Bee Sting by Paul Murray Have finished listening to the entire back catalogue of The Rest is History podcast, which I highly recommend.

Content Preview

June 2024 Culture: Books I’ve enjoyed since November: The Mating Season by P.G. Wodehouse Clash of Kings by George R. R. Martin Storm of Swords by George R. R. Martin His Bloody Project by Graeme Macrae Burnet Demon Copperhead by Barbara Kingsolver The Finckler Question by Howard Jacobson Dune by Frank Herbert The Sense of an Ending by Julian Barnes The Narrow Road to the Deep North by Richard Flanagan The Blind Assassin by Margaret Atwood The Bee Sting by Paul Murray Have finished listening to the entire back catalogue of The Rest is History podcast, which I highly recommend.

Tweets

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/tweets/

I have only ever said two funny things on Twitter: Desirable developer skills: 1 Ability to ignore new tools and technologies 2 Taste for simplicity 3 Good code deletion skills 4 Humility — David Winterbottom (@codeinthehole) December 3, 2014 On your first day at the new job, squash every commit from the repo into a single commit with message "Legacy code" and force-push to master. — David Winterbottom (@codeinthehole) August 15, 2018

Content Preview

I have only ever said two funny things on Twitter: Desirable developer skills: 1 Ability to ignore new tools and technologies 2 Taste for simplicity 3 Good code deletion skills 4 Humility — David Winterbottom (@codeinthehole) December 3, 2014 On your first day at the new job, squash every commit from the repo into a single commit with message "Legacy code" and force-push to master. — David Winterbottom (@codeinthehole) August 15, 2018

Videos

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/videos/

Good technical talks that I can refer people to. Misc: Inventing on Principle by Brett Victor. An inspiring talk demonstrating new UI concepts for creation. All talks (and articles) by James Mickens are essential: Computers are a Sadness, I am the Cure by James Mickens. “Say word count one more time…”

Content Preview

Good technical talks that I can refer people to. Misc: Inventing on Principle by Brett Victor. An inspiring talk demonstrating new UI concepts for creation. All talks (and articles) by James Mickens are essential: Computers are a Sadness, I am the Cure by James Mickens. “Say word count one more time…”

Writing

Published: Mon, 01 Jan 0001 00:00:00 +0000
Updated: Mon, 01 Jan 0001 00:00:00 +0000
UTC: 0001-01-01 00:00:00+00:00
URL: https://codeinthehole.com/writing/

Below is a list of all posts from this site.

Content Preview

Below is a list of all posts from this site.