Posts categorized under: programming

What's in a (file)name?

Seeking a name for an obvious un-named thing

Here's a simple conundrum: Given a file, say:

/foo/bar/baz.exe

what are the names for the different parts of it?

The directory (part) is /foo/bar. The non-directory or file part is baz.exe. The extension is .exe.

So what is baz?

Frequently, I've written code ...

The "verbose" Python

PYTHONPATH, PYTHONVERBOSE and other infuriations

The symptoms

“Suddenly” (as is traditional to commence these sort of stories), I started having problems with some of the Python installations on my local machine. As is the way of things, I had three distinct sources of Python:

  • The native / builtin Python that came with my Mac
  • The Python ...

"Indexing and slicing"

Coming from other (arguably saner) programming languages, R's indexing behaviour kept catching me off guard. When should I use [] and when [[]]? What's the difference between accessing a member by name or using that name as an index? Why is iterating over a list so fiddly? Here's an ...

Logging PHP errors

Somehow, this feels like a personal failing.

The scenario

I try to avoid PHP as much as possible (just not my thing) but I'd installed a PHP-based webapp (REDCap) and there was something occuring that seemed like it should have generated an error. However I could not find any PHP errors logged anywhere on the machine ...

Porting this site to Pelican

On making a simpler website in a slightly complicated way.

Background

Way, way back, this site ran on various permutations of PHP-based frameworks. Getting sick of ugly URLs and having to vigilantly update the software so as to avoid hacks (and still periodically falling victim to various hacks), I went in search of a Python-based framework that I could customize ...

Primitive R objects and S3

Objects are just lists and class is just an attribute.

Primordial R objects

The first thing to know is that nearly every object in R is really just a list with named elements. For example, the results returned from "summary" are returned in an object:

x <- summary(c(1:10))
x
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
##    1 ...

An introduction to objects in R

The various types of OOP in R, whatever that is.

Overview

Doing object-oriented programming (OOP) in R can be complicated. This is because when you look in R for the usual idioms and features of OOP:

  • some are missing
  • some are present but implemented in unusual albeit valid ways
  • some ...

Reference classes

Also sometimes wittily referred to as S5 or R5.

A more recent development in R is Reference classes. These promise, at last, fully blown objects and classes like those of C++ and Java. Naturally, this innovation comes with some downsides:

  • Reference classes are still, as yet, lightly documented
  • They use their own form of syntax and idioms
  • The C ...

Using Paperclip for attachments

Doing something more than a toy example

This is not as straightforward as it seems.

In development, it works fine. Simply declare that the model has an attachment:

class Download < ActiveRecord::Base
        ...
        has_attached_file :attachment

and stuff will work automagically. A directory “system” will be created in your public folder and the attachments will be created in there ...

Rdoc, the essentials

A short memory jogger.
= Top level title

== 2nd level / sub-title

----

A horizontal rule is at least 4 dashes

----

Text can be _italic_, *bold* or +monospaced typewriter+.

   Verbatim / literal text is indented. But look out
   if it's after a list, as they must be idented to a
   different extent

----

   * Indent your lists and use ...

Unicode and HTML entities

In which we struggle with a cacophony of characters for the web.

Buried in the Python standard library, unicodedata contains most the information needed to interrogate and translate unicode characters. Unfortunately, it's underdocumented. More accurately, the docs are a terse list of what it does, but not why you might want to use it or how you use it. Unfortunately it ...

008 - paths across grid

Project Euler problem #8.

See Problem 8: Find the greatest product of five consecutive digits in a 1000-digit number.

This is fairly straightforward. First we define the number, using Python's implicit string continuation:

numstr = '73167176531330624919225119674426574742355349194934'
'96983520312774506326239578318016984801869478851843'
'85861560789112949495459501737958331952853208805511'
'12540698747158523863050715693290963295227443043557'
'66896648950445244523161731856403098711121722383113'
'62229893423380308135336276614282806444486645238749'
'30358907296290491560440772390713810515859307960866'
'70172427121883998797908792274921901699720888093776'
'65727333001053367881220235421809751254540594752243'
'52584907711670556013604839586446706324415722155397'
'53697817977846174064955149290862569321978468622482'
'83972241375657056057490261407972968652414535100474'
'82166370484403199890008895243450658541227588666881'
'16427171479924442928230863465674813919123162824586'
'17866458359124566529476545682848912883142607690042'
'24219022671055626321111109370544217506941658960408'
'07198403850962455444362981230987879927244284909188'
'84580156166097919133875499200524063689912560717606'
'05886116467109405077541002256983155200055935729725 ...

A useful .irbrc file

Tweaking irb.

For better or worse, programming in Ruby means that you will be spending a lot of time in irb. You can customise its behaviour in your .irbrc file, usually found in your home directory. The tweaks in mine were gathered from various places across the 'net:

require "rubygems"
require "wirble ...

Euler 035 - counting circular primes

How many circular primes (primes that when rotated are still primes) are there below one million?

See Problem 20.

Again, this rescues a lot of code from previous problems, but the solution is less than perfect. The Euler guidelines say that every problem should be computable within a minute, but despite much tweaking, this solution takes significantly longer.

For speed, primes should be generated only once ...

Euler in Groovy 1: sum of numbers

Print the sum of numbers less than 1000 that are divisble by 5 or 3.

The problem is:

Print the sum of numbers less than 1000 that are divisble by 5 or 3.

Already, we're exposed to a few nifty features of Groovy - ranges, and the functional methods on arrays like findAll:

// print the sum of numbers less than 1000 that are divisble by ...

Exercise 3: finding prime factors

Exercise 3: finding prime factors

What is the largest prime factor of the number 600851475143?

// What is the largest prime factor of the number 600851475143? target = 600851475143 factors = [] residue = target residue_is_prime = false // NOTE: ``#`` is not a comment character! // While the residue is not prime, keep extracting factors and adding to ...

Exercise: 4 palindromic numbers

Exercise: 4 palindromic numbers /* A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 x 99. Find the largest palindrome made from the product of two 3-digit numbers. */ // The only way to work this out is to try ...

Exercise 5: smallest divisible number

Exercise 5: smallest divisible number

What is the smallest positive number that is evenly divisible by all of the numbers from 1 to 20?

/* 2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder. What is the smallest positive ...

Mysteriously a tuple

When a result is inexplicably of the wrong type

Not the first time I've been stung by this, so it's worth recording the problem.

Symptoms

A function or call inexplicably returns not as the expected type but as a tuple containing the said type. After a furious amount of debugging, you determine that everything is acting as ...

Naming an association in Rails

Naming an association in Rails

In which we thwart the magic of Ruby on Rails.

The automagically generated and named rails associations (:has_many, :belongs) are handy, but I ran into problems when trying to give an association a name other than the default based on the class name (e.g ...

Django form fields

A pointer on how to process them.

Django's form generation machinery for the most part is fine and straightforward, but on a recent project I ran into problems with a form that accepted files. The documentation is all there, but changes in how things work and an emphasis on forms for models make things harder to ...

Can I use RMagick?

A simple question, with the answer simply laid out for you.

RMagick is necessary for a lot of Ruby packages, so the restrictions on using it are critical (and very annoying). Here are the constraints on what can be used where:

     
Ruby 1.8 RMagick 1 RMagick 1
Ruby 1.9 RMagick 2

Or to put it another way: if you ...

Gotcha: map keys in Groovy

Key equality is tricky, let's go shopping.

Maps are great quick-and-dirty data structures or caches in many programming languages. But there's a least one way that Groovy maps aren't so friendly. Observe:

groovysh> m1 = [1:1]
===> {1=1}
groovysh> m1[1]
===> 1
groovysh> m1[(int) 1]
===> 1
groovysh> m1[(long) 1]
===> null

So, map keys ...

Page-specific assets in Rails

A common task that is surprisingly obscure.

A common use-case in web development is including some asset (e.g. a stylesheet or javascript file) only for certain pages. The situation gets trickier if the inclusion has to be in the page header. Finding how do do this in RoR is surprisingly tricky, due to outdated or bad ...

Setuptools and the single file

Setuptools and the single file

Problems with making an installer for a simple module

Although setuptools is a complex thing, and there has has been widespread complaints about its architecture, for the most part, I've had few problems with it. However, I recently encountered a crusty and under-documented corner ...

Simple string interpolation

Safe and straightforward markup, for renaming files.

It's often useful for a script to allow the specification of how an output file is named, e.g. myscript.rb -o file.out. At the same time, it would be nice to have the output name based on the an input file name, and have easy and consistent ...

Universal binaries with Netbeans

How to compile a MacOS executable from C++ using the Netbeans IDE.

Universal binaries are one of Apple's clever innovations that paper over the differences between architectures by including several distinct executables within a single file. When run, the OS examines the file and decides which binary to execute ...

Upgrading to 3.0

In which, again, Plone makes something simple complicated.

Despite the Plone community's tendency to dump everything and jump onto the latest version, I'd resisted the move to Plone 3.0. One good reason for this is that I'd repeatedly installed Plone 3.0 and it had - repeatedly - failed to startup. The symptoms weren't identical ...

Illogical or

'Or' or something else.

In which we consider the difference between or and || in Ruby. The secret is ... it depends. On context.

Behold:

>> foo = nil or "foo"
=> "foo"
>> bar = nil || "bar"
=> "bar"

That makes sense. But!:

def testor (&block)
   foo = block or "foo"
   bar = block || "bar"
   return foo, bar, block
end

>> testor()
=> [nil, "bar ...

Ruby 1.9, keyword arguments, WTF

Amongst the many entries Ruby has in the "you have got to be kidding me" stakes, this is a doozy.

Due to Ruby's lack of explicit support for keyword arguments, it's traditional to use a quirk of its argument parsing that pushes named arguments into a hash:

def ...

Surprises

In a way, tuples are one of the most magical parts of Python. How does the interpreter distinguish between a method call, a bracketed portion of a mathematical expression and an actual tuple?:

calc_midpoint (1, None, [2, 3], False)
x = w + (x * y)
a = (b, c)

The really smart thing ...

Parsing Excel

Some (very) quick notes on libraries for manipulating (mostly reading) Excel files in Ruby.

Parseexcel

  • This is the usual / traditional method for reading Excel
  • Doesn't work on XSLX (modern Excel)
  • There are a number of versions handing around the web, look on rubygems for latest version
  • Appears to not ...

Creating a Hobo project

A step-by-step guide to setting up Hobo development.

Hobo is a nifty thing, a better Rails than Rails, but there are a number of small roadbumps to getting a project started. This is a list of one possible way through the maze.

Assumptions

  • You're working in a Unix-like environment
  • For development purposes, you will be building a ...

write argument must be string

When wsgi won't work.

So I was writing a web application using WSGI and - after working for a long time - it started erroring out with:

AssertionError: write() argument must be string

from deep inside the WSGI handler. Now, this usually means that you're returning unicode, which WSGI can't handle. However, it's ...

Plotting & Graphs for the web

Or, "there must be simpler way to do graphs in Rails"

In several recent projects, I've needed to produce graphs of various epidemiological & bioinformatic datasets on demand in web applications. Unfortunately, in the world of Ruby and Rails, the choices of toolkits aren't as straightforward as hoped. This is a summary of the available choices and what I found ...

Rubyvis

Some problems, some solutions.

If you need to draw a custom graph type in Ruby, there's a few problems:

  1. There's only a modest number of graphing libraries
  2. You don't have to go to far for your graph type to not be covered by these

Which leaves you few choices. You could ...

Can't use print

Who would of thunk it? print is a reserved word everywhere.

Perhaps this is buried in some specification, but it seems that you can't use print as the name of a method of a class. (It makes sense you can't override the global name, but in an object ...

Getting the name of something

From the Department of the Blindingly Obvious.

Recently, I had the need to get an informative name from a range of Python objects for generating a useful error message. Where problems started was that these objects could include classes (new-style and old), functions, lambdas and built-in types. And here the logic started getting tricky.

  • To get the ...

Epydoc go boom

Making a good but abandoned documentation tool work.

So, epydoc used to be the neatest and bestest documentation tool available for Python. It produced documentation from introspecting code and thus was the easiest way to the most accurate API documentation. However, it looks like the project has been abandoned. Attempts to run it over code result in sometimes ...

How to stop output and printing

The essential problem is how to not get output from a program.

Let me explain: Ete2, a Python module for representing phylogenies, has a number of dependencies (MySQLdb, Numpy, PyQt, etc.) that it doesn’t necessarily need and it can be installed without them. If you don’t use the associated functionality, you won’t need these dependencies. But, irritatingly, ete2 tries ...

Applescript via Python

Controlling Mac applications with appscript

Over the years, Apple has fallen in and out of love with Applescript, its "official" scripting language for MacOS. True, Applescript isn't going to go away any day now and true, it is a very simple language and easy to use. But if you don't want to have ...

The etree tail quirk

How to make a good XML solution for Python better.

Python was late to adopt XML and even then the libraries provided weren't as powerful or easy to use as those available elsewhere. Fortunately, with Python 2.5, a subset of the ElementTree library was incorporated into the standard ...

Tips & tricks

Some brutal experience gained while learning to love Zope.

Archetypes

Ah, archetypes. How I adore thee. Other ways of constructing Plone content are as nothing before me. You are quick to develop, flexible in your approach, sensible in your defaults. But cryptic in your errors.

A lesson or two on breaking archetypes: First, not all fields accept all widgets ...

Changing a user's password

When "reset password" fails.

How do you give a user a new password? Well, in the "User & Groups" control panel, you select reset password for them. What if they've lost or changed their email address? In the same place, set the new one. What if "that doesn't work"? You can assign them ...

Matplotlib

Notes on installing and using.

Matplotlib is cool. Did I say it was cool? I meant very cool. While principally a plotting library, it can be used for image manipulation and drawing.

draw_lines

Which is where our first problem occurs (as of the 0.8.4 version of matplotlib and possibly earlier). Should you try ...

Installing PloneRSS

Notes for the less persistent.

One of Plone's marked faults has been the lack of an out-of-the-box RSS aggregation product. Although there have been a number of attempts (CMFSin, CMFNewsFeed, CMFFeed), these are unfinished, quirky or only semi-functional. Fortunately, PloneRSS is a robust solution for gathering and publishing newsfeeds on your site. Unfortunately, installation ...

One great big bag of Plone problems

Assorted bugs, weirdness and some solutions.

Whenever I get a cryptic error from Plone (which is all too frequently), I google to see if anyone else has it. This is me paying back into the system.

Can't add type: 'NoneType has no attribute get'

After fiddling with the schema on a type, adding it then ...

Implicit typename error

"Iterator is implicitly a typename ... warning: implicit typename is deprecated"

While compiling some code with gcc (that had previously compiled and run without complaint under CodeWarrior), the following warning was reported:

SblNumerics.h:334: warning: 'typename
std::iterator_traits<_Iterator>::value_type' is implicitly a typename
SblNumerics.h:334: warning: implicit typename is deprecated, please see
the documentation for details

The offending ...

On the debugging of page templates

... and form controllers and macros and webforms ...

Try embedding something like this in your code:

<dl class="collapsible inline collapsedOnLoad">
<dt class="collapsibleHeader">Title</dt>
<dd class="collapsibleContent">
<span tal:replace="structure python: here.printRequest(request,
errors)"/> </dd>
</dl>

and supply the necessary printRequest somewhere in your code or as a script:

def printRequest (self, request, errors ...

Encoding types in etree

Setting encoding in the header of ElementTree generated XML.

Not that I'm picking on ElementTree or anything but ...

Normally XML documents will declare their character encoding in their opening tag. And, normally, this will be utf8. (This may be a standard.) So this is a common sight:

xml version='1.0' encoding='utf8'?>

An oddity in ElementTree is ...

Installing MySQLdb

When unresolved linker symbols attack.

Good news: you can get reliable and easy access to a proper, fully-featured database from within Python.

Bad news: you may have to tinker to get it to (a) install properly. And if not done properly, this may cause it to (b) not work properly.

Good news: Download a .dmg ...

Installing Pydee on OSX

Short notes on getting Pydee (and PyQt) to run on a Mac.

Pydee is new and nifty graphical frontend for using Ipython. Unfortunately, there's no monolithic binary package available for installing (at least not on the Mac), and so users have to install and compile all the prerequisites. Given the the number of steps and small but important details, it's ...

Parsing dates in Python

One of those weird things that always slips my mind and always seems less obvious than it should be.

Parsing a date (object) from a string representation in Python seems oddly neglected or cumbersome as compared to the rest of the standard library that surrounds it. (Witness the number of ...

Restoring viewlets

How do you get a Plone theme to uninstall and restore viewlets to the usual way. That is the mystery ...

The symptoms

In a theme I recently built, I moved or hid several viewlets around for artistic effect. The problem didn't show itself until I uninstalled the theme. The ...

The incident of the missing form script

The incident of the missing form script, another Plone mystery.

Symptoms

While developing a series of forms using CMFFormController, suddenly one of the components couldn't be found.

To recap (and serve as an example to those neophytes), CMFFormController structures your forms as a series of form templates or scripts with metadata files determining how execution moves between them. In ...

Non-class type

"... request for member ... which is of non-class type"? What?

After a long hiatus, we return with a traditionally opaque C++ error message:

error: request for member 'close' in 'out', which is of non-class
type 'std::ofstream*'

which like all the best C++ error messages is long, detailed and completely unhelpful. It could be triggered by a number of things ...

Upgrading to 2.1.x

Some notes on the process of updating your Plone site software.

The process of upgrading a Plone site can be intricate. The 2.0.x to 2.1.x jump is no different. (Anyone who has followed the series of quirks and bugs in Agapow.net over the early part of 2006 will have seen the evidence.) While you could just ...

Doxygen cheatsheet

Doxygen is a free tool for documenting code. With a single command it can generate cross-referenced HTML documentation from any C++ or Java code. Furthermore, if the code is commented in a particualr styled, Doxygen can leverage that to enhance the documentation. The below is a selection of the most ...

Version control systems for Mac OS X

Some quick notes from my search for decent source code control.

This document was originally written in late 2003 / early 2004. The scene has changed some in the meanwhile, with various offshoots of Arch and Subversion gaining a lot of mindshare, and some of the URLs are probably now invalid. However, the broad conclusions still hold, and this is left here ...

VisualWidget, don't do It

More accurately, can't do it. A schema for Archetypes Visual Widget that works.

Archetypes Visual widget - the WYSIWYG HTML editor that shows up on Pages in Plone 2.5 - is pretty damn nice. I've had problems with Epoz and this is the first embedded web editor that makes me think about giving up on restructured text.

Getting it to work in your ...

Installing and using PIL

Overcoming the sometimes tricky installation of the Python Imaging Library and improving image quality.

While some people have reporting installing PIL on MacOSX without problems, there are legion reports that say otherwise. Frequently this revolve around PIL installing without JPEG support. There are a myriad of solutions (involving editing make files, configure flags, using fink to install dependencies, using package managers, casting goat entrails ...

Creating attributes on classes

Programmatic addition of methods.

I've gotten used to manipulating members of Python objects with getattr, setattr etc. But a recent similar problem had me stumped. I had a class that I wanted to create a large number of similar behaving properties on (they would all manipulate an internal dictionary):

>>> f = Foo()
>>> f.contributor ...

How to get a multiply-defined symbol

"symbol X is multiply-defined"

There are many reasons why the "multiply-defined" error can occur at link-time, most of them annoying but mundane: failure to include compilation guards on header files, the compiler finding more than one version of a file (perhaps a backup?) on the file-search path, lack of the extern qualifier and so ...

Looser throw specifier error in C++

Looser? Virtual? What?
looser throw specifier for 'virtual Error::~Error()'
... overriding 'virtual std::exception::~exception() throw ()'

While compiling some code with gcc (that had previously compiled and run without complaint under CodeWarrior), the following fatal error was reported:

Error.h:83: looser throw specifier for `virtual Error::~Error()'
/usr/include/gcc/darwin/3 ...

Useful products

Recommended extensions and add-ons for your Plone installation, including some advice and hacks.

ListingPages

ListingPages at PloneProducts

In theory Topics can do just about everything you would ever need to do as regards dynamically aggregating content. In practice, they seem to be rarely used. (Neither of the three Plone books ...

Diagnostic printf

A better basic debugging tool.

The crude "lets drop a print statement in" approach keeps being useful even with advanced dynamic languages. Here's a (slightly) improved version for ruby that pretty prints any number of passed objects and (importantly) where the print call was made from:

def dbg (*args)
   print("dbg: #{caller()[0]}: ")
   args ...

Slots

A new, and poorly explained, feature of Python classes.

Around version 2.2, Python rejigged its classes with some useful extensions. Unfortunately these enhancements have been explained so poorly that they appear in little published code.

One such enhancement is __slots__. An attribute of this name in a class restricts what attributes can be created in objects of that ...