ground- water, geo- statistics, environmental- engineering, earth- science

Archive for January, 2014

Fun with Python: Script for Daily Newspaper

without comments

This post has to do neither with water nor with statistics. But because it’s been christmas break, I found some time to do some fun stuff with python:

I wrote a python script that downloads the issue of my choice of a pdf on my iPad. Partcularly, the script

  • lets me chose which issue of the paper (relative to the current day) I want to download
  • checks if that file might already exist on a relevant dropbox folder – checks if that file exists on the newspaper’s server (there is no paper on a holiday, f.ex) – if checks are ok, then the pdf is downloaded locally – saves the pdf in a selected folder within my dropbox – opens Goodreader, downloads the current file from dropbox (the proper url of the file needed to be determined) – deletes the downloaded pdf locally within pythonista

The relevant modules are the dropbox module of pythonista and the requests module to download the pdf with authentication.

The rationale behind attempting this was two fold:

  • using python on my iPad: I have downloaded pythonista a little while ago, but haven’t had time to play with it
  • since it has been possible, I have been subscribing to a German daily newspaper, which can be download as a pdf. I guess they are trying to foster the use of iPaper. I tried that for the last couple of weeks using iBooks on the iPad, and I don’t like it for two reasons:

    1. the iPaper files are larger than the pdfs, hence they take noticeably longer to load.
    2. I found it to be almost impossible to get the iPaper files out of iBooks on the iPad (to some storage place, for example).

The tipping bucket that led me to actually do this was that since a little while it is not possible anymore to download the pdfs directly from within Goodreader

Goodreader came out with an updated version two days ago. This makes the script even more powerful. This update solved issues related to the rendering of pdfs. I am not getting money from Goodreader, but I consider it to be the most useful software I have on my iPad. It particularly integrates awesomely with directories on an OSX Server.

Downloading the pdf

This is surprisingly easy using the requests package, that I just found out that it existed. Basically, downloading a pdf from a known url comes down to one line:

r = requests.get(url, auth=('XXX', 'YYY'))

Saving to Dropbox

This is step is a little bit more involved:

  • Before you start you have to register an ‘app’ in the developer part of dropbox. This will get you an APP_KEY and an APP_SECRET. In the python API you also need a token, which is from what I understand a string consisting of the name of your app in dropbox concatenated with '.token'.
  • When you’ve got that, I found some code on github that deals with the authentication process.
  • Finally, within the python module within pythonista, saving a file comes down to another one liner basically (with some preparations):

    sess = dropbox.session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE) 
    client = dropbox.client.DropboxClient(sess) 
    f = open(filename) 
    response = client.put_file('/Apps/Pythonista/' + filename, f)

Open Issuess

  • the link between pythonista and good reader via dropbox works reasonably well. Would be nice to have a direct link (along the lines of ‘open in’).
  • it’s possible to create little home scree icons that call your code, also with command line arguments. As far as I can tell, you have to fix a certain value of the command line argument to one icon… that seems a bit against the flexibility of command line arguments
  • the time delta doesn’t treat boundaries of months correctly fixed by the first commit
  • not sure if raise Exception is really the way to go if a) the file exists already on dropbox or b) if there doesn’t exist a paper on the desired date


  • Pythonista offers a tool to achieve things that otherwise would be next to impossible on the iPad – at the same time, one has to wonder, why such things are forced to be impossible to begin with – writing code on the iPad without a dedicated external keyboard is… challenging. Despite the good editor of Pythonista.

The script is available on github.

Written by Claus

January 9th, 2014 at 4:23 pm

Posted in

Uncertainty in Daily Life

without comments

Welcome 2014!

I’ll start this year by pointing to two articles from the New York Times that point out the importance of uncertainty:

1. Monte Carlo to try and Find Fisherman who Fell off his Boat

This story in the New York Times Magazine describes the rescue for a fisherman who fell off his boat into the Atlantic while his colleagues were sleeping. The human aspects play the most important part, but one section describes how the US Coast Guard uses a Monte Carlo based simulation tool to predict the most likely locations where to search for the fisherman floating in the ocean. I found this story via John Gruber.

2. Inconvenient Uncertainties

In this comment, on another article published in the New York Times, Gernot Wagner tries to make the point that science in general, particularly environmental science is inherently uncertain . The title of the original article that he refers to is “By 2047, Coldest Years May Be Warmer Than Hottest in Past, Scientists Say.” Wagner tries to make the point, that science in general and climate science in particular, are uncertain. Will this threshold be passed exactly in 2047 or might it be around 2050? He continues to state that

The scientific method imposes some order, but in the case of climate change, that order is probabilistic. For the sake of science and the planet, we should not become distracted by a false sense of certitude. Imprecise truths are the most inconvenient ones. We know enough to act now. What we don’t know should prompt us to even more decisive action.

Written by Claus

January 4th, 2014 at 12:44 pm

Posted in