Green Building Forum - RaspberryPi

RaspberryPi

Ed Davies — Sat, 22 Feb 2014 11:43:37 +0000

Example of sending email via GMail which seem to use STARTTLS (ie, it opens a normal HTTP connection then upgrades it to use TLS once it's talking):

http://www.doctormonk.com/2014/02/pi-fridge-minder-receive-email-when.html

RaspberryPi

SteamyTea — Mon, 17 Mar 2014 09:41:23 +0000

Right, I have ordered some new temperature and humidity sensors. That mean I have to write some new code, but that is not the question.
I want to log (and possibly do some control) temperature, RH and solar data. I will probably have a minimum of 4 sensors for each, so 12 in all (may stick some more temp ones on as temperature is so useful).
The question is, should I write just one bit of Python Script that runs from start to finish i.e. reads the temp sensors, then the RH ones, then the solar ones, then the RTC and then write that to a file, or write 3 separate scripts for each and then use a 4th one to pull all the data together and write to a log file.
The first one is my preferred choice as I can probably do that, but do not know if it is the best way to do it.
The same applies to any outputs, should I incorporate them in a script and act on them as I go along or wait till all the data is collected and then flick the switches as needed?

RaspberryPi

Ed Davies — Mon, 17 Mar 2014 10:37:05 +0000

Don't try to overthink things, just do whatever seems simplest. Keeping the code to read the sensors and do the logging all in one process is probably less confusing in the long run - you don't finish up with separate shell scripts to coordinate them and so forth.

Having said that, keeping the sensor reading code, the logging code, and so on reasonably separate (either in different source files which are imported or just in individual functions) seems like it would make future modification easier.

Off the top of my head, I'd probably have one main file which contains a list (tuple of dictionaries probably) of the necessary details of the various sensors (1-wire addresses, ADC ports or whatever plus name to go in the log file) and code to loop round and call out to separate imported files to do the readings and build a dictionary of read values. It'd then write that to the log file and do any output required via calls to other scripts.

Think about the mostly likely modifications and make them easy. Here it's adding or removing sensors and/or outputs so making the management of that as isolated to one part of the code as possible seems the way to go.

RaspberryPi

SteamyTea — Mon, 17 Mar 2014 10:46:33 +0000

Thanks Ed. As I will probably be ditching the 1-Wire stuff for this project, it is probably easiest to keep it in one file. Shall see when the DHT11s get here how easy they are to read.

RaspberryPi

Ed Davies — Mon, 17 Mar 2014 13:46:13 +0000

Just re-read the DHT11 datasheet. I'd have thought it's a bit tricky to do the timing required reliably in a user-space process as it'll get interrupted rather unpredictably. Have you a kernel-space driver in mind? Nice job for an Arduino, perhaps.

RaspberryPi

SteamyTea — Mon, 17 Mar 2014 13:52:02 +0000

There are a couple of bits of code on the internet about making them work on the RPi, one one is Python and the other is C.
I don't really want to try and understand C (tried that in 1986) but may have to.

What is a kernal-space driver. I think I came across something about a driver for these sensors.

RaspberryPi

Ed Davies — Mon, 17 Mar 2014 14:40:37 +0000

The kernel is the core of the operating system (the bit that Linus Torvalds wrote, as opposed to all the user space stuff from GNU, etc). It deals with the low level access to hardware, managing memory, switching between processes, etc. A kernel-space driver is one which is loaded into the memory space of the kernel and can access hardware directly. Generally you use the insmod (or modprobe) command to load them if it's not done automatically for you somehow.

E.g: http://www.tortosaforum.com/raspberrypi/dht11driver.htm

I did look at some users-space C code to read from these things from Adafruit on github. It's frighteningly crap - it looks to me like they have a half-plausible idea but left their debug code in in such a way that it'll crash nastily if you try to read too many sensors.

RaspberryPi

SteamyTea — Mon, 17 Mar 2014 17:45:39 +0000

From the little reading I have done about it, existing code seems to keep reading until the flag and/or the CRC give the correct result and then returns that line as correct. Shall have a better idea when I get them. I may just use the existing C stuff and then use the Python to log the data from it.

RaspberryPi

PeterW — Thu, 20 Mar 2014 20:29:56 +0000

ST

You can bit-bang a DHT11 to make it give up its data using python and a couple of functions - got it on mine currently and I can extract the code if you like..?

Cheers

Pete

RaspberryPi

SteamyTea — Thu, 20 Mar 2014 21:31:45 +0000

Pete
That would be useful, thank you.
Not got mine yet, they said they would take between 7 and 21 days.

RaspberryPi

PeterW — Fri, 21 Mar 2014 23:24:44 +0000

ST

Finally got onto my pi - long story - and I've used the Adafruit C library to drive the DHT11, mine is plugged into GPIO#4 hence the code that gives '11' and '4' in the line.

First off, download the code from github here

$ git clone git://github.com/adafruit/Adafruit-Raspberry-Pi-Python-Code.git
$ cd Adafruit-Raspberry-Pi-Python-Code
$ cd Adafruit_DHT_Driver

Code is as follows - its purely writes out to screen at 5 second intervals, its an extract of more detailed code but works.

#!/usr/bin/python

import subprocess
import re
import sys
import time
import datetime

dateString = '%Y/%m/%d %H:%M:%S'

# Continuously extract data
while(True):
# Run the DHT program to get the humidity and temperature readings!

output = subprocess.check_output(["./Adafruit_DHT","11", "4"]);

# print output
matches = re.search("Temp =\s+([0-9.]+)", output)
if (not matches):
time.sleep(3)
continue
temp = float(matches.group(1))

# search for humidity printout
matches = re.search("Hum =\s+([0-9.]+)", output)
if (not matches):
time.sleep(3)
continue
humidity = float(matches.group(1))

# write out the data to the command line
timestamp = datetime.datetime.now().strftime(dateString)
print timestamp, "Temperature: %.1f C" % temp , "Humidity: %.1f %%" % humidity

# Wait 5 seconds before continuing
time.sleep(5)

The re.search looks for the reading in the buffer - basically if it doesn't see it then it sleeps 3 seconds and tries again. I've had a look at the C code and it sets the GPIO high to trigger the DHT11 to respond and then waits for a response - I have stripped a lot of the code out from the C file but its not really worth it unless you are looking to tune the code to be very responsive....

Have fun with it, don't forget that the python script will need to be run under root or sudo to work too !

Cheers

Pete

RaspberryPi

SteamyTea — Sat, 22 Mar 2014 06:06:49 +0000

Thanks Pete
I am still waiting for them to be delivered, so shall report back as soon as I have them wired up.
One thing that I may need to change is the GPIO pin as I have a real time clock fitted and can't remember which pins it uses.

RaspberryPi

Ed Davies — Sat, 22 Mar 2014 09:19:27 +0000

If you wrap a code sample in a

 element (and select Format comment as Html, of course) the indentation doesn't get so mangled. More than a little helpful for Python, particularly if it's more complicated than this code where the indentation is pretty obvious.

# Continuously extract data
while(True):
    # Run the DHT program to get the humidity and temperature readings!

    output = subprocess.check_output(["./Adafruit_DHT","11", "4"]);



RaspberryPi
SteamyTea — Thu, 07 Aug 2014 23:11:02 +0100
This is more about Python than the RPi.
I have a large CSV file and I want to get useful data out of it.  Excel is a bit slow, so how easy is it to write a bit of script to open the file and do some conditional 'sums' on it.
Basically it starts with a date, then the data in a few columns.  I want to get the usual statistical stuff out, mean, median, mode, standard deviation, count, skew, kurtosis etc.
I need it to be pretty easy to understand.
Thanks


RaspberryPi
Ed Davies — Fri, 08 Aug 2014 09:41:53 +0100
Concentrating just on the CSV file reading - statistics I leave to others.

Depends on the CSV file. If it's just literally numeric values separated by commas then it's easy. Where things get complicated is if you have strings (e.g., labels) particularly if they might include commas, quote marks and so on.

If you want something general look at module csv.

import csv
help(csv)

Generally, though, something more pragmatic based on looking at the actual data you have works fine. Picking a random .CSV file out of my working directory (EGPC_HDD_12.0C.csv) which starts:

Description:,"Celsius-based heating degree days for a base temperature of 12.0C"
Source:,"www.degreedays.net (using temperature data from www.wunderground.com)"
Accuracy:,"Estimates were made to account for missing data: the ""% Estimated"" column shows how much each figure was affected (0% is best, 100% is worst)"
Station:,"Wick (3.09W,58.46N)"
Station ID:,"EGPC"

Date,HDD,% Estimated
2008-12-01,9.1,0
2008-12-02,10.4,0
2008-12-03,11.3,0
2008-12-04,7.9,0
2008-12-05,8,0
2008-12-06,9.3,0

In this case it's probably simplest to just ignore the header lines and concentrate on the data:

#!/usr/bin/python3

for line in open('EGPC_HDD_12.0C.csv').readlines():
    line = line.strip()
    if (len(line) >= 1) and (line[0] in '12'):
        date, hdd, estimate = line.split(',')
        hdd = float(hdd)
        y, m, d = map(int, date.split('-'))
        print(y, m, d, hdd)

produces output starting:

2008 12 1 9.1
2008 12 2 10.4
2008 12 3 11.3
2008 12 4 7.9
2008 12 5 8.0
2008 12 6 9.3



RaspberryPi
SteamyTea — Fri, 08 Aug 2014 13:05:36 +0100
Thanks Ed, I think I follow most of that, shall see if I can get something out of my data later, looks like I am in for a wet weekend.


RaspberryPi
SteamyTea — Sat, 09 Aug 2014 09:56:03 +0100
Going to see if I can understand this bit of code, I shall put my comments before the code:

#opens the csv file so that all lines can be read. if I put a number in the () after the readlines would that limit the lines it reads, either to or from?
for line in open('EGPC_HDD_12.0C.csv').readlines():

#creates a working file somewhere in memory that is used and manipulated, in this case stripping out unwanted stuff
line = line.strip()

#only works on lines that have more than 1 character in them, but don't understand what the (line[0] in '12') does, what does the [0] and '12' do?
if (len(line) >= 1) and (line[0] in '12'):

#creates columns that were separated by a comma and gives them names
date, hdd, estimate = line.split(',')

#makes the data in the hdd column have a floating decimal point
hdd = float(hdd)

#all about the date format, I assume that map is some clever thing that puts parts of the date that are separated by a - into a form that does not have the -, but a space instead and only prints the integer.  So if my date and time info is in the form 27/07/2014 01:10 then I will have to split it into two, the date and the time, then recombine the date part and the time part.
y, m, d = map(int, date.split('-'))

#prints out the result with a blank space between each element.
print(y, m, d, hdd)

Hope that makes sense.


RaspberryPi
Ed Davies — Sat, 09 Aug 2014 19:22:37 +0100
Bit more of a Python tutorial than is really appropriate to the forum but I think one or two others than just Steamy might have some interest:

Posted By: SteamyTeaGoing to see if I can understand this bit of code, I shall put my comments before the code:

#opens the csv file so that all lines can be read. if I put a number in the () after the readlines would that limit the lines it reads, either to or from?
for line in open('EGPC_HDD_12.0C.csv').readlines():
Not quite. Actually readlines reads all of the lines from the file and puts them in a list. If you had a *really* big file (bigger than the GB or so of memory you've probably got to hand) then a different approach would be needed. The for loop then loops over the whole of that list.

Because readlines returns a list you can index it as you please. For example, to read just the 11th, 12th and 13th lines use:

for line in open('EGPC_HDD_12.0C.csv').readlines()[10:12]: (oops, changed the example text but didn't change the code)
for line in open('EGPC_HDD_12.0C.csv').readlines()[10:13]:

bearing in mind that the first line of the file is line zero and that array slices go from the first mentioned index up to but not including the second.

To read the last 10 lines of the file:

for line in open('EGPC_HDD_12.0C.csv').readlines()[-10:]:

The alternative approach if you want to deal with very large files would be to leave out the readlines() bit:

for line in open('EGPC_HDD_12.0C.csv'):

Then you get an iterator over the lines and only the one you're processing at the moment is in memory (plus a small bit of buffering hidden in the i/o system). You can't then do the indexing I mentioned above.

#creates a working file somewhere in memory that is used and manipulated, in this case stripping out unwanted stuff
line = line.strip()
Yes. Removes whitespace from the beginning and end of the line. In particular, removes the newline character which readlines leaves on the end of most lines.

#only works on lines that have more than 1 character in them, but don't understand what the (line[0] in '12') does, what does the [0] and '12' do?
if (len(line) >= 1) and (line[0] in '12'):
A string is an indexable list of characters. line[0] selects the first character in the line in the same way an index like this would select the first element in a list or tuple. The "in '12'" part tests to see if the first character in the line is in the string '12', ie, if it's a '1' or a '2' so a line from the second or third millenniums AD.

We need to check for at least one character in the line first otherwise the line[0] bit will blow up on empty lines. 

#creates columns that were separated by a comma and gives them names
date, hdd, estimate = line.split(',')
Yes. Also checks there are exactly three columns - if there were more or fewer it would blow up.

#makes the data in the hdd column have a floating decimal point
hdd = float(hdd)
Not quite. The date, hdd and estimate column values split out of the line are strings of characters. Using the float type as a function like this causes it to convert that string to a number.

#all about the date format, I assume that map is some clever thing that puts parts of the date that are separated by a - into a form that does not have the -, but a space instead and only prints the integer.  So if my date and time info is in the form 27/07/2014 01:10 then I will have to split it into two, the date and the time, then recombine the date part and the time part.
y, m, d = map(int, date.split('-'))
date.split('-') returns a list of strings: the three separate fields which make up the date. Map is a function which takes a function and a list (or other iterable) and applies the function to each element of the list making up a new list of the results. As with float above, int called as a function takes a string and returns an integer value parsed from it.

(Map can also take multiple iterables but let's not go there.)

#prints out the result with a blank space between each element.
print(y, m, d, hdd)
Yep.


RaspberryPi
djh — Sat, 09 Aug 2014 21:21:26 +0100
Posted By: Ed Daviesfor line in open('EGPC_HDD_12.0C.csv').readlines()[10:12]:

bearing in mind that the first line of the file is line zero and that array slices go from the first mentioned index up to but not including the second.

Hrmmf! That offends my mathematical sensibilities. The established syntax for a half-open interval like that would be [10,12)  The notation [10,12] would include both endpoints. ISO 80000-2:2009 apparently but they want money to let me see how to write mathematics :(  Hrmmmf! What is the world coming to?


RaspberryPi
Ed Davies — Sat, 09 Aug 2014 22:11:44 +0100
Luckily this is Python, not mathematics. There's no line 11.999 (or platform 9Â¾). Having [m:n] mean from m up to but not including n is convenient as you know you'll get (n-m) elements back (if m and n are positive and the addressed elements are present).

It also allows special cases like extracting zero elements at the start of a list with [0:0] whereas if the end was included that would extract one element and to get zero you'd have to write [0:-1] which wouldn't work because negative indexes count back from the end of the list so that'd actually give you all but the last element (which is what the index -1 references):

>>> ('a', 'b', 'c', 'd')[0]
'a'
>>> ('a', 'b', 'c', 'd')[-1]
'd'
>>> ('a', 'b', 'c', 'd')[0:-1]
('a', 'b', 'c')
>>> start = 2
>>> length = 2
>>> ('a', 'b', 'c', 'd')[start:start+length]
('c', 'd')

Why would you want to extract zero elements? Maybe it's just a special case. Say you're splitting a CSV line and the first field is empty then the line would start with a comma. Rather than having to do special code for this case you can just remember that the field length is zero and extract away:

Also, the poor old parentheses are already horribly overworked in Python being used for the normal mathematical grouping in expressions, for parameter lists and for tuples. Overloading them for indexing as well would probably result in a nasty accident.


RaspberryPi
SteamyTea — Sat, 09 Aug 2014 22:15:54 +0100
Thanks Ed.
I hope the pass mark is 40%, I may just scrape though.

Shall see if I can play a bit tomorrow.


RaspberryPi
djh — Sun, 10 Aug 2014 18:46:24 +0100
Ed Davies asserted: "Luckily this is Python, not mathematics."

Well, yes, and I regard my primary knowledge as computing rather than mathematics. But I still think that Python convention is very counter-intuitive and flies in the face of convention:

http://en.wikipedia.org/wiki/Comparison_of_programming_languages_%28array%29#Slicing

I confess I'm prejudiced. I learnt to hate languages using tab at the start of lines with make and I'm probably not going to overcome that hatred enough to ever love Python.


RaspberryPi
SteamyTea — Sun, 24 Aug 2014 13:20:20 +0100
Bella
Any RPi would do you , but the latest B++ is the same price now.
You will also need an SD or MicroSD card, a power supply (at least 1 amp, it is a phone charger one), a monitor that is suitable for an HDMI connection, or an HDMI adapter to your monitor.  A wireless keyboard and mouse are useful, as is a wireless network adaptor, though using the network port via CAT5 cable is more reliable.

Software you will need several, depending on what you want to do.

SDFormatter
ODIN
Win32DiskImager
Those are to prepare and write Linux to the disk

To connect a PC to an PRi I use:

WinSCP
putty
TightVNC

Start by looking here and see if it makes sense.
http://www.raspberrypi.org/help/quick-start-guide/

I got my DHT11s from Amazon
http://www.amazon.co.uk/gp/product/B008D8RI42/


RaspberryPi
bella — Sun, 24 Aug 2014 14:15:58 +0100
OK, thank you for that summary. I have saved it and will work thro' step by painful step starting with the link to the quick start guide. It will take me weeks and I may (!) need some prompts along the way. Perhaps some of the content of this thread will clarify as I proceed.


RaspberryPi
SteamyTea — Sun, 24 Aug 2014 14:44:05 +0100
I still need my hand holding when I do something new, but it is amazing how quick things fall into place.
There are some really good tutorials online about how to do things.  But like most things, they assume basic knowledge.
Too me ages to realise that software is installed in a totally different way than on a Windows machine, and the Linux file system is just horrible.

I had a look at the spec of the DHT11s and they only go down to 0Â°C, so may get some of the more expensive DHT22 for outside measurements, they go down to -40Â°C I think.  DHT11s are fine for inside use.


RaspberryPi
borpin — Tue, 26 Aug 2014 07:39:26 +0100
Posted By: SteamyTeaLinux file system is just horribleOh no, can't let that pass. Unix file systems are a thing of beauty.  Microsoft/DOS files systems are just horrible 


RaspberryPi
Seret — Tue, 26 Aug 2014 08:35:51 +0100
Posted By: borpinOh no, can't let that pass. Unix file systems are a thing of beauty.  Microsoft/DOS files systems are just horriblehttps:///forum114/extensions/Vanillacons/smilies/standard/bigsmile.gif" alt="" title="" >


100% agreed. They're certainly different from what a Windows user might be used to, but once you've made the switch you realise what an unholy mess Windows is.


RaspberryPi
SteamyTea — Tue, 26 Aug 2014 09:50:47 +0100
So after 30 years of finding an .exe file to start a program, generally in the 'program files' directory, I have still not worked out how to do that in Linux.


RaspberryPi
Ed Davies — Tue, 26 Aug 2014 11:16:58 +0100
To run a program on Linux:

1) you need to give the path explicitly (e.g., if you want to run a file called â€œscriptâ€ in the current directory you have to say â€œ./scriptâ€, not just â€œscriptâ€) or 

2) it's in one of the directories named by the $PATH environment variable or

3) it's built in to your shell.

To be executable it has to have the â€œxâ€ flag set in the file mode bits for whoever you are relative to it (owner, group or world). Set that mode for everybody with, eg, â€œchmod a+x scriptâ€.

That's about it, I think. It's a bit different from Windows but reasonably consistent and sensible.

Some less essential information:

a) The main reason for having to give the path explicitly when trying to run programs not in the $PATH is to avoid conflict between system commands and filenames in the current directory which have (maliciously or accidentally) the same name.

b) Filename extensions are not used (much) for identifying file types - it's mostly done my reading the file contents, particularly signature bytes (â€œmagic numbersâ€) at the start. The â€œfileâ€ command can be helpful in giving this and more information.

c) To find which program will actually be run for a particular command use the â€œwhichâ€ command:

edavies@bill:~$ which thunderbird
/usr/bin/thunderbird


RaspberryPi
Seret — Tue, 26 Aug 2014 11:20:24 +0100
That's because you're expecting things to be as difficult as they are on Windows Steamy! Generally on Linux you don't need to know where the individual files are. A competent OS should know this kind of thing so you don't have to! You either click an icon or launch it from a command line.

For example, the command to launch Firefox from the command line is:

firefox

No need to know where the executable is at all, just type that from any location.

If you're just running scripts on an RPi you might need to click on or run individual executables from their install location, but that works exactly the same way as Windows. Just go to wherever you dumped the application (/home? /opt?) to and run it. On the command line you do:

./script_name