Green Building Forum - RaspberryPi

RaspberryPi

borpin — Tue, 26 Aug 2014 15:44:14 +0100

As requested - "I have one of these http://www.maplin.co.uk/p/silver-usb-wireless-touchscreen-weather-forecaster-n96fy connected to a Pi running pywws to log the data. "

Pywws is the software http://jim-easterbrook.github.io/pywws/doc/en/html/guides/getstarted.html regular updates and an active google group for support.

http://jim-easterbrook.github.io/pywws/doc/en/html/guides/getstarted.html does what it says but read the first line carefully and follow the 'dependencies' link before doing anything else. I suspect the software has changed since I installed it so I won't say any more.

I do use monit to monitor the process as it was falling over occasionally and monit simply restarts it. I also wrote a proper daemon for it so monit could monitor it. This may have now been implemented but I have not been following recently.

RaspberryPi

SteamyTea — Tue, 26 Aug 2014 15:53:45 +0100

Thanks
Shall wait till mine arrives and see how I get on.
Still need some scaffolding

RaspberryPi

djh — Tue, 02 Sep 2014 18:27:13 +0100

Posted By: SteamyTea
Still need some scaffolding

I just gave three lorry loads back to the people that own it! (Well, they came and spent two days collecting it) So we can finally see some of the house. SWMBO thinks it looks like a barn Hopefully she'll change her mind when we can see the rest of it!

RaspberryPi

SteamyTea — Tue, 02 Sep 2014 22:46:11 +0100

A house is a barn really. I have never been bothered by the exterior of a house, I would much prefer a view.
Friends of mine bought a modern house that overlooked a thatched cottage. made them feel as though they were part of a pretty village.

RaspberryPi

djh — Sun, 07 Sep 2014 21:59:02 +0100

Posted By: SteamyTeaFriends of mine bought a modern house that overlooked a thatched cottage.

Our new house is next to a thatched cottage. It doesn't overlook it though. Planning conditions are wonderful.

RaspberryPi

borpin — Wed, 01 Oct 2014 20:07:57 +0100

Anyone using a Pi really must do an update and an upgrade to patch for the shellshock 'feature'.

RaspberryPi

SteamyTea — Wed, 01 Oct 2014 20:33:37 +0100

Done mine.

It is easy to check.

copy and paste (right click) into a terminal

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If it prints

vulnerable

this is a test

You have a problem
If it prints

this is a test

You should be OK

RaspberryPi

djh — Thu, 02 Oct 2014 10:32:56 +0100

Posted By: borpinAnyone using a Pi really must do an update and an upgrade to patch for the shellshock 'feature'.

Patching it is sensible but I think a more important observation was made by jojopi at

http://www.raspberrypi.org/forums/viewtopic.php?f=66&t=87812

If your Pi gives bash unsanitised environment variables from the internet, in contexts where the remote user is not trusted to run arbitrary commands, I suggest disconnecting it to fix that issue and audit your code.

That's why perl has taint mode! Don't use shells in web servers.

RaspberryPi

cbatjesmond — Thu, 02 Oct 2014 22:21:12 +0100

Posted By: djh
That's why perl has taint mode! Don't use shells in web servers.

And don't have bash as /bin/sh so that it's used by any program using system() or popen() functions.

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Feb 19  2014 /bin/sh -> dash

So safe from _this_ problem, if not the next, even if this bash is vulnerable.

(I've just had a week+ of people running round at work like headless chickens: yes we've dozens of machines
with vulnerable versions of bash, but /bin/sh is just sh or perhaps dash. So we need to patch but you'd think everyone's credit card#, bank a/c, and naked selfies had been stolen!).

RaspberryPi

SteamyTea — Wed, 14 Jan 2015 11:11:48 +0000

Probably a simple solution to this, but going to ask before I try it out.

If I have one of my energy monitoring RPi's connected to a network and I want to save the data it collects to a remote server (back at office). Is it just a matter of getting the port forwarding sorted out on the routers and then changing the path where the file is saved (so a couple of IP addresses, one for the router and one for the file server and something to make it log in).
I intend to just send the data as soon as it is collected and cleaned up. So will only be a short line with a timestamp and an amount. The hard work will go on in the office.

So my Python script will do something like this:

Load all the necessary modules/libraries

Read the data from the USB port

Strip out unwanted data

Add timestamp

Save to remote location

Then do it all again and again.

I may add in a second save to a local file that saves all the data to the RPi in case there is a connection loss.

RaspberryPi

Ed Davies — Wed, 14 Jan 2015 12:35:27 +0000

Posted By: SteamyTea: â€œSave to remote locationâ€

Depends what you mean by this. You can't just write a file (open, print, printâ€¦, close) to some magic filename to make it write remotely without setting up something possibly a bit complicated. If you want to do it that way then probably SSHFS is the best approach:

https://en.wikipedia.org/wiki/SSHFS

Set up properly that would be very secure and convenient for your logging script. I've not tried it myself so wouldn't know - particularly on the Raspberry Pi. The SFTP bit should be easy but fuse (which provides the mounted file system bit) could be a bit of a pain - not sure.

Posted By: SteamyTea: â€œI may add in a second save to a local file that saves all the data to the RPi in case there is a connection loss.â€

That's a good idea. You could write a local file then just scp to the remove server.

http://linux.die.net/man/1/scp

You could even write whatever files you want to a local directory then rsync that to the server. That would have the advantage that if the connection is lost for a while then when it next picks up again the missing data would get transferred in one go. Also, you could do something like write data every 5 minutes but only rsync every half hour or whatever. rsync is clever at only sending the parts of files which have changed so your local logging could be appending to a file then, when transfers happen, only the changed parts on the end get sent.

http://linux.die.net/man/1/rsync

(PS, I'm assuming here that the server is a Unix-like system: Linux, BSD, OS X or whatever, perhaps another Pi. If it's Windows I'll leave suggestions to somebody else.)

RaspberryPi

gravelld — Wed, 14 Jan 2015 14:32:00 +0000

To avoid firewall (is "back at the office" not on the same network?) and other networking issues with your office server you could always store the data in the cloud for later retrieval by your office server, this could be via saving files (e.g. using S3), or write a simple servlet that accepts the data as HTTP messages and stores the data.

RaspberryPi

skyewright — Wed, 14 Jan 2015 15:08:25 +0000

Posted By: gravelldor write a simple servlet that accepts the data as HTTP messages and stores the data.

I think that's another way of describing something I was wondering about too, i.e. maybe the requirement is simple enough to send the data URL encoded via http with perl, php, or some such, catching & saving the data at the other end?
e.g.

http://www.fileserver/pidata.cgi?timestamp=201501141502&data=1001

RaspberryPi

ringi — Wed, 14 Jan 2015 17:25:49 +0000

What about sending the data over SMTP (email) to your ISP server address to a gmail account?

RaspberryPi

Ed Davies — Wed, 14 Jan 2015 17:29:50 +0000

Steamy's whispered that they want to react to changes quickly, within a minute, not just do logging as I assumed.

From the security point of view having a Pi at the office receiving from the various sensors would probably make sense.

Whatever, for quicker reactions having a socket interface at the office for the remote systems to connect to might be best. That could be over HTTP (or better HTTPS) but even if it's a Windows system that doesn't knock out SSH.

https://en.wikipedia.org/wiki/Comparison_of_SSH_servers#Platform

I know nothing about them but Apache Mina SSHD and PowerShell Server catch my eye.

The sensor Pis could execute the equivalent of â€œssh someaccount@server.example.org program 1234 5.6"

where program is whatever program (or script or batch file) is supposed to react to the readings (or decide if a reaction is needed) and the numbers are the latest set of readings passed to the program on its command line.

RaspberryPi

SteamyTea — Wed, 14 Jan 2015 21:23:58 +0000

I need to have a serious think about this, thought it would be tricky.
Better see if I can get my ports opened so I can remotely connect first (after I have disconnected my back-up server and main PC).

What is it they say, write a little, test alot.

RaspberryPi

gravelld — Wed, 14 Jan 2015 21:43:51 +0000

I wouldn't call it tricky, more like intricate. I don't understand your network(s?) setup.

RaspberryPi

SteamyTea — Wed, 14 Jan 2015 21:53:35 +0000

I don't know the network setup where it is for yet.

My home one is simple though.
ADSL line, wireless/4port router/modem, jumble of RPi's, phones, the odd Mac (mates when he calls around), neighbours Hudl (I think they have put their phones on it too) and my laptop PC. And the useless Buffalo NAS.

Initially I shall try and set something up to send the data to my a spare RPi. Not going to get the chance to do much before Sunday though.
First thing to do is set all my wireless stuff to static IPs, that will make my life easier. I hate dynamic IPs.

RaspberryPi

djh — Thu, 15 Jan 2015 19:47:29 +0000

Posted By: SteamyTeaFirst thing to do is set all my wireless stuff to static IPs, that will make my life easier. I hate dynamic IPs.

Something's wrong there.

RaspberryPi

SteamyTea — Thu, 15 Jan 2015 22:43:12 +0000

How come? I am only talking about my own Intranet, not my router (though I think I have had the same IP address for 9 years now, may have mentioned that I wanted a static one when I signed up)

RaspberryPi

gravelld — Fri, 16 Jan 2015 10:41:54 +0000

It's just more state to record and maintain.

RaspberryPi

djh — Sun, 18 Jan 2015 17:25:20 +0000

Posted By: SteamyTeaHow come?

I don't know anybody who sets static addresses for wireless stuff. Since wireless devices can connect to several different networks, they're all designed to just pick up an address from whichever network they connect to. There's no reason for you to know the address or worry about it.

RaspberryPi

SteamyTea — Sun, 18 Jan 2015 17:29:34 +0000

Without knowing the IP address how can I connect remotely? I would like to use 'friendly names', is there an easy and reliable way to do that. Also I would set this up to just be wired, wireless (not G3/4) is not much use in a field

RaspberryPi

cbatjesmond — Sun, 18 Jan 2015 18:45:26 +0000

Posted By: djhI don't know anybody who sets static addresses for wireless stuff.

I do!

It depends on what the client device is: if it's a phone/tablet/laptop - a media consumer, yes dynamic all the way. But if the device is a data/service provider and wireless because that's only cost-effective way of getting it on the network: static addressed!

RaspberryPi

SteamyTea — Sun, 18 Jan 2015 19:45:12 +0000

So say I have a server that wants to contact a number of remote clients and ask fro the data.
The server sends out the message 'Here I am, give me what I want'.
The clients rummage though their files, finds what is wanted, and sends them back.

If the server tries to contact the last place it saw the client, and the client has moved (the IP address has changed), the all seeing internet will just say 'Return to sender, address unknown', see, Elvis is alive and with the CIA now.

Now I know there there are services like No-IP, that allow the client to connect via a static server, but not sure how fast they react to calls for data (I want to get a response within a few seconds).

There is an application that uses the Tor network to shift data about (I like Tor and wish everyone would set up a relay), but my experience of that is slowness. Not surprising as it has to wrap and unwrap data all the time and goes though at least 5 computers to do it. Though I was using it on my HTC Desire C, which is about as powerful as an RPi.

So I think I have to set the clients up with a static IP (not actually my part of the project, if it comes off), unless there is a way around it. I know that the RPi Foundation have a list of Pi's that are connected via something or other, but I am reluctant to give out a location to a load of school kids.

RaspberryPi

borpin — Sun, 18 Jan 2015 20:16:14 +0000

Posted By: SteamyTeaSo say I have a server that wants to contact a number of remote clients and ask fro the data.

Better if the clients simply send the data regularly or by exception to a 'static' server.

RaspberryPi

SteamyTea — Sun, 18 Jan 2015 20:21:49 +0000

Yes, just been chatting to Ed about that. I was going to work on it a bit today, but it was sunny, so I went out instead.
I think I need to write things down and get it clear what I need to do and then work out how to do it. Save confusing myself.

RaspberryPi

cjard — Sun, 18 Jan 2015 21:03:49 +0000

Posted By: SteamyTeaSo say I have a server that wants to contact a number of remote clients and ask fro the data.
The server sends out the message 'Here I am, give me what I want'.
The clients rummage though their files, finds what is wanted, and sends them back.

Technically what you're calling the "server" in that example is a client

Clients make requests of a server and servers respond. If you have a RasPi that requests data from your fridge, washing machine, xbox media centre and some weather website, then it's a client, not a server..

No-IP style things update reasonably quickly. Most things that use a dynamic dns service are programmed suc that they send the service an update when they notice their IP changing (or when they've been rebooted)
Some DNS services cache requests they see, but the dynamic services have short expiry times to keep the caches fresh.

Ideally though, as others have said, you don't send your "server" out to places it last saw clients to try and get data, you have your server sit in a known place and clients post data to it when they can

RaspberryPi

SteamyTea — Sun, 18 Jan 2015 21:28:00 +0000

Posted By: cjardTechnically what you're calling the "server" in that example is a client

Clients make requests of a server and servers respond. If you have a RasPi that requests data from your fridge, washing machine, xbox media centre and some weather website, then it's a client, not a server..

That makes sense, even if the files are permanently stored on the client (back at the office).
Think this is why I need to write it all down and draw a few pictures.

RaspberryPi

cjard — Mon, 19 Jan 2015 08:47:55 +0000

Every time I get to a particularly think-intensive part of a piece of software I'm developing I write everything out in English comments and drawings first - system design is so low level that it's always better to design it high level in whatever language you natively think in, then translate

Feel free to post your drawings up so we can see/comment .

Also, come to appreciate that to a computer, everything is a file - just a stream of bytes that is either sitting somewhere or being transmitted somewhere. I can better describe what I mean if I can get an idea of the context you're operating in, but essentially it's really handy when working with computers, not to put boundaries and constraints on the way you think about a system. There was a comment earlier, for example, about having several sensors and should there be several python scripts or just one - several scripts needs some coordination via e.g. a bash script, but then you step back and realise that really, there's no difference between these approaches - either one script has several subparts that do stuff, directed by the main script, or there are several scripts, directed by.. another script. It's all essentially a programming language/environment and it matters not that it's one file, several files.. ultimately it's just a block of data in the computer's memory so the way to decide on the approach is for best future readability, maintainability and modularisation - it's the human element that needs to have its life made easy, not the computer's :)