All posts by Scat

I'll post a lot about computer stuff, esp programming, but I am a general purpose geek with interests in music, cooking and long walks on the beach. Hope you check back frequently, and enjoy the posts.

Intelligent Agent #2
(ninja turtles)

As promised, here is a super simple example of an Intelligent Agent program using NetLogo.

The goal is to find the boundaries within an image. There are powerful algorithms for locating boundaries, but I wanted to do it another way, I wanted to see if Intelligent Agents could be used. As I said in my earlier post, NetLogo provides a rapid prototyping means for creating Agents and offers a visual development and execution environment. Below is the image I started with. Note the low resolution. NetLogo could handle a much higher resolution image, but the image would be too large to fit in this post, so I had to lower the resolution so I could display it here. (now someone who is an expert in NetLogo might point me to a setting to make the ‘patches’ smaller, that would help).

Screen shot before Agent run.
Contour1

Screen shot after Agents have run a bit.
Contour2

As you can see, the Agents did a fair job of locating the color contours, and thus the edges.
My approach is bone headed simple. Agents wander around randomly keeping track of the color of recently visited patches. If the new patch is enough different from the prior patch, then it marks the patch White to indicate a boundary.
The approach has a serious flaw, and a weakness. The flaw is that the boundary is dependent on the direction of the Agent at the time it encounters a boundary. For example, an Agent moving from Blue to Red will mark the Red patch, while a neighboring Agent moving from Red to Blue will mark the Blue patch. This causes the edge to be much more ragged than it really is. This could be solved by adding a rule such as always mark the lighter colored patch.
The weakness is the algorithm could be much more efficient. Rather than wander aimlessly, the Agent could try to determine the direction of the edge and follow it. If two Agents collide, one could jump to a new location to let the other complete the boundary.

I am sure you are eager to see some code. OK, here goes:
extensions [bitmap ]

turtles-own [last-color second-last-color hunt-color]

to setup
let img bitmap:import "C:\\Temp\\Desert.jpg"
;;set img bitmap:to-grayscale img
bitmap:copy-to-pcolors img false

create-turtles 10 [fd 10]
ask turtles [
set last-color pcolor
set second-last-color pcolor
set hunt-color pcolor
]
end

to hunt
ask turtles[
rt random 50
lt random 50
fd 1
if pcolor = 0
[jump-elsewhere]
set second-last-color last-color
set last-color pcolor
if isContour second-last-color last-color
[set pcolor [255 255 255]]
]
hunt

end

to-report isContour [color1 color2]
let retVal false
foreach [0 1 2]
[
;; show item ?1 color1
if abs (item ?1 color1 - item ?1 color2) > 60 ;; or item ?1 color2 - item ?1 color1 > 60
[set retVal true]
]
if approximate-rgb item 0 color1 item 1 color1 item 2 color1 = white or approximate-rgb item 0 color2 item 1 color2 item 2 color2 = white
[set retVal false]
report retVal
end

to jump-elsewhere
set xcor random 40
set ycor random 40
set last-color pcolor
set second-last-color pcolor
set hunt-color pcolor
end

Intelligent Agents
(ninja Turtles)

Most of us in IT are interested in Artificial Intelligence and leveraging AI techniques to solve current problems.
Intelligent Agents is one branch of AI and although not as sexy as Neural Nets and other branches, it is perhaps the most widely used.
Agents are everywhere, from viruses, to web crawlers, there are many bits of code with a degree of autonomy.
Definitions of Agents can be found on Wikipedia.
In a nutshell, an Intelligent Agent receives data from its environment and makes decisions based on that data.
Agents work for someone, so there is usually a communication and reporting aspect.
If this sounds like the proxy pattern, than I have done a poor job describing it, please see the Wikipedia article which goes into much more detail.

A classic simplified example of Intelligent Agents is called Sheep and Wolves. In the most simple form, you have Sheep agents, Wolf agents and an environment made up of grass. Sheep eat grass, Wolves eat Sheep. Sheep wander around aimlessly as do Wolves. When a Wolf bumps into a Sheep, it eats it. If all the Sheep are eaten, the Wolves all die. If the Sheep over populate, they die of starvation.

Even this simple model can be expanded to make the problem more ‘real’ and interesting.
For example – the sheep could learn when wolves hunt. The wolves could learn when sheep graze. There could be different species of grass that grow at different rates and have differing nutritional value. Food is not the only resource that animals need, so you could introduce water and shelter. Sheep don’t clone themselves, they must find a mate etc. Sheep have a notoriously low learning rate, so if one sheep discovers a new resource, the others are slow to learn about the new resource, but some do eventually learn.

You can see how fun this could get even for a contrived example.

Northwestern University has provided NetLogo, a great, easy to use and visual way to develop models such as described above.
Go here: NetLogo for more info and to get started.
MIT has a similar system: StarLogo

Before you go all Donatello on me, download NetLogo and explore some of the 100 or so included models.

Soon I will post a simple model which can locate edges in an image. I call it color contouring.

Screen shot of Sheep and Wolves model running in NetLogo.
sheepwolves

ecobee thermostat review

The new scat pad came with thermostats that look like 1970’s surplus thermostats.

oldthermo
Totally analog. I think they even had mercury in them.
So I started a search for more appropriate units, with a high geek factor. I looked at the Nest 2, which would be OK . . . for my mom. The Nest 2 does not get good reviews, but they look great. People who buy them love them for their design. Read more than one review before making a decision. I also considered the Honeywell WiFi units sold at Home Depot. These units are nice and get good reviews. However, they lack the cool factor of the Nest 2, and the geek factor of the Ecobee.
The Ecobee:
Stat2, http://www.ecobee.com/solutions/home/smart/ has the geek factor and function I was looking for. I can program as many time periods as I want. You are not limited to: Sleep, Awake, Away of typical programmable thermos. So I have Sleep, Awake, Workout, Work, Home. You can also program Vacations. It has a “Quick Save” button, so if the weather is great, touch a button on the way out the door to go surfing and it will set the temp back 4 degrees until you return. The units are WiFi enabled, so they can be programmed via Chrome, or iPhone or Android.
Any geek can install these units themselves. The installation is slightly more complex than the competitors because the Ecobee is two pieces of hardware. There is a smart board that contains the relays etc which connects to your HVAC units. This can be installed anywhere, but is usually close to the HVAC unit itself. The other piece is the thermostat unit, which connects to the smart board. The thermostat unit is a nice sized, color touch screen with a neutral design that is skinable. The smart board can accept a module to connect additional sensors (temperature and humidity). I suppose Ecobee or a hacker could create additional modules. The smart board has a zillion connections to connect to and control about any type of HVAC unit you need to control.
The scat pad has three zones, and thus three of these units. The units are all tied to one account on the web, which means, I can easily control each unit from my phone or whatever.
Ecobee provides extensive reports on your energy usage. If the reports are not enough, they provide an API so you can go nuts creating your own.
Now you are asking yourself, “wouldn’t a real geek build their own system”?
You could probably could build your own, but why? Not to mention, HVAC units are expensive to replace if you get it wrong.
After all, we are just talking about a thermostat.
There are many competitors. So take a look and consider upgrading to a connected thermostat.

Comparing huge files using Python

I had the need to compare two huge files.

None of the file compare tools that I tried could handle large files, so I decided to write my own compare utility.

Comparing large files is not a common case. I could have solved my issue by loading each file into a database, then using the excellent RedGate DataCompare  against the two tables. Heck for that matter – load both in one one table and do a GROUP BY. But I had several files and loading the database would have been tedious.

Use cases could include replacing an ETL process with a new process or upgrading a web service that generates large datasets. For me, it was a system migration of an ETL process.

The script was done ‘quick and dirty’, but really came in handy.

What this does is opens the two files, and starts reading them with a configurable line buffer (set to 10 currently). It looks to see if lines from the second file are in the buffer of the first file. It records misses and increases the buffer. It outputs misses for later review and will stop processing if there are too many misses (configurable).

That is it – easy

### Program to compare two large files
### fails fast – order of rows is important
### smart enough to look ahead for matching rows
fs2 = “C:/Temp/File2.txt”
fs1 = “C:/Temp/File1.txt”
ofs = “C:/Temp/diffb.txt”
maxerrors = 500
Lq1 = []
Lq2 = []
rcount1 = 0
rcount2 = 0
isFound = False
#setup – read first n rows
n = 10
f1pointer = n
f2pointer = n
f1 = open(fs1)
f2 = open(fs2)
of = open(ofs, ‘w’)
for x in range(0,n):
    Lq2.append(f2.readline())
# important stuff
errorsFound = 0
rowcounter = 0
goodcounter = 0
row = f1.readline()
while errorsFound < maxerrors and len(row) > 10:
        isFound = False
        if (rowcounter%10000 == 0) :
            print “.”,
        rowcounter = rowcounter + 1
        for y in range(0,len(Lq2)):
            if (row == Lq2[y]):
                isFound = True
                Lq2.pop(y)
                Lq2.append(f2.readline())
                break
        if  not isFound:
            errorsFound = errorsFound + 1
            of.write(row)
            for x in range(0,n):
                Lq2.append(f2.readline())
        row = f1.readline()
print “”, “done – rows processed: “, rowcounter
f1.close()
f2.close()
of.close()
print len(Lq2)