Using crosstool-ng on MinGW

March 29, 2014 Maksym Shyte Leave a comment

Hello sadistic friends.

I have given up.

I am leaving the instructions below but this is plan too difficult and I am moving on to Cygwin. This is will remain here for simply… I don’t know… warnings to others that hair loss is not a good trade-off for getting this to work under MinGW.

We are going to try to build a compiler on Winderz now. This is a follow up to installing crosstool-ng on MinGW on Winderz. You should have MinGW already installed.

Open up your msys bash shell the batch file – C:\MinGW\msys\1.0\msys.bat as administrator. We will assume that you installed MinGW on C:\MinGW. We will be working from the instructions here. I installed mine at /home/maks/crosstool. Always, I repeat, always run the terminal as administrator or some of the decompression of packages will not occur due to permission errors.
Export your tool location by typing
```
export PATH="${PATH}:/home/maks/crosstool/bin"
```
Substitute your own path because I don’t think your name is Maks. Now you should be able to run ct-ng now. In fact, if you create a .profile file in your home directory that includes this line, you will never need to type it again.
Make a directory for your cross compiler. I called mine linux64 and I placed it in my home directory. Remember your name is not Maks. I decided that I am going to build an x86_64_unknown-linux-gnu because I want it to run on an Ubuntu built server 13.10. So in the linux64, type
```
ct-ng x86_64-unknown-linux-gnu
```
You can read about using ct-ng from the link above. Then through
```
ct-ng menuconfig
```
I moved to all the latest versions of everything, eglibc, the 3.10 kernel, binutils, etc. Also, make sure that your compiler is not linking statically. This doesn’t work on Windows (and I haven’t tracked down why).
Ok, now run
```
ct-ng build
```
You will get errors. We will work through them.
The first error will be that the filesystem is not case sensitive. We will remedy this by adding the key to the registry.
```
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive
```
Set this value to zero.
The second will be that the OS MINGW32_NT-6.2 is not supported. We will add that. Edit ~/crosstool/lib/ct-ng.1.19.0/scripts/functions and look for a function called CT_DoForceRmdir. Find the case statement and add MINGW32* to the line. It should look like this.
```
Linux|CYGIN*|MINGW32*)
```
You may need to create a source directory where all of the tarballs are cached. Simply
```
mkdir ~/src
```
The next error relates to downloading the tarballs and the use of certificates. I determined this by looking at build.log. The utility wget is attempting to validate certificates. For now, we will simply remote the check. Edit ~/crosstool/lib/ct-ng.1.19.0/scripts/functions and look for CT_DoGetFile. On the line that starts with “if CT_DoExecLog ALL wget”, add –no-check-certificate to the command line. Be mindful for split lines in script using the \.
Oh geez, there is way too much. Headers aren’t there… ugh. Too much. I have given up. I am going to now try my hand at Cygwin.

Python

Grepping any type of file encoding in Python

March 27, 2014 Maksym Shyte Leave a comment

Let’s take handling any encoding of files one step further.

We need to look for specific text in in files in a directory regardless of encoding. Here is one way in Python.

#! /usr/bin/python
import sys
import os.path
import os
import re
import fnmatch

def DecodeBytes(byteArray, codecs=['utf-8', 'utf-16']):
  for codec in codecs:
    try:
      return byteArray.decode(codec)
    except:
      pass

def ReadLinesFromFile(filename):
  file = open(filename, "rb")
  rawbytes = file.read()
  file.close()
  content = DecodeBytes(rawbytes)
  if content is not None:
    return content.split(os.linesep)

# this came from http://stackoverflow.com/questions/1863236/grep-r-in-python
# with a substitution of ReadLinesFromFile and a file name match filter
def RecursiveGrep(pattern, dir, match):
  r = re.compile(pattern)
  for parent, dnames, fnames in os.walk(dir):
    fnames = fnmatch.filter(fnames, match)
    for fname in fnames:
      filename = os.path.join(parent, fname)
      if os.path.isfile(filename):
        lines = ReadLinesFromFile(filename)
        if lines is not None:
          idx = 0
          for line in lines:
            if r.search(line):
              yield filename + "|" + str(idx) + "|" + line.strip()	
              idx += 1

lines = RecursiveGrep("needle", "\yourpath", "*.cs")

The will recurse all subdirectories, looking in all .cs files to find needed returning the data in this format (pipe separated):

full file path|line number|line content

Very useful on Windows with multilingual files.

Python

Getting lines of a file of any encoding type in Python

March 26, 2014 Maksym Shyte Leave a comment

I really don’t want to know the encoding. I only want the data. In other words, I don’t want to think. I don’t want to open notepad++ and convert between types of encoding.

My old standby doesn’t work on various file encodings that aren’t ansi (ascii, cp1252, whatever):

f = open("poo.txt", "r")
lines = f.readlines()
f.close()
for line in lines:
  dosomething(line)

I have had enough. (I am also venturing into Python 3 as I have been on Python 2 forever but that is a different story.)

The following code will read a file of different encoding and split them into lines:

import os

def DecodeBytes(byteArray, codecs=['utf-8', 'utf-16']):
  for codec in codecs:
    try:
      return byteArray.decode(codec)
    except:
      pass

def ReadLinesFromFile(filename):
  file = open(filename, "rb")
  rawbytes = file.read()
  file.close()
  content = DecodeBytes(rawbytes)
  if content is not None:
    return content.split(os.linesep)

lines = ReadLinesFromFile("poo.txt")
for line in lines:
  dosomething(line)

If you need to add encodings, simply add them to the codecs default assignment (or make it more elegant as you deem).

Utilities

Useful DLL utilities on Windows

March 26, 2014 Maksym Shyte Leave a comment

A current project involves moving from a 32 bit to 64 bit system. Some self contained exes are required to remain 32 bit while the rest of the system will move to 64 bit. Some .NET assemblies are also “any cpu”.

As in all cases where the projects get complicated and you inherit code, it is easy to lose what gets installed where especially of different types (32 bit, 64 bit, any-cpu).

DLL Information using Dumpbin.exe

To get the dump of all the DLL headers recursing all subdirectories, the following is useful in a command prompt.

for /f "tokens=*" %i in ('dir /s /b /a-d *.dll') do call dumpbin.exe /headers "%i"

If you use dumpbin.exe and need to move it to a target machine, you will also need to copy link.exe and mspdb100.dll. (This version is Visual Studio 2010 and located in C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64. I was targeting a 64 bit machine.)

Notice that a .NET assembly set to “any cpu” will look like a 32 bit assembly.

Part of the output looks like this:

Dump of file C:\turd.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
14C machine (x86)
3 number of sections
4F6A184B time date stamp Wed Mar 21 14:04:59 2012
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL

OPTIONAL HEADER VALUES
10B magic # (PE32)
8.00 linker version
2A000 size of code

.NET Assembly Type using Corflags.exe

An equally useful feature is to find out if an .NET assembly is 32 bit, 64 bit, or any cpu (or ILASM only) using corflags.exe. (This version is located here on my machine: C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools.)

for /f "tokens=*" %i in ('dir /s /b /a-d *.dll') do call echo "%i" >> out.txt & corflags.exe /nologo "%i" >> out.txt

The output looks like this:

"C:\stinky.dll"  
Version   : v4.0.30319
CLR Header: 2.5
PE        : PE32
CorFlags  : 3
ILONLY    : 1
32BIT     : 1
Signed    : 0
"C:\smelly.dll"  
Version   : v2.0.50727
CLR Header: 2.5
PE        : PE32
CorFlags  : 1
ILONLY    : 1
32BIT     : 0
Signed    : 0
"C:\uhoh.dll"  
corflags : error CF008 : The specified file does not have a valid managed header

To interpret the results…

PE	32BIT	Type of Assembly
PE32	0	Any CPU
PE32	1	x86
PE32+	0	x64

Assembly Versions

To get the version of assemblies recursing all subdirectories, the following is useful in a power shell. This will not truncate the line.

ls -fi *.dll -r | % { $_.versioninfo } | Ft -autosize | out-string -width 4096

The output looks like this:

ProductVersion FileVersion  FileName
-------------- -----------  --------
1.0.0.0        1.0.0.0      C:\poo.dll
4.3.0.4        4.3.0.4      C:\caca.dll

So there you go. Some useful utilities.

Cross Compiling

Cross compiling with MinGW and crosstool-ng

March 16, 2014 Maksym Shyte Leave a comment

I am sadistic.

I have given up.

I am leaving the instructions below but this is plan too difficult and I am moving on to Cygwin. These instructions will definitely get cross tools built but it is a serious uphill battle to get it to compile a compiler.

My goal is to cross compile on MinGW (on Winderz) for a Linux target – both 64 bit Ubuntu 13.10 or an ARM (such as a Beagle Bone). Why? Because I am sadistic. We covered that.

The real purpose is to take a Windows GUI that generates C code and compile it for a different platform (hence cross compiling). Should be easy right? well… let’s find out. This is the steps on how I did it (in my own little hacky way).

The first step will be to use crosstools-ng and get it to run under MinGW. This post will deal with that. A subsequent post will deal with creating the cross compiler.

Note that as information on the web becomes quickly out of date, realize that this is the middle of March in 2014.

Download and MinGW from here. We will assume that you installed it on C:\MinGW.
In the installation manager, mark the following packages to add and apply them. The installer is really easy to use and should be self-explanatory. (While applying them, it may be good to refill your coffee mug.)
This is not obvious now but later we will need Subversion for eglibc and gcj.exe (Java) for crosstool-ng. First copy gcj.exe from /MinGW/bin to /MinGW/msys/1.0. It is included with MinGW but not with msys. Second, install subversion onto your Windows machine. In C:\Program Files (x86)\Subversion\bin\, copy svn.exe to /MinGW/msys/1.0.
Open up your msys bash shell the batch file – C:\MinGW\msys\1.0\msys.bat.
Download and build crosstool-ng following the steps here. Substitute your version. There are listed below with my version and some other things I did.
wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.19.0.tar.bz2 tar xjf crosstool-ng-1.19.0.tar.bz2 cd crosstool-ng-1.19.0 ./configure --prefix=/home/maks/crosstool
A couple of problems now. First there was a strange tar error where it couldn’t copy a symbolic link. The second time it ran fine. The second issue is that gperf doesn’t exist. We will build that from source as MinGW doesn’t include it. Get that from GNU source here. Grab the bz2 file, uncompress, and copy the source to your home directory. Inside that directory, do the old standby.
cd gperf-3.0.4 ./configure --prefix=/usr make make install cd crosstool-ng-1.19.0 ./configure --prefix=/home/maks/crosstool
Now it can’t find the curses header. Originally, I thought I had pdcurses package installed and was good to go. The package is missing some important things like a DLL and header. So onto source we go. This page has all the information. Even though it appears for mingw-w64, it works for 32 bit as well. In the source directory, do the following.
./configure --enable-term-driver --enable-sp-funcs --prefix=/usr make make install
Surprise! More issues. Here is a list of corrections then redo the previous step.
Now back in crosstools-ng, run ./configure –prefix=/home/maks/crosstool. Yea!!! It creates a make file. Now on to making it. Run “make” now.

Next issue? The gnu extension strcasestr doesn’t exist in MinGW. The file ./kconfig/nconf.c uses it. I am a bit surprised that the configure script didn’t check for that. After much research, I decided to simply implement it inside the file that needed it. Add the prototype to the top of the file:

// Added for support in mingw. This ought to be checked and enabled with autotools.
const char *strcasestr(const char *s1, const char *s2);

Add the following to the buttom of the file.

// Added for support in mingw. This ought to be checked and enabled with autotools.
const char *strcasestr(const char *s1, const char *s2)
{
 // if either pointer is null
 if (s1 == 0 || s2 == 0)
  return 0;
 // the length of the needle
 size_t n = strlen(s2);
 // iterate through the string
 while(*s1)
 // if the compare which is case insensitive is a match, return the pointer
 if(!strncmpi(s1++,s2,n))
  return (s1-1);
 // no match was found
 return 0;
}

One more thing. If you really wanted to get rid of all warnings, cast the first argument to every bzero call to a void pointer.

Finally, a make and a make install will work. You will see a bin and lib and other directories in /home/maks/crosstools (or wherever you placed your crosstool.

Next we are finally ready to build our cross compiler. We will discuss this in our next post. Why? Because we are sadistic.

Things we like

Right Sized Project Management Toolset

March 13, 2014 beans Leave a comment

I have been using Assembla now for a few years and want to give a shout out to the company for making a great product.

If you have not heard of Assembla – it is a Scrum / Kanban / Agile project management system on the web. It is cheap, with prices ranging from free for small public projects to more expensive for multi-project many user service levels. My experience has been with the $490 / year level which gets several private project spaces, a dozen users or so and quite a bit of storage. Check out the web site for current prices and plans.

Swiss Army Knife: Assembla is one stop shopping for managing a project. Features include: Tickets, CardWall, Wiki, Messages, Version Control. Really, everything you need, all in one place, accessible from every place.

Let’s take a look at some of the features in more detail.

Tickets:

Tickets or cards are the most important part of Agile project management. Because they are used so frequently – they should be quick and easy to enter. Ideally – I should be able to just mail in a ticket. Ticket priorities change frequently, so I should be able to drag and drop them. I’ve had to work with systems where entering a ticket is like filling in a tax form. Assembla has just the essentials, nothing more and nothing less. You can add fields, but it is a simple effective system without the scripting that other systems have. I like the simplicity and SPEED.

Collaboration:

Agile is all about collaboration. To me, that means everything is out there and visible to the people who need to see it. The wiki is a great place to put documentation, brainstorming, processes, standards, knowledge base articles etc. Messages encourage brainstorming and provide a stream of consciousness on a topic. Files provide a place to upload Word documents if you are into that kind of thing. You can also link to Google docs. Snippets let people comment and collaborate on a block of code. There is a StandUp tool so that if your team is distributed, you can have not real time stand up meetings. There is a Twitter feed.

Version Control:

Git, Mercurial, Perforce, Subversion, GitHub, BitBucket are all supported.

Assembla can be backed up using Amazon Backup, or you can tell the system to generate a backup set as a file you can download and archive.

There are many charts, and an API if you want to create custom reports off of the raw data.

Ticket views can be customized with filters and adding / removing fields. This is fantastic as it lets lets me get a quick and detailed view of the status of tickets.

Weak areas: The system security is not very granular. For example, you can’t give a consultant access to the source code repository without also giving her visibility into the whole project. Normally visibility is a strength, but in one case, we just wanted to give a consultant access to the repository without giving visibility into the total scope of the project. Time tracking could be improved. For example; if you make a mistake, like entering that you worked 16 hours when you meant to say 6, you can’t change it once it is saved.

Certainly for each individual tool, you could find a better tool somewhere else. However, having everything in one place, in essentially one tool, with cross link capability, Assembla has been a huge productivity boost.

Things we like

ecobee thermostat review

March 13, 2014 Scat Leave a comment

The new scat pad came with thermostats that look like 1970’s surplus thermostats.

Totally analog. I think they even had mercury in them.

So I started a search for more appropriate units, with a high geek factor. I looked at the Nest 2, which would be OK . . . for my mom. The Nest 2 does not get good reviews, but they look great. People who buy them love them for their design. Read more than one review before making a decision. I also considered the Honeywell WiFi units sold at Home Depot. These units are nice and get good reviews. However, they lack the cool factor of the Nest 2, and the geek factor of the Ecobee.

The Ecobee:

Stat2, http://www.ecobee.com/solutions/home/smart/ has the geek factor and function I was looking for. I can program as many time periods as I want. You are not limited to: Sleep, Awake, Away of typical programmable thermos. So I have Sleep, Awake, Workout, Work, Home. You can also program Vacations. It has a “Quick Save” button, so if the weather is great, touch a button on the way out the door to go surfing and it will set the temp back 4 degrees until you return. The units are WiFi enabled, so they can be programmed via Chrome, or iPhone or Android.

Any geek can install these units themselves. The installation is slightly more complex than the competitors because the Ecobee is two pieces of hardware. There is a smart board that contains the relays etc which connects to your HVAC units. This can be installed anywhere, but is usually close to the HVAC unit itself. The other piece is the thermostat unit, which connects to the smart board. The thermostat unit is a nice sized, color touch screen with a neutral design that is skinable. The smart board can accept a module to connect additional sensors (temperature and humidity). I suppose Ecobee or a hacker could create additional modules. The smart board has a zillion connections to connect to and control about any type of HVAC unit you need to control.

The scat pad has three zones, and thus three of these units. The units are all tied to one account on the web, which means, I can easily control each unit from my phone or whatever.

Ecobee provides extensive reports on your energy usage. If the reports are not enough, they provide an API so you can go nuts creating your own.

Now you are asking yourself, “wouldn’t a real geek build their own system”?

You could probably could build your own, but why? Not to mention, HVAC units are expensive to replace if you get it wrong.

After all, we are just talking about a thermostat.

There are many competitors. So take a look and consider upgrading to a connected thermostat.

C# and nullable value types

March 11, 2014 Maksym Shyte Leave a comment

In C# you can declare nullable value types. (This isn’t really what happens but the syntax looks like it.) This is an unusual construct for those who come from other “lower-level” languages such as C++.

For example, the following declaration of “i” can not be set to null.

int i = 42;

However, in the following example, the following can be done.

int? j = 42;
int? k = null;

This blows my C++ brain. In reality, “j” and “k” are not ints. They are objects of type System.Nullable<T> as Microsoft describes here.

So, to safely coerce to int, you will need to check the value to be sure it is not null. One approach is to use the HasValue method to make decisions or the GetValueOrDefault to simply get the default base type value. For that matter, you can assign in a try-catch block but that seems so evil.

Finally, the cleanest type is the null-coalescing operator which is another foreign concept to C++ type brains.

Check out each example below.

int? i = null;
int j;

if (i.HasValue)
   j = (int) i;
else
   j = 42;

// j = 42 as i is null
Console.WriteLine("j = " + j.ToString());

// y is set to zero 
j = i.GetValueOrDefault();
// j = 0 as i is null
Console.WriteLine("j = " + j.ToString());

// an exception? really?  
try
{
    j = i.Value;
}
catch (InvalidOperationException e)
{
    Console.WriteLine("Why would anyone do this?");
}

// my favorite
j = i ?? 42;
Console.WriteLine("j = " + j.ToString());

The output you ask? (That’s right, I heard you.)

j = 42
j = 0
Why would anyone do this?
j = 42

Using an exception is like having only one square of single-ply toilet paper. If it is all you have, you use it. However, we both know there are better methods.

Python, Remote Access

Hysteria with PYROmania (special URIs)

March 10, 2014 Maksym Shyte 2 Comments

I had a need to modify code written using Pyro so that objects on localhost could be exposed remotely. I have never worked with Pyro before so I was in some hysteria. I was ready to try though.

There is no definitive guide anywhere using the special names in the URI (actually I found something here and here later). Love bites.

If you look at Pryo servers, the URI from the daemon is a very long string like the following:

PYRO://127.0.0.1:62100/c0a8006516bc7752e7526becdb059ce9

That is a rather long URI and the number changes on every service start up (obviously like a GUID or time based id). So on the client side, is this my URI? Is it too late for love?

Well, no. Here is a quick guide for how to use the special URI strings and you can be a Pyro Animal.

For name servers, you can use:

PYRONAME://<hostname>[:<port>]/<objectname>

For straight remote access (otherwise called the regular method):

PYROLOC://<hostname>[:<port>]/<objectname>

So, it isn’t as bad as it initially seems. No Foolin’. Finally, Armageddon It.

Python

Comparing huge files using Python

March 6, 2014 Scat Leave a comment

I had the need to compare two huge files.

None of the file compare tools that I tried could handle large files, so I decided to write my own compare utility.

Comparing large files is not a common case. I could have solved my issue by loading each file into a database, then using the excellent RedGate DataCompare against the two tables. Heck for that matter – load both in one one table and do a GROUP BY. But I had several files and loading the database would have been tedious.

Use cases could include replacing an ETL process with a new process or upgrading a web service that generates large datasets. For me, it was a system migration of an ETL process.

The script was done ‘quick and dirty’, but really came in handy.

What this does is opens the two files, and starts reading them with a configurable line buffer (set to 10 currently). It looks to see if lines from the second file are in the buffer of the first file. It records misses and increases the buffer. It outputs misses for later review and will stop processing if there are too many misses (configurable).

That is it – easy

### Program to compare two large files

### fails fast – order of rows is important

### smart enough to look ahead for matching rows

fs2 = “C:/Temp/File2.txt”

fs1 = “C:/Temp/File1.txt”

ofs = “C:/Temp/diffb.txt”

maxerrors = 500

Lq1 = []

Lq2 = []

rcount1 = 0

rcount2 = 0

isFound = False

#setup – read first n rows

n = 10

f1pointer = n

f2pointer = n

f1 = open(fs1)

f2 = open(fs2)

of = open(ofs, ‘w’)

for x in range(0,n):

Lq2.append(f2.readline())

# important stuff

errorsFound = 0

rowcounter = 0

goodcounter = 0

row = f1.readline()

while errorsFound < maxerrors and len(row) > 10:

isFound = False

if (rowcounter%10000 == 0) :

print “.”,

rowcounter = rowcounter + 1

for y in range(0,len(Lq2)):

if (row == Lq2[y]):

isFound = True

Lq2.pop(y)

Lq2.append(f2.readline())

break

if not isFound:

errorsFound = errorsFound + 1

of.write(row)

for x in range(0,n):

Lq2.append(f2.readline())

row = f1.readline()

print “”, “done – rows processed: “, rowcounter

f1.close()

f2.close()

of.close()

print len(Lq2)

Geek Droppings

Monthly Archives: March 2014

Using crosstool-ng on MinGW

I have given up.

Grepping any type of file encoding in Python

Getting lines of a file of any encoding type in Python

Useful DLL utilities on Windows

DLL Information using Dumpbin.exe

.NET Assembly Type using Corflags.exe

Assembly Versions

Cross compiling with MinGW and crosstool-ng

I have given up.

Right Sized Project Management Toolset

ecobee thermostat review

C# and nullable value types

Hysteria with PYROmania (special URIs)

Comparing huge files using Python

The poop on being a geek