Category Archives: Uncategorized

process table size in erlang

Whats the cost of increasing the process limit?

I was curious:

Ender@PROSPERO /j/git/erlang_decorators
$ erl +P 32768
Eshell V5.7.5  (abort with ^G)
1> io:format("~p",[erlang:memory()]).
[{total,4081272},
 {processes,351700},
 {processes_used,344924},
 {system,3729572},
 {atom,297141},
 {atom_used,266009},
 {binary,317744},
 {code,1836583},
 {ets,129660}]ok
2> init:stop().
ok
3>
Ender@PROSPERO /j/git/erlang_decorators
$ erl +P 10000000
Eshell V5.7.5  (abort with ^G)
1> io:format("~p",[erlang:memory()]).
[{total,43949344},
 {processes,40220628},
 {processes_used,40213852},
 {system,3728716},
 {atom,297141},
 {atom_used,266009},
 {binary,317744},
 {code,1836583},
 {ets,129660}]ok
2> init:stop().
ok
3>

So (on my 32 bit vm on windows), you pay 4 bytes per process whether you use it or not.
Pretty compact. Feels like its probably implemented as an array of pointers. Yep

People always seem to run into the default limit. Its quite low.
I wonder if any large Erlang project has ever been done without increasing the limit.

google test with static libraries in msvc

Using google test with tests in static libraries under msvc has historically been a pain.
Often you would be left scratching your head wondering why some tests didnt run. This entry presents a tool that ensures that all tests from your static library will get run by your test runner.

Disclaimer: I don’t actually recommend this approach.
You will have cleaner interfaces if your tests are written in your test runner program, which then links against your production code library.
Plus it wont be possible for test cases to sneak into the production app.
This was purely done as an exercise, but it may be of use to some people.

If you have ever tried to use gtest to test code in static libs you have probably run into this problem.
See the gtest wiki for an overview of the problem.
The wiki ends up suggesting that you don’t put tests in static libraries.

Generally id agree with that.
But if you have a lot of code that doesn’t have external linkage then testing that code can only be done indirectly.
Sure, this may be a sign that either
* it hasnt been designed for testability
* the author hasent followed TDD
* havent broken the module down enough. (SRP)

But I still found myself wondering “why cant I put a test case in a static library?”.
Clearly the fact that its on gtests wiki means that other people wonder too.

Its common for people to put eunit tests at the bottom of their erlang modules and similar practices exist in other languages. They arent best practices. But if its so common in other languages why does it suck so much in msvc?

A walk through of the problem

The common idiom for using gtest (and other test frameworks) is to have a library where you put your code, a minimal program that uses that code, and a program that acts as a test runner. This is what im going to do in this article.

For this example we would be writing our tests in our static library.
(Perhaps we are testing a piece of code with internal linkage.)

// internal linkage function we are testing
static int plus(int a, int b) { return a+b; }
// test case
TEST(MathTest, TwoPlusTwoEqualsFour) {
	EXPECT_EQ(plus(2,2), 4);
}

Compile it. Everythings fine and we get our .lib file.

Then we have our test runner. It links against our lib and gtest

#include <gtest/gtest.h>
 
int main(int argc, char **argv) {
	::testing::InitGoogleTest(&argc, argv);
	return RUN_ALL_TESTS();
}

Time to run it and see our test passing.
But wait… we see this instead

What happened?

The msvc linker does not link an obj file from a static library into the main program unless there is an unresolved symbol in the main program that resolves to that obj.
This is fair enough. You dont want code in your exe that you dont need.
But heres the kicker: static initialization code that exists in an obj wont cause it to be linked in either.
gtest works by constructing classes that are initialized and linked into the test framework during static initialization.
GCC has a –whole-archive option, which can be used to link in everything, but msvc doesnt have anything like this.

Our test runner doesnt refer to ANY code in the static library, so our test doesnt get linked in.

The solution

heh. solution. get it? we are talking about visual studios… oh nevermind…

The solution is gen_msvc_test_header.py.
You can find it here.

If you have a static library with tests in it you can generate a header file which can be included in your test runner that will force the tests to run.

run like so:

python gen_msvc_test_header.py mylib.lib generated_test_syms.h

then have your test runner include the header file

#include <gtest/gtest.h>
#include <generated_test_syms.h>
 
int main(int argc, char **argv) {
	::testing::InitGoogleTest(&argc, argv);
	return RUN_ALL_TESTS();
}

compile and run and:

It works!

How it works

This python script runs dumpbin against the lib, it then runs a regex over the output to pull out symbols that match the constructors for gtests generated classes.
Once we have that its easy to emit a header file that forces the linker to include a reference to that symbol.

here is the header we generated above

#ifndef generated_5312bcde_fd17_4e3b_bbca_99f20116304a
#define generated_5312bcde_fd17_4e3b_bbca_99f20116304a
// Generated by gen_msvc_test_header at 2011-02-10T04:22:47.397000
// do not modify 
 
#pragma comment(linker, "/include:??0MathTest_TwoPlusTwoEqualsFour_Test@@QAE@XZ")
 
#endif // generated_5312bcde_fd17_4e3b_bbca_99f20116304a

If you were going to actually use this I suggest you set it as a post build step for your static lib.

This approach can also be used for other cases where you rely on static initializers being run, but you are using static libraries.

trying to write more

Ive been meaning to write more. For a long time really. The problem is I really don’t like writing. Id rather be programming, playing xbox, or drinking beer. Sometimes all 3 at once.

Looking at the behavior grid Im trying to achieve a Green Path behavior change.
So here are some things Im going to try:

  • Boost motivation.
  • Couple the trigger to an existing behavior.
  • Reduce demotiviation by making the behavior more familiar.

Recently my brother proposed that a few of his blogger friends start a Write Club, in order to encourage more regular blogging. Basically (if i understood it correctly) every week you need to write a blog post on a certain day. You get points based on how close to the target day you wrote your post.
I guess the theory is that there will be social pressure acting to increase motivation (leaderboards!).
Plus the structured nature should help make it more familiar.
I think its a neat idea. Im definitely in.
I think itll probably need some kind of “slow car catchup” so that if someone does fall behind by missing a week that they aren’t permanently behind in the score board. Otherwise the competitive pressure of a score system would lose some impact (in my opinion).

My plan is to write short blog post every day for a week or 2, to get more used to blogging.
To boost motivation Im going to try keeping a Seinfeld Calendar. I found a nice pdf here. Alex said I should take it to OfficeWorks and get it printed up massive. I might do that.

Now I just need to work out how to trigger the behavior when motivation and opportunity is there.
An obvious technique is to couple it to an existing behavior. I dont know what though.

quickfix logon support for username password

QuickFix is an open source FIX engine.
Ive been doing a little simulator along the lines of this (but using zeromq+quickfix) to become familiar with FIX.

The ASX supports the Username and Password fields for Logon messages. Quickfix normally takes care of Logon for you, but does not support Username and Password as session settings. Im only interested in FIX 4.4 (as thats the highest the ASX supports), so ill restrict my code to dealing with that.

Because we are modifying outgoing Logon messages, normally we would use a message cracker from our Application::toAdmin() override.
But there is a bug in quickfix (1.13.3) that means that cracking non const message references calls the wrong code. It ends up calling the onMessage for the message const ref.
ie it calls

void MQFeederApplication::onMessage( const FIX44::Logon& m, const FIX::SessionID& sessionID)

instead of

void MQFeederApplication::onMessage( FIX44::Logon& m, const FIX::SessionID& sessionID)

Rather than showing how to fix that bug and the code you will use ill just code the message type check right into toAdmin(). If you were doing a real program youd fix the bug and do it properly.

Firstly the obvious. We need to inject the settings class into our application so that we can pull the username and password from the config file.

class MQFeederApplication
	: public FIX::Application,
	private FIX::MessageCracker
{
public:
	MQFeederApplication(const FIX::SessionSettings&);
// ...
private:
	const FIX::SessionSettings& settings;
// ...
};

and pass it to our Applications ctor:

int main(){
		FIX::SessionSettings settings( "../mqfeed.cfg" );
		MQFeederApplication app(settings);
//...

Then in toAdmin we check the message type and pull the appropriate username and password from the settings

void MQFeederApplication::toAdmin( FIX::Message& message, const FIX::SessionID& sessionID)
{
	// when we send Logon make sure we add Username and Password if we have it configured for this session
	// I would have like to use a message cracker for this, but theres a bug in quickfix 1.13.3
	// wrt cracking non const messages
	const FIX::Dictionary& session_settings = settings.get(sessionID);
	if (FIX::MsgType_Logon == FIELD_GET_REF( message.getHeader(), MsgType)) {
		FIX44::Logon& logon_message = dynamic_cast<FIX44::Logon&>(message);
		if (session_settings.has("Username")) {
			FIX::Username username = session_settings.getString("Username");
			logon_message.set( username );
		}
		if (session_settings.has("Password")) {
			FIX::Password password = session_settings.getString("Password");
			logon_message.set( password );
		}
	}
}

So far working with quickfix has been pretty easy.
I was initially surprised by the use of little class wrappers for each field type, but it means the compiler can check that you arent trying to set fields that arent supported.
eg

		FIX44::Logon& logon_message = dynamic_cast<FIX44::Logon&>(message);
		logon_message.set(FIX::Symbol("GOOG"));

will result in a compile time error. neat.
If you do want to add custom fields you can either use Message::setField, or regenerate the code from a modified data dictionary.

Fizzbuzz

I stumbled across an old article on coding horror about how people are having to make job applicants sit a simple filter test.
The article is here.
The test is of course fizzbuzz. Its super simple to explain (but i wont) and only takes a handful of lines of code to solve, even in the most verbose languages.

At first I couldn’t believe the stats presented in the article.
Surely their resume would filter out the people that cant program?
Surely you can tell a programmer within 5 seconds of a phone interview?
Surely theres no need for such a simple test?
But then one of my minions went for a job interview at some place and they actually gave him the fizzbuzz test.
One of the local game company CEOs also says that 30% of the programmers they get in to interview cannot program to save themselves.

Then a workmate sent me an amusing fizzbuzz solution: Enterprize Fizzbuzz
This cheered me up. Its good to see people can take a light hearted view on this sorry state of affairs.

So I decided to do some ridiculous solutions to fizzbuzz as a bit of fun.
Ive put them on github, and will add to them as I think up more ways to mess up fizzbuzz.
My current favorite is the template meta-programming version. Its succinct and quite beautiful, but a complete waste of time.

What other ways can we over-engineer fizzbuzz? Maybe a webmachine web service that streams the results back…
or using llvm to generate a machine “optimized” [1] solution at runtime.

Aside: One thing I like doing when interviewing people is getting them to show me code from their own projects, and then asking them if there is anything that they would change or improve.
It tells you a lot about the sort of programmer they are.
It would be interesting to do this with peoples fizzbuzz solutions. Sadistic maybe. But interesting.

[1] which will still be dominated by IO, hence the quotes around “optimized”.