Review: ZeroMQ [Packt Publishing]

ZeroMQ book

TL;DR: Despite its flaws, this book could still make a good printed primer for new hires. Experienced people should look elsewhere.

I am a big fan of the ZeroMQ library. When Packt asked me to review their ZeroMQ book I was very interested to see how it turned out. ZeroMQ is a networking library which is commonly described as “Sockets on steroids”. Many successful projects use ZeroMQ to handle network communication and IPC, for example ZeroRPC and the Mongrel2 web server.

I strongly believe that if you are writing networked software that you should become familiar with the ZeroMQ library.

Standard disclaimer: I have no affiliation with the author or publisher of this book. I do not benefit from sales of this book. I received an electronic copy for review from the publisher.

ZeroMQ’s intended audience is people with (some) C coding experience, but no ZeroMQ experience is assumed. It is a very short book. Amazon lists it at 108 pages, but the pdf I have is really only about 90 pages.

Already I know that this is probably not a book for me… but I will try to be fair.

The book is broken up into 4 sections.
Chapter 1 covers the request-reply pattern, plus some other minor details.
Chapter 2 covers publish-subscribe, and pipeline patterns, as well as some very brief coverage of resource management and using Valgrind to find leaks.
Chapter 3 covers the difference between standard sockets and ZeroMQ sockets, as well as covering the high(er) level CZMQ API
Chapter 4 covers advanced patterns, and techniques needed for real world scenarios (eg how to handle slow subscribers).

The Good:

  • Its cheap. At the moment the ebook is around 13 bucks on Packt’s site. Even if its terrible (and it does have some issues), for 13 bucks you might consider taking the plunge if you are new to ZeroMQ.
  • Covers the common messaging patterns that ZeroMQ implements. REQUEST-REPLY, PUBLISH-SUBSCRIBE, PIPELINE (aka PUSH-PULL)
  • Briefly covers the CZMQ API. This is one time where its good to be brief. The book covers just enough to show you that it exists, and start you off.
  • The diagrams are really good. They are similar to the diagrams in the zguide, but nicer. They are easy to understand, plentiful, and professional looking.

The Bad

  • Coverage is actually very brief. A lot of sections are overly brief and would benefit from a little more explanation and exploration.
  • The writing is very halting. It doesn’t flow very well. I’m not saying I could do much better, but its certainly noticeable while reading.
  • There’s sections that don’t make sense or are incomplete. For example, there’s a section on using Valgrind to detect memory leaks. The author shows a tiny C program (which doesnt use ZeroMQ) with a trivial leak and shows Valgrind’s output. Then the author shows a sample suppression file. But the author never explains why the suppression file is needed, or uses Valgrind on a ZeroMQ program. The section (like a lot of others) feels half written.

The Ugly

  • A lot of the code in this book is pretty bad quality. Almost eye-stabbingly bad. There’s misleading variable names, missing sections of code, sections of code that do nothing constructive, and even downright bugs. Luckily the code quality picks up a bit for the second half of the book and it’s still pretty clear what the author was actually intending. Its jarring, but these issues won’t be a large barrier to understanding.
    I sincerely hope these issues are fixed in a second edition.
  • The books site claims that it is example rich, but in reality the examples are very short. I would prefer meaningful examples that could occur in the real world over lots of “hello world” snippets. No big deal though.
  • There’s some minor mistakes. For example, in the PUB-SUB section, the book states that filtering is done on the subscriber side. In ZeroMQ, this changed in 3.x, so now for TCP and IPC endpoints, filtering is done on the publisher side. Completely understandable, because this changed between versions. But it should have been picked up in editing.

The lingering taste

Overall, the coverage is ok, but a little brief. The book feels half finished, and looks like it could have benefited from more time in editing. Its not a bad book though. Content-wise, it manages to cover everything a beginner needs to know.

A second edition (with more pages) should be able to fix these problems and turn this into a truly good book.

Who should buy this book?

This definitely isn’t a book for everyone. Its too brief for my tastes, and the quality issues are jarring. However there are some times when a short book is better than a long book, and when a comprehensive overview is actually overkill. For example, if you hire new programmers you will want something that will quickly introduce them to the concepts and techniques behind ZeroMQ. I think this could be a book that would quickly bring them up to speed.
Anyone looking for detailed coverage of ZeroMQ should skip it though.

What supplements/alternatives exist?

The most obvious supplement is the Z Guide. This is pretty much a must read if you are serious about developing with ZeroMQ.

For people that want larger books with comprehensive coverage, Pieter Hintjens (the author of the ZeroMQ library) has written Code Connected Volume 1. This covers the first 5 chapters of the ZGuide. There is a free ebook version.
There is also an O’Reilly book called ZeroMQ: Messaging for Many Applications, also by Pieter Hintjens.

For people wanting advanced tips and uses of ZeroMQ, I highly recommend Pieters blog. There’s a lot of really great stuff on there.

Python style decorators in Erlang

Alan Perlis said that “A language that doesnt change how you think about programming isnt worth knowing”.
My favourite languages are Erlang, Python and C++ (and Lua, but these days Python is taking more of that market share). I like them all for very different things, and they all have vastly different strengths,weaknesses and feature sets.
I also like thinking about how/if you can apply idioms that are common in one language to another language, and if theres any benefit in doing so. Today Im going to look at Python style decorators implemented in Erlang.

The example code is here

Decorators overview

Firstly: Python decorators are unrelated to the GoF Decorator pattern.
Decorators are syntactic sugar in python for applying a function to another.

def some_expensive_func(n):
   return some_expensive_computation(n)

is the same as

def some_expensive_func(n):
   return some_expensive_computation(n)
some_expensive_func = memoize(some_expensive_func)

This works because Pythons functions are first class values.

Decorators are a common idiom in Python and one that I really like.
I personally think that when used appropriately they make functions easier to read, by moving small details out of the main logic.

You can read more about the origin of the decorator syntax here.
You can see some of the neat things people do with decorators here.
Decorators are also very common in one of my favourite web frameworks; Django.

Decorators in Erlang

Following the goals laid down in the PEP I decided to set the following goals:

* Allow decorators to run code before and after a call to the function they are decorating. Eg to commit db transactions, cache values, trace or time execution etc.
* Allow decorators to alter the argument list that gets passed to the function they are decorating.
* Have a simple, non intrusive syntax. (Within the confines of erlang syntax rules.)
* Support multiple decorators on a single function
* Support passing custom arguments to the decorator.

Erlang does not support reassignment of variables, so no monkey business (monkey patching) here folks.
It also doesnt support executing code at module scope. Though on_load can often achieve similar results to module scope.
This complicates the implementation a little.

Id been intending to play with parse transforms for a while, and thought they could help.

The Plan

transform something like

-decorate( {decorator_module, decorator_function} ).
foo(N) -> N.


foo(N) -> foo_arity0_1([N]).
foo_arity0_1(Args) ->
   F = decorator_module:decorator_function( fun foo_arity0_0/1, Args),
foo_arity0_0([N]) -> foo_original___(N).
foo_original___(N) -> N.

This would let each decorator do whatever it wanted. It could call a completely different function or charge the arguments.

Our simple first test case

As a fan of TDD I wrote a very basic first test that I could use to start development

-compile([{parse_transform, decorators}]).
% example decorator that replaces the return value with the atom 'replaced'
	fun() -> 
		_R = apply(F, [Args] ),
-decorate({ ?MODULE, replace_return_value_decorator }).
replace_ret_val_decorated() -> ok.
	?assertEqual(replaced, replace_ret_val_decorated() ).

Getting the abstract forms

I did the simplest thing possible and printed the forms that were passed to the parse transform.

	io:format("~p~n=======~n",[ Ast ]),

We get the following info out.
You can read a bit about the format here

                        {string,20,"replace_ret_val_decorated ( )"}]},

Pretty Printing

I think the abstract forms are quite easy to follow, but it would be nicer to be able to see what we are producing as if it was real Erlang code.
Luckily erl_pp has this.
If you are writing your own parse transforms this will be very useful for debugging.

pretty_print(Ast) -> lists:flatten([erl_pp:form(N) || N<-Ast]).

for the untransformed code we see

-file("test/decorator_test.erl", 1).
-file("c:/PROGRA~2/ERL57~1.5/lib/eunit-2.1.5/include/eunit.hrl", 1).
-file("test/decorator_test.erl", 3).
replace_return_value_decorator(F, Args) ->
    fun() ->
           _R = apply(F, [Args]),
replace_ret_val_decorated() ->
replace_ret_value_test() ->
    fun(__X) ->
           case replace_ret_val_decorated() of
               __X ->
               __V ->
                                    "replace_ret_val_decorated ( )"},
test() ->

The transform

We need to walk the forms collecting any decorators, and apply collected decorators to a function when we meet it.
For the application step I figured it would be easy to output a nested list for when 1 form (the original function) expands to many, and clean it up later.
We also need to remove the decorator attributes, as you really are only meant to have attributes before all functions.

	{ExtendedAst2, RogueDecorators} = lists:mapfoldl(fun transform_node/2, [], Ast),
	Ast2 = lists:flatten(lists:filter(fun(Node)-> Node =/= nil end, ExtendedAst2)),
% transforms module level nodes
% see
% outputs nil (to swallow the node), a single node, or a list of nodes.
% nil nodes are removed in a subsequent pass and the lists flattened
transform_node(Node={attribute, _Line, decorate, _Decorator}, DecoratorList) ->
	% keep a list of decorators but dont emit them in the code.
	% this is important as you arent meant to have attributes after functions in a module
	{nil, [Node|DecoratorList]};
transform_node(Node={function, _Line, _FuncName, _Arity, _Clauses}, []) ->
	% pass through decoratorless functions
	{Node, []};
transform_node(Node={function, _Line, _FuncName, _Arity, _Clauses}, DecoratorList) ->
	% apply decorators to this function and reset decorator list
	{apply_decorators(Node,DecoratorList), []};
transform_node(Node, DecoratorList) ->
	% some other form. (the only other valid forms are other attributes)
	{Node, DecoratorList}.
apply_decorators(Node={function, Line, FuncName, Arity, _Clauses}, DecoratorList) when length(DecoratorList) > 0 ->
		% output the original function renamed
		% output a trampoline into our decorator chain
		function_form_trampoline(Line, FuncName, Arity, DecoratorList),
		% output the funname_arityn_0 function to unpack the arg list and forward to the original 
		% output our decorator chain
		| function_forms_decorator_chain(Line, FuncName, Arity, DecoratorList)

I wont go to much into the details of what each of these do, because frankly its pretty simple.
They just fill in various absform templates.. very basic code gen..

function_form_decorator_chain(Line,FuncName,Arity, {DecMod, DecFun}, DecoratorIndex) ->
	NextFuncName = generated_func_name({decorator_wrapper, FuncName, Arity, DecoratorIndex-1}),
	{function, Line, 
		generated_func_name({decorator_wrapper, FuncName,Arity, DecoratorIndex}), % name
		1, % arity
		[{ clause, Line,
			emit_arguments(Line, ['ArgList'] ),
			emit_guards(Line, []),
				% F = DecMod:DecFun( fun NextFun/1, ArgList),
				emit_decorated_fun(Line, 'F', {DecMod, DecFun},	NextFuncName, 'ArgList'),
				% call 'F'
				{call, Line,{var,Line,'F'},[]}

Sure theres some emit_* funcs in there to make it easier to read, but anyone that even glances as the absform docs and the absform for some sample functions could piece this together in short time.

I often find myself thinking “I cant believe how nice Erlang is”. Python too.

custom warnings and errors

Oh yeah, I also wanted to warn if there was any decorators at the end of the file that werent associated with a function.
To do that I just add warning nodes to the end of the file for each element in RogueDecorators.

	{ExtendedAst2, RogueDecorators} = lists:mapfoldl(fun transform_node/2, [], Ast),
	Ast2 = lists:flatten(lists:filter(fun(Node)-> Node =/= nil end, ExtendedAst2))
		++ emit_errors_for_rogue_decorators(RogueDecorators),
	[{error,{Line,erl_parse,["rogue decorator ", io_lib:format("~p",[D]) ]}} || {attribute, Line, decorate, D} <- DecoratorList].

We also need to do it when we hit the eof node. Otherwise the decorators would get applied to the function that eunit generates after the eof.
So I added the following clause to transform_node

transform_node(Node={eof,_Line}, DecoratorList) ->
	{[Node| emit_errors_for_rogue_decorators(DecoratorList) ], []};



to the end of the file makes it emit the following

Ender@PROSPERO /j/git/erlang_decorators
$ ./rebar compile eunit
==> erlang_decorators (compile)
==> erlang_decorators (eunit)
test/decorator_test.erl:22: rogue decorator {f,f}

the result

The code is here.
It could be developed more and cleaned up a little.

Heres the code after transformation (ie pretty_print(Ast2))

-file("test/decorator_test.erl", 1).
-file("c:/PROGRA~2/ERL57~1.5/lib/eunit-2.1.5/include/eunit.hrl", 1).
-file("test/decorator_test.erl", 3).
replace_return_value_decorator(F, Args) ->
    fun() ->
           _R = apply(F, [Args]),
replace_ret_val_decorated_original___() ->
replace_ret_val_decorated() ->
replace_ret_val_decorated_arity0_0([]) ->
replace_ret_val_decorated_arity0_1(ArgList) ->
    F = decorator_test:replace_return_value_decorator(fun replace_ret_val_decora
replace_ret_value_test() ->
    fun(__X) ->
           case replace_ret_val_decorated() of
               __X ->
               __V ->
                                    "replace_ret_val_decorated ( )"},
test() ->

As we can see it did the transform that we planned.

and the result

$ ./rebar compile eunit
==> erlang_decorators (compile)
Compiled src/decorators.erl
Compiled src/decorators_app.erl
Compiled src/decorators_sup.erl
==> erlang_decorators (eunit)
Compiled src/decorators.erl
Compiled src/decorators_app.erl
Compiled src/decorators_sup.erl
Compiled test/decorator_test.erl
  All 4 tests passed.

Thats enough for now.

Is it useful?

Its just syntactic sugar (The same is true in python).
If you are wondering where decorators are useful take a look at django.

But It was interesting to play with parse transforms, and it does provide an interesting syntax option.
I personally like this form of decorator, but beauty (especially in programming languages) is in the eye of the beholder.
I feel that the things that decorators *feel right* for is things that are a bit of a cross cutting concern, so its nicer to not need to pollute the code in the function.

After this Im thinking of doing a more general purpose attribute system.
I think it could be neat to eg automatically expose certain functions as RPC methods based on this.
Pretty atypical Erlang, but its interesting to see how you can apply constructs and idioms from other programming communities to achieve goals and save time.

Limitations and future extensions

I didnt end up implementing passing arguments to the decorator, but its a relatively simple extension.

Erlang is a functional language, with single assignment. Unless you use something like meck, you cannot reassign a function.
So these decorators don’t execute any code when the module is loaded, so we cant use this to eg track test coverage of certain annotated functions.
We could extend this in future to support generating on_loaded function, or to allow us to query a module for its absforms at runtime without requiring debug_info compiler option. Both of these approaches would go someway to addressing this limitation.

We are still bound by Erlang syntax rules. So eg you cant put decorations on individual clauses

bar(1) -> ok;
bar(N) -> bar(N-1).

If I was serious about using this Id also like to extend it to make it easier/prettier to define custom decorators in a header, which could use a simplified syntax. And of course to support passing extra information to the decorator.
something like

-log_entry_exit( [{logger, logger_proc }] ).
foo() -> 1.

process table size in erlang

Whats the cost of increasing the process limit?

I was curious:

Ender@PROSPERO /j/git/erlang_decorators
$ erl +P 32768
Eshell V5.7.5  (abort with ^G)
1> io:format("~p",[erlang:memory()]).
2> init:stop().
Ender@PROSPERO /j/git/erlang_decorators
$ erl +P 10000000
Eshell V5.7.5  (abort with ^G)
1> io:format("~p",[erlang:memory()]).
2> init:stop().

So (on my 32 bit vm on windows), you pay 4 bytes per process whether you use it or not.
Pretty compact. Feels like its probably implemented as an array of pointers. Yep

People always seem to run into the default limit. Its quite low.
I wonder if any large Erlang project has ever been done without increasing the limit.

Erlang R14B03 out

See the readme here

Things Im excited about:

  • lots of halfword vm fixes and improvements
  • Open ssl now staticly linked to crypto on Windows.
  • Majority option for mnesia tables.
    This is one convenient way to handle network splits. I have to give it a go with a large distributed+fragmented table

        OTP-9304  Add {majority, boolean()} per-table option.
    	      With {majority, true} set for a table, write transactions
    	      will abort if they cannot commit to a majority of the nodes
    	      that have a copy of the table. Currently, the implementation
    	      hooks into the prepare_commit, and forces an asymmetric
    	      transaction if the commit set affects any table with the
    	      majority flag set. In the commit itself, the transaction will
    	      abort if it cannot satisfy the majority requirement for all
    	      tables involved in the transaction.(Thanks to Ulf Wiger)
  • better cover support for tests
    OTP-9204  add user specified compiler options on form reloading
    	      In order to be able to test non-exported functions from
    	      another (test) module it is necessary to compile the specific
    	      module (at least during the test phase) with the export_all
    	      compiler option. This allows complete separation of testing
    	      and productive code. At the moment it is not possible to
    	      combine this with a test code coverage using the cover
    	      module. The problem is that when cover compiling a module
    	      using cover:compile_* the code is reloaded into the emulator
    	      omitting/filtering the passed user options. In my example
    	      above the export_all option would be removed and the
    	      non-exported functions cannot be called any more. (Thanks to
    	      Tobias Schlager)
  • Table viewer can now display small binaries!
    I tend to use binaries for strings so this bugged me on a number of occasions (obviously not enough for me to fix it myself).

        OTP-9153  tv: Allow table viewer to display refs, ports and small
    	      Table viewer displayed #Port, #Ref, or #Bin as place holders
    	      for their respective object types in ets and mnesia tables.
    	      This can make table viewer difficult to use when viewing
    	      tables containing those data types. It doesn't make sense to
    	      render large binaries so #Bin will still be used for binaries
    	      that exceed 100 bytes. (Thanks to Blaine whittle)
  • Fixed a display problem with the Event Tracer on Windows.
    OTP-9238  The popup window 'contents viewer' did not display properly on Windows.

Hiding IO latency in generators by async prefetching

Tricks with generators

Hiding IO latency by async prefetching


Its very common for interesting problems to not fit into memory.
In cases like this being able to process a subset of the problem at once is necessary to solving the problem.
Streams present a nice way of processing a limited subset. They let you program as if you were working with the full list, but only work with a single element in memory at a time. Alex recently wrote about streams.
You may already be used to implementing streams in this fashion with generators (python), iterators(C++), coroutines (Lua).
Ill use the term “Generator”, because ill be using Python.

If the time to produce/fetch an element is high, you can easily find yourself alternating between working with an element, and fetching an element.
Often fetching an element is a high latency operation, so your program ends up waiting on IO a lot of the time.
There are a number of ways of addressing this issue. Some that spring to mind are

  1. For certain tasks, use a library that supports non blocking operation. eg eventlet
  2. Explicitely parallelize the processing code (actors, worker thread pools, multiprocessing.Pool)
  3. Fetch elements in the background while processing the current element.

All 3 techniques are suited to specific situations.
I will discuss the third one in this article.

Its possible to wrap any generator such that it prefetches from the source generater in the background.
This has the benefit of overlapping the fetching of elements with normal execution.
This works best when the cost of processing the item is high, and the cost of fetching is not quite as high.
This will ensure that there is always an element ready for us when we finish processing one.

However this will not overlap fetching of individual elements.
It cannot do this with generators in the general case as they are stateful.
So this technique is not as helpful when the time it takes to process an element is tiny compared to the time to fetch.
For example, Ive seen FQL queries take over 5 seconds.

The Code

import functools
import Queue
import threading
def async_prefetch_wrapper(iterable, buffer=10):
	wraps an iterater such that it produces items in the background
	uses a bounded queue to limit memory consumption
	done = object()
	def worker(q,it):
		for item in it:
	# launch a thread to fetch the items in the background
	queue = Queue.Queue(buffer)
	it = iter(iterable)
	thread = threading.Thread(target=worker, args=(queue, it))
	thread.daemon = True
	# pull the items of the queue as requested
	while True:
		item = queue.get()
		if item == done:
			yield item
def async_prefetch(func):
	decorator to make generator functions fetch items in the background
	def wrapper(*args, **kwds):
		return async_prefetch_wrapper( func(*args, **kwds) )
	return wrapper

Pretty straight forward.


So does it work?

First Ill do a simple test that creates a bunch of files and md5s them
heres the code

def test_setup():
	files = []
	lines = 1000000
	for i in xrange(100):
		filename = "tempfile%d.txt"%i
		with open(filename, "w") as f:
			f.write( ("%d\n"%i)*lines )
	return files
def test_cleanup():
	for f in glob("tempfile*.txt"):
def contents(iterable):
	for filename in iterable:
		with open(filename, "rb") as f:
			contents =
		yield contents
def test():
	files = test_setup()
	for c in contents(files):
from timeit import Timer
t = Timer("test()", "from __main__ import test; gc.enable()")
print t.repeat(5, 1)

To use the async prefetch we change the contents generator to the following:

def contents(iterable):
	for filename in iterable:
		with open(filename, "rb") as f:
			contents =
		yield contents

Here are the results

without async prefetch 11.282730626378491 5.430316841944997 3.947590615567062 4.129772568860009 4.102539568576454
with async prefetch 6.155376451476375 3.790340392424177 3.384881807604039 3.436732252283459 3.415469144821479
without async prefetch and no md5 6.315582636899382 3.342140062493976 3.197938983865267 3.102182470118409 3.2230784219782134

Not a spectacular improvement, but not bad.
The best speedup we can hope for is reducing the total runtime by N*min(fetch time, work time).
You can see from the 3rd row that the total runtime is dominated by IO still. Ie this is a case with tiny workload vs slow IO.

Lets try it by replacing md5 with zlib compression.

without async prefetch. zlib compress 12.757559200959898 12.518354357886267 16.015608687696343 12.331753337365505 12.05284226067839
with async prefetch. zlib compress 10.578236130569667 9.156545245275586 12.359309772610764 9.072958714026505 8.881738391331858

not bad.

So this technique can give you a speedup when you have producer style generators that take time to produce elements.


google test with static libraries in msvc

Using google test with tests in static libraries under msvc has historically been a pain.
Often you would be left scratching your head wondering why some tests didnt run. This entry presents a tool that ensures that all tests from your static library will get run by your test runner.

Disclaimer: I don’t actually recommend this approach.
You will have cleaner interfaces if your tests are written in your test runner program, which then links against your production code library.
Plus it wont be possible for test cases to sneak into the production app.
This was purely done as an exercise, but it may be of use to some people.

If you have ever tried to use gtest to test code in static libs you have probably run into this problem.
See the gtest wiki for an overview of the problem.
The wiki ends up suggesting that you don’t put tests in static libraries.

Generally id agree with that.
But if you have a lot of code that doesn’t have external linkage then testing that code can only be done indirectly.
Sure, this may be a sign that either
* it hasnt been designed for testability
* the author hasent followed TDD
* havent broken the module down enough. (SRP)

But I still found myself wondering “why cant I put a test case in a static library?”.
Clearly the fact that its on gtests wiki means that other people wonder too.

Its common for people to put eunit tests at the bottom of their erlang modules and similar practices exist in other languages. They arent best practices. But if its so common in other languages why does it suck so much in msvc?

A walk through of the problem

The common idiom for using gtest (and other test frameworks) is to have a library where you put your code, a minimal program that uses that code, and a program that acts as a test runner. This is what im going to do in this article.

For this example we would be writing our tests in our static library.
(Perhaps we are testing a piece of code with internal linkage.)

// internal linkage function we are testing
static int plus(int a, int b) { return a+b; }
// test case
TEST(MathTest, TwoPlusTwoEqualsFour) {
	EXPECT_EQ(plus(2,2), 4);

Compile it. Everythings fine and we get our .lib file.

Then we have our test runner. It links against our lib and gtest

#include <gtest/gtest.h>
int main(int argc, char **argv) {
	::testing::InitGoogleTest(&argc, argv);
	return RUN_ALL_TESTS();

Time to run it and see our test passing.
But wait… we see this instead

What happened?

The msvc linker does not link an obj file from a static library into the main program unless there is an unresolved symbol in the main program that resolves to that obj.
This is fair enough. You dont want code in your exe that you dont need.
But heres the kicker: static initialization code that exists in an obj wont cause it to be linked in either.
gtest works by constructing classes that are initialized and linked into the test framework during static initialization.
GCC has a –whole-archive option, which can be used to link in everything, but msvc doesnt have anything like this.

Our test runner doesnt refer to ANY code in the static library, so our test doesnt get linked in.

The solution

heh. solution. get it? we are talking about visual studios… oh nevermind…

The solution is
You can find it here.

If you have a static library with tests in it you can generate a header file which can be included in your test runner that will force the tests to run.

run like so:

python mylib.lib generated_test_syms.h

then have your test runner include the header file

#include <gtest/gtest.h>
#include <generated_test_syms.h>
int main(int argc, char **argv) {
	::testing::InitGoogleTest(&argc, argv);
	return RUN_ALL_TESTS();

compile and run and:

It works!

How it works

This python script runs dumpbin against the lib, it then runs a regex over the output to pull out symbols that match the constructors for gtests generated classes.
Once we have that its easy to emit a header file that forces the linker to include a reference to that symbol.

here is the header we generated above

#ifndef generated_5312bcde_fd17_4e3b_bbca_99f20116304a
#define generated_5312bcde_fd17_4e3b_bbca_99f20116304a
// Generated by gen_msvc_test_header at 2011-02-10T04:22:47.397000
// do not modify 
#pragma comment(linker, "/include:??0MathTest_TwoPlusTwoEqualsFour_Test@@QAE@XZ")
#endif // generated_5312bcde_fd17_4e3b_bbca_99f20116304a

If you were going to actually use this I suggest you set it as a post build step for your static lib.

This approach can also be used for other cases where you rely on static initializers being run, but you are using static libraries.

ZeroMQ + The Active Object Pattern

I almost missed todays blog post. I was in bed when I remembered.

Today we will implement the Active Object pattern to manage a ZeroMQ Pub socket.
This will allow us to easily send on the socket without worrying about manual thread synchronization.
Note: current versions of ZeroMQ require that only the thread that created the socket interact with the socket.
I will not cover message encoding. I recommend protocol buffers or BERT or JSON, depending on the requirements. Avro and Thrift also look interesting. Especially Avro (I personally think versioning should be part of the handshake)

First a bit of background on the mini project this is for.
Its market data streamer. It talks to an upstream FIX server, subscribes to market data for certain instruments, and encodes and forwards any received market data over ZeroMQ. So basically a FIX to ZeroMQ proxy, but just for market data.
A program does not exist in a vacuum, so as part of this example I have also done an upstream FIX server for “simulating” price ticks.
And Ive done a Python zmq subscriber. This just prints the ticks as the come in. Maybe ill make it do a pretty graph, but thats not important at the moment.

Im not doing this for work. Im between jobs at the moment and enjoying the break.
Im doing this to become more familiar with FIX, and the quirks of QuickFix and ZeroMQ.
Plus its nice to do a small project to keep the mind ticking (heh. ticking. get it?).

Now for some background on ZeroMQ.
ZeroMQ is a lot of things. The usual saying is its sockets on steroids.
One of the nice things it does is provides a uniform interface for various transport types such as TCP, interprocess communication, and pgm/epgm multicast. It also has some nice/common built in messaging patterns (like all MQs). The interesting pattern here is Pub/Sub.
Unlike all MQs, ZeroMQ is brokerless. This is occasionally a bit of a shift in thinking.
ZeroMQ also boasts extremely low latency, probably due to brokerless nature. Financial guys tend to froth at the mouth about latency.

Zed Shaw’s mongrel2 uses zeromq. Mongrel2 looks really interesting.
I havent really been looking that closely at it but I have to admit I got interested when he talked about Tir.
Im happy using django (python) or webmachine (erlang) for my web dev, but I know a lot of people that will be interested in Tir.
At my old job we had *quite* a few Lua developers.
It should be interesting to see how people cope with the Lua GC. IMO Tir will suffer from issues as node.js apparently does. This will effect the types of applications its usable on.

One of the things about QuickFix is that its very much a threaded program. Each session runs in its own thread.
One of the things about zeromq is that (currently) you can only interact with a socket from the thread that created it, regardless of locking.
As I currently only have 1 session in the proxy I could have just created the socket in the Application::onCreate( const FIX::SessionID& ) override. However there is nothing in quickfix or my program that stops a user specifying multiple initiator session in their config file.
Say if they wanted to subscribe to market data from the ASX’s Market Point service, as well as subscribing to data from HKEX using the one proxy instance.
If I went with the above approach that would not be possible to use the same endpoint for more than one FIX session.
So I needed a thread that owned the zmq socket that each session thread communicated with somehow.
And having done threading in the past I want to avoid having manual locks over the place as much as possible.

This is a perfect case for the Active Object pattern.
A nice side effect from this is that it makes it clearer what parts of the code are responsible for what functionality, makes it more modular, and eases implementation. Message Passing baby. Aww yeah.

While TBB and boost both provide some C++0X compatible threading libraries, neither provides an active object class, and only TBB provides a concurrent queue.
Yeah I know. Imagine boost not having something as useful as that. They have everything else.
I prefer using boost for C++0x style threads, so we will need to implement our own message queue and active object classes.

Firstly we need a message queue

// multiple writer, multiple consumer
// based on Anthony Williams implementation (with added support for bounded size)
// Anthony Williams is the current maintainer of boost::thread
template<typename T>
class concurrent_queue {
	concurrent_queue():max_elements(0) {}
	explicit concurrent_queue(size_t max):max_elements(max) {}
	// pushes an entry onto the queue.
	// if the queue is at maximum, the current thread waits
	// this helps us avoid producers outpacing the consumer(s) and causing OOM
	void push(const T& v) {
		boost::mutex::scoped_lock l(m_mutex);
		while(max_elements!=0 && m_queue.size() >= max_elements )
	// pops an element off the queue and returns it
	// if there are no elements in the queue the current thread waits
	void pop(T& v) {
		boost::mutex::scoped_lock l(m_mutex);
		// we cant return by value and maintain strong exception safety because copy ctors can throw
		// if it throws on the return we would have already done the pop. 
		// see
		v = m_queue.front();
	// no guarantee that this is accurate as soon as its returned.
	// but may be useful for diagnostics
	bool empty() const {
		boost::mutex::scoped_lock l(m_mutex);
		return m_queue.empty();
	// no guarantee that this is accurate as soon as its returned.
	// but may be useful for diagnostics
	size_t size() const{
		boost::mutex::scoped_lock l(m_mutex);
		return m_queue.size();
	size_t max_size () const {
		boost::mutex::scoped_lock l(m_mutex);
		return max_elements;
	mutable boost::mutex m_mutex;
	std::queue<T> m_queue;
	size_t max_elements;
	boost::condition_variable m_cond;

We can create a helper to ease implementing active objects.

// helper for the Active Object pattern
// see Sutters article at
class active_object_helper {
	active_object_helper():m_exit(false) {
		m_thread.reset( new boost::thread( boost::bind(&active_object_helper::run, this) ) );
		send( boost::bind(&active_object_helper::exit, this) );
		// wait for queue to drain and thread to exit
	void send(const boost::function0<void>& f) {m_queue.push(f);}
	// gets run on the launched thread
	void run(){
		boost::function0<void> f;
		while (true){
			if (m_exit)
	// a message we use to exit the thread
	void exit() { m_exit = true; }
	concurrent_queue< boost::function0<void> > m_queue;
	boost::scoped_ptr<boost::thread> m_thread;
	bool m_exit;

Now we have our utility classes out of the way, onto the publisher implementation.
Our tick_publisher becomes

class tick_publisher {
   virtual void tick(const MarketData& md) = 0;
   virtual ~tick_publisher() {}
class zmq_tick_publisher: public tick_publisher {
	zmq_tick_publisher(zmq::context_t& ctx, const std::string& bind_address) {
		m_active_object.send( boost::bind(&zmq_tick_publisher::init, this, boost::ref(ctx), bind_address) );
	virtual ~zmq_tick_publisher(){
		m_active_object.send( boost::bind(&zmq_tick_publisher::deinit, this) );
	virtual void tick(const MarketData& md) {
		m_active_object.send( boost::bind(&zmq_tick_publisher::tick_, this, md) );
	void init(zmq::context_t& ctx, const std::string& bind_address) {
		// setup socket
		m_socket = new zmq::socket_t(ctx, ZMQ_PUB);
	void deinit(){
		// teardown socket
		delete m_socket;
	void tick_(const MarketData& md){
		// encode and broadcast on socket
		zmq::message_t msg;
		encode(md, msg);
		bool success = m_socket->send(msg);
	active_object_helper m_active_object;
	zmq::socket_t* m_socket;

and using it is as simple as creating it and just calling methods on it. easy.

main() {
 // ...
 std::string bind_address = settings.get().getString("BindAddress"); // eg "tcp://*:5000"
 boost::shared_ptr<tick_publisher> tick_pub(new zmq_tick_publisher(zmq_ctx, bind_address) );
 MQFeederApplication(settings, tick_pub);
 // ...
MQFeederApplication::MQFeederApplication(const FIX::SessionSettings& s, boost::shared_ptr<tick_publisher> publisher) 
	:m_settings(s), m_publisher(publisher)
void MQFeederApplication::onMessage( const FIX44::MarketDataSnapshotFullRefresh& m, const FIX::SessionID& sessionID)
	FIX::Symbol s = FIELD_GET_REF(m, Symbol);
	MarketData md(s);
	// fill in market data
	// ==snip==
	// publish

Feels a bit like a gen_server, except in this example we only use the equivalent of gen_server:cast. To implement active objects returning results we can either block the caller thread (gen_server:call style) or return a boost::unique_future. But thats for another day.

Some awesome things about this:

  • No locking in user code. all the locking is in the message_queue. And it doesnt call any unknown code while it holds the lock. Deadlocks are impossible to accidently introduce.
  • The caller of tick() doesnt even need to know about threading. We can call it from any thread without messing up our resources.
  • The zmq socket is completely managed by the active objects thread. No awkward resource sharing. just very simple message passing.

trying to write more

Ive been meaning to write more. For a long time really. The problem is I really don’t like writing. Id rather be programming, playing xbox, or drinking beer. Sometimes all 3 at once.

Looking at the behavior grid Im trying to achieve a Green Path behavior change.
So here are some things Im going to try:

  • Boost motivation.
  • Couple the trigger to an existing behavior.
  • Reduce demotiviation by making the behavior more familiar.

Recently my brother proposed that a few of his blogger friends start a Write Club, in order to encourage more regular blogging. Basically (if i understood it correctly) every week you need to write a blog post on a certain day. You get points based on how close to the target day you wrote your post.
I guess the theory is that there will be social pressure acting to increase motivation (leaderboards!).
Plus the structured nature should help make it more familiar.
I think its a neat idea. Im definitely in.
I think itll probably need some kind of “slow car catchup” so that if someone does fall behind by missing a week that they aren’t permanently behind in the score board. Otherwise the competitive pressure of a score system would lose some impact (in my opinion).

My plan is to write short blog post every day for a week or 2, to get more used to blogging.
To boost motivation Im going to try keeping a Seinfeld Calendar. I found a nice pdf here. Alex said I should take it to OfficeWorks and get it printed up massive. I might do that.

Now I just need to work out how to trigger the behavior when motivation and opportunity is there.
An obvious technique is to couple it to an existing behavior. I dont know what though.

quickfix logon support for username password

QuickFix is an open source FIX engine.
Ive been doing a little simulator along the lines of this (but using zeromq+quickfix) to become familiar with FIX.

The ASX supports the Username and Password fields for Logon messages. Quickfix normally takes care of Logon for you, but does not support Username and Password as session settings. Im only interested in FIX 4.4 (as thats the highest the ASX supports), so ill restrict my code to dealing with that.

Because we are modifying outgoing Logon messages, normally we would use a message cracker from our Application::toAdmin() override.
But there is a bug in quickfix (1.13.3) that means that cracking non const message references calls the wrong code. It ends up calling the onMessage for the message const ref.
ie it calls

void MQFeederApplication::onMessage( const FIX44::Logon& m, const FIX::SessionID& sessionID)

instead of

void MQFeederApplication::onMessage( FIX44::Logon& m, const FIX::SessionID& sessionID)

Rather than showing how to fix that bug and the code you will use ill just code the message type check right into toAdmin(). If you were doing a real program youd fix the bug and do it properly.

Firstly the obvious. We need to inject the settings class into our application so that we can pull the username and password from the config file.

class MQFeederApplication
	: public FIX::Application,
	private FIX::MessageCracker
	MQFeederApplication(const FIX::SessionSettings&);
// ...
	const FIX::SessionSettings& settings;
// ...

and pass it to our Applications ctor:

int main(){
		FIX::SessionSettings settings( "../mqfeed.cfg" );
		MQFeederApplication app(settings);

Then in toAdmin we check the message type and pull the appropriate username and password from the settings

void MQFeederApplication::toAdmin( FIX::Message& message, const FIX::SessionID& sessionID)
	// when we send Logon make sure we add Username and Password if we have it configured for this session
	// I would have like to use a message cracker for this, but theres a bug in quickfix 1.13.3
	// wrt cracking non const messages
	const FIX::Dictionary& session_settings = settings.get(sessionID);
	if (FIX::MsgType_Logon == FIELD_GET_REF( message.getHeader(), MsgType)) {
		FIX44::Logon& logon_message = dynamic_cast<FIX44::Logon&>(message);
		if (session_settings.has("Username")) {
			FIX::Username username = session_settings.getString("Username");
			logon_message.set( username );
		if (session_settings.has("Password")) {
			FIX::Password password = session_settings.getString("Password");
			logon_message.set( password );

So far working with quickfix has been pretty easy.
I was initially surprised by the use of little class wrappers for each field type, but it means the compiler can check that you arent trying to set fields that arent supported.

		FIX44::Logon& logon_message = dynamic_cast<FIX44::Logon&>(message);

will result in a compile time error. neat.
If you do want to add custom fields you can either use Message::setField, or regenerate the code from a modified data dictionary.

C++ tricks: lazy mans operator<

Disclaimer: This may seem less interesting when its not 2am.

So Im writing this very simple class representing market data subscriptions.
Its really just an aggregate of some data.
Session id, symbol to subscribe to, and subscription request id.

It basically looks a little bit like

class Subscription {
      // ...
      FIX::SessionID session;
      std::string symbol;
      std::string req_id;

At some point I wanted to put these in an STL set.
This means I need a function that is a model of Strict Weak Ordering

All the members already have operator< but its still tedious to write a conforming func when theres more than a few members. Typically you end up with things like

inline bool lt_compare(const Subscription& a, const Subscription& b) {
   return a.get_session() < b.get_session()
      || a.get_session() == b.get_session() && a.get_symbol() < b.get_symbol()
      || a.get_session() == b.get_session() && a.get_symbol() == b.get_symbol() && a.get_req_id() < b.get_req_id();

Which isnt pretty and is easy to make a typo in. Adding more members just makes it uglier.

So I started thinking how Id do it in Python and came up with something like

def lt_compare(a,b):
   return (a.session,a.symbol,a.req_id) < (b.session,b.symbol,b.req_id)

This works because python’s tuples support __lt__ in the obvious manner.

I also recalled that std::pair provides operator<. Seeing as boost::tuple is an extension of std::pair I realized I could leverage that.

inline bool lt_compare (const Subscription& s1, const Subscription& s2) {
	return boost::make_tuple(s1.get_session_id(), s1.get_symbol(), s1.get_req_id())
		< boost::make_tuple(s2.get_session_id(), s2.get_symbol(), s2.get_req_id());

Quite close to the python version. IMO less prone to typos than the original C++ version.

Now for the problem: It does copy the members when it makes the tuple. This means its still not suitable for classes with heavy copy constructors. Like most versions of string.
We can fix that by wrapping all the arguments in boost::cref.

inline bool lt_compare (const Subscription& s1, const Subscription& s2) {
	return boost::make_tuple( boost::cref(s1.get_session_id()), boost::cref(s1.get_symbol()), boost::cref(s1.get_req_id()))
		< boost::make_tuple( boost::cref(s2.get_session_id()), boost::cref(s2.get_symbol()), boost::cref(s2.get_req_id()));

but it starts to look ugly again.
In the end I didnt use it anyway. I didnt need to store them in a set after all.

It should be pretty easy to write a template or macro thats even easier to use.
For 1 mins work i thought it was a neat trick.