Friday, 7 February 2014

Twisted Names – now with a (private) EDNS(0) message parser

This is the first in a series of articles describing my progress towards introducing EDNS(0) and DNSSEC support to Twisted Names.

In this article, I'll describe two new private classes designed to parse EDNS(0) messages.

But first a little background.

The EDNS protocol extensions depend on the use of a special DNS record which crucially contains the advertised maximum datagram size, extended return codes, a DNSSECOK flag, and a payload which contains zero or more extensible headers.

This special record is called an OPT record. It isn't a real record though; it doesn't convey any domain names or domain records. It's a hack to work around the lack of space in a standard 512 byte DNS message. All the header flags and fields that couldn't be shoehorned into the 96bit header section of a DNS message are instead encoded into an OPT "pseudo" record.

EDNS clients and servers append one OPT record to the "additional" records in a sent DNS message. This record is then extracted and interpreted by an EDNS aware receiver.

Twisted already has a class for parsing DNS records and it already has a class for parsing DNS messages. I've now created some wrappers which add the EDNS special behaviour to those existing APIs.

The first wrapper is a class called twisted.names.dns._OPTHeader, which wraps an instance of twisted.names.dns.RRHeader. This allows us to re-use the existing record parsing code, and means that the code in _OPTHeader is purely concerned with the special behaviour of an OPT record. Which seems neat! It also made the unit tests much more focused.

The second wrapper is a class called twisted.names.dns._EDNSMessage which is in trunk but not yet in a released version. This wraps an instance of twisted.names.dns.Message. And again, this allows us to re-use the existing parsing code while clearly separating the EDNS specific behaviour.

So Twisted Names can now interpret OPT records and EDNS messages, but we've decided to keep these private for the time being; until we are satisfied with the APIs.

This first version of _EDNSMessage has been written as a drop in replacement for dns.Message. But as a result it has inherited some of the ugliness of that class, namely:

Tom has some ideas about introducing a new Interface for message constructors and I've done some work on improving the abbreviated names, but I got stuck trying to figure out how to introduce the changes while maintaining backwards compatibility.

I'd originally thought those things were blockers, because I intended that users would pass curried message constructors / factories to the various protocols and client APIs, as a way of configuring / overriding EDNS flags and fields in their requests. But that's not a good API (as Tom points out in another comment).

So I've decided that those cleanups would be nice to have, but probably not necessary right now.

A much simpler strategy – and my current priority – is to just switch all Twisted names code (client, protocols, server) to use the new dns._EDNSMessage class in place of dns.Message. This shouldn't introduce any backwards incompatibilities and I'll explicitly disable the use EDNS during encoding (for now). That way Twisted Names clients and servers should continue to behave exactly as before.

But I'll then be free to start work on a DNS client which can be configured to send EDNS(0) requests and which is capable of negotiating with EDNS(0) servers. That's the key requirement of the first project milestone.

(twisted.names.server and the twistd dns plugin will remain non-EDNS for the timebeing.)

In fact I've already started that work, but first I had to fix a bug in the way twisted.names.server generates response messages. My proposed fix includes new functions and methods for generating response messages from a request message. (Previously, the request message was being re-used as the basis for the response, and inevitably, client specific flags and fields were leaking into the response.)

Happily, these new APIs will also allow me to explicitly disable EDNS(0) in twisted.names.server, until such time as we implement server side EDNS(0) support.

So that's all for now. In my next post I hope to be able to report that Twisted Names is using the new _EDNSMessage class internally and summarise some of the other work that I've done on Twisted Names in support of the EDNS(0) and DNSSEC project.

Thanks once again to The NLnet Foundation for funding this project.

Monday, 3 February 2014

The Bristol Twisted Sprint – a belated thankyou

I'd like to thank Luke, Rob and everyone at HybridCluster for organising the Bristol Twisted Sprint in December last year and for their hospitality on the day.

I got quite a bit of work done and it was a rare opportunity to meet Tom and JP, who had flown in from the Americas, and Hynek who had flown in from Berlin.

Earlier that week I'd been amazed to receive an email from Tom with code-reviews of three of my EDNS branches. He'd done them during his flight from Canada! Not sure I'd have had the stamina, given the lack of elbow room and the temptation of in-flight entertainment and complimentary booze. But Tom's a pro and it meant that we had lots to discuss on the day of the sprint. Tom had raised some concerns about my proposed mechanism for setting EDNS parameters from the high level twisted.names.client API – with which I agree. But I'm still struggling to implement the alternative mechanism that we discussed on the day. More on that in a later blog post.

It was also nice to see Hynek, who showed me an interesting custom DNS server which he'd written despite the lack of documentation and despite having to subclass and override the private _lookup method of twisted.names.authority. He laid it on pretty thick, and guilt–tripped me into writing some decent documentation for twisted.names.server. Conscience is what motivates me, so I'm pleased to report that we now have a little more narrative documentation for Twisted Names. And it looks pretty nice too, thanks to khorn's and glyph's work on porting the Twisted documentation to ReST and Sphinx.

JP adopted a "headmasterly" role; supervising things from a lectern at the front of the room (an improvised standing desk), and intimidating us (the pupils) with astonishing Typespeed scores between builds – the keyboard noise was deafening! But I'm exaggerating, JP was as helpful and as thoughtful as usual. Switching context effortlessly to assist us with all areas of Twisted — from development environment tips, to Twisted OpenSSL integration, to EDNS, to PyPy tests and benchmarking, to Tun/Tap APIs. Very impressive!

Let's hope there are more UK Twisted sprints in 2014.

Saturday, 1 February 2014

Twisted Names EDNS — a progress update (or lack thereof)

Apologies to those of you who have been eagerly awaiting news of EDNS and DNSSEC support in Twisted Names. I haven't really got a good excuse; just that I'm a bad communicator and I've been distracted by some mundane (and some exciting) non-computer activities. Sorry.

So, a progress report is long overdue and progress has been a little slower than expected, but let me try and compensate by promising a series of short updates and demos which I'll publish over the next few weeks, demonstrating what has been achieved so far.

Stay tuned!

Tuesday, 26 November 2013

EDNS(0) and DNSSEC Coming Soon to Twisted

I'm happy to announce that I've won some funding from The NLnet Foundation - DNS Security Fund, to add EDNS(0) and DNSSEC (and possibly DANE) support to Twisted.

The current project plan contains the list of milestones and tickets.

If there are any DNS enthusiasts / experts reading this, I'd appreciate your feedback; on the plan and on the implementation (as it evolves). Please get in touch.

Tom Prince has kindly agreed to do some "express" code and design reviews for me. That'll be a great help given the current length of the review queue.

But the more eyes on the code the better, so please consider helping out with code reviews if you can. (I'm happy to trade reviews if you've got your own branch / patch waiting to be reviewed.)

I've been given a head start in this project by the patches contributed by Bob Novas and Phil Mayers, so thank you both. I hope you'll be able monitor what I'm doing and steer me in the right direction.

Thanks also to Itamar who encouraged me to apply for the funding and to Tom Prince and everyone who helped me draft the proposal.

I'll be working on this at the Twisted Sprint in Bristol, UK on December 7th; where I'll be delighted to discuss the project and demonstrate what I've been up to.

Hope to see you there!

Tuesday, 26 July 2011


  • Start bind-9.7.3-P1
$ bind-9.7.3-P1/bin/named/named  -g -c named.conf -4
26-Jul-2011 00:19:17.201 starting BIND 9.7.3-P1 -g -c named.conf -4
26-Jul-2011 00:19:17.212 listening on IPv4 interface lo,
  • Dynamic DNS delete message with update section type "ANY"
$ sudo scapy
[sudo] password for richard:
Welcome to Scapy (2.1.0)
>>> conf.L3socket=L3RawSocket
>>> sr1(IP(dst="")/UDP(dport=10053)/DNS(opcode=5, qd=[DNSQR(qname="", qtype="SOA")], ns=[DNSRR(rrname="", type="ANY", rclass="ANY", rdata="", ttl=0)]), verbose=1, timeout=5)
Begin emission:
Finished to send 1 packets.

Received 8 packets, got 0 answers, remaining 1 packets
  •  Oh dear :(
26-Jul-2011 00:24:42.740 ncache.c:343: INSIST(remaining.length >= 5) failed, back trace
26-Jul-2011 00:24:42.740 #0 0x805ac28 in assertion_failed()+0x48
26-Jul-2011 00:24:42.740 #1 0x81bfd37 in isc_assertion_failed()+0x27
26-Jul-2011 00:24:42.740 #2 0x80e093e in dns_ncache_towire()+0x57e
26-Jul-2011 00:24:42.740 #3 0x813f05f in towiresorted.clone.0()+0xcf
26-Jul-2011 00:24:42.740 #4 0x80d6b76 in dns_message_rendersection()+0x1a6
26-Jul-2011 00:24:42.740 #5 0x804ffcc in ns_client_send()+0x2bc
26-Jul-2011 00:24:42.740 #6 0x8081b7a in respond()+0xaa
26-Jul-2011 00:24:42.740 #7 0x8086ef9 in ns_update_start()+0x79
26-Jul-2011 00:24:42.740 #8 0x8051c98 in client_request()+0x1118
26-Jul-2011 00:24:42.740 #9 0x81dd0f9 in isc__taskmgr_dispatch()+0x149
26-Jul-2011 00:24:42.740 #10 0x81dffc2 in evloop()+0x92
26-Jul-2011 00:24:42.740 #11 0x81e026f in isc__app_ctxrun()+0x7f
26-Jul-2011 00:24:42.740 #12 0x81e07f2 in isc__app_run()+0x12
26-Jul-2011 00:24:42.740 #13 0x805be62 in main()+0xd32
26-Jul-2011 00:24:42.740 #14 0x272e37 in ??
26-Jul-2011 00:24:42.740 #15 0x804bab1 in _start()+0x21
26-Jul-2011 00:24:42.740 exiting (due to assertion failure)
Aborted (core dumped)
Further reading: