Planet Ubuntu-Au

# nano /etc/udev/rules.d/51-android.rules

SUBSYSTEM=="usb", ATTR{idVendor}=="0502", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0B05", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="413C", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0489", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="091E", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="18D1", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="109B", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0BB4", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="12D1", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="24E3", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="2116", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0482", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="17EF", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="1004", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="22B8", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0409", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="2080", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0955", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="2257", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="10A9", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="1D4D", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0471", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="04DA", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="05C6", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="1F53", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="04E8", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="04DD", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0FCE", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="2340", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="0930", MODE="0666", GROUP="plugdev" SUBSYSTEM=="usb", ATTR{idVendor}=="19D2", MODE="0666", GROUP="plugdev"

# chmod +x /etc/udev/rules.d/51-android.rules

Reference:
Using Hardware Devices

“Steam”, “Ubuntu TV”, “Ubuntu Phone”, “Ubuntu Tablet”, “Ubuntu for Android”.

12 months ago these keywords were merely ideas, possibilities that were seemingly unclear. Skip forward to today and we have a totally different perspective. Steam is on Ubuntu. Developer previews have been released for the phone and tablet. Unity is mature on the desktop. Fast, functional, easy-to-use and visually appealing.

With the increased use of Google+ Hangouts, ala Ubuntu OnAir, I think it’s certainly worth trying to use the technology to host more frequent sprints and community-wide discussion.

Unfortunately, for the dubbed UDS-1303, the sessions are being held between 1am-7am local time. In effect, I’ll be able to participate (read: watch) the summaries and lightning talk sessions.

Hopefully the next UDS-(1305?) will be scattered along different time zones.

I’ve been hitting the limits of gigabit ethernet at home for quite a while now, and as I spend more time working with cloud technologies this started to frustrate me.

I’d heard of other folk getting good results with second hand Infiniband cards and decided to give it a go myself.

I bought two Voltaire dual-port Infiniband adapters – a 4X SDR PCI-E x4 card. And in a 2 metre 8470 cable, and we’re in business.

There are other, more comprehensive guides around to setting this up – e.g. http://davidhunt.ie/wp/?p=2291 or http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html

On ubuntu the hardware was autodetected; all I needed to do was:

modprobe ib_ipoib sudo apt-get install opensm # on one machine

And configure /etc/network/interfaces – e.g.:

iface ib1 inet static address 192.168.2.3 netmask 255.255.255.0 network 192.168.2.0 up echo connected >`find /sys -name mode | grep ib1` up echo 65520 >`find /sys -name mtu | grep ib1`

With no further tuning I was able to get 2Gbps doing linear file copies via Samba, which I suspect is rather pushing the limits of my circa 2007 home server – I’ll investigate futher to identify where the bottlenecks are, but the networking itself I suspect is ok – netperf got me 6.7Gbps in a trivial test.

I don’t like to post about politics much, because it’s usually all BS, but I unfortunately read this tweet by AJ while reading about a girl badly burned at soundwave:

How much debt we’ve been left in?

The problem with this is that it’s just propaganda and firmly not based in reality, here for AJ is the reality of the situation:

 

“Whether the government has a small surplus or a small deficit is not important for the rating.”

So the major economists and economic powerhouses of the world reckon our debt levels are ok, our unemployment is in an unbelievably fantastic situation compared to those hit hardest by the GFC, we’re slowly having our internets being upgraded to the fastest in the world, we’ve got our carbon being taxed in line with the rest of Europe and Asia, and our treasurer responded sensibly to dramatically lower tax revenue and has chosen not to blindly get a surplus by slashing public jobs and projects ala Campbell Newman, as if getting a surplus would suddenly make everything better, and yet Australians feel sad and even angry about our situation in the world? what?

I blame the liberal media.

 

StreamResult, covered in my last few blog posts, has panned out pretty well.

Until that is, that I sat down to do a serialised version of it. It became fairly clear that the wire protocol can be very simple – just one event type that has a bunch of optional fields – test ids, routing code, file data, mime-type etc. It is up to the recipient at the far end of a stream to derive semantic meaning, which means that encoding a lot of rules (such as a data packet can have either a test status or file data) into the wire protocol isn’t called for.

If the wire protocol doesn’t have those rules, Python parsers that convert a bytestream into StreamResult API calls will have to manually split packets that have both status() and file() data in them… this means it would be impossible to create many legitimate bytestreams via the normal StreamResult API.

That seems to be an unnecessary restriction, and thinking about it, having a very simple ‘here is an event about a test run’ API that carries any information we have and maps down a very simple wire protocol should be about as easy to work with as the current file or status API.

Most combinations of file+status parameters is trivially interpretable, but there is one that had no prior definition – a test_status with no test id specified. Files with no testid are easily considered as ‘global scope’ for their source, so perhaps test_status should be treated the same way? [Feedback in comments or email please]. For now I’m going to leave the meaning undefined and unconstrained.

So I’m preparing a change to my patchset for StreamResult to:

• Drop the file() method altogether.
• Add file_bytes, mime_type and eof parameters to status().
• Make the test_id and test_status parameters to status() optional.

This will make the API trivially serialisable (both to JSON or protobufs or whatever, or to the custom binary format I’m considering for subunit), and equally trivially parsable, which I think is a good thing.

One of the hardest things about putting together Angry Beanie isn't chasing interviews. Nor is it building the hosting infrastructure, or in fact resisting the urge to keep checking the downloads stats.

No, it's working out how to get more people to listen without appearing to be either:

• Desperate for validation
• Pushy 
• Or a complete smeg (Social Media Expert Guru)

By nature I'm not someone who does the self marketing thing well. I tend to shy away from anything that I think might be big noting myself too much, this applies both in my private life as well as professionaly. One of the reasons I decided to start podcasting in the first place was that it allowed me to combine a passion of mine (multimedia content creation) with something that would help me deal with one of my failings (dealing with people I don't know).

So for me, this marketing lark could be seen as the next step in "learning how to do things with people".

So with that in mind I've decided to set myself a couple of marketing goals.

Downloads per episode:

I would love to increase the number of people who listen to Angry Beanie content. Currently For Science is averaging around 200 - 250 downloads per episode (I'm only counting those downloads in the fortnight following the release of the ep) while Purser Explores The World is getting between 170 and 220.

These aren't bad numbers, but I would like to see more.

So I'm going to set myself the goal of an average of at least 500 downloads per episode period.

Community

No content producer survives without a community behind them. Angry Beanie is no different. I want to build up that community, get people commenting on the stuff we make, both negative and positive (so long as it's constructive) and so on. 

I have no idea how I'm going to do this but I'm open to suggestions :)

So there you go, my two marketing goals for Angry Beanie. It's going to be an interesting learning period, so bear with me.

Blog Catagories: angry beaniemarketing foo

My last two blog posts were largely about the needs of subunit, but a key result of any protocol is how easy working with it in a high level language is.

In the weekend and evenings I’ve done an implementation of a new set of classes – StreamResult and friends – that provides:

• Adaption to and from the existing TestResult APIs (the 2.6 and below API, 2.7 API, and the testtools extended API).
• Multiplexing multiple streams together.
• Adding timing data to a stream if it is absent.
• Summarising a stream.
• Copying a stream to multiple outputs
• A split out API for instructing a test run to stop.
• A simple test-at-a-time stream processor that makes it easy to just deal with tests rather than the innate complexities of an event based interface.

So far the code has been uniformly simple to write. I started with an API that included an ‘estimate’ function, which I’ve since removed – I don’t believe the complexity is justified; enumeration is not significantly more expensive than counting, and runners that want to be efficient can either not enumerate or remember the enumeration from prior runs.

The documentation in the linked pull request is a good place to start to get a handle on the API; I’d love feedback.

Next steps for me are to do a subunit protocol revision that maps to the Python API, both parser and generator and see how it feels. One wrinkle there is that the reason for doing this is to fix intrinsic limits in the existing protocol – so doing forward and backward wire protocol compatibility would defeat the point. However… we can make the output side explicitly choose a protocol version, and if we can autodetect the protocol version in the parser, even if we cannot handle mixed streams we can get the benefits of the new protocol once data has been detected. That said, I think we can start without autodetection during prototyping, and add it later. Without autodetection, programs like TestRepository will need configuration options to control what protocol variant to expect. This could be done by requiring this new protocol and providing a stream filter that can be deployed when needed.

Background

This post is the story of my first practical look at Junos on Juniper EX-series switches.

One day last December, Skeeve Stevens from eintellego opened a can of worms by offering a deal on Juniper equipment to all network engineers on the AusNOG mailing list.  I had been looking for an excuse to try out Juniper switches for a while; my office switch, an HP ProCurve 3400cl (J4905A), was very loud, and the EX2200-C was the only affordable fully-managed Gigabit fanless switch which i could find from a major networking vendor.  So i ordered one and waited patiently.  I expected it to take 6 weeks or so to arrive (since apparently all the cool kids ordered one too), so i was pleasantly surprised when it arrived the day after Boxing Day.  (For those non-Commonwealth inhabitants reading this page, Boxing Day is the day after Christmas Day; it's always a holiday here in Australia.)  Over the Christmas/New Year break, i spent three days learning and playing with Junos.  I don't claim this as a fair, balanced, or reasonable review - it's simply a recounting of my experiences.

I wanted to get a good general idea of the Junos platform and set it up as my main office switch, moving the HP to my test lab (which is only turned on when i'm using it).  So the basic task i set myself was to learn enough Junos to replicate the functions of my ProCurve as completely as possible.  I knew ahead of time that it wasn't going to be a 100% fit - the 3400cl was a great value switch for its day (and still is, in many ways), and had a lot of features included out of the box (like OSPF and PIM-DM multicast routing).  Junos requires an additional license to implement OSPF, and it wasn't discounted (buying it would nearly have tripled the cost of the switch), so i knew that i'd be eliminating that part of my config.  My other OSPF devices are VMs running Debian Linux with Quagga, and Cisco 1700 & 1800 series routers running IOS, so i simply connected them to the same VLANs and phased out the ProCurve as a router. 

Hardware

My first few impressions of the hardware were positive:

• The switch has a solid metal case with a nice heavy feel to it.
• It was well packed, and came with a very well-packed rack mount kit (perhaps a little too well-packed):
  
• It also came with a rather longer-than-expected, but solid, power cord retainer:
  
• It sports wall-mounting holes as well as a rack-mount kit (hat tip to Chris Lathem for pointing this out):
  

The biggest turn-off for me on the hardware side was the realisation that Juniper switches use zero-based port numbers.  Seriously?  Being an old C programmer, i think of this as "proper" from a programming perspective, but try telling an end-user (troubleshooting an issue in a branch office over the phone with tech support) that port 1 is the second port not the first.  I can't think of any reason other than compatibility with "the way they've always done it" on their routing platforms, and i consider this self-indulgent behaviour on an equipment manufacturer's part.  Other vendors manage to start unit or chassis numbers at zero without being bloody-minded about starting port numbers at zero (although HP's Comware switches start their port numbers at the bottom left rather than the top left, which is almost as frustrating).

Software

After my inspection of the hardware, i moved on to plugging it in and inspecting the software.  Bootup took a bit longer than I expected (the system uses a 1.2 GHz Marvell ARM CPU - the same as in my QNAP TS-219P), but it looked reassuringly Unixy: Junos uses a FreeBSD-based management plane.  After seeing the switch booted through to a login prompt, i logged in as root and had a poke around.  Typical Unix commands like ls, df, and ps all had their usual functions, and one runs "cli" to get into the Junos configuration system.  I worked for the first day or so from the serial console.

I noticed some disconcerting dates in the initial copyright notice:

"This product includes FreeBSD software developed by the University of California, Berkeley, ... 1979, ... 1994. "

Does this mean that 1994 is the most recent date they synchronised with the FreeBSD project?  I hope this is not true, and that Juniper are actively engaging with the Open Source project upon which their code is based.

"GateD software copyright © 1995, the Regents of the University. All rights reserved. Gate Daemon was originated and developed through release 3.0 by Cornell University and its collaborators. Gated is based on Kirton’s EGP, UC Berkeley’s routing daemon (routed), and DCN’s HELLO routing protocol. Development of Gated has been supported in part by the National Science Foundation. Portions of the GateD software copyright © 1988, Regents of the University of California. All rights reserved. Portions of the GateD software copyright © 1991, D. L. S. Associates."

It has been a very long time since i've seen a Unix distribution old enough to contain a gated implementation, and i sincerely hope that no vestiges of this code remain.

During boot an alarm light appeared on the front panel.  I was afraid i had a hardware fault and would have to return my shiny new switch!  After some investigation in the relevant manuals, i found the command to view the active alarms ("show chassis alarms") and found that it was due to the management interface being down.  I already had a management VLAN in my network, and i was expecting this switch to host it.  I investigated the management LAN (at this point a shout-out to the helpful folks in #juniper is due) and found that it was not VLAN-capable at all.  It's more like a NIC for the switch's management plane.  So it's a true out-of-band network, and can't be bridged onto any part of the data plane.  I quickly decided not to use it (and disabled the alarm with "set chassis alarm management-ethernet link-down ignore").  I can understand why the management port is provided, but i'd much prefer that it could be bridged (in software, if necessary) onto the relevant VLAN for installations that aren't large enough to have a dedicated physical out-of-band management network.  I also think enabling alarms for the management LAN being disconnected in the factory default configuration violates the principle of least surprise.

Software Updates and Support

When commissioning a new switch, router, server, storage array, etc., one of the first things i do is look for the latest firmware/software to ensure i'm up-to-date with security patches.  This proved to be a bit of an issue for Junos.  As a newly-arrived customer, i didn't know what to expect; HP offer all software updates for their switches on their public web and FTP sites; with Cisco it's a bit more involved, requiring a login to their site, and with routers you need to know the right things to say to TAC if you don't have a support contract (with switches they're more generous - could this have anything to do with the existence of GNS3?).

With Juniper, it seems one must have a login on their site that is specifically granted access to register devices.  I had already registered a user account on their site in order to download the documentation and view their certification materials, and i was somewhat perplexed and frustrated to find that i couldn't even register my switch until Juniper had changed my account to have "real customer" status.  Their support staff were first class, though, and got it sorted out pretty quickly.

Documentation

My reading of documentation began with Juniper's "Day One" guides, some of which i had downloaded prior to receiving the switch.  These guides range from the 66-page "Exploring the Junos CLI" to the massive "Complete Software Guide for Junos OS for EX Series Ethernet Switches, Release 11.1", weighing in at 4628 pages.  Suffice it to say i only used the latter as a reference.  If you've used Vyatta at all, you'll find the former guide a fairly dull read.  A quick skim to see the differences between Junos and Vyatta was all that was necessary for me.  (Vyatta is a deliberate clone of the applicable parts of Junos' interface.)  "Configuring Junos Basics" was a useful guide to the initial setup of the switch, some parts of which have made it into my Network Engineer's Rosetta Stone - initial setup wiki page.  At 84 pages, it's pretty digestible in a short CLI session.  I used the free PDF format downloads, but most of the Day One guides are available for purchase in print format also.  (Curiously, they cost US$0.99 to purchase in Kindle format, even though Kindle can read PDFs just fine.)  The Day One guides (and the follow-on series "This Week") are generally excellent in quality, and formatted for easy comprehension, although i thought that the headings could have been clearer in some cases.  ("This Week: Hardening Junos Devices" devices is the one that i made mention of in my notes.)

Configuration

My checklist for features to convert from ProCurve to Juniper looked like this:

• Initial setup: user logins, hostname, management IP address, default gateway, DNS, NTP, time zone
• Essentials: VLANs, Rapid Spanning Tree Protocol (RSTP), LACP trunk to VM server, port labels, syslog, SNMP, ssh public keys
• Hardening: login inactivity timer, login banner, disable unused services (including telnet and web management), management ACLs, eliminate LLDP data leakage on Internet interfaces, Spanning Tree BPDU protection/BPDU filtering
• Nice-to-have: QoS on VLANs, port mirroring, IGMP querier, DHCP relay

Initial setup was pretty straightforward, especially given the aforementioned Day One guide.  Having worked with Vyatta for a few months, i found it easy to adjust to the Junos CLI style.  It's a simple and elegant configuration system which made instant sense to me the first time i used it, and Junos worked very much as i expected.  When logged in via ssh, responsiveness of the EX2200-C at the CLI (in terms of character echo delay) is noticeably snappier than the ProCurve 3400cl it replaced (and many other similar devices).

Junos has a number of powerful configuration features.  The ones i came to appreciate most were configuration groups (i used this to set up SNMP) and interface ranges (which i used to set up my VLAN assignments).  The latter is very powerful: a range's name can be used in place of an interface name almost anywhere, and it lives in the configuration as a real entity, as opposed to the temporary nature of the equivalent on IOS.  (Recent HP Comware versions allow a port list to be permanently associated with a name, but store the configuration in the individual ports rather than the interface range itself.)

Configuration is very slow to commit on the EX2200-C compared with Vyatta on an Intel server (understandable, given the low-end CPU); "commit | display detail" set my mind at rest that it was doing useful things, so i just got used to waiting for my commits to complete.  When working in configuration mode, using "run <operational command>" works similarly to IOS' "do <operational command>" with the added bonus that tab and ? completion works.  This was very helpful when i found that "run show interfaces brief" is not very brief, and "run show interfaces terse" is probably more like what i expected of briefness.

There were a number of features in the configuration process that struck me as sub-optimal, or at least unpleasantly surprising:

• There are separate NTP servers during boot and normal operation, with separate configurations for each.  (Presumably under the covers the former are used as arguments to ntpdate during boot whereas the latter are used to configure ntpd.)  This strikes me as a rather obscure feature which probably resulted from a large government department insisting on its inclusion in a large RFP.  The very strange thing about this is that only one boot NTP server is allowed, whereas multiple NTP servers are allowed during normal operation.
• The Junos shell interprets the pipe character (vertical bar) in a counter-intuitive fashion: the pipe is a well-established mechanism on Unix/Linux, Windows, IOS and Comware (amongst others) that sends the output of the first command to the input of the second command, which is a filter.  Junos seems to use it as a generic way to affect the display of the output of the first command, which would more regularly be done by flags or additional arguments on other operating systems (e.g. "ls -l" on Unix/Linux, "dir /b" in DOS/Windows, "show ip interfaces brief" on IOS).  This is another departure from the principle of least surprise, and i'm glad that Vyatta didn't follow Junos in this respect (thus "show | compare" in Junos is simply "compare" in Vyatta).
• The Junos best practice (i can't remember which guide i read this in) is to set the IP address of lo0.0 to 127.0.0.1/32 rather than 127.0.0.1/8, as it is on Linux and Windows.  This is more likely an issue when using Junos for routing rather than switching, but i still couldn't see a reason for it.
• Entering multi-line banners is annoying: one must use \n within the string instead of just pressing enter and finishing the string on the next line.
• VLAN setup is a little tricky.  I initially tweeted: "#Juniper #EX2200-C impression #9: #Junos makes more sense as a router OS than a switch OS. VLAN setup is almost as painful as #Mikrotik."  But after spending more time setting up VLANs on Mikrotik recently, i can now formally rescind that remark.  :-)  VLANs are configured in two different ways: under the interface (or interface range) and under the VLAN definition.  Juniper recommends using the former for trunk (tagged) interfaces, and the latter for access (untagged only) interfaces.  Because i often play around with my VLAN assignments, i ignored this advice and defined them all as if they were trunks.
• Junos includes a concept of an "edge" port, but i couldn't find a clear definition of it in the manuals i was working with, and as soon as i tried to apply BPDU filtering using "bpdu-block-on-edge" in interface context, i discovered its inflexibility, and discarded it in favour of "set ethernet-switching-options bpdu-block interface NAME".

Features

Almost everything i had on my checklist to convert from my 3400cl was done at the end of my three-day lab.  The exact mechanisms for each feature (on both the Juniper EX2200-C and the HP 3400cl it replaced) can be found in the config files attached to this post.  A few features seemed to be missing or deficient (although this could be due to my poor research skills rather than a feature gap):

• There is seemingly no equivalent of IOS' "switchport trunk allowed vlan remove ID".  I couldn't find a way to set up a VLAN trunk such that all VLANs except a particular list were allowed.
• There's also seemingly no equivalent of ProCurve's "lldp admin-status PORTNUMBER RxOnly" (for allowing receive but not send of LLDP info); Julien Goodwin suggested that ACLs could be used to fix this; i simply chose to disable LLDP entirely on my ISP uplinks.
• More perplexingly still, there's seemingly no equivalent of IOS' "no ip routing". Seems very strange that there is a Junos kernel option to drop all TCP packets (i can't think why anyone would want to turn on this option, since it would prevent remote management via ssh), but no option to turn off IP routing.  Workarounds suggested by the IRC crowd include:
• blocking traffic between vlans by policy
• using multiple routing instances and putting each VLAN in its own instance
• create only L2 VLANs with no L3 interface (which is the option i chose in this case)
• There's no way to enable all interfaces for sFlow - it appears one must add each interface individually.  And further, there's no SNMP write access at all, so i couldn't just configure authentication and leave the heavy lifting to sFlowTrend.
• Probably the feature i spent the most time investigating and discussing on IRC (hat tip especially to Kurt Bales) was VLAN QoS priorities.  Junos' QoS capabilities are extensive, but configuring them is not as straightforward (some would say "simplistic") a matter as ProCurve makes it (vlan X; qos priority N).  Because my home office switch is never congested, i decided it wasn't worth spending the time to learn this part of Junos until later.

Conclusion

Junos is a mostly-logical network OS with an elegant configuration system and excellent freely-available documentation.  The quality of the EX2200-C hardware is first rate, and i suspect this only increases in subsequent models in the series.  I'm glad i took the time to get familiar with the basics of Junos, and i'll probably try to be certified to their entry-level standard if time permits.  I don't think Juniper's offering is unique enough to cause me to choose them ahead of Cisco or HP (the latter's Comware switches are still my go-to choice), but they're certainly an excellent choice for quality managed switches, especially for those organisations with an investment in Juniper's routing platform.

AttachmentSize hp3400cl.txt5.41 KB ex2200c.txt10.33 KB

Hi all,

The Griffin Powermate is a great little device and I use it daily, (and when I’m on my pc I use it pretty much hourly), the biggest problem is getting the thing installed and setup correctly.

I’ve got myself a new SSD so I’ll need to reinstall Ubuntu, I’ve used the packages from the ubuntu forums as a base, many thanks to the hardwork of the people who put the original packages together: http://ubuntuforums.org/showthread.php?t=1346654

Since I use my powermate strictly for changing the audio on my system up and down (my reasons here), my changes involve turning the device into basically an easy to reach volume changer.

If this interests you feel free to download and install the package: here

Currently it is reporting as a package of bad quality, I shall fix that as soon as I’m on my new install.

When:
16th and 17th March 2013 at 9.00 am

Where:
“The Shack”, Behind Blackwood Girl Guides Hall
Hannaford Road
Blackwood
SA 5052

Contact:
Sasi Nayar (VK5SN)
0417 858 547
vk5sn@wia.org.au

refer to flier for more details

References:
AHARS

Of course, as happens sadly often, the scope creeps..

Additional pain points

Zope’s test runner runs things that are not tests, but which users want to know about – ‘layers’. At the moment these are reported as individual tests, but this is problematic in a couple of ways. Firstly, the same ‘test’ runs on multiple backend runners, so timing and stats get more complex. Secondly, if a layer fails to setup or teardown, tools like testrepository that have watched the stream will think a test failed, and on the next run try to explicitly run that ‘test’ – but that test doesn’t really exist, so it won’t run [unless an actual test that needs the layer is being run].

Openstack uses python coverage to gather coverage statistics during test runs. Each worker running tests needs to gather and return such statistics. The current subunit protocol has no space to hand this around, without it pretending to be a test [see a pattern here?]. And that has the same negative side effect – test runners like testrepository will try to run that ‘test’. While testrepository doesn’t want to know about coverage itself, it would be nice to be able to pass everything around and have a local hook handle the aggregation of that data.

The way TAP is reflected into subunit today is to mangle each tap ‘test’ into a subunit ‘test’, but for full benefits subunit tests have a higher bar – they are individually addressable and runnable. So a TAP test script is much more equivalent to a subunit test. A similar concept is landing in Python’s unittest soon – ‘subtests’ – which will give very lightweight additional assertions within a larger test concept. Many C test runners that emit individual tests as simple assertions have this property as well – there may be 5 or 10 executables each with dozens of assertions, but only the executables are individually addressable – there is no way to run just one assertion from an executable as a ‘test’. It would be nice to avoid the friction that currently exists when dealing with that situation.

Minimum requirements to support these

Layers can be supported via timestamped stdout output, or fake tests. Neither is compelling, as the former requires special casing in subunit processors to data mine it, and the latter confuses test runners.  A way to record something that is structured like a test (has an id – the layer, an outcome – in progress / ok / failed, and attachment data for showing failure details) but isn’t a test would allow the data to flow around without causing confusion in the system.

TAP support could change to just show the entire output as progress on one test and then fail or not at the end. This would result in a cognitive mismatch for folk from the TAP world, as TAP runners report each assertion as a ‘test’, and this would be hidden from subunit. Having a way to record something that is associated with an actual test, and has a name, status, attachment content for the TAP comments field – that would let subunit processors report both the addressable tests (each TAP script) and the individual items, but know that only the overall scripts are runnable.

Python subtests could use a unique test for each subtest, but that has the same issue has layers. Python will ensure a top level test errors if a subtest errors, so strictly speaking we probably don’t need an associated-with concept, but we do need to be able to say that a test-like thing happened that isn’t actually addressable.

Coverage information could be about a single test, or even a subtest, or it could be about the entire work undertaken by the test process. I don’t think we need a single standardised format for Coverage data (though that might be an excellent project for someone to undertake).  It is also possible to overthink things . We have the idea of arbitrary attachments for tests. Perhaps arbitrary attachments outside of test scope would be better than specifying stdout/stderr as specific things. On the other hand stdout and stderr are well known things.

Proposal version 2

A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag – one of (estimate + number, stdin, stdout, stderr, file, test), test-id, runnable, test-status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes.

The std/stdout/stderr content tags are gone, replaced with file. The names stdin,stdout,stderr can be placed in the attachment name field to signal those well known files, and any other files that the test process wants to hand over can be simply embedded. Processors that don’t expect them can just pass them on.

Runnable is a boolean, indicating whether this packet is describing a test that can be executed deliberately (vs an individual TAP assertion, Python sub-test etc). This permits describing things like zope layers which are top level test-like things (they start, stop and can error) though they cannot be run.. and it doesn’t explicitly model the setup/teardown aspect that they have. Should we do that?

Testid is for identifying tests. With the runnable flag to indicate whether a test really is a test, subtests can just be namespaced by the generator – reporters can choose whether to be naive and report every ‘test’, or whether to use simple string prefix-with-non-character-seperator to infer child elements.

Impact on Python API

If we change the API to:

class TestInfo(object): id = unicode status = ('exists', 'inprogress', 'xfail', 'xsuccess', 'success', 'fail', 'error', 'skip') runnable = boolean class StreamingResult(object): def startTestRun(self): pass def stopTestRun(self): passs def estimate(self, count, route_code=None, timestamp=None): pass def file(self, name, bytes, eof=False, mime=None, test_info=None, route_code=None, timestamp=None): """Inform the result about the contents of an attachment.""" def status(self, test_info, route_code=None, timestamp=None): """Inform the result about a test status with no attached data."""

This would permit the full semantics of a subunit stream to be represented I think, while being a narrow interface that should be easy to implement.

Please provide feedback! I’ll probably start implementing this soon.

Subunit is seven and a half years old now – Conrad Parker and I first sketched it up at a CodeCon – camping and coding, a brilliant combination – in mid 2005.

revno: 1 committer: Robert Collins <robertc@robertcollins.net> timestamp: Sat 2005-08-27 15:01:20 +1000 message:  design up a protocol with kfish

It has proved remarkably resilient as a protocol – the basic nature hasn’t changed at all, even though we’ve added tags, timestamps, support for attachments of arbitrary size.

However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format.

Existing design

The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of ‘make’ or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined.

This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity – but also requires that state be buffered until the end of a test, as two tests cannot be executing at once.

Dealing with debuggers

The initial protocol was intended to support dropping into a debugger – just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious – the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like.  All in-principle fixable, right ?

When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else.

We’ve added some idioms in testrepository that control test execution by a similar but different format – one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying – you need two parsers, and two output formats.

Good points

The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility).

Pain points

The debugging support was a total failure, and the way the parser depraminates it’s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout).

Dealing with concurrency

The original design didn’t cater for concurrency. There are three concurrency issues – the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues?

Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we’ve lost information. Or at best muddled it all up.

Secondly, buffering – imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1′s test is taking more than 5 seconds or something – but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer.

Thirdly, scheduling – if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time.

Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn’t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn’t help when it is W1 and W2 reading from stdin.

So additional requirement over baseline subunit:

1. make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well – the protocol doesn’t need to care about that, just make it possible.
2. make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
3. make it possible (but optional) to pass tests to run to a running process that accepts subunit.
4. make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don’t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)

Dealing with corruption

Consider the following subunit fragment:

test: foo starting serversuccess:foo

This is a classic example of corruption: the test ‘foo’ started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn’t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like ‘make’ runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn’t finish it. That looks like:

test: foo test: bar success: bar

One way that this sort of corruption can be mitigated is to put subunit on it’s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn’t solve the failing test script case.

I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work – so the ‘make -j2′ case can be ignored at the wire level – though we should create a simple way to safely mux the output from such tests when the execute.

The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path.

Dependencies

Various suggestions have been made – JSON, Protobufs, etc…

A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal.

High level proposal

A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag – one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes.

The content tags:

• estimate – the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn’t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate ‘TestSuite’ objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
• stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
• test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.

Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed.

Attachment names must be unique per routing code + testid.

So how does this line up?

Interleaving and recovery

We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won’t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data.

listing

Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once.

Selecting tests to run

Same as for listing – while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band.

Debugging

Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append ’1′ or ’2′ to select which of W1/W2 the content came from, and then forward the packet. S would append ’1′ or ’2′ to indicate I1/I2 – the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question.

Multiplexing

Very straight forward – unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for).

Parsing / representation in Python or other languages

The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams.

Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:

• Concurrent tests (e.g. in multiprocessing) wouldn’t need multiplexers and special adapters – a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
• The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
• The listing facility and incrementally running tests would be useful too I think – we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult

The API might be something like:

class StreamingResult(object): def startTestRun(self): pass def stopTestRun(self): pass def estimate(self, count, route_code=None): pass def stdin(self, bytes, route_code=None): pass def stdout(self, bytes, route_code=None): pass def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None): pass

This would support just-in-time debugging  by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process – a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently

Generation new streams

Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. ‘test foo completed + some attachments’ or ‘test foo starting’.

I ran across an infected site the other day…

Just to confirm decided to run a quick security scan on it:

Ok so it’s definitely infected, I’m going to check with McAffee SiteAdvisor and Norton Safeweb

oh?

Brilliant!

So a site definitely infected and spreading malware is a-ok according to site advisor and norton. What a waste of bandwidth and time installing useless toolbars that clutter up google for no reason.

The last couple of months have been interesting for me.

I've had a couple of short contracts working with Drupal and Joomla and I have to say, I'm still very much a Drupal person. Joomla just doesn't seem to have the same polish, either in the core system or the module set around it. You may disagree but meh, everyone's different.

I attended lca2013, which of course rocked. On Monday there was MobileFOSS. We had an excellent range of talks (admittedly almost half came from the Serval Project), including a demo of the OpenPhoenux and FireFoxOS platforms. A good time was had by all and when the videos go up I'll post them up here.

Angry Beanie is getting back up to speed, I've released two episodes of Purser Explores The World - Tomorrows Geek and My Kingdom For A Horse - so far this year and tomorrow night the crew from For Science! is going to be recording the first episode for the year.

In other Angry Beanie news, I'm looking at ways to raise funds for various projects, including upgrading the gear I use to record and put everything together. I'll be posting more about that once I've got it sorted out (and frankly once I've got the guts up to do it). Needless to say it could all be sorted if some podcast loving billionaire decided to give me n+lots monies to work things out.

So to summarise, still looking for work, Angry Beanie coming along and Joomla still (for me) sucks.

Blog Catagories: angry beanieforscience!purserexplorestheworlddrupaljoomla

About:
The Old Reynella Festival of History includes re-enactors in uniforms and historic clothing, the firing of old cannon and mock battles and huge model soldier re-enactment of the Battle of Leipzig. There will be a major street parade featuring vintage cars, old military vehicles and bands

When:
Saturday 9th Sunday 10th March 2013

Where:
Old Reynella South Australia

Attached is a pdf explaining the whole event, the groups participating, and some the history of Old Reynella.
2013 Festival of History

Well we're only a couple of weeks into the official unofficial election campaign and already the derp is knee deep. I can confidently predict that by the time the actual poll rolls around, we're not going to be able to get to a polling station due to the fact that we're all going to be trapped in a great steaming layer of derp several miles deep.

Let's have a look at some of the highlights of the past two weeks shall we?

Virtual Caretaker Mode derp

As soon as the PM announced the date of the election, the Opposition went into full derp mode, demanding that the government and associated bodies essentially shut up shop for the next eight months and enter into a "virtual caretaker mode"

For those of you who aren't politics wonks, the caretaker period is that bit of time between the actual launch of the election period (when writs are issued) and the election itself. During that time the government is not allowed to make new policy and essentially the idea is to just keep things running as they were on the day the writs were issued.

So, the Coalition in effect demanded that the Government just put the handbrake on and idle for the next eight months, stop passing laws, stop making policy, and just generally stop.

Right.

The two stand out examples of this sort of derp for me were Malcolm Turnbulls demand that the NBN Co stop entering into any new contracts until the election and Greg Hunt demanding that the  Climate Change Fund not hand out any funds when it starts operating in July.

Stop the boats derp

We all know how fond the Coalition is of the Stop the boats campaign. It reaches to that mean little nugget of bile that drives a certain demographic.

Well they managed to reach a new low/high (depending on your point of view I guess) this week when they declared that they would have the Navy forcibly turn back boats from Sri Lanka before they hit Australian territorial waters. I don't know if this moves into the territory of state sanctioned piracy or not, but at the very least it seems to be a complete abrogation of our treaty obligations.

They also claimed people are leaving Sri Lanka in crappy boats to travel thousands of miles across the ocean purely for economic reasons.  This, despite the mass of evidence that suggests that Sri Lanka is less than the harmonious and healing nation that it might be portrayed by the Sri Lankan government.

Northern Australia derp

Of course the latest in the constant rain of derp is the leaked Coalition "discussion paper" for developing North Australia.

This appears to be an adaption of the plans put forward both by Gina Reinhardt and the Citizens Electoral Council. It also has the backing of patron Saint and founder of Katter's Australia Party, Bob Katter.

I'm just going to leave that particular pedigree there. 

As to actual details? Well the document above recommends a lot of working parties and inquiries into various areas, including the following:

• Moving government departments from Canberra to towns and cities such as Townsville, Karratha and Darwin
• Increased "incentives" to encourage migration from the southern states. These include tax incentives, grants and so on.
• Creating a working party with south east asian neighbours to plan out a special economic zone in the north.
• Oh and they want to re-direct 800 million dollars from the Foreigh Aid budget to the building of tropical disease research and care facilities
• Establish a Water Project Development Fund. No word yet as whether this will involve turning the rivers back.

All this is and we're only two weeks in.

We are so going to be swimming in it by the time the polls roll around.

Blog Catagories: Politics

AIM:
Estimate the acceleration of an object due to gravitational force.

Equipment:

1. mass (weights)
2. ticker tape
3. ticker machine
4. PSU
5. tape measure
6. sticky tape

Method:

1. Connect sticky tape to one end the ticker taper and thread the other end through the ticker tape machine.
2. Connect the ticker tape machine to the AC connectors on the PSU and set the PSU to ~6VAC.
3. Connect weight (mass) to the end of the ticker tape with the sticky tape on and turn on PSU and drop mass from min. of one meter.
4. Check for dots made by the ticker machine and pic a section of ~500mm of ticker tape and measure between the first and last dots in the section, then count up the intervals between the dots.
5. Work out the acceleration now by doubling the intervals and divide by 100 to get a decimal value (t)

using the formula

S = length of tape section

The hackers tried to cloak the source of the attacks on The Times by first penetrating computers at United States universities and routing the attacks through them, said computer security experts at Mandiant, the company hired by The Times. This matches the subterfuge used in many other attacks that Mandiant has tracked to China.

http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?smid=re-share&_r=0

While this feels like war, the cynic in me questions what the American military is doing right now to Chinese business and government systems as well.