Rhys Goodwin's Weblog
I AM the system administrator. Who do I call?
I AM the system administrator. Who do I call?
Apr 7th
In an effort to better manage our finances I decided to ditch my self-written ASP.NET budgeting tool and adopt GnuCash, an excellent open source accounting application. As well as being a true double entry accounting system, one of the great things about GnuCash is its ability to import a set of transactions in various formats. The idea here is that you import an OFX or CSV from your bank and allocate transactions to various accounts.
After almost 4 years of manually entering every single transaction into my crappy home-grown tool I was on the verge of giving up altogether. I decided that whatever new system I went with would need to be as automated as possible. So partly for the challenge and partly because I’m efficient (lazy) – I decided to automate downloading of transaction files from my bank accounts at Kiwibank.
Now it would be really nice if KiwiBank provided a webservice API to pull these transactions down – of course that would be too good to be true. With an API ruled out that only leaves the front end.
The first option I looked at was a Python based web scraping tool called Scrapy. It’s a really flexible powerful tool for parsing html. As I started getting a grip on the syntax of Scrapy it became clear that it wasn’t going to do the job due to the JavaScript-heavy interface that Kiwibank uses.
The second option was browser automation. To me this seemed like a less elegant option but after finding Selenium I soon forgot about that. Selenium is a web testing and automation suit. It consists of a number of components including a pretty extensive set of development libraries and interfaces. The two tools I used were Selenium Server and Selenium IDE (Integrated Development Environment) for Firefox.
The Selenium IDE Firefox extension allows you to create, record, edit and test Selenium automation scripts.
Start off by creating a new test suit and then a new test case within that suite. Hit the record button and start recording your browser session. Every action you perform in the browser will be recorded as a step in the script. This will give you the basis for the automation. Once you’re done recording you might need to manually edit, add or remove some steps to make the script more robust, or fix bits that don’t play back correctly. You can play the script back with the buttons on the toolbar or you can execute one step at time by selecting the step and pressing ‘x’.
Another extremely useful tool to help analyze page elements is Firebug for Firefox, it’s an excellent compliment to the Selenium IDE.
In an attempt to make their site more secure Kiwibank employ a two step authentication process. The first being AccessNo./Password and the second, a question/answer system which asks you to click the missing letters from the answer. This adds a slight level of security because it means an attacker needs to have a logger that’s a little more extensive than just logging keys.
Now it’s probably possible to get Selenium to read the question, work out which letters are missing and look up a table to determine which JavaScript should be called to complete the answer. And I may end up having to do that if Kiwibank reads this post! But fortunately for me Kiwibank allows you to set your own questions and answers. The questions all have to be different but the answers don’t. Simply setting all the answers to the same five letters means that I always call the same JavaScript.
To be honest it felt good the be the user, circumventing the security for a change!
Selenium allows you specify a file with your own JavaScript functions. The file must be named user-extensions.js. It’s location can be configured in the IDE under options/options. I don’t think these scripts can interact with elements on the page though. Someone please correct me if I’m wrong here.
I created a custom JavaScript function that returned the current date less x number of days given as a parameter.
I used this function to get the last 28 days when specifying the “from date” on the export selector.
The whole purpose of this exercise is to automate downloading of transaction files so we need to tell Firefox to automatically save files of a certain type instead of prompting. We’d also like to save them in a specific location.
The best way to handle this is create a custom Firefox profile for Selenium to use just for this automation. There’s a great post here which details the optimum profile settings for use with Selenium.
The last thing you’ll need to do to the profile is make sure that it handles your chosen export file type correctly. In my case I’m using .OFX so I needed to tell Firefox to always download .OFX files without prompting. This is done through the mimeTypes.rdf file in the profile. Details on this file here.
If you keep getting the add-ons popup every time you use the custom profile I found the following fix:
To disable add-ons window which appears every time when Selenium scripts are run on Custom Firefox Profile.
Close all instances of Firefox browser and delete the following files from the Custom Profile folder
extensions.cache
extensions.ini
extensions.rdf
compatibility.ini
This should reset Extension Manager and disable add-ons pop-up.
Now with a fully working script and customFirefox profile in hand we can set about scheduling this automation with the Selenium server and the Windows task scheduler. The Selenium server would normally be stated and left running like any other server application. In our case we’ll just start it, run our script and then exit.
Once you get the command working at the command prompt you can then use it in a scheduled task running under it’s own user account. If you do this, everything will run in the background and you won’t see any windows pop up and it will run even if no one is logged on to the PC.
Here is the command I use to run my Kiwibank automation:
java -jar selenium-server.jar -singlewindow -htmlSuite "*firefox" "https://www.ib.kiwibank.co.nz" "D:\Selenium\Kiwibank\SeleniumSuitKiwibank.html" "D:\Selenium\Kiwibank\SeleniumSuitKiwibankResults.html" -firefoxProfileTemplate "D:\Selenium\FirefoxProfiles\Kiwibank" -userExtensions "D:\Selenium\JavaScript\user-extensions.js"
Notes about the command:
That’s it. Fully automated transaction file download! Make sure you observe good security practices with this sort of stuff – Principle of Least Privilege.
Nov 19th
Now for something completely different….after shelling out for one of these awesome microchip cat doors to be installed I thought I’d DIY the opening in my security mesh door. – As the weather warms up hopefully I’ll get more DIY stuff up here.

Mark the bars you'll need to cut to get make a square just a little bigger than the cat door opening

Cut where you marked using some hefty bolt cutters - a good excuse to buy tools, it would cost more to get a guy out to do the job, right? For now only cut the bars not the screen.

Make up 2 "picture frames" using mitre saw. The inner dimension should be about the same size as the cat door opening or just slightly bigger. I glued and tacked mine together with small nails.

Put the screws into the drilled frame and push through the mesh screen to hold it in place. Cut out the mesh, and put some dabs of glue to hold the mesh
Nov 8th
This one cost me some time!
Symptoms
After installing and configuring OpenAM you’re unable to log on to the admin console with the amAdmin account and password you set during the install. It doesn’t give an error message, just drops you back to the login page.
Cause
When you go through the custom configuration wizard you get asked for the cookie domain. If your OpenAM server is openam.mydomain.co.nz then your cookie domain should be .mydomain.co.nz but by default the wizard just takes the trailing two domain components from the server name – i.e. .co.nz. Unless you specifically set the cookie domain correctly you’ll get the issue described above. As you can imagine this issue wouldn’t occur if your OpenAM server was called openam.mydomain.com.
This means that if you have a domain name with more than 2 domain components then you’ll always need to run the custom config wizard.
Oct 11th
I’ve been meaning to get this out there for a while now. I’m not going to go into great detail on ADFS but you can get more background on ADFS and federation in these posts:
Salesforce SSO with ADFS 2.0 – Everything You Need to Know
ADFS 2.0 Choose Your Attributes Wisely
SAML WebSSO Federation Made Easy
My scenario is as follows:
Here are the high level hoops I had to jump through to get this working:
setspn -a HOST/federate.internal.com stjohn\Svc.ADFS
C:\Program Files\Active Directory Federation Services 2.0\
Run the following command to configure and new ADFS 2.0 farm
FSConfig.exe CreateFarm /ServiceAccount "INTERNAL\Svc.ADFS" /ServiceAccountPassword "somebiglongpassword" /AutoCertRolloverEnabled /FederationServiceName "federate.internal.com
Notes

Add-PSSnapin Microsoft.Adfs.Powershell Set-ADFSProperties -AutoCertificateRollover $true Update-AdfsCertificate -Urgent
This might not be the only way to get this working and I haven’t tested it thoroughly – your mileage may vary! But as always I’m keen to hear how you get on and happy to field questions.
Jul 24th
In my last post I showed you my RG6 coax patch panel build which included cabling up the 4 outputs of my LNB. Well there was a reason I risked life and limb on the roof in high winds.
TBS Technology of Shenzhen China have only been making TV cards for about 5 years but they’re starting to build a good reputation amongst HTPC enthusiasts, and for good reason – as you’ll see the 6984 is a solid performer. This review will focus on using the TBS 6984 with MediaPortal TV Server.
The TBS6984 really is the grand-daddy of DVB-S cards! It’s a DVB-S/S2 PCI Express card with 4 tuners allowing you capture from 4 different satellite transponders simultaneously. At $249USD the price is right. That’s about $62 per tuner – considerably cheaper than buying 4 separate DVB-S2 cards and much more convenient.
TBS list the following specifications:
Receiving Frequency: 925~2175 MHz Tuning Range
Input Level: -69~23dBm
4x Advanced DVB-S2/DVB-S 8PSK QPSK Demodulator
Symbol Rates:
DVB-S QPSK: 1-45 Msps
DVB-S2 8PSK/QPSK: 2-36 Msps
Code Rates:
DVB-S: 1/2, 2/3, 3/4, 5/6, 7/8
DVB-S2 QPSK: 1/2, 3/5, 2/3, 3/4. 4/5. 5/6, 8/9, 9/10
DVB-S2 8PSK: 3/5, 2/3. 3/4. 5/6. 8/9, 9/10
I’m not going to pretend that I could explain all the technical aspects of these specs, but suffice to say this card will handle pretty much anything you can throw at it.
The build quality of the card is excellent – all the soldering looks clean and solid, and the components are well aligned. The bracket has labels for the tuners ‘A’ through ‘D’ stamped on it – a nice touch. The chipset consists of the following:
The remote is pretty basic; but let’s be honest, if you’re looking for a quad tuner card you’re probably an HTPC enthusiast in which case you’ll already have an advanced remote. If not you’ll need to get one! It has TV and navigation buttons, but lacks buttons for the advanced features you’d typically find in media centre packages like MediaPortal. That said, it will do just fine to get you up and running with basic TV software.
As for the driver CD, I haven’t even put it in my PC. Personally I never use the driver CDs which come with any hardware. I prefer to go straight to the web and download the very latest version. It would be nice if TBS released a white-box version of this product, which included just the card and the power cable.
TBS state that the additional power cable is only required when you need extra current for driving things like dish positioning motors and some LNBs. In general, you shouldn’t require it which is good – the less cables floating around the better when it comes to an HTPC that you’re trying to keep cool with a minimum of fans.
Installation
I’d love to go into great detail about the installation but there really isn’t much to say. The hardware side is obvious – unplug your PC and install the card into a spare PCIe 1x slot.
As for the driver, TBS keeps it simple which I really appreciate. You don’t have to run an installer (although there is one). You can simply let Windows detect the card and then tell it where to find the latest driver files and the device installs without any fuss. I wish more manufacturers would take this simple clean approach. With an installer, you don’t really know what you’re getting and what’s being changed on your system. I’m running Windows 7 x64. Once the driver is installed, you’ll see a single “TBS 6984 Quad DVBS/S2 BDA Tuners” device listed in device manager. The driver is a BDA driver, which means it conforms to Microsoft’s broadcast driver architecture so the card will be compatible with any TV software which supports BDA devices.
MediaPortal
MediaPortal is a free and open source media centre package for Windows.
You can get a wealth of information and support at the Team-MediaPortal site, but these are the basic components that you’ll need to get TV up and running with the TBS 6984. The remainder of this review will focus on the TV Server component of MediaPortal.
Once you’ve got the driver installed TV Server will detect the card. You’ll have to restart the TV Service and TV Server configuration tool if they are already running.
One thing I really like about this card is the way it identifies itself. In the Windows device manager it just shows a single device, but once you open up the TV Server configuration tool you’ll see all 4 tuners and they’re actually labelled A, B, C and D – unlike some other dual cards I’ve seen, which just show 2 identical tuners so you can’t tell which one is which.
Scanning speed is impressive – just over 6 minutes to scan 41 transponders. Both DVB-S and DVB-S2 channels are found correctly.
Now the part you’ve been waiting for – recording 4 channels at once. In fact, with MediaPortal TV Server you can record even more than that because it allows you to record all the channels on a given transponder at the same time. The TBS 6984 can tune into 4 separate transponders, so if each of those transponders carries 6 channels that would mean you could record 24 channels simultaneously! Below you can see I’m receiving 12 channels quite happily and the 6984 doesn’t skip a beat! “Just try that Windows 7 Media Center!”
The driver seems to report the signal quality and strength much more accurately than a lot of other cards I’ve seen, and also updates these quite frequently – which is great.
Channel Change Speed
The most common question I hear when discussing various TV cards with HTPC enthusiasts is “How fast can it change channels?”. There are a number of things that can affect this – system hardware, TV card, TV card driver, TV software, media codecs, etc. It also depends on where you take the measurement. The following results are taken from the TV Server logs, and indicate the time it takes for the TV card to switch channels.
Start DVB-S Channel (Same channel/transponder for each tune) 1st Tune: 437ms 2nd Tune: 218ms 3rd Tune: 203ms 4th Tune: 281ms 5th Tune: 124ms
Start DVB-S Channel (Different channel/transponder for each tune) 1st Tune: 390ms 2nd Tune: 718ms 3rd Tune: 531ms 4th Tune: 796ms 5th Tune: 609ms
Change DVB-S Channel (Same transponder for each tune) 1st Tune: 437ms 2nd Tune: 453ms 3rd Tune: 374ms 4th Tune: 281ms 5th Tune: 593ms
Change DVB-S Channel (Switch transponder for each tune) 1st Tune: 475ms 2nd Tune: 374ms 3rd Tune: 468ms 4th Tune: 468ms 5th Tune: 687ms
Change DVB-S2 Channel (Same transponder for each tune) 1st Tune: 234ms 2nd Tune: 296ms 3rd Tune: 234ms 4th Tune: 343ms 5th Tune: 171ms
Change DVB-S2 to DVB-S (Switch transponder for each tune) 1st Tune: 656ms 2nd Tune: 390ms 3rd Tune: 718ms 4th Tune: 656ms 5th Tune: 687ms
Change DVB-S to DVB-S2 (Switch transponder for each tune) 1st Tune: 2159ms 2nd Tune: 2124ms 3rd Tune: 2156ms 4th Tune: 2218ms 5th Tune: 2187ms
Very impressive, with all tests sub-second, except DVB-S to DVB-S2 switching which takes a little longer.
DiSEqC
The TBS 6984 supports DiSEqC 2.x. MediaPortal TV Server doesn’t yet support DiSEqC for this card, but I’ve spoken to a member of the MediaPortal development team who has informed me that they will be adding it soon, and has asked me to be a tester when the time comes.
All-in-all, my only criticism is that such a high-end card should be matched with a high-end remote. I think the best solution is a white box version of the product so the user can choose their own remote.
The channel change speed tests speak for themselves; that, combined with the solid driver and excellent build quality, makes the 6984 an excellent choice for anyone looking to build or expand an HTPC. In fact, unless you’re certain you won’t need more than 2 tuners, I would say just go straight for the 6984 because you’ll end up saving money in the long run.
As for TBS support – while I haven’t needed any technical support, from what I’ve read elsewhere they seem to have a reputation of being very responsive as well as being happy to interact with the MediaPortal development team. They also make their SDK (software development kit) freely available.
Jul 11th
Here’s a quick update on my structured cable at home. Hopefully it will give you some ideas if you’re looking to do something similar. The main goal here was to run all 4 LNB outputs from my dish and my UFH antenna back to a single point.
RG6 quad-shield run up from the floor through the existing data cable channel.
I used a piece of powder coated aluminum which I cut from a 2U server rack blanking panel and ran the RG6 cables through the wall and terminated them with ‘F’ joiners.
I came across a great free 2D CAD application called DraftSight which I used to create a template for drilling the plate and the wall.
Completed and all back together – featuring the aptly named ‘Patch’. The cables connected to the completed patch plate all go back up the channel to the TV Server PC in the cupboard above. I haven’t cabled any of the rooms (except the lounge) because everything is delivered over IP, however I have pre-drilled at the back of the plate and half drilled the plate for future expansion.
I’d like to thank Godfey who supplied all the RG6, F connectors, and tools. And also took the time to show me how to do PPC compression fittings. Kiwi’s – if you’re in need to any of the gear to do this stuff Godfrey trades through TradeMe and gives the best service and prices around!
Jun 25th
Ok this is one of those “if you need it, you’ll know what I’m talking about posts”!
I recently started using oscam and being that I don’t like sever applications that need to run in the foreground I wrote a small windows service wrapper to handle oscam for me.
Just drop it in your oscam folder and install it by running oscamSVC -install
Using oscam as a system service also overcomes the issue that some people have found with some USB card readers such as the Omnikey disconnecting or ejcecting when a remote desktop connection is made to the PC running oscam.
Version History & Changes
0.3.5.0 – Current version
0.3.0.0
Requires .NET 2.0
Enjoy.
Apr 25th
Project boxes available off the shelf always seem to be just too smahttp://blog.rhysgoodwin.com/wp-admin/post.php?post=973&action=edit&message=1ll or way too big! This is especially true here in New Zealand where the options between JayCar and SurplusTronics are fairly limited. I needed a specific size to house a project I’m working on so I decided to cast my own in Polyester Resin. I hope the details which follow will prove helpful.
Draw up a design – I used Google Sketch up. Make a box of which the inner dimensions represent the outer dimensions of your final enclosure. I used blocks of pre-dressed pine. You’ll want to use something reasonably solid and screw it down to a base board so you get nice square vertical sides. For a base I used melamine board – it’s nice and smooth and the resin won’t bond to it. Your local kitchen builder will give you off cuts for free if they’re nice.
Wrap the blocks with masking tape. This provides three benefits:
Put a mark on the side of at least one of the walls to indicate the height of the enclosure, this is where you will pour the resin up to.
Accuracy when cutting the wood is important if you want a professional looking result. Decide on a tolerance and stick to it. If you cut a length and it’s not within tolerance then re-do it. If you don’t, you’re sure to be disappointed with the end result. Resist the “She’ll be right” temptation – it won’t be right! Errors are amplified at each stage of the process. I worked to 0.5mm. For me this was an excellent practice exercise in hand-saw and measuring accuracy.
Make a shape of which the outer dimensions will represent the inner dimensions of your finished enclosure. This is where you decide on the thickness of the enclosure walls. I made over-sized corners so that I had solid pillars to screw into.
Unless you’ve got a dead level work bench you’ll probably need to set up a little platform that you can level off with screws – like this:
Mix up some resin and pour it into the mold up to the height you marked. Mix the resin and MEKP as per the instructions. I mixed towards the higher end of the 1%-2% ratio, about 1.7%. And please be careful with the MEKP. Don’t even think of going near it without eye protection and gloves. MEKP is a severe skin irritant and can cause progressive corrosive damage or blindness.
Now the fun bit. After a few hours the resin will be hard enough to remove it from the mold. Unscrew and remove the inner blocks and as many outer blocks as you need to get the enclosure out.
It will be a bit hard to get out because the resin shrinks a little bit as it cures. It will probably also still be a bit sticky when it comes out.
Now on to the top and bottom sections. You could just cut some flat plexi-glass for this but while we’re at it we might as well just cast them. Clean up the enclosure with sand paper. I used wet/dry from 120 grit up to 400.
Drill and tap the corners.
Use making tape to mask off the areas where the resin will touch. Screw in 4 countersunk lid screws leaving them out to the height you want the thickness of top lid to be (I made mine 4mm). Make sure they are all exactly the same height.
Reassemble the outer box of the mold on a new piece of melamine and place the enclosure back in, screws down. The enclosure will have shrunk since it was last in the mold so use multiple layers of masking tape as packers to center it (use an even number of layers on each opposing side).
Make sure your platform is dead level and pour the resin into the mold so it just comes up over the sides of the enclosure. This will make a locking lip on the lid.
After a few hours you can de-mold the box with lid attached. It will be stuck to the board but just slowly ease it away, you almost have to peel it up. Don’t try to knock or tap it. With a bit of luck you’ll be able to remove the screws from lid and take the lid off without too much difficulty. Making the lid this way takes care of the screw holes and countersinking.
Now repeat the process for bottom lid. For the bottom I didn’t mask it off because I wanted it sealed on – I don’t need to remove it. I also made it a bit thicker so it’s nice and solid for mounting to.
Once you’re all done you can sand/polish the enclosure as much or as little as you like. I left it with a frosted look but you could shine it up to be completely transparent if you wanted. I also removed the thread from the holes in the lid.
Well that’s quite a process! But the result is good and it’s good practice for accuracy, woodwork and resin casting. I tried a number of methods before I came up this and it’s by no means perfected – as always I’m keen to hear your ideas.