Archive | Windows Desktop

Auto-download [Kiwi]bank Transactions with Selenium

Intro

In an effort to better manage our finances I decided to ditch my self-written ASP.NET budgeting tool and adopt GnuCash, an excellent open source accounting application. As well as being a true double entry accounting system, one of the great things about GnuCash is its ability to import a set of transactions in various formats. The idea here is that you import an OFX or CSV from your bank and allocate transactions to various accounts.

After almost 4 years of manually entering every single transaction into my crappy home-grown tool I was on the verge of giving up altogether. I decided that whatever new system I went with would need to be as automated as possible. So partly for the challenge and partly because I’m efficient (lazy) – I decided to automate downloading of transaction files from my bank accounts at Kiwibank.

Now it would be really nice if KiwiBank provided a webservice API to pull these transactions down – of course that would be too good to be true. With an API ruled out that only leaves the front end.

The first option I looked at was a Python based web scraping tool called Scrapy.  It’s a really flexible powerful tool for parsing html.  As I started getting a grip on the syntax of Scrapy it became clear that it wasn’t going to do the job due to the JavaScript-heavy interface that Kiwibank uses.

The second option was browser automation.  To me this seemed like a less elegant option but after finding Selenium I soon forgot about that.  Selenium is a web testing and automation suit.  It consists of a number of components including a pretty extensive set of development libraries and interfaces. The two tools I used were Selenium Server and Selenium IDE (Integrated Development Environment) for Firefox.

 

Selenium IDE

The Selenium IDE Firefox extension allows you to create, record, edit and test Selenium automation scripts.

Start off by creating a new test suit and then a new test case within that suite.  Hit the record button and start recording your browser session.  Every action you perform in the browser will be recorded as a step in the script. This will give you the basis for the automation. Once you’re done recording you might need to manually edit, add or remove some steps to make the script more robust, or fix bits that don’t play back correctly. You can play the script back with the buttons on the toolbar or you can execute one step at time by selecting the step and pressing ‘x’.

Another extremely useful tool to help analyze page elements is Firebug for Firefox, it’s an excellent compliment to the Selenium IDE.

 

Getting Creative with Kiwibank Security

In an attempt to make their site more secure Kiwibank employ a two step authentication process. The first being AccessNo./Password and the second, a question/answer system which asks you to click the missing letters from the answer. This adds a slight level of security because it means an attacker needs to have a logger that’s a little more extensive than just logging keys.

Now it’s probably possible to get Selenium to read the question, work out which letters are missing and look up a table to determine which JavaScript should be called to complete the answer. And I may end up having to do that if Kiwibank reads this post! But fortunately for me Kiwibank allows you to set your own questions and answers. The questions all have to be different but the answers don’t. Simply setting all the answers to the same five letters means that I always call the same JavaScript.

To be honest it felt good the be the user, circumventing the security for a change!

 

Custom JavaScript

Selenium allows you specify a file with your own JavaScript functions.  The file must be named user-extensions.js. It’s location can be configured in the IDE under options/options. I don’t think these scripts can interact with elements on the page though. Someone please correct me if I’m wrong here.

I created a custom JavaScript function that returned the current date less x number of days given as a parameter.

 

 

 

 


I used this function to get the last 28 days when specifying the “from date” on the export selector.

 

 

Auto-downloading Files & Firefox Profiles

The whole purpose of this exercise is to automate downloading of transaction files so we need to tell Firefox to automatically save files of a certain type instead of prompting. We’d also like to save them in a specific location.

The best way to handle this is create a custom Firefox profile for Selenium to use just for this automation. There’s a great post here which details the optimum profile settings for use with Selenium.

The last thing you’ll need to do to the profile is make sure that it handles your chosen export file type correctly. In my case I’m using .OFX so I needed to tell Firefox to always download .OFX files without prompting. This is done through the mimeTypes.rdf file in the profile. Details on this file here.

 

If you keep getting the add-ons popup every time you use the custom profile I found the following fix:

To disable add-ons window which appears every time when Selenium scripts are run on Custom Firefox Profile.
Close all instances of Firefox browser and delete the following files from the Custom Profile folder

extensions.cache
extensions.ini
extensions.rdf
compatibility.ini

This should reset Extension Manager and disable add-ons pop-up.

 

Selenium Server

Now with a fully working script and customFirefox profile in hand we can set about scheduling this automation with the Selenium server and the Windows task scheduler. The Selenium server would normally be stated and left running like any other server application. In our case we’ll just start it, run our script and then exit.

Once you get the command working at the command prompt you can then use it in a scheduled task running under it’s own user account. If you do this, everything will run in the background and you won’t see any windows pop up and it will run even if no one is logged on to the PC.

Here is the command I use to run my Kiwibank automation:

java -jar selenium-server.jar -singlewindow -htmlSuite "*firefox" "https://www.ib.kiwibank.co.nz" "D:\Selenium\Kiwibank\SeleniumSuitKiwibank.html" "D:\Selenium\Kiwibank\SeleniumSuitKiwibankResults.html" -firefoxProfileTemplate "D:\Selenium\FirefoxProfiles\Kiwibank" -userExtensions "D:\Selenium\JavaScript\user-extensions.js"

Notes about the command:

  • Specify the test suite not the test case
  • Specify a results file. (I haven’t need to look in it though)
  • Specify the user extensions file if you have one
  • Specify the location of your Firefox profile

That’s it. Fully automated transaction file download! Make sure you observe good security practices with this sort of stuff Principle of Least Privilege.

4

Service Wrapper for oscam

Ok this is one of those “if you need it, you’ll know what I’m talking about posts”!

I recently started using oscam and being that I don’t like sever applications that need to run in the foreground I wrote a small windows service wrapper to handle oscam for me.

Just drop it in your oscam folder and install it by running oscamSVC -install

Using oscam as a system service also overcomes the issue that some people have found with  some USB card readers such as the Omnikey disconnecting or ejcecting when a remote desktop connection is made to the PC running oscam.

 


 

Version History & Changes

0.3.5.0 – Current version

  • Added settings form (Loaded by starting oscamSVC.exe without any parameter)
  • Added start-up delay option
    Use this option if you see card detection errors in your oscam log. I did – even though I made oscamSVC dependent on the smartcard service.
    Adding the delay just lets Windows start a bit more which for what ever reason seems to help. It will be dependent on your reader.
    During the delay the service is kept in the ‘starting’ state so you can still add service dependencies using regedit.
    e.g. MediaPortal TVServer depends on oscamSVC.
  • Changed command line options from slash to dash
    (i.e oscamSVC.exe -install instead of oscamSVC.exe /install)

0.3.0.0

  • Initial public release

 

 

 

Requires .NET 2.0

Enjoy.

oscamSVC
Version: 0.3.5.0
Updated: 2011-06-25
Download: oscamSVC.exe - 23.5 kB
9

Fingerprint Logon with hp ProtectTools -Fixed

 

I just got a new hp 8440P EliteBook(Windows 7×64) and wanted to use fingerprint logon. To cut a long story short:

  1. Installed latest ProtectTools security manager v.5.1.1.744
    Result: ProtectTools is installed but no fingerprint options.
  2. Installed Validity Fingerprint Sensor Driver v.4.0.15.0
    Result: Fingerprint options are visible in ProtectTools and fingerprint enrolment is successful but no option at windows logon to unlock with fingerprint
  3. Un-installed ProtectTools v.5.1.1.744
  4. Installed ProtectTools v.5.0.4.669 (The version I have on my other Elitebook which works)
    Result: Windows logon now displays the swipe to logon option but when you swipe it throws an error: “the system cannot find the specified file“.
  5. Installed ProtectTools v.5.1.1.744 again over the top of v.5.0.4.669 which upgrades v.5.0.4.669 to v.5.1.1.744
    Result: Everything is working as it should!
8

Vmware Player Unity (Trackpad Scrolling Works!)

I like to use VMwaer Player to compartmentalise different tasks on my laptop.  For example I have a Visual Studio development virtual machine.  It’s got all the SDKs and tools I need for development.  This has a couple of advantages:

  1. It keeps things tidy, I don’t have to have all sorts of SDKs and tools on my local OS,  at the same time my development environment is also kept clean
  2. The development environment is very portable.  If I want to rebuild my laptop I can just backup the entire VM and I don’t have to worry about setting it up all over again

This only issue I’ve had is that I’ve never been able to get trackpad scrolling to work within the virtual machine.  This makes VS  development a bit painful!

One of the new features of VMWare Player 2.x is “Unity”, it allows you use apps within the guest VM as if they were running on your local OS.  Once you enter Unity mode you will see the Virtual Machine window minimnise and all the apps that you had running inside the VM appear in your local taskbar. It’s a bit like the way Citrix seamless window app publishing works.  In Unity trackpad scrolling works!

Now you get the best of all worlds – compartmentalisation of environments with a native OS feel and trackpad scrolling!

0

SPDIFKeepAlive 1.2

I finally got around to fixing the auto play on start-up problem. The reason it took so long was because I was trying to perfect the auto restart option:

Sometimes when an AC3 file is played or something that takes control of the sound card SPDIFkeepAlive gets stopped. The auto restart option restarts playback every 3 seconds. This means that shortly after AC3 playback stops SPDIFKeepAlive resumes automatically.

Unfortunately I’ve still not got it working properly, at least not for me, depending on your sound card / driver you might get better results.  But be warned, on my system after a random amount of time (~1 hour) I get a terrible noise produced by SpdifKeepAlive and I have to exit, hence I don’t use this feature! I’ll try to fix it sometime. I think I’ll need to use threading and play 2 files that continuously overlap.

You can download the new version at the bottom of the SPDIFKeepAlive Post

6

Dual Cloned Windows Install with Linux GRUB

Ok this might be old news but I had some trouble finding a simple straight forward guide.

I just got a new laptop at work and I wanted to run to two completely separate copies of Windows XP using GNU GRUB as the boot loader.

Grub Boot Screen

These are the steps I eventually followed:

1. Completely clear the drive, all windows partitions, recovery partition etc.

2. Run XP setup, create a new partition leaving enough room for the second copy of windows; install Windows as per usual.

3. Install all drivers, updates and any software that you want to be on both copies of windows.

4. In windows Disk management create a new NTFS partition in the remaining space on the disk, leaving enough space for GRUB bootloader, I left 100MB,

5. Run Symantc ghost (or some other disk cloning tool) and clone partition 1 to partition 2. You will now have 2 identical copes of Windows on the same drive.

6. After cloning don’t reboot into windows, instead boot from CD to you’re favourite Linux distro. I used Ubuntu 6.1 which boots live off CD as part of it’s install process; I’m sure you could use knoppix or whatever.

7. Create a Linux partition in the remaining space using fdisk or cfdisk; flag this partition as the bootable partition. Assuming the disk is /dev/hda; XP1 and XP2 are /dev/hda1 and /dev/hda2 then this partition will be /dev/hda3

cfdisk /dev/hda

.

8. Format the new Linux partition with

mkfs.ext3 /dev/hda3

.

9. Make a new directory and mount the partition to. For this example e.g we’ll mount it to /mnt/tmp

mkdir /mnt/tmp
mount /dev/hda3 /mnt/tmp

.

10. Install Grub to the partition. This Will install grub to the root of /dev/hda3 and to the master boot record of /dev/hda

grub-install --root-directory=/mnt/tmp /dev/hda

.

11. Create a grub menu list file in in /mnt/tmp/boot/grub/
Use vi or nano to create a file in this location called menu.lst This will contain a list of operating systems you wish to boot. The file should look like this:

timeout 60
default 0

title Windows XP - Work
unhide (hd0,0)
hide (hd0,1)
rootnoverify (hd0,0)
chainloader +1 

title Windows XP - Home
unhide (hd0,1)
hide (hd0,0)
rootnoverify (hd0,1)
chainloader +1

.

12. Now unmount /dev/hda3, remove the linux CD and reboot. You should now get a grub boot menu where you can choose which copy of XP you want to load. The the hide and unhide commands for each OS entry in grub mean that the that each copy of windows won’t be able to see the other.

13. (Optional) To add a nice background to the to the grub menu, boot back into your live Linux distro and use firefox to download a grub splash screen. I got one from here, they also have a guide to create your own. Again mount /dev/hda3 and copy the splashimage to /mnt/tmp/boot/grub. Edit the menu.list to include the following line:

splashimage=(hd0,3)/boot/grub/myfile.xpm.gz

.

That’s it! 2 Copies with windows compleatly hidden from each other wih a nice menu using the renound GNU GRUB bootloader!

*UPDATE*

DON’T USE HIBERNATE WITH DUAL BOOT.
At first it seems neat to be able to choose which OS you want to resume but then the disk corruption starts!

5

SPDIF KeepAlive for Home Theatre PC

Optical SPDIF ConnectorAfter building a new Home Theatre PC I’ve discovered that the onboard IDT audio has a problem with the SPDIF output, or at least my Sony receiver has a problem with it! Every time a sound is played it causes the SPDIF input on the receiver to initialise which takes about 500 milliseconds, after the sound has finished the SPDIF goes back to sleep. As a result the first 500ms is lost off every sound that is played; not really a problem if you’re watching a movie but for applications that have little blips as you navigate around these sounds tend to get missed altogether; such is the case in MediaPortal the HTPC application I use.

My old motherboard with Nforce sound didn’t have this problem the SPDIF remained “active” all the time.

After much searching I did find a few other people with the same problem but no solution so I’ve written a small .NET application called SPDIFKeepAlive. It does just that. It sits in the system tray and continuously plays a silent wave file to keep SPDIF port active.

SPDIFKeepAlive Settings

Hope this is helpful for others, leave comments for bugs / feature requests etc


Version History

Version 1.0

  • Initial release

Version 1.1

  • Added new output options

Version 1.2 (Current Version)

  • Fixed auto play on startup
  • Added auto-restart option
SPDIFKeepAlive

Version:
Updated: 2010-11-08
Download: SPDIFKeepAlive-1.2.exe -

(.Net 2.0 Required)

Source Code – Feel free to do what you like with this (no license)

SPDIF Keep Alive Source
Version:
Updated: 2017-03-12
Download: SPDIFKeepAlive-1.2.exe - 2.42 MB

48