Data Analysis and Visualisation

Sin vs Sin

With the direction I'm seemingly headed in with Python at work; evaluating large amounts of survey data in GIS for reporting back to engineers and superintendents – I've recently started a new Udemy course “Learning Python for Data Analysis and Visualisation“.

I am currently more interested in the visualisation and presentation side of things than the analysis as I've already sorted the analytical side out  – just need to find a more efficient way to prepare and present it. Naturally I am still quite happy to learn more data analysis skills, especially with Python so not skipping any pieces of the course that's for sure!
The course itself is easy to follow, the instructor uses an iPython notebook, unsure exactly how this works but given it requires the user to visit a localhost web page; I'm just going to assume it is running some sort of local server – which means I cannot use it at work without breaking one of four hundred IST protocols, so, spent a bit of time getting it all working in PyCharm; which wasn't so easy – particularly the installation of NumPy where I had to download and install the Microsoft Visual C++ Compiler for Python before it would work. A few videos in, and I now have a colour representation of Sin vs Sin and, am about to embark on Pandas.
A few links in here, be sure to check them all out 🙂

Stuck on something simple!

After a full day of coding and achieving some pretty cools stuff, I'm stuck on the simplest of things;
I want to append the new data to the end of the file, (easy enough) but I also want to move the current last word to the very end of the file after the append.

My plan is simple (enough);

  1. Open file in read / write mode (r+)
  2. Find the string “</EMBR_TRANSACTIONS>”, it's the last word on the last line of the file.
  3. Delete string
  4. Close file
  5. Open file in append mode (a) – so if it all goes west the new data is always at the end of the file and not inserted randomly in the middle
  6. Append new string data to end of file, new string data – through good planning and not coincidence – ends with the string “</EMBR_TRANSACTIONS>
Initially I had some issues with the correct file settings to use, once I deleted the entire contents of my (testing!) file. Now I am seemingly unable to delete the embr_transaction closing tag, and, when it appends the data, it's appending it like four times to end of file. I've done a 12 hour session so I'm just going to give it a rest for the night and see if the error kicks me in the eye first thing tomorrow!

One other thing; since updating my PyCharm; continuous memory issues – cleared cache, no good. Will look to alternatives solutions for that too I suppose.

Quick update;
managed to get what I wanted to achieve thanks to this post the result is below, final code (for the night) also. Unsure why, but it's still duplicating the entire file… fak.
Fixed the duplication problem; pretty special sorta error. Was well too complicated for what it needed to be. The answers on this StackExhange question were very helpful.

1:    def write_the_data(self, data_to_write):  
2: # Just appending the transaction to the end of the file.
3: self.f = open(LOCAL_DBS_FOLDER + mpc_constants.D_SLASH + self.ember_file_to_update[0:-4] + '.embr', 'r+')
4: self.string_data = self.f.readlines()
5: self.f.writelines([item for item in self.string_data[:-1]])
6: self.f.close()
8: self.f = open(LOCAL_DBS_FOLDER + mpc_constants.D_SLASH + self.ember_file_to_update[0:-4] + '.embr', 'a')
10: self.f.write(data_to_write + "\n</EMBR_TRANSACTIONS>")
11: self.f.close()

Doing quite well with Python… for work

But still haven't made a game (in Python, other than Flappy Richard!)
I've created a fairly extensive quantification and reporting tool for use in Pipeline industry, for this to work as I needed to I created my own .embr file format, it started as a fairly unique file but now closely resembles an .xml file – which means the only logical thing to do is to complete the transition and save the world an additional file type!

The .embr (or ember file as we affectionately call it) acts as a sort of bank transaction file for geodatabases. We had some fairly big problems to solve and this was / is the best way to do it. Specifically; we work with many geodatabases that store information about the pipelines and the infrastructure fittings. Other than the content each database is the same. Iterating through the database features allows us to export a snapshot of data at a specific point in time – exporting it to the ember file then allows us to easily present this data to the interested stakeholders – in any required format.

Is it just a bridge? No. The embr file also holds additional metadata / operational data that can follow the geodatabase throughout its lifecycle. An example of this is the alternate coordinate system. A simple fix to a buggy problem. Our work area covers two zones; 55 and 56. Our geodatabases are in GDA 94, a client specification to allow their data to span large areas and safely across multiple zones. The problem we encountered is simple, human, and removable. When doing calculations that require re-projecting to localised coordinates, often the incorrect zone was selected – having an alternate zone embedded in the ember file now means this tiny little step has been removed from all processing, and that particular human error has been removed.

I'm still working on the ember file, long term it is likely to be absorbed as an xml. It's still very much what I'd consider to be beta however we are using it in a production environment and this use is the driving force behind the direction the ember progresses; initially it contained a simple entry highlighting 2d vs 3d length to aid in mass error checking of survey data across multiple geodatabases in a short amount of time – now, as stated earlier, it is the backbone of an extensive quantification and reporting tool for multiple geodatabases.

November Update

Been very busy at work, needless to say Python has definitely made my life easier! Built some great reporting / quantification toolsets for the unique datasets we use, and, also started building a land drone and some Python based website projects – nothing major, just some skill-building stuff.

One thing I haven't worked on for a while is game building, however the skills I've learnt so far I'm applying to some GUI building for the work based Python applications I've whizzed up. My arcade machine is working but still lacks arcade buttons, joysticks and super duper light show – it's going to get some more attention during the Christmas heat!

Arcade frame arrived

My custom arcade arrived the other day from – for those wondering, it is _exactly_ as I expected!
It took a few hours to put together but was simple enough with clear instructions.
I still have to buy the buttons, controllers, lights and, fit it three guts of it. Am just planning on taking all that slow rather slowly for now. I've ordered green t-moulding from which I expect week be here by the end of the week.
Photos soon!

Building an arcade

As much fun as it is to be programming, I've made the move to start putting together my own home arcade machine. Today I order a two player upright, MDF cabinet from Arcadeworx 
I opted for the pre-built version simply because I already have enough on and don't want to get caught up in restoring a 'classic' cabinet.
It's anticipated this arcade will run on a Raspberry Pi – that's my initial thoughts anyway, I'm intending to be able to play some of the original old school games and also load it up with my own game(s)(which are currently just ideas).

Virtual Pet Game is finished.

Put in the hard yards and finished the virtual pet game, Module 3 of the Complete Mobile Game Development Course from Pablo over at Zenva Academy (Of course I'm going to advertise the hell out of them, look at the skills they're giving me dagnabbit!) – I'm very happy with the results so far. I seem to think the course is running a bit fast, but then there is so much repetition (which is a good thing) I am learning more than I know. I get to the end of a mini-lecture and think “wow, that's going pretty fast” but get to the end of 13 of them and it's all sunk in pretty deep.

Module 3, the virtual pet game, is built on the popular Phaser Javascript library, and runs in any browser. The module itself covered a lot of interesting techniques, including load, boot, gamestate screens, pre-loading images, displaying scores etc; you know, all those little things you wonder how-they-work? when playing a game. Setting up the background and pre-loading the images was easy, as was making Blobby Head move. In fact, the most part was quite easy with the tutorials and lectures (like this do that) – I am still coming to grips with the whole 'this' ie…. the this's made an appearance 105 times in the GameState.js file alone, you would think that would be enough to make it perfectly clear how and when to use 'this', let's just pretend for a second I have average intelligence and have a little trouble grasping new concepts!

Another really cool thing worth a mention; last night I was checking out the Disqus forums on Zenva and right up the top there was a short post from a guy called Ramón Lence Martínez introducing his app, “Plane!“. The bit that excited me, this is Ramon's first app, he learnt how to build apps from the same guys I'm learning from (Zenva, in case you missed it the first 800 times) and this proved a couple of things to me

  1. I've selected the right course,
  2. I've selected the right tutors,
  3. all I need to do is put in the time and the effort and shouldn't be too long until my own indie app hits the stores.
Anyway, head on over to Obsidibox to play the blobby head game; leave a comment below if you like it.

Blobby head

Made it back to the mobile game development course I'm doing on It's interesting as fuck and quite easy to follow. Currently at Module 3, creating a virtual pet – because a rabbit, cats, goldfish and guinea pigs just isn't enough for me, I need a fake pet as well!

You can see from the image, it's a pretty groovy little game. So far the blobby head can be dragged about and the items can be clicked. Not bad for half an hours work and 82 lines of code; important to note here, the reason there is 82 lines of code is because this game uses the Phaser Javascript library, making (what I would assume to be) technical challenges really quite simple.

The five lines of code above are all that is needed to add the blobby head, and make him draggable. There's also a couple of other parameters included that hold the values for fun and health levels.
Probably won't post another update until this is finished (Expecting Sunday, Minecraft pending)

Still coding

I use the term 'coding' in the casual manner, certainly not implying I'm coding in any professional scene, just thought I'd clear that up!
Work has been demanding lately, which is both good and bad. Bad because I haven't had a chance to pursue my own programming goals (for all the good that can be done with programming, I just want to make arcade games!). Good, because I've been able to use my coding abilities to further enhance my own productivity at work. ArcPy (ArcGIS Library for Python) mainly. This has also had flow on effects for my own programming abilities. I've had to rapidly learn about Github to enable versioning in a live development environment, my favourite tool at first was the “revert” tool – at least until I got the hang of branches.

Clearly, for me to say I've been too busy at work to be achieve any self improvement, would just be a massive lie! Should be getting back to some more fun coding soon.

cruising around the languages…

Going through a phase where I'm exploring languages and their uses. C, C++, back to javascript and HTML… no real purpose other than to entertain myself at the moment. Still keen to make the RetroPy arcade – feel i need a little more exposure to different levels of programming first.

Here's a little something I've been working on today, nothing much (other than an awful lot of self praise!). Just a little frogger clone called Hoppy 🙂