Tools and Automation

Project Name: Tools / Automation
Language: C# / PowerShell / Misc

GitLab - Multiple Tools: gitlab.com/immersivegamer/tools
GitLab - LANSA Tools: gitlab.com/lansa-tools
GitLab - Node Monitor: gitlab.com/immersivegamer/node-monitor
GitLab - Snippets: gitlab.com/dashboard/snippets

Snapshot: I enjoy creating tools to help with my day to day jobs or help others. Above are the links to source code you can check out or read below where I talk about some of them.

Tools Repo

This is just a repo with a bunch of tools I've created. Some have been used in production for automation (for example csv-to-xml) and others for personal use (csv-to-input).

These are my favorites of these tools:

csv-to-input

This tool I created for my wife. Their office was enrolling in a new service that would automatically capture debt from any clients that had debt with their agency, however they were daunted with the volume if they had to manually input all the data into the website's form entry.

They considered hiring me to create a tool to interface with the service's FTP drop off which would require XML files. Taking a look at the specs and timeline it was quite out of scope. I asked how many records they needed to port and how many new records they would get each period. It was ~500 records to port and < 10 per period.

So instead of doing the full integration I switched gears and created a tool to read a CSV file which would then for each column do inputs, tabs, and enters to automatically enter the form. I made the configuration editable via a JSON file. Gave the program to my wife with a quick into on JSON syntax, and what would have been tens of hours of manual entry was reduced to a couple hours.

mysql-restore

At Suttle-Straus we started leveling up our development practices and starting doing local development on people's machines instead of stepping on each other's toes on a single dev server that we had to SCP our files into.

Another co-worker wrote up instructions on how to do the database restore on our local machines. The problem was that when I needed to do a restore I would sometimes make a mistake causing a wasted hour or two(we did a pretty much full production restore due to multiple tenancy setup of the website). Or I would forget to change a configuration like the base URL for the website so when testing I would pop over the live site or an integration would fail!

So I started with a simple PowerShell script to do the commands for me. Eventually I expanded it to take an XML configuration and to allow pre and post script execution, and do variable replacement. I also did some research on limiting dumps speeding up the process to ignore certain unended tables. With the script I got a consistent 30 minute refresh whenever I wanted.

I also started using this script to refresh lower server environments which before hand was a longer manual process. I've also used this script to do automated restores from backups at other places for reporting reasons.

xml-distinct

This was a simple tool. But a fun one to write with C#'s LINQ XML libraries. There was a legacy system that outputted XML files that we processed. However, we had no accurate documentation on what the schema was. We were modifying/creating a new integration and needed to know all possible nodes. While we did not have correct documentation we did have years and years worth of log files with the XML data saved.

The tool reads each XML file and then combines it with the next to build up a full XML. So if one XML used a node but another one didn't the final XML output would have both, plus an example of data.

So while we didn't get a real schema, we got as close as we could which helped with the project.

LANSA Tools

At Demco we use LANSA for the ERP development. LANSA has their own IDE but it can be slow and sometimes buggy. I enjoy using Visual Studio Code so I created some tools to help.

The first one is a formating tool in PowerShell. When you copy RDMLX code out of the LANSA IDE it copies it with out formating! So this is just a simple script to indent code based on key words and remove duplicate lines. I wrote it in PowerShell as I wanted to be able to use it across multiple programs.

The second was RDMLX code highlighting in VS Code. I didn't feel like creating a textmate grammar syntax from scratch, so I edited an existing language. It works actually very well, tempting me to do full coding in VS Code. It has actually helped me catch a bug as string values (ALPHANUMERIC) can be unquoted.

The bug was like this:

* set variables
Change Field(#FLAG) To(2)
Change Field(#STATUS) To(3)

If Cond((#FLAG *EQ 2) *AND (STATUS *EQ 3))
    * do process
Endif

#FLAG is a variable, being compared to the number 2. STATUS is supposed to variable too, it has the same name as an existing variable in a program but because it doesn't have a # it is actually just a string value and so will never process as true! But with syntax highlighting it looks like this:

Now clearing showing that STATUS is actually a string value and not a variable, which the current LANSA IDE does not support.