Pete's Wordshop

It’s summertime, summertime, sum, sum, summertime…..

Right!  Time for fun in the sun!  Given the time lapse since my last post, you’d think that I had gone on vacation.  Not so!  I have been busy, mostly with some challenging projects both in the tech world and outside it.  I have been to Italy and back attending the Common Europe Congress in Milan.  I have written a lengthy magazine article which forced me to grow my mobile app development skills.  I have been invited to present a mobile application development workshop at the Common Fall Conference along with presenting on Open Source Report Writing tools and using Open Office with IBM i data.  So, it looks like there is no rest to come this fall.  Add travel, a couple of pro-bono jobs (like I can afford to give away my time…) and I can see the summer gone in a flash.

Anything new besides the same ol, same ol?  Yes.  I have been playing with Liferay which is a pretty cool and easy to use portal application (if there is such a thing).  I have been trying to sort out the CMS and the portal plugin pieces because they would be of the most use to me directly.  Redesigning the valadd.com and the petesworkshop.com sites along with assap.com and the opensource4i.com (and it’s variations) sites is HIGH on my list, I just can’t afford to hand rewrite all of them. I was hoping to “portalize” them and consolidate them into a few portlets.  We’ll see.  The summer is quickly fading…

IBM seems to have fixed the SWMA mysteries and I am very hopeful that my next round of renewals (next year) will be problem free…

Still much to do.  I have an exam to pass to get certified as a GIAC Secure Software Programmer-Java and my time is running out.  I’ll update you by August 12th (if not sooner).

Before I begin this rant, let me say that IBM, the IBM I grew up with, has been an amazing company to have a developer relationship with.  I have said it before and I’ll say it again, nobody does support or supports it’s developer community better than IBM.  But Rational, well, not so much…My mantra with HP products is: “Great hardware, crappy software”.  With IBM (ir)Rational my mantra is “Great tools, crappy processes”.

If you haven’t been an IBM i developer then you have missed out on some of the most marvelous marketing missteps on the planet.  Our hardware has gone through multiple name changes (not SO unusual) and our Operating System name has had, to paraphrase the late Al Barsa, “more names than Elizabeth Taylor had husbands”.  We have also experienced the thrill of “unbundling” where the heretofore “all you can eat buffet of developer tools” was reduced to an A la Cart sampling.  This was done, as IBM claims,  to benefit us developers and this is “what we asked for” (yeah, right).  Add to that a set of PC based, GUI development tools that evolved from WDSc (Websphere Development Studio Client), to RDi (Rational Developer for i), to RDp (Rational Developer for Power) and next (yes, rumor has it maybe another name change)…As those names evolved so have the components that have been removed, added, deprecated and enhanced. It is really difficult to keep track of what the current version of tools are and what they include.  You couldn’t design a more daunting environment to navigate when it comes to development tools (And, I am not even going to try to explain the compilers that must be purchased separately.)

So I have come to the conclusion that developing software for IBM i isn’t for wimps.  The tools are great, if you can keep up with the names and the changing features.  But when navigating the SWMA path, things get ugly (I’ll get to SWMA in a minute)

But wait, there is more. IBM also has an awesome developer discount program so that you can purchase the hardware and software at low rates.  In most cases the OS and software components are next to free. All,that is, except the Rational Tools.  For that you pay real money.  The process of configuring and purchasing through the developer discount program is really a breeze. The folks who put together the configurations are helpful and patient and the order and delivery is nearly flawless.  A couple of years ago, though we ordered a Bladecenter S and a JS12 running IBM i, everything we needed showed up properly configured and ready to go.  All the hardware was there and all the software was there which is no mean feat when the hardware itself has changed so radically and IBM i on the blade runs in virtual operating environment.  Even support was OK to navigate, given that the hardware and software came from the “p” (Power) world and the IBM i world.  The REAL challenge was yet to come.  SWMA renewal…

SWMA is the software maintenance agreement that keeps support and software updates current.  Now, I have been on the phone with IBM probably only a dozen times over the 15 years I have been in the software developer program with IBM.  I measure *years* between calls, that is how stable and trouble free the i is.  For me SWMA allows me to stay current with software updates, period. But staying current on SWMA is not easy in the developer program.  IBM doesn’t notify you when SWMA will expire (actually it does, NOW) and the process of renewing SWMA is cumbersome and confusing.  In a perfect world, you would receive a notice which lists all the software and charges and has a link that says “Click here to renew”.   Not so with DD (developer discount)  SWMA.  First you contact IBM.  Next they send you a listing of all the hardware and software you have that is associated with the machine serial number (a mind numbing list of part numbers and ID’s) and then that configuration is priced and a quote sent to you.  The challenge is that if you assume the listing is complete, you may have missed something important. [Quick aside: When it was all bundled together, the SWMA was quoted on the bundle which made it easy.  Now SWMA is quoted separately for some of the unbundled items].  In my case, because the Rational tools had gone through a renaming and product ID change, it didn’t show up on the renewal list so my Rational SWMA didn’t renew (sigh)…

So let’s get to the meat of it.  IBM (used to) not notify of renewals but at least when it was renewed, it was simply renewed as a bundle.  Not so any more. The SWMA renewals are A la Cart.  Here is my specific scenario when it came to the attempt to renew SWMA on RDp (and this truly illustrates the irrationality of the process)

1. The SWMA had expired.  I had to pay the after license rate (a 200% premium) in order to ‘renew’ which came to $535.
2. I could buy a “retail” version of RDp for $886 and then *next* year my SWMA renewal would be $117.
3. Even if the SWMA hadn’t lapsed, I’d still have to pay the $535 to renew because the configuration tool that IBM uses (e-config) doesn’t have a product code for RDp *renewal*, only the after license upcharge.
4. If I had a license for the ‘OLD’ version of RDp (which is RDi) then I *could* ‘renew’ that for $159 even though it has been withdrawn from marketing.

So I pay a penalty for being on the developer discount program when it comes to renewing SWMA on Rational developer tools.  I pay no such penalty on any other software product on my IBM i.  Something doesn’t seem quite right with that.  All of this comes after hours of talking with my IBM contact, the person who configures the i and SWMA, and a very helpful Rational Sales rep (on the *retail* side of the house).

So what am I going to do?  Well, turns out that when we bought the JS12 we bought TWO licenses of RDi.  One license was upgraded to RDp (within the SWMA window) but the other license was not upgraded.  Turns out that RDi license can be upgraded to the new version of the Rational Developer tool (whatever it will be called) for $159.00.  So my inclination is to pay the $159.00 to upgrade the RDi license and just live with the older RDp 7.6 software we have since it would cost $535.00 to upgrade that (Yes, I know, the older, expired software license is cheaper to renew than the newer license and that doesn’t make sense).  But, as the guy using the configuration tool said “I can only do what the tool lets me do”.

Apparently this particular, very helpful person said that Rational is working to resolve the issue, too late for my current SWMA renewal but maybe BEFORE my SWMA expires on “Open Access RPG” product we own that is, alas, on yet another separate SWMA contract.

A Smarter Planet?  A Rational Tool?  I don’t think so.

A long while ago, maybe 4 years ago, I installed Asterisk, an open source voice mail and call management system.  It is a little bit of overkill for a small office like mine and although I also host my home voicemail on it, it really isn’t a necessity.  But I installed it to get familiar with it in anticipation using it as the IVR engine in my attendance management system. It has worked well but after two hacking experiences in the past 6 months I figured it was time to upgrade and perhaps find an easier way to install and manage it.  AsteriskNow is a “pre-packaged” installation of Asterisk and a Linux distro called CentOS and is supposed to make the whole experience easy and painless.

In my first attempt I chose to install Asterisk with a web GUI called FreePBX.  The installation was a no brainer.  Skills required are the ability to push the open/close drawer button on the CD/DVD drive and be able to use a keyboard. If you can’t do that you shouldn’t be near a computer in the first place.  I will say this: AsteriskNow is by far the easiest way to get an Asterisk installation installed on Linux.  Period.  It is perfect.  But the EASY part stops there.

First a little background on configuring Asterisk.  Everything is handled by configuration files. Everything!  For Linux geeks this is nirvana.  Grab your favorite text editor and away you go.  The default installation of AsteriskNow includes no desktop application so you will be hacking files with a command line text editor (whoopee!).  My first go around with Asterisk four years ago took just that approach.  Tediously configuring Asterisk using a text editor and a lot of luck.  The good news here is that there is plenty of good documentation on how the files all hang together.  The bad news is that if you fat-finger something it can take hours to sort out just where you went wrong.   Enter FreePBX and Asterisk GUI!  These web front ends are designed to make the whole thing quick and easy.

After my first install of AsteriskNow/FreePBX I brought up the web interface and then did a quick search to find the step by step documentation on how to use the GUI to configure Asterisk.  Nada! Everything I came across was incomplete or out of date so after poking around for an afternoon, I decided to give up and reload, this time with Asterisk GUI as the web front end.

At least Asterisk GUI uses the base Asterisk config files AND the Asterisk GUI has an easy way to get logged in and change the admin’s password (which was almost impossible to do with ease using FreePBX).  So I was off and running!  The menu layout from top to bottom is basically the ‘step by step’ so I moved down the menu tree working through the config.  The context help is good but again there is VERY little documentation on how to use the GUI and the few places I got stuck I really was stuck.

I finally got most of the steps completed but I couldn’t make any outbound calls (I *could* call other extensions) and none of my inbound calling was working either.  In frustration I posted some of my questions to the Asterisk forum (which happened to be unusually responsive) and got to the point where I just decided to reload.  Which I did.  I made a couple of tweaks, broke it again, and reloaded again (see, it really IS easy to install, thank goodness).  And then using all I had learned in the prior attempts, made one last effort to sort out my issues.  Here is what it came down to:

I was thrown by the Module Name on the hardware config page.  It had ‘wctdm24xxp’ which seemed to point to a different card than I had installed (TDM400p).  Based on the Asterisk forum I changed it to ‘wctdm’ which was supposedly the correct module name for my card but every time I did that and rebooted it basically hosed up my install.  So, DON’T change it!

I tracked down my issues with the inbound and outbound calls to an incorrect configuration that the Asterisk GUI was placing in the extensions.conf file. So it was back to editing the config files manually!  In the globals section, the GUI had configured the two trunks I had as trunk_1 = DAHDI/1 and trunk_2 = DAHDI/2 I think those should have been either DAHDI/G1 and DAHDI/G2 OR I just set them to DAHDI/3 and DAHDI/4 which should point them directly to my FXO ports 3 and 4.  The G1/G2 syntax is to automatically chose the highest available ports but something is broken there.

The second issue I had with the inbound calling was that two entries that Asterisk GUI added to the contexts broke the inbound answering.  Those two entries are:

exten = s,1,ExecIf($[ “${CALLERID(num)}”=”” ]?SetCallerPres(unavailable))
exten = s,1,ExecIf($[ “${CALLERID(num)}”=”” ]?Set(CALLERID(all)=unknown <0000000>))

I just commented them out.  Not sure what those commands do, except break the inbound contexts I had.

With those things out of the way, the rest of Asterisk GUI really DOES make it easier to manage your Asterisk installation but, MAN!, you really have to stick with it.  I haven’t had a chance to explore any other features yet; I have already flushed the better part of two days sorting out the GUI.  But, all is well now. So, call me (and you’ll get my voicemail!)

The next steps are to go back and secure the server again before I open it back up to the Internet.  The bad guys are always knocking….

No.  It isn’t a reference to the 1951 science fiction movie, it is a reference to last Friday when my Internet connection slowed to a crawl and despite the best efforts of Comcast, I continued to experience outages and slowness.  Time to take matters into my own hands.

I started with taking a look at the firewall, in my case Small Business Server 2003 (ISA 2004 I think). Plenty of activity there with Network bandwidth utilization running at about 5% but as I watched I saw a weird pattern emerge.  My pings to an external server began to time out as the network utilization spiked and then I got long latency on returns. Then the network utilization dropped for a minute or so and began to ramp up again. A weird pattern.  Of course, figuring out what was causing the traffic spikes is next to impossible with the tools ISA has but through trial and error I discovered my Asterisk server was the source.  Looks like I was hacked. And even through only SIP traffic was allowed to and from the box, there must have been an exploit that someone used to push files on and off the server.  So I pulled the plug (I have rebuilt with AsteriskNow).

On Monday, late in the afternoon we had a wind storm come through Salt Lake (69 mph nearby) and the lights flickered for only a second.  I really didn’t give it a second thought, my servers have redundant power supplies and UPS’s.  A few moments later I lost Internet connectivity and thought that I had been hacked again (I have another Asterisk server that isn’t exposed to the outside world…but, you never know) so I went down to the server room to take a look.  SBS 2003 was rebooting but the blade center was merrily humming along.  Looks like I need a new UPS battery.

Tuesday morning I discovered that the JS12 blade in the blade center had dropped as well and my IBM i wasn’t running (nor the Linux instance running Asterisk and Nagios) so I am a bit baffled by that.  A almost instantaneous drop in power dropped two servers, both of which had UPS’s attached (one of them with TWO UPS’s attached).

So the day(s) stood still while I diagnosed and fixed dropped and hacked servers.  So much for productivity increases due to technology.

Since I am a developer, I don’t usually go out of my way to make disparaging remarks about other software companies.  Oh, we programmers ARE an opinionated bunch, but we ALL have been humbled by the complexity of the environment we work in and the difficulty of accommodating the non-standard “standards” that apply.  But our basic mantra is: “Do no harm”.

I have encountered the egregious breach of that mantra just today.  Read on and have caution!

About a month ago a purchased a new laptop, as my Gateway was LONG in the tooth and was lumbering under the bloat of the latest development tools and frameworks.  It is an Acer Travelmate 8572.  With an Intel i7 64 bit processor (Quad Core!) I expected some great gains in speed and productivity.  Alas, I am not completely happy with it, even at 64 bits it won’t support more than 4GB RAM and the graphics resolution isn’t as high as I want  (but that is all another story).

I had installed Windows 7 Ultimate and have had very good luck with it.  Some incompatibilities and hassles, but overall, a good experience.  However, in the last few days my bluetooth headset has been acting flaky so I decided to install some updated bluetooth drivers.  No joy, I still had intermittent loss of connection.  So I decided to remove the device drivers, reboot and then reinstall.

Removing the drivers, no problem.  I rebooted and then attempted to reinstall the software.  It couldn’t find the hardware. DOH!  Function-F3 activated the hardware and the software started to install,  but first it began by uninstalling the software.  Progress bar got to about 75% and then just sat, and sat, and sat.  I saw plenty of hard drive activity but no progress.  So I decided to check email while I was waiting.  The shortcut said it couldn’t find the thunderbird.exe file that was referenced.  That’s weird!  I followed the link to the folder for Mozilla Thunderbird.  Not there! NO FOLDER!!!! Then my icons started to disappear off of my desktop.  So, I immediately shut down the system.  I restarted and now the entire desktop was gone and I was down to just some basic folders and programs on the start menu.  Basically nothing left in the Program Files folder.  The system was hammered.  After pondering what I should do and thinking I had picked up some nasty malware I started a System Restore.  It ran for about 45 minutes.  While I was running the restore, I decided to Google WIDCOMM and uninstall and delete and I found this:

http://social.answers.microsoft.com/Forums/en-US/w7performance/thread/ffe5f236-da04-4c3c-8df9-696080ec9d0c

Basically the WIDCOMM uninstall will go rogue and delete the data on the C drive.  This is confirmed by Microsoft (signed driver!)  and so MS is an accessory to hard drive murder IMHO.  The negligence of WIDCOMM in producing such a flawed POS (piece of software) is unconscionable.  These folks apparently are clueless software developers and, no doubt, the software was probably developed off shore by low cost programmers in order to maximize their profits.

The System Restore did very little to get things functioning again.  So, now I will have to reformat and reinstall EVERYTHING!!  Thanks Widcomm!

Anyone game for a Class Action Suit against Widcomm?  I’ll lose at least a day of productivity.  That is worth a few bucks to me….

You probably won’t notice but I have changed the server running this blog.  We now have Zend Server running our PHP applications rather than Zend Core.  Lost on you but not on me is that the configuration is now MUCH simpler to maintain.  I am not managing multiple Apache instances and no longer have to deal with PASE.

As an FYI to others that go this route, just treat the http.conf file like any other Apache configuration file and all will be well.  Very pleased that the move was so trouble free (well, once I figured it out).

Next challenge is the upgrade WordPress itself.  I have usually ended up running WP updates manually because of of authority issues with the way the Automatic Update routine runs, but you can pretty much guess that the next post will be on WP 3.0.  Now that the management of the i instance is so much easier, I plan to move the rest of the PHP and Apache applications to the i.  If I could get Glassfish to run on i, I’d move my Glassfish apps as well.  Apache, PHP, Ruby, JRuby and Tomcat all run nicely on i.

The final piece of the IBM “it’s broke please fix it” list was getting Websphere Application Server Express Edition V7 installed and running.  What triggered this “simple” activity was that there was some discussion of source management and version control  tools on the midrange.com list and Rational Team Concert was mentioned as something folks didn’t have much experience with.  So I decided to check it out, especially since RTC is free to use for Open Source projects which was exactly what I had in mind.

However, RTC relies on WAS and my past experiences with WAS have been anything but pleasant.  I have run Tomcat and Glassfish as servlet containers and they install smoothly and run great. WAS, on the other hand, has always been a challenge to install and run.  But, if it is required it is required, so I started down the simple path of getting WAS 7 Express installed.

WAS on i installs multiple ways.  I first took the approach of running the installer from Windows, which *seemed* the easiest.  The installer ran until about the 40% mark on the configuration step, and, 8 hours later, still hadn’t progressed but had not errored out either.  So I shot the installer and restarted.  I quickly learned that it wasn’t quite so simple to end and restart the install.  The installer complained about a .lck file in the ‘tmp’ folder that had to be removed.  However, the installer wasn’t intelligent enough to give me the full path as to where to find this file (as an aside, this ALWAYS aggravates me about installers.  They rarely tell you where they are stuffing things even though they may ask you where to install the software).  After a full scan on my laptop hard drive looking for the mysterious file, it occurred to me that since the installer was installing to the i, perhaps I should look there.  Voila! So I cleaned up the installer files in the IFS and restarted the installer.  Same result. So I cleaned up after install #2 and decided to do some more reading and perhaps download the CD images from the IBM web site to make sure I had the latest and greatest.

And yes, I DID review the prerequisites and made sure I was current on PTF’s and groups.  I had downloaded the WAS Group as well, ready to install it once I had the product installed.  I tried the GUI install one more time with the new CD’s I had downloaded, same result (#3). The next approach I tried, this time using the new CD images, was to attempt an install from PASE.  Although not as sexy as the GUI, I was hoping I would get better feedback on what was going on.  This install (#4) again reached the 40% mark on the configuration step and, after about an hour, dumped an error message to the console (at least I got an error!).  Install fails on configuration action 80Fos400.

Googling turned up very little.  I again revisited the requirements and double checked them.  After one more try (#5) I then submitted a PMR and began  to work with IBM trying to solve the problem.  I will spare you the details, but the next 5 attempts also failed (so now we are up to 10 install attempts).  Finally, an IBM’er jumped on the box which led to a couple of revelations:

1) Even though all the licensed programs indicated that they were happily installed, a command called CHKPGMOPT indicated otherwise.  That lead to two PTF applies that wouldn’t have been uncovered any other way.

2) Although I had downloaded the WAS group PTF, my attempt to apply it resulted in a error about the product not being installed (DUH – I hadn’t been able too).  However, it turns out I was taking the wrong install approach on the group.  I need to take option 8 on the PTF  menu.

3) My path environment variables weren’t correct.  I sorted them out and I doubt it contributed to the fix but, what the hey…

With those actions taken, the install just zippity-do-dah’ed along.  WAS is now installed and happily sucking up memory and CPU cycles.  I guess that the lesson here is that even if you RTFM, the devil is in the details of carrying out the steps.  Counter intuitive as it was, installing the WAS group BEFORE installing the product was the way to go.