Showing posts with label release management analysis. Show all posts
Showing posts with label release management analysis. Show all posts

Tuesday, July 22, 2014

Work: Enterprise Defect Management (rant)

I have worked with defects my entire professional life in support, integration, project management, and release management. I found that the size of the team makes a large difference in the management of defects. In smaller companies, there were few factors to determine the priorities of defects. Simply, the client's most urgent issue is the highest priority. The reason for this is because typically there is only a single person working on each project and each project has a single client. Although a person could have multiple projects, project priorities are typically managed by the boss (i.e. manager).

As companies grow in size, there are more people involved and more priorities to manage. Management of defects move towards another team that does not have easy access to development, testing, and deployment planning. Unlike its smaller company counterpart which typically will only have 2-3 roles involved, now I have to deal with several parties. Depending on the size of the projects, there may now be up to 10 different groups each with their own resources and priorities.

One of the biggest problem that I run into is finding a single method on how to manage the defects. The largest factor to my problem is that there is no single entity that makes the final call. Because of this, there is a lot of delays to the decision because the work cannot "start" until the decision is made. Although the official start is not determined, some teams take it upon themselves to work on certain items. This actually causes more problems to managing the defects. Because a change is simple for a development group, the amount of testing is not necessarily simple nor the deployment.

A simple change in a more fundamental code base could trigger a lot of tests required. Because the lab environment only has a single instance, production may have to deploy over a span of time because it has several environments. And so far, only a single path for a single change.

If multiple development groups provide multiple solutions, there are a lot of resource conflicts that can occur across many teams especially testing. For example if ChangeA is put into testing and requires 2 weeks to deploy to production, ChangeB cannot be put into testing until ChangeA has completed testing. ChangeC cannot be put into testing until ChangeB has completed testing, but ChangeB cannot leave testing until ChangeA is completed in production.

So far it is manageable although it does not address how it can be caught up yet, but another factor on issues from deployment or testing is not considered at this point. If ChangeA faces a problem in production and needs to be backed up then what should be done to ChangeB? Depending on the criticality, ChangeB could be backed out or merged with ChangeA. If ChangeA cannot be replicated in test, then what is the path forward?

If ChangeA works but ChangeB is unable to pass testing, ChangeC is now on hold. How should the group handle if a very high critical issue now comes up? Should everything be backed out then readdressed? Should it be merged to current change?

So far I have only addressed the main actors to SDLC, but there are other actors that make this management more complex: management and product planning. Because they have their own plans resources are now being used to manage unexpected work that comes up, budget and promises need to be considered.

This is comes to another factor to the resource consumption. Regular/planned/scheduled releases are impacted or impacting to defect deployments. Although rules are made to what changes should go into a emergency/point release or a planned/scheduled release, there are always exceptions that throws the process for a loop.

There seems to be several red lights to the chaotic process that I am currently in.

  1. The most obvious request from release management is to skip a few weeks of releases. And obviously, no one can stand missing fixes for 2-4 weeks. If the product cannot remain stable for a month, there is something seriously wrong with the product (IMO). What customer wants a solution that require weekly updates? I get quite irritated with weekly updates for changes that do not even impact me to begin with!
  2. Similar defects keep showing up. No one wants to manually fix out-of-sync issues, while also unable to replicate or completely fix the problem. There is something seriously wrong fundamentally with the product if no one is able to figure out syncing issues that keep coming up for over 3 months (for us two years even with some fixes that didn't do anything).
  3. Every team is requesting additional resources or time. Obviously something is wrong with planning if everyone (or even the majority of groups) cannot accomplish the simplest of tasks within a regular plan.
  4. Always in catch-up mode. There is seriously something wrong if product/application planner cannot realign the plan given the current situation.
  5. No COO. There is not a single point person who can just put their foot down and just move forward. I do not how many times we spent more time "negotiating" with everyone when the worst plan was still faster than the "planning" of the changes.
Rant over

Friday, May 2, 2014

Work Life: Enterprise Release Management for Hardware Dependent Solutions

I see a greater need for release management. Now with agile development becoming more of the norm for corporate/enterprise level solutions, there is even more need to validate what is going out into production.

To me, the main driving force is because software solutions have finally deviated (in my opinion) from the traditional workflow of hardware development, a very traditional waterfall model. Even with agile development, agile also follows a very cyclical version of the waterfall model.

In the past, software was deployed in its own instance at a client system. Thus each client can be on different versions thus maintaining an overlapping waterfall model if one was to build a model on an instance level. In other words if we tracked each individual client, we can create a waterfall for that client even though the SDLC will be shared among the clients with the same version.

But with introduction of cloud, now there is only a single solution or a single client install that everyone shares. Add this to a corporate/enterprise level, you have many development cycles that need to be deployed to a single instance.

With a single instance, a software company can no longer just push out changes whenever they want to. Planning and scheduling is also more complex. Basically, software releases now must follow a more traditional waterfall process while software development is moving towards a more agile process. There are two bottlenecks in an enterprise cloud solution: system testing and production deployment.

System testing has to continuously run regression testing, but as enterprise solutions grow larger, regression testing becomes longer and longer. When regression testing starts to get around 2+ weeks to complete, this becomes a major bottleneck for changes to be deployed within a normal agile schedule.

With more components, there are also more defects. More defects means more emergency change windows. With more change windows, there are more regression test needed. Not only do changes become longer, but they also become more frequent.

A simple solution for non-hardware dependent solutions, is to create more replications of the production environment. This allows multiple regression testing to overlay. Although this does not help with decreasing the cycle time, it allows a higher frequency of deployments into system test.

Putting all the changes into a single "patch" can alleviate a lot of stress on deployment but adds stress to production support. With more changes, it becomes more difficult for production support to know the source of the problems whether it already existed or coming from one of the other changes. By isolating changes to only a few select changes, then a queue will build up.

Priorities also require different expectations and management when multiple changes are bundled together. If one of the changes is a high priority, but then a low level change fails in production deployment, all changes should be rolled back. General management must then make the decision to go with the change with a regression test that did not test that specific scenario or to roll back a high priority change that may delay other factors by several days if not weeks.

System test environment will be under even more stress than production because production only has to maintain the latest completed and tested code. System test has to maintain three different environments. In an enterprise level, this causes a lot of complications.

First, it is difficult to financially maintain three instances of relatively the same thing. One is needed for the latest major release, one to test emergency changes (aka point releases, patch releases, etc.), and last to be on the same version as production to replicate errors and bugs.

Second because there are three different environments, not all vendors are capable of existing in multiple instances on the same network much less different versions, etc. Then if you want to have different versions of a vendors application, then more labs will be needed.

Third, changes are coming from many different groups. Agile causes even more iterations to be introduced into system test. Eventually, a quiet period (aka code freeze, soak period, etc) is required to do a comprehensive regression test. Rarely will there not be an issue, then scheduling becomes an issue because quiet period is no longer possible to fix changes.

This does not even include all the politics that occur in enterprise level solutions, nor the management of defects across different groups, nor the different policies of different groups. There are also many things out of the control of process like OS maintenance updates, vendors no longer support older versions, bugs from vendors (ie Heartbleed), and so many other variables that interrupt a process that is already very stressed.

Evident of current software development and trending agile development, having more formal processes are nearly impossible to implement much less enforce. I believe there will be a greater need for skilled engineers and technical managers, because they are required to maintain processes that require less common abilities. Formal processes has corrupted the workforce with personnel that cannot keep up with the pace.

In corporations, processes have been so dummiefied (ie idiot-proofed) that people who can follow instructions call themselves engineers. These people are unable to keep up the changes. Sadly, they are so tenured and senior in the company, that they are also the source of the inertia to any change to the culture.

The age of corporations to make human resources a commodity is in the recession. This is why the technical sector has a need for "skilled" engineers yet unemployment still remains relatively the same. This is why small companies can take over corporations.

Returning back to the topic on release management, release management is needed to make sure that changes that are introduced in the different environments are maintainable. For example if there is a backlog of changes into production, new changes should not be introduced into system test. If they must, then dependencies must be evaluated. If there is a dependency, then many other factors must be considered.

Monday, November 18, 2013

Work Life: HealthCare.gov - Search Returns SQL Query-like Results

I didn't think this would actually work after reading this on a blog. What you see as the results are some sql queries. Although it looks kind of dangerous at first, these results are likely from other users entering these terms likely trying to hack into the website. Or also very likely (and hopefully), these could also be from third-party vendors who do website security checks because these are fairly common tables that you would try if you were going to guess at what the table names are.

Even if the attempts were legal, the results are very poor. I even tried one just to see if there were any results. After a few more times, I also noticed that the suggestion results changed. This may potentially just be other people trying the search terms after seeing the post thus skewing the results as well.

My interest in this is primarily on the software release and software testing side. After these spotlight issues, these are the type of things you would consider in a typical software test. I've learned my lessons in this same area over a decade ago. Even then, I've had preventions to these types of attacks. So although I cringe at the thought that something like this could come about, I am also somewhat relieved that this can be overlooked even at such a product with that level of scrutiny. 

In this case, the suggestion feature still works. The results are just not very good quality for your standard insurance seeking web visitor. Even as an software engineer, this probably wouldn't help me find what I want although it would be neat to customize my own search query. I could perhaps write something that would be specific to my needs instead of a common person's template.

But let us say that this does happen to your company, how do we prevent this from happening in the future? My quick take on this is that there is no good process to prevent something like this because it is a type of issue where once you've identified it, the issue will be fixed thus highly unlikely this will reoccur. For regression testing, you would add this to your set of search terms to look out for. I am sure that they most likely have sql injection prevention implemented in the back-end, but that does not necessarily mean that the search results would be ok. In the regression test, it would be important to make sure that the results are also good.

But how do you test result quality? At first glance, this does not appear to be something that can be automated because there are just too many ways a user could enter information. The search does not have a clear measurable method to evaluate what a good result is. These are things that search engineers battle everyday and still try to improve. Even if you find the right formulas for the common users, you may find someone trying to exploit the algorithm thus throwing the results off.

From what I can come up with at the moment, the best method is to investigate the more likely sql injection queries and have exceptions to those searches. This would not prevent other poor searches, but at least it would prevent some bad publicity while the program team figures a way to improve the algorithm.




Reference:
https://plus.google.com/111405772080232969785/posts/P1ZEXFo681C

Wednesday, November 6, 2013

Complaint: Google Analytics - Demographics and Interest Reports - New Tracking Code

I was playing around with Google Analytics and stumbled across the Demographics and Interest Reports which seemed interesting at first but it brings me to a page to say that I need to make a change to the scripts to have this work.

Why didn't they just give me the proper scripts to begin with? I have the option already enabled. This is quite a hassle even though I know how to do the work. Then again, after trying to follow the instructions I could not find what they needed me to change.

The instructions is to change:
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
To:
ga.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + 'stats.g.doubleclick.net/dc.js'; 

And no where in the script do I have these lines. The code surrounding the lines are not supposed to change but even those are not what the script was provided by Google Analytics. What is provided to me is the follow:
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
  ga('create', 'UA-xxxxxxx-x', 'xxxxxxxxxxxx.blogspot.com');
  ga('send', 'pageview');
Figuring what the changes are, I could deduce that I change google-analytics.com/ga.js to stats.g.doubleclick.net/dc.js. But then I checked the privacy policy and noticed that that this also required me to update my privacy policy. At this point, this is just too much of a headache so decided not to move forward this change until I figure out what to do about the privacy policy part. Whatever reason for not implementing the change, I find it odd that the instructions to update the script is not correct for me which I would expect from a company like Google.

Release Management

Who's responsibility was this issue? For my team, this would appear to be needed from our product team. It is also odd that no one has a checklist to make sure that documentation should be updated since the instructions appear to have been for an older version (here's an example to publish versions for those people who believe version numbers are practically obsolete due to new agile processes). None of the steps after product would really check on this change because it wouldn't exist in the requirements since it was missed. This wouldn't have a chance to be caught until much later in the process like User Acceptance Test (UAT)... possibly System Test (ST) if they did some free-style testing which I find unlikely since most corporations would have the team follow test scripts which would exist since there was no requirement. Also odd is that even after all that, a user like myself should have reported this by the time I got to this (although I am making a huge assumption that this feature has been out for while). But Google typically had something like a "did you find this helpful" and of course, it does not exist for the one time I looked for it.

So even after all that, there is a chance that I have missed a step somewhere. I am not sure where I skipped a step and if it wasn't that intuitive then something at least needs to be improved.

Reference
Instructions to change - https://support.google.com/analytics/answer/2444872?hl=en&utm_id=ad
Policy requirements - https://support.google.com/analytics/answer/2700409
Google Analytics - https://www.google.com/analytics