Douglas Lee: software engineering

Showing posts with label software engineering. Show all posts

Friday, March 6, 2020

Interview: What Project Were You Most Proud Of? (Software Engineering role) (Failed)

I have had several interviews in the last couple months. This was probably one of the most popular questions for me (January - February 2020), "Describe to me about one of your proudest projects that you worked on?"

Because I was a release manager applying for a software engineering role, this question was quite tricky because I did not work on projects. I had other projects that I would like to share but they were not really programming projects or projects where I produced something. In most cases, I managed and delegated so I gave those samples.

Thinking back on those interviews, I do not feel that I gave good responses to the question. I also tried to provide a side project that I worked on as a release manager. The proof-of-concept did not get enough traction to be approved for actual use, although everyone liked the idea. Environment and management of an application was not something management wanted to handle, as they rather just purchase a third-party system or none at all.

Perhaps, I should have used a project that I worked on back when I was a programmer. I am not sure what a good response would be even after seeing what our developers do. Especially in an agile environment, the developers no longer work on projects in the same sense as a waterfall project. They are constantly just writing code per user-stories, test, and refactor. Most of them do not know the full project scope. Maybe they should, but from a procedural point-of-view, this method has been rather effective for the company.

Perhaps more effective for the company in that the developers will have trouble finding new jobs because they may also struggle with the same question. Without the experience, they will not know how to position themselves in such a way where they can also gain something that they can market.

Given that I was not able to even get a second interview with any company that asked this particular question makes me really think that I am definitely failing this question at some level. There have also been other weak parts of my interview that may have also contributed like not being the local vicinity (even though I recently moved to the area) and no in a current programming role which a couple HR managers have told me directly.

Tuesday, February 10, 2015

I am not a fan of agile / scrum (Software Methodology)

I am not pro-agile, but also not anti-agile. I did just enjoy reading a post by Giles Bowkett on "Why Scrum Should Basically Just Die In A Fire." I think agile methods (including Scrum) can work, but time morphs any processes and agile is no different. Giles presents a lightly-humored touch on how Scrum can go wrong.

For me the biggest issue with agile (at least today) is its inability to weed out poor-performing members. Even weeding out bad-performing members is pretty difficult. The second biggest issue with agile is the ability to game the company system. Another big issue that most sites do not point out is that agile is not good for high risk solutions, especially solutions that may determine real life and death risks like medical solutions or an airplane.

There are other issues that agile also do not manage well is regression test. It requires that it to be automated which is fine, but how would it regression test features that take up real physical time like scheduled jobs that are triggered daily or even weekly. Agile also loses performance when dealing with multiple components or multiple environments, where the project essentially becomes traditional methods to coordinate all the different pieces.

Poor-Performing Teams

My biggest problem with poor-performing members is that we have the misconception that those members are dumb. In reality they are just not good at their role, but are smarter than people credit them to be. So this allows a lot of freedom for people to game the system or start politicking.

The easiest excuse is that agile is new, so they still need to work out the kinks. Blame the other group, blame the requirements, blame coding, blame testing. Everyone just points fingers at everyone else. Sadly, these poor-performing teams come out looking better because they are more prepared to blame others while the good teams were working how to fix the issues.

Even if you can isolate the team, identify the individual is even more difficult. There is less information as there is less focus on documentation (if there is an documentation at all). Is the problem with the scrum master not leading properly or the developer not coding properly or the tester not testing sufficiently? Or the scrum master does not have proper support or the requirements are consistently poor or test cases ill-defined? Finding the high-value and low-value members is like a game of mastermind.

High Risk Solutions

I find it odd that I have not come across any sites, posts, or forums (even anti-agile fanatics) mention that agile is not advisable for solutions that have high risks if something does fail. I surely do not want to be on an airplane that went through an agile process... unless maybe if it were on its one millionth iteration and the last 500,000 iterations had no deaths. Or use the laser eye surgery. With my luck, I will probably be that one test case that no one anticipated.

I am also sure that financial companies would not want to risk millions of dollars due to a flaw in the system. I doubt explaining that they didn't provide the proper requirements would make financial companies feel any better. Telling them that the next iteration would correct the problem would not be much consolation.

Or an online bank losing your life-savings. An individual losing $100 of life-savings will likely be just as angry as another person with $1,000,000 of life-savings. Sure there is insurance for that too but who really wants to go through that experience.

Sure some solutions do not carry such risks like Candy Crush or Farmville (there have been extremes with some video gamers that could probably prove that wrong too). The problem is defining where that line is like a cloud solution that promises 99.99% uptime. Is the risk worth all the customers possibly switching to your competitor?

Documentation

How many people do you know that the value to agile methods is "comprehensive" documentation in its core value: working software over comprehensive documentation? How many of those people even know what that mean? Most people hear "no documentation is needed" (at least the people I know). Not sure how many times I had to point out that we still need some documentation after hearing that they are not supposed to provide documentation anymore because they went agile.

There is definitely something wrong if that value even has to be stated. What company would not love to drop documentation as part of the process if the software just worked? Writing documentation takes up time, and to our corporate world, time is money. I mean they will drop testing if that would save them money too. Besides time and money, I think traditional methods can also drop comprehensive documentation so making it a value of agile is kind of redundant.

Comprehensive documentation came out of technical writers wanting to keep their job and managers wanting to make people as replaceable as possible which means that the documentation also needs to be continuously updated (mostly the latter which coincidentally didn't get complaints from technical writers). If it is faster to read code to learn what it does than it is too read documentation, the documentation is too comprehensive. I have personally read some old corporate documentation that went into such details as even providing illustrations on single-clicking versus double-clicking or how to move the mouse or how to insert a disk (although the younger generation probably would need that step today).

But my main issue with agile is that people seem to take this to mean "no" documentation. Although it is not stated that way in the manifesto, many people perceive it that way. Developers don't want to document, managers don't want people to waste time documenting, so it appears like a win for everyone.

That is until 6 months later and the whole development team has changed. New updates are needed on the old code which no one knows about. No documentation, so more time is spent figuring it out. Code is eventually updated and passes the end-user acceptance test. Deploy the change then realize an old process has changed which broke something else. Maybe the change was a comma instead of a semi-colon which passed regression test but fails for the client. The point is more time was spent than was necessary. Management is going to put this as a lesson learned or retrospective, then will require more documentation. 10 years later and we are all going to be back with the issue of comprehensive documentation for agile and traditional methods (or whatever new method name it will be in the future).

I haven't even touched on audit which is already a nightmare with traditional methods. Surprisingly I have not had to experience audit with agile project (yet), but I would guess that it would be just as difficult as traditional in its best case. More likely it would be more painful due to lack of documentation which is a majority of the problems I face in traditional methods. But I supposed it could be easier just to tell the auditor that it just does not exist than trying to hunt down the document, then let someone else worry about what process need to be put into place to "correct" the audit problem.

Solution Complexity

Assuming a great and awesome agile team, the agile method will slowly become similar to traditional methods as the solution grows in complexity and scale large enough to partner with other groups. Having to coordinate times, priorities, and many values to traditional methods, agile cannot avoid certain decision gates needed for multiple team coordination. Agile can work purely in the requirements through development phase, but once it reaches integration another branch of code is required. Then the company will also need another set of testers and developers to fix issues because the agile teams are to be working on the next iterations. If the same team is used, then most of the agile iterations are pretty moot.

Assuming another team is available to support code that is being prepared for production, processes need to be available to merge code to any code that is still open. Further regression test is required. And this is probably the easiest and most regular process.

How will the group manage if there is a production issue that needs to be corrected while in the middle of an iteration? What if it happens to be in the middle of an iteration and in the middle of integration? Is there a separate team? Does integration stop to work on production issue? Now there are at least 3 branches of code being developed. That is only if the code is for one group.

Working with multiple groups, departments, partners, etc., there is bound to scenario where there is not enough resources. In traditional world, all the requirements are coordinated. For agile, this may be more difficult because iterations now have dead-lines which is almost counter to its values. One of the groups may keep its agile schedule, but the others have to coordinate accordingly to their dependencies. Working with other partners may even mean coordinating different flavors of agile methods.

And this is becoming more and more commonplace, as companies have software solutions for billing, accounting, finances, manufacturing, sales, and other departments. Then you want to interact with other companies.

Conclusion

I think agile is basically the same as traditional but just from a different view angle. I think traditional has always been trying to go agile, and many companies probably have made it work. Most of those methods are probably more agile than traditional compared to the corporate world. I think some would argue that it was more or less lean processes like six sigma.

I spent most of my career with small to medium sized companies which basically are traditional methods in a very fast paced world which forced us to be very agile-like. So when I learned agile, most of it was already very familiar except without all the fancy terms and ceremonies. The learning curve was actually harder when I worked with a corporation sized company where the bureaucracy was so daunting.

So, I feel that agile is either the leanest process of traditional method or a traditional method that goes through many prototyping that just never gets refactored. In the smallest of worlds, developers at the very beginning is the truest form of agile where we our own client, product, end-user, developer, and tester. Came up with an idea, tried to program, test, check with initial idea, come up with new idea, develop, test, accept the result, repeat if not acceptable, then becomes final product when accepted.

Reference

1 - http://gilesbowkett.blogspot.com/2014/09/why-scrum-should-basically-just-die-in.html

2 - http://www.mountaingoatsoftware.com/blog/removing-team-members

Sunday, December 8, 2013

Work Life: Software Versioning - Understanding the basics

The main reasons that someone would be interested in the versions is when they need to identify which features are in a particular version or identify which code to update for remedy a defect. Otherwise, version id is rather useless except for maybe as a marketing gimmick which really does not serve any functional purpose to software development.

At the most fundamental level, version id allows the developer know exactly which code base to identify or modify. In other words, the version id does not have to follow any sort of schema, rules, or process. You can use numbers (version 1), groups of numbers (Version 1.2.3.4), letters (version A), mixed (version 1.2.a.3.b), pictures, names (Windows XP, Vista, Google Jellybean, Ice Cream Sandwich). Basically anything you want it to be as long as you know how the version id goes with which code base.

To expand on the numbers and letters, you will commonly see that there is some incremental process. This is also not required on the fundamental level. You can jump from (v3 to v1.1 to v2.0.4 to vA to vB) like how Microsoft jumped from (v3.1 to 95 to 2000 to ME to XP to Vista to Win 8). No rhyme, no reason needed for version id except that it should at least be unique. I use 'should' because you could technically use the same version id but then that really complicates the matter which I won't cover here because who knows why some people do that (but it really does exist).

But because our memory with versions work well in chronological order, many companies use an incremental pattern type. Not only is it easier to remember, it is also easier to automate. Every time you have an update, you increment by 1 (version 1, then 2, then 3).

Eventually, you'll find it cumbersome to remember difference between version 15 and version 32 and version 75. So many companies compartmentalize the changes by adding a sub-group (Version 1.1, then 1.2, then 1.3, etc). When you reach a arbitrarily major update, you will increment the super-group from 1.X to 2.0 (or 2.1 depending on your preferences or even 3.X if you want people to go slightly insane). Here, I use X because it can be any number. You can find that you go from 1.9 to 1.10 which does not seem to make mathematical sense, primarily because it is not a mathematical number (technically two numbers thus 1.10 is not the same as 1.1). This can be understood more easily if another sub-group is added like 1.10.20 which completely breaks any relationship to a mathematical number.

You'll find that 2 to 3 level system works for most groups and companies. But for software managed by many people (20-100 developers), you'll notice that there are more levels and that they do not increment in the normal fashion. The reason for this is because of branching of code. Because there are so many people developing at the same time, there is a root application that they all work off of. So everyone could be working off of code version 1.2.1. Developer A will be writing code version 1.2.1.5 while another developer B will be writing code version 1.2.1.8 (maybe he has an intern working on 1.2.1.8.2). Someone will eventually decide that it is time to bring some of the changes together to create version 1.2.1.22. If they find that to work, they could publish version 1.2.1.22. Of course, you could also have a policy that just increments the third-level whenever you merge which then should have been version 1.2.2.

But let's say we kept with 1.2.1.22 because it is a patch update and not really an application update. While testing, this version did not work and deemed to difficult to troubleshoot. Version 1.2.1.22 can be discarded. After the major items are addressed, eventually someone will merge a different set of versions to create Version 1.2.1.45. Then, repeat until they have a working version. If 1.2.1.45 works, they may just publish that so the public will never see version 1.2.1.22. If you use the other policy, you wouldn't see 1.2.2 but a 1.2.3.

At this point, you can also see from the end-user (i.e. customer) that may concern them especially for new vendors. Why did they skip a number if they always had an incremental system? This is why larger corporations that invest in PR develop their own version schema. Company A may say that we don't want to confuse the users with an update from 1.2.1 to 1.2.3. They are going to stick with version 1.2.2.

But on the development side, you cannot simply rename 1.2.3 to marketing's 1.2.2 because it was already used. Thus a cross-reference table needs to be maintained to tracking marketing's version to the development's version.

On a higher level, you can easily see where development and marketing would deviate in version id when there are impacts to technology versus usage. For example, the development major version (level 1) needs to be incremented because of a new software design that does not impact function. To marketing this is not a major change. So development could move to version 2.0.1 while marketing may choose to remain in version 1.x.x format. On the other hand, marketing may see one of the new features that will change how they sell the product but was not a major change to code (like backup and restore, or moving to SSO). For this marketing will sell it as version 3.0.1 while development keeps their 2.x.x format.

After some time, marketing will then start to simplify or complicate the version id towards their market research. That is why you see so many odd versions in today's world. You can have Google's candy theme that increments by the first letter. Apple's animal theme with the same incremental scheme. Or Windows random scheme. Primarily, their versions are easier to remember maybe because we can relate more things to it than just numbers which we see everyday. Also it makes it easier to search because all applications use numbers so searching for version 1.1 will return several results while version Vista is more unique. Or you can complicate it which then many people will not remember the older versions.

So in conclusion, version ids can be anything. Best practices are good guidelines. Choose and/or adapt one that fits your business model. I do not recommend random version id for development as there is little value to that, but who knows perhaps it has value if you are trying to make it difficult for other people the chronological progression of your software development (like for a spy-world tv show).

Sunday, December 1, 2013

Life: Decoupling Economic Dependencies on Government Laws

I was going through some articles about parking ticket scams where people were given parking tickets where most have never been in the state before or was clearly not there during the time of the violation. Although there were so many things wrong with the system, what really got me thinking was when did our justice system put the burden of proof upon the victims when all they have is the hearsay of a traffic officer. Somehow, the victims now have to provide documents, lawyer fees, etc., for something they did not commit in the first place yet the law does not even produce a picture of the violation even especially after dispute.

On a similar note... if a sign was missing or a meter was broken, the burden is on the victim to prove that it was missing or broken, yet not the burden of the government to prove that the meter was working and the sign was there. On top of that, even if you pay the fine because it is cheaper than proving your innocence a $2 convenience fee is added for paying online.

Where did our society go wrong? Yet when come to changing the laws, some of the arguments is on how the government will compensate for the lost revenue if the law changes. What does that even have to do with the justice system? Then to kick the average citizen while they're down, they let many criminals free because of technicalities that average citizens cannot exercise due to severity of crime or too poor.

This is not an easy task for the government to balance but I think there are benefits to directing our budget to segregate where the revenue is used. Economically, I believe the problem is because they pay for programs using revenue from another program. In this case, parking tickets and violations produce a lot of revenue for the government (local, state, or federal) to pay for many programs.

To simplify the laws, I think it would be wiser to start to focus the penalties of laws towards the costs of misdemeanor crimes. Revenue produced from parking tickets can only be used towards the cost of parking issues. Taxes on tobacco or other illegal substances should only be used on programs that deal with anti-substance abuse.

This is not necessarily simpler but possible as insurance companies are able to make profit on similar type disasters. Parking issues should include some of the cost of the chance that the parking may cause the building to burn down. For example, the car parked in front of the fire hydrant so the fire engine could not effectively save the house. Part of the revenue from that parking ticket should pay for the damages on the burning house scenario. Thus inversely, the proportional percentage of the cost should be built into the ticket fine. Thus if the damages was $500k and car caused $10k of the damage, then $10k divided by number of expected fire hydrant parking violations should be the minimum to the ticket. Of course there are several other variables that needs to be included but that should be the gist of the concept.

By this same concept, income tax should be used towards our universal programs that impacts all citizens, like National Security, Social Security, National Healthcare (if it continues to exist), public education, all the court fees for defending the constitution, etc. Property tax used on programs for the property.

If we ever reach a point where one of the programs "makes a profit" then that should be saved in the government "bank" that can be "borrowed" for other programs when a disaster does occur but there is not enough funds for that program yet. If there was an unexpected hiccup, like Superstorm Sandy, recovery programs can borrow money from the "bank" to pay for the costs now. But eventually, that "debt" needs to be repaid. In the case of Sandy, since it is the welfare of East Coast residents, that should at least come out of the state income tax of the states impacted. If the US citizens want to assist, a part of the federal tax revenue can also be used. By doing this, it sets our expectations that our income tax is used to protect our welfare.

By not meeting expectations. this puts a lot of burden on the government for many things like PR, managing risks, and handling a very complex system. For example, the $2 convenience fee. We all know that online payments save the government money over hiring processors to receive mail, open mail, manually enter data, send check to bank, and all the other exceptions like overdrafting, canceled checks. Why leave such a sour taste in their own citizens by stating that it is for their convenience when most people know it is not for our convenience? The government can probably get less gripe by just by saying "because we want to fee" instead of convenience fee.

The purpose of all this is to decouple the dependencies between programs due to economic reasons. We should not be prevented in correcting a particular social issue just because it impacts the revenue, thus preventing ourselves from passing a law that is more justified.

Side Note

First, I'd like to apologize if the ideas do not flow together. I started with one idea then it morphed into something a bit bigger and just wanted to type it down as the thoughts came to me. I figured I will still go ahead and publish in-case someone happens to stumble upon this article and has some additional thoughts or reading material. I would like to revisit this thought again sometime in the future.

Another interesting thought that came to mind was my use of a software engineering term, decoupling. This got me into thinking how the government should be structured similar to a large application with a controller (federal government) with maybe something like interfaces (for state government). Basically, I think we need a plan that would refactor the code (laws) so that it is more effective.

If there are any PhD candidates looking for a thesis (probably in Political Science or maybe Computer Science or Software Engineering), I would definitely be interested in such a topic.

Very last thought, isn't it a very terrible way to learn (or get clarifications) on certain laws by accidentally breaking them? Seriously... blocking the box, no right turns in the city proper violations, are not common knowledge nor is it common for people to check on all the laws on everything. I tell that to anyone visiting to nyc. Or not having to come to a complete stop for right turns in California. How about a warning system that fines people the minimum amount for the time of the officer to give me the warning? Instead of having your average citizens more afraid of the law than criminals. http://www.dmv.org/fun-stuff/bizarre-driving-laws.php

As if to prove that traffic laws in themselves are confusing, most areas do not allow U-turns yet I have encountered intersections with No U-Turn signs even when it is already a violation but still have random intersections without the sign within eye-sight of the previous intersection. If you do not read up on the laws, you may be tricked into thinking that it is legal to make a U-Turn at an intersection that does not have the sign. So make sure to remember that even though there may be a sign explicitly stating that violation, that does not necessarily mean that it is not a violation when a sign does not exist.