Open Source, why organizations can’t afford to ignore it?

A couple of weeks ago I ended up here and it occurred to me that for the last year or so I had published posts ranging from WSO2 product related how to’s(what this site was initially supposed to be about), linux how to’s that bordered on immaturity to opinionated tech event reviews from my local community. This thought led to me changing the blog’s name to "The FOSS Merchant".

FOSS stands for Free Open Source Software, think Apache(no, not this guy[1]) and Ubuntu. “Free” in in this context mean something closer to “free speech” than “free beer”. Exercise your linguistic demons here, if you must[2]. Since “FOSS Merchant” sounds like an oxymoron, I thought I should write a few words on the matter.

…..

Open Source Software, as it’s name suggests is free to be used and worked on by anyone. Meaning, most open source software projects are maintained by a community spanning organizations and geographical boundaries. This facet of open source software makes it more sensitive to industry demands and needs, it makes it evolve faster. In brief, organizations that tap into this quality of open source could, in turn, use it to increase their technological agility. 

The days when open source software was considered a plaything for the geeky and not a contender for the professional is far behind us. At present 1 in 3 servers[2] powering the internet is run on Linux and the number keeps rising every year. In fact one survey found Linux to be preferred by 78% of participants(including some fortune 500 companies) for their cloud-based endeavors[3], this figure makes sense considering the popularity of server crafting tools such as docker and the extended support it provides to Linux. Furthermore, there are plenty of examples of product companies that have successfully worked open source into their business models, from Red Hat estimated to be worth 14 billion USD, Oracle’s acquisition of MySQL at 1 billion USD, last week’s news of MuleSoft’s 100 million IPO to WSO2’s 11 year track record of providing leading middleware solutions and driving digital transformation[3] for organizations.             

A common accusation made against FOSS is that it is not as polished or as user friendly as it’s  closed source counterpart. Although this is true on occasion it is my personal opinion that this characteristic of open source software attracts the kind of engineer that end up having a bigger say in technological decisions at ground level, thus they drive organizational changes in terms of technology in the long run(common sense dictates its better to ride this wave than to expend energy redirecting it). This hypothesis is backed by some empirical evidence. In one survey done by The Linux Foundation[4] 52.7% participants cited in house expertise as a key driver for OSS adoption. In another study done on the adoption and transition from closed to open source software, based on the Technology Acceptance Model(an information theory dealing with technology adoption) found a negative correlation between the perceived ease of use and adoption of OSS in the public sector[5]. It is also easy to assume this hypothesis, as one of the factors for the phenomenal success of GIT version control system, when one takes into account the fact that it was considerably different(conceptually and in usage) from other version control systems of the time and was often accused of not being user friendly.

Other than the agility and ground level traction that can be expected by selecting OSS(as discussed in preceding paragraphs), organizations may also take comfort in knowing they have complete access to the code base of the software they are procuring, this may not even be an option with some proprietary vendors. This disclosure coupled with in house expertise on the code base which can be cultivated, in time can amount to technological independence and assurance for the organization. 

However, OSS posses its own novel set of concerns which organizations should address before making the transition. As most open source projects are maintained by volunteers, they lack documentation and other knowledge resources which organizations need to make informed decisions on suitability. Tailoring the software to exact organizational needs and maintenance afterwards may prove to be difficult,  as expertise on the software cannot be readily accessed. As this is the case for many open source projects, it is of paramount importance that organizations select a commercial OSS vendor/partner that can bring in value by addressing these concerns. Furthermore, commercial OSS vendor/partner’s more often than not provide useful features that cannot be found in the original projects. 

[1] - https://www.youtube.com/watch?v=kZzBd41NuZw

[2] - http://ell.stackexchange.com/questions/2516/free-as-in-free-speech-not-as-in-free-beer

[3] - http://wso2.com/whitepapers/an-iterative-approach-to-digital-transformation/

[4] - http://www.ossdirectory.com/knowhow/2012_LinuxAdoptionTrends2012.pdf

[5] - https://www.google.lk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjartfZibDSAhWJl5QKHTVqADsQFggaMAA&url=https%3A%2F%2Fwww.bcu.ac.uk%2FDownload%2FAsset%2F38bcddbb-4c73-e511-80ce-005056831842&usg=AFQjCNFNWf65F8-TwXtNJxIxX8ALlImF1w&sig2=0u52M_FiBONeLxV_j8AV1A     

A Caveman's Bash Editor

WSO2 is in the process of improving its CI/CD pipeline in preparation for a bunch of new products(btw did you hear about our exciting new integration language? find more information here: http://ballerinalang.org/) and yours truly was given the awesome responsibility of creating a jenkins post build shell script inline with this goal. During this process I noticed a serious lack of tools for shell/bash script writers and so i wrote this simple helper script.

bash script
vim script

How to get it working,

1. Install vim if you don’t have it in your box.
2. Copy the vimrc file to $HOME/.vim/
3. Move CavemansBashEditor.sh to a location of your liking and run it providing the bash script that needs to be syntax checked as a script parameter e.g. ./CavemansBashEditor.sh /home/dumiduh/myscript.sh
4. Leave the script running and refer to it while editing your script(Refer screenshot below).

Notes,

• The script uses ‘bash -n’ linux command and will only pick up syntax errors it picks up.
• The vimrc file changes the vim editors behavior so that it automatically saves changes, this behavior can be a pain at times so remove the file from $HOME/.vim/ when the script is not used.
• The script was tested on Ubuntu 14.04 with vim 2:7.4.052-1ubuntu3

Thoughts on the SLASSCOM Blockchain Event

When will I find a better place to keep these write-ups?

Blockchains and bitcoins are two words that have been thrown around for while, I know little about either and so to fix the situation I decided to attend a blockchain knowledge sharing session organized by SLASSCOM.

To be honest, I have tried figuring out bitcoin when it first popped up as a buzzword but the effort didn’t really lead to any substantial gain in my knowledge on the subject. Now, in retrospect, I can see that it turning out the way it did had more to do with the way in which I went about figuring it out than any other reason.

Here’s the deal, what I figured out during the session was that 'blockchains' is one of those topics that should be looked at conceptually than from an implementation point of view when you first go about trying to wrap your head around it. If you focus on the underlying implementation technologies too much you miss the point! From a technological point of view, blockchains are not doing anything radical or new for that matter. As far as I could understand they are mostly using batch processing technologies and proven security concepts that are old as the hills. Maybe it was because of this reason that I found it a little hilarious when two audience members tried to make it look like it’s just about security, in my humble opinion they added noise to the session(more rants on a similar subject here)

What one should understand instead is what it tries to accomplish and what benefits it has over what’s existing, cash money(yo!) One needs to understand concepts such as what is money? What does a 100 rupee note really represent? Concepts that don’t come to us engineers too naturally.

Blockchains are a huge one up on paper currency because it provides absolute traceability and improvements in authenticity. All four speakers were competent on the subject and delivered their segments well. I was especially impressed with the segments Jayamin Pelpola from Alfinity Ventures and Tharindu Dissanayake from MillenniumIT delivered as they were focused more on real world implications/experiences, areas of future growth and were driven more towards getting the concept across, which I felt was best suited as the topic is a lot more novel conceptually than technologically. Though it must be stated that Omega Silva from VirtusaPolaris did a great job of explaining the technical nitty gritty.

Bottomline, Blockchains can be used as a tool for organizations to optimize situations where they need to process money or other tokens that needs to be held in the same light and add an additional level of integrity and authenticity to transactions. But for you and me Blockchains wouldn’t really make a difference either way for many many years to come. It is only if and when blockchain systems such as bitcoin become adopted(or projected to be adopted) by a majority would you and I really reap benefits of trading in the metal coins in our wallets for their digital counterparts, regardless of how accurately they can be tracked down to their origins or significance of their processing improvements.

Yet another great event by SLASSCOM.

Thoughts on the 6th Colombo IoT meetup

I suppose I have to leave this post here for the lack of a better place.

The meetup was titled Rise with Cloud Intelligence and staying with the theme the talks revolved around IoT PaaS offerings. The first couple of speakers(employed at Persistent Systems and Virtusa Polariz respectively) talked about their experiences working with IBM Bluemix, their segments leaned more towards demonstrations and explanations towards the work they had done with the platform. I felt considering the time constraints of the event it would have been more useful had some of these talks been done at a conceptual level because at times it seemed like time was wasted explaining simple nitty gritty. Rounding up, the segment gave a holistic image of the capabilities of IBM Bluemix.

The segment on Bluemix was followed with an interesting talk on an IoT enabled PaaS solution by two gentlemen with thick american accents. The first speaker(one Dean Hamilton) started off sharing some statistics on IoT space(published by an organization going by the name of ABI research). The space according to the speaker can be divided to 3 subcategories, the device manufacturers(hardware), connectivity(business with getting the sensor data to the servers) and the server side of things(control and analytics). Common sense and the statistics shown to us said most of the big IoT money(everyone is talking about) is expected to be made in software and service category.

Interestingly, the solution Dean presented to us was aimed at the first category. An IoT enabled PaaS that device manufacturers can use to monetize sensor data pushed back to them, to get a piece of the IoT pie. He took the example of how a manufacturer of agricultural machinery (harvesters and such) could enable the sensor data to be push on to a PPaaS, be enriched with other services(aligning with the manufacturer's interests) and to be sold to any interested party(such as fertilizer manufacturers). Their solution tries to cater to this need.

As an engineer with the aim of attending such events as a way to keep in beat with happenings of the local community I walked away satisfied.          

How to find more needles

If you're someone like me who needs to find needles in log haystacks then bash and grep are surely your friends. Here are two tricks I've come to use to make my work easier.

Making errors scream!

Okay so you have four or five terminal tabs going tailing different but related logs and the logs keep on pilling up. A script like the one I got below should be useful (given that you have headphone on, unless you getting weird looks from people around you). 

The script tails a log periodically and checks for the existence of a word(or phrase). If a match is made it plays an audio file(wave) to get your attention.



Usage,

sh logchecker.sh <NO_OF_LINE_TO_TAIL> <LOG_FILE> <WORD_TO_LOOK_FOR> <WAV_FILE> 

Redirect errors to a file

You have a server with debug log level enable at the root. The logs are putting on MBs instead of KBs. Grep out what you need to a separate file to make analysis more manageable.

Usage,

sh grepout.sh <LOG_FILE> <NUM_LINES_BEFORE_MATCH> <NUM_LINE_AFTER_MATCH> <WORD_TO_MATCH> 

How to hide the secure vault decryption password

whats covered: embedding the secure vault password for the linux startup script.

WSO2 servers ship with the capability of encrypting and securing plain text passwords used in configuration files, find more information about this feature here[1]. As the final step of this process the decryption password will need to be provided at server startup by entering it or placing it as a temp text file. When needing to start the servers as background services the password can be embedded into the server startup script as shown in the post  to make the process more secure.

1) encode the password to base64

run the command below to get it encoded

echo 'put your password here' | base64

2) modify the wso2server.sh script to generate the password file at runtime.

Include the following line at the start of the elif [ "$CMD" = "start" ]; then block. Refer[2]

echo <the encoded base64 string goes here> | base64 -d | tee $CARBON_HOME/password-tmp

the password-tmp file will get generated before the carbon bootlstrapper is run. The file will get deleted after it is read.

Note that this would only work on Linux distros where the base64 program is available.

[1] - https://docs.wso2.com/display/Carbon420/Securing+Passwords+in+Configuration+Files

[2] - https://gist.github.com/handakumbura/f4496c7578c36775086059751e03861d

How to front a bearer token secured endpoint using WSO2 API Manager

whats covered: fronting a bearer token secured endpoint using a mediation policy for APIM 1.10.0

If a requirement arises to front a bearer token secured API while maintaining API manager authentication mechanism(though this is unlikely and should probably be avoided) it could be met using a mediation policy.

1) Create a mediation policy with the logic

The mediation policy should be such that it takes in the bearer token(of the back-end service) passed in as a custom transport level header value and passes it on to the backend service with correct formating. This can be achieved using a property mediator[1], header mediator[2] and a few synapse built in functions.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="bearersequence">
   <property xmlns:ns="http://org.apache.synapse/xsd" name="btoken" expression="$trp:token" scope="default" type="STRING"></property>
   <header xmlns:ns="http://org.apache.synapse/xsd" name="Authorization" scope="transport" expression="fn:concat('Bearer ', get-property('btoken'))"></header>
<header name="token" scope="transport" action="remove"></header>
</sequence>

download the example mediation policy from here[3]

2) Attach the mediation policy to the API In flow

Start creating an API with required HTTP methods etc, select Manage API from implementation and from Message Mediation Policies section upload the mediation policy to the in flow. Publish the API.

3) Invoke

Invoke the API with the bearer token of the backend service set to a header named "token" (as this is the header name that we have configured in the mediation policy).

 

[1] - https://docs.wso2.com/display/ESB490/Property+Mediator

[2] - https://docs.wso2.com/display/ESB490/Header+Mediator

[3] -  https://drive.google.com/file/d/0B9oVIeyHJKBXTkpBNTY2c3Rqa2s/view?usp=sharing