How to find more needles

If you're someone like me who needs to find needles in log haystacks then bash and grep are surely your friends. Here are two tricks I've come to use to make my work easier.

Making errors scream!

Okay so you have four or five terminal tabs going tailing different but related logs and the logs keep on pilling up. A script like the one I got below should be useful (given that you have headphone on, unless you getting weird looks from people around you). 

The script tails a log periodically and checks for the existence of a word(or phrase). If a match is made it plays an audio file(wave) to get your attention.



Usage,

sh logchecker.sh <NO_OF_LINE_TO_TAIL> <LOG_FILE> <WORD_TO_LOOK_FOR> <WAV_FILE> 

Redirect errors to a file

You have a server with debug log level enable at the root. The logs are putting on MBs instead of KBs. Grep out what you need to a separate file to make analysis more manageable.

Usage,

sh grepout.sh <LOG_FILE> <NUM_LINES_BEFORE_MATCH> <NUM_LINE_AFTER_MATCH> <WORD_TO_MATCH> 

How to hide the secure vault decryption password

whats covered: embedding the secure vault password for the linux startup script.

WSO2 servers ship with the capability of encrypting and securing plain text passwords used in configuration files, find more information about this feature here[1]. As the final step of this process the decryption password will need to be provided at server startup by entering it or placing it as a temp text file. When needing to start the servers as background services the password can be embedded into the server startup script as shown in the post  to make the process more secure.

1) encode the password to base64

run the command below to get it encoded

echo 'put your password here' | base64

2) modify the wso2server.sh script to generate the password file at runtime.

Include the following line at the start of the elif [ "$CMD" = "start" ]; then block. Refer[2]

echo <the encoded base64 string goes here> | base64 -d | tee $CARBON_HOME/password-tmp

the password-tmp file will get generated before the carbon bootlstrapper is run. The file will get deleted after it is read.

Note that this would only work on Linux distros where the base64 program is available.

[1] - https://docs.wso2.com/display/Carbon420/Securing+Passwords+in+Configuration+Files

[2] - https://gist.github.com/handakumbura/f4496c7578c36775086059751e03861d

How to front a bearer token secured endpoint using WSO2 API Manager

whats covered: fronting a bearer token secured endpoint using a mediation policy for APIM 1.10.0

If a requirement arises to front a bearer token secured API while maintaining API manager authentication mechanism(though this is unlikely and should probably be avoided) it could be met using a mediation policy.

1) Create a mediation policy with the logic

The mediation policy should be such that it takes in the bearer token(of the back-end service) passed in as a custom transport level header value and passes it on to the backend service with correct formating. This can be achieved using a property mediator[1], header mediator[2] and a few synapse built in functions.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="bearersequence">
   <property xmlns:ns="http://org.apache.synapse/xsd" name="btoken" expression="$trp:token" scope="default" type="STRING"></property>
   <header xmlns:ns="http://org.apache.synapse/xsd" name="Authorization" scope="transport" expression="fn:concat('Bearer ', get-property('btoken'))"></header>
<header name="token" scope="transport" action="remove"></header>
</sequence>

download the example mediation policy from here[3]

2) Attach the mediation policy to the API In flow

Start creating an API with required HTTP methods etc, select Manage API from implementation and from Message Mediation Policies section upload the mediation policy to the in flow. Publish the API.

3) Invoke

Invoke the API with the bearer token of the backend service set to a header named "token" (as this is the header name that we have configured in the mediation policy).

 

[1] - https://docs.wso2.com/display/ESB490/Property+Mediator

[2] - https://docs.wso2.com/display/ESB490/Header+Mediator

[3] -  https://drive.google.com/file/d/0B9oVIeyHJKBXTkpBNTY2c3Rqa2s/view?usp=sharing

How to create OData Services with WSO2 DSS

whats covered: creating and consuming an odata conforming service with WSO2 DSS 3.5.0.

Liberating the data layer with services is an important step in converting a monolithic system to one that conforms to Service Oriented Architecture(SOA). WSO2 Data Services Server[1] is a solution targeted at this very requirement. In the past exposing a data-source(such as a RDBMS) as a service involved configuring the data source and mapping service operations(SOAP) or resources to predefined SQL queries(time consuming). With DSS version 3.5.0 onwards you can use Odata to provide more flexibility to data access and reduce the need of specialized SQL queries/operations.

Open Data is a protocol that specifies how data can be exposed conforming to RESTful architecture and how it can be queried using HTTP calls.

The anatomy of an Odata service URL

extracted from the Odata Spec[2]

 

Creating The Service

1) Create a Carbon Data-source

 

Navigate to Configure > Datasources > New Datasource and fill in the RDBMS details appropriately to create a data-source[3]. 

 

2) Generate a service and enable Odata

Generate a service[4] using the data-source created in step 1 . Edit the generated service, enable Odata from the datasource configuration page and publish the service. The Entity Model will be generated and the odata service will be available at,

https://<host>:<https port>/odata/<service name>

find the sql scripts and data service file used in this example here[5]

Consuming the service

The service is in conformance with the ODATA Protocol(version 4). You can access entities [6] and use query options[7] to filter and query the data as needed according to the protocol spec.

Example Invocations using CURL[8]

Select all data items belonging to BOOK resource.

curl -X GET -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK' -k

Select a single data item belonging to BOOK resource by PK,

curl -X GET -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK(1)' -k

Remove a data item belonging to BOOK resource by PK,

curl -X DELETE -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK(1)' -k

Select all data items belonging to BOOK resource where TITLE equals Cook.

curl -X GET -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK?$filter=TITLE%20eq%20%27Cook%27' -k

Select all data items belonging to BOOK resource where TITLE contains Coo.

curl -X GET -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK?$filter=contains(TITLE%2C%27Coo%27)' -k

Select all data items belonging to BOOK resource where TITLE contains Coo, result limited to 9 and ordered by TITLE

curl -X GET -H 'Accept: application/json' 'https://localhost:9443/odata/RestaurantOfTheMindService/default/BOOK?$filter=contains(TITLE%2C%27Coo%27)&$orderby=TITLE&$top=9' -k

note that the HTTP requests must be appropriately URL encoded.

[1] - http://wso2.com/products/data-services-server/

[2] - http://docs.oasis-open.org/odata/odata/v4.0/errata02/os/complete/part2-url-conventions/odata-v4.0-errata02-os-part2-url-conventions-complete.html#_Toc406398071

[3] - https://docs.wso2.com/display/DSS350/RDBMS

[4] - https://docs.wso2.com/display/DSS350/Generating+Data+Service+using+a+given+Database+Structure

[5] - https://drive.google.com/file/d/0B9oVIeyHJKBXOUEzUUExbTBqa2s/view?usp=sharing

[6] - http://docs.oasis-open.org/odata/odata/v4.0/errata02/os/complete/part2-url-conventions/odata-v4.0-errata02-os-part2-url-conventions-complete.html#_Toc406398076

[7] - http://docs.oasis-open.org/odata/odata/v4.0/errata02/os/complete/part2-url-conventions/odata-v4.0-errata02-os-part2-url-conventions-complete.html#_Toc406398092

[8] - https://curl.haxx.se/docs/manpage.html

How to extend prototyping capabilities of WSO2 API Manager

whats covered: creating API prototypes using mediation policies for APIM 1.10.0

WSO2 API Manager comes with API prototyping capability OOTB. However if you are in need of advance prototyping capabilities or feel restricted by the available implementation for the time being you could tap into the underlying mediation engine (WSO2 ESB) to meet your prototyping need.

1) Create a mediation policy with the prototype logic

The mediation policy should be such that it respond back to the client with configured response rather than passing the message back to the backend. We can achieve this requirement using Respond[1] and Payload Factory mediators[2].

 <sequence xmlns="http://ws.apache.org/ns/synapse" name="prototypesequence">
   <header name="To" action="remove"></header>
   <header name="CustomHeader" scope="transport" value="test123"></header>
   <property name="RESPONSE" value="true"></property>
   <property name="NO_ENTITY_BODY" action="remove" scope="axis2"></property>
   <payloadFactory media-type="json">
      <format>               {"id":"101","name": "dumiduh","desc": "hard coded json"}            </format>
   </payloadFactory>
   <class name="org.wso2.carbon.apimgt.usage.publisher.APIMgtResponseHandler"/>
   <respond></respond>
</sequence>

The example above is using a hard coded response body and headers, you could also populate the response with variables as required[3]. Find the example mediation policy here [4]

 

2) Attach mediation policy to API in flow

Start creating an API with required HTTP methods etc, select Manage API from implementation and from Message Mediation Policies section upload the prototype mediation policy. Publish the API.

3) Invoke

Invoke the API as you would any other managed API.


[1] - https://docs.wso2.com/display/ESB490/Respond+Mediator
[2] - https://docs.wso2.com/display/ESB490/PayloadFactory+Mediator
[3] - https://docs.wso2.com/display/ESB490/PayloadFactory+Mediator#PayloadFactoryMediator-Example3:Addingarguments
[4] -  https://drive.google.com/file/d/0B9oVIeyHJKBXb0xkTGUwSmlJc0E/view?usp=sharing

Running a Jmeter script through Java

It may be useful to run a jmeter script through java and take actions depending on assertions in the script.

1) Creating the project

generate a java project using maven and add following dependencies,

• ApacheJMeter_core
• ApacheJMeter_http
• jorphan

further dependencies[1] may need to be put if the jmeter script being run uses samplers other than http sampler or other comps such as pre-post processors.

 

2) Code

        ....
        ....   
        StandardJMeterEngine jmeter = new StandardJMeterEngine();
       
        //Initialize Properties, logging, locale, etc.
        JMeterUtils.loadJMeterProperties(JMETERPROPFILE);
        JMeterUtils.setJMeterHome(JMETERHOME);
        JMeterUtils.initLocale();
        SaveService.loadProperties();
               
        // Load existing .jmx Test Plan
        FileInputStream in = new FileInputStream(JMETERSCRIPT);
        HashTree testPlanTree = SaveService.loadTree(in);
        in.close();
       
        // Runing JMeter Test       
        Summariser summer = new Summariser();
        String testLog = JTLHOME+new Date().getTime()+".jtl";
        MyResultCollector resultCollector = new MyResultCollector(summer);
        resultCollector.setFilename(testLog);
        testPlanTree.add(testPlanTree.getArray()[0], resultCollector);      
        jmeter.configure(testPlanTree);
        jmeter.run();
        results = resultCollector.getResults();
        ....
        ....

ResultCollector class is extended so that actions can be taken based on the assertion result.

    ....
    ....   
    @Override
        public void sampleOccurred(SampleEvent e) {
        super.sampleOccurred(e);
        SampleResult r = e.getResult();
       
        ResultDTO result = new  ResultDTO();
        result.setSamplerName(r.getSampleLabel());
        result.setResponseCode(r.getResponseCode());
        result.setResult(r.isSuccessful());               
        results.add(result);
    }
    ....
    ....



Find the full example here,
https://github.com/handakumbura/jmeterrunner

Java Regex

Putting this up for my own reference more than any other reason. Java Regex are useful in manipulation strings. A regex consist of pattern blocks and optional quantifiers.

Pattern Blocks

some patterns are built in to java,

\d - digit

\w – word character

\s – whitespace

if a range of characters need to be used as a pattern it should be put as follows,

[a-z] – a to z simple

[a-zA-Z] - a to z capital and simple

[1-9] – 1 to 9

Quantifiers

these add meaning to the patterns. They should be added immediately after the pattern.

* - match pattern 0 or more times

+ - match pattern 1 or more times

{n} - match pattern exactly n times

{n,m} – match pattern between n to m times

Regex Examples

[a-zA-Z]* - a to z capital and simple 0 or more times

[a-zA-Z]+ - a to z capital and simple 1 or more times

\d{2,5} – digits 2 to 5 times

[.]*[a-zA-Z]{1,5} - . 0 or more times in the beginning followed by a to z capital and simple 1 to 5 time, the whole pattern once.