DenisTek

Mutual Authentication with Client Certificate over HTTPS/SSL using Java

2010-05-07T12:20:00.000-07:00

This blog is about SSL/TLS mutual authentication using Java. I am on the client side with a client certificate signed by an intermediate issuer and finally by Verisign. (aha, a certificate chain is here to make the situation not vanilla already.) The server requests a client certificate and recognizes Verisign as a Certification Authority (CA). You can safely ignore this blog if you have a self-certified certificate.

During the implementation, I got tripped more than a couple of times over error messages like:

403.7 Forbidden: Client Certificate Required.
javax.net.ssl.SSLException: HelloRequest followed by an unexpected  handshake message

To save somebody some time in the future, a step by step instruction is provided below:

I assume you have a valid certificate or a chain of certificates, whose root is acceptable by the server. The valid certificate contains its private key. Run the following command to verify:
```
keytool -list -v -keystore "your certificate file"
...
Entry type: PrivateKeyEntry
```
Import your certificate and intermediate certificates into a browser like IE or Firefox and test out the https URL. This step will validate the certificates and save you a lot of troubles down the road. Java version of the SSL implementation is not as simple/mature as the browsers'. Please make sure all the certificates have not expired.
Backup your keystore located at /your_home_directory/.keystore by default and the truststore located at somewhere similar to \Java\jre6\lib\security\cacerts
Use not-yet-commons-ssl utility to import your certificates into the Java keystore format. Sample command is:
```
java -cp not-yet-commons-ssl-0.3.9.jar org.apache.commons.ssl.KeyStoreBuilder 
```

Customize the following java code, replace the static final Strings to fit in your needs. Note that this implementation forcefully use a specific alias to present the corresponding certificate/certificate chain to the server. Somehow the default KeyManager simply disqualifies my certificate to be presented to the server.

public class Main {

private static final Logger logger = Logger.getLogger(Main.class.getName());
private static final String LINE_BREAKER = System.getProperty("line.separator");

private static final String CERTIFACATE_FILE = "your keystore location";
private static final String CERTIFACATE_PASS = "changeit";
private static final String CERTIFACATE_ALIAS = "your alias";
private static final String TARGET_URL = "https://xyz.com";


public static void main(String[] args) {
   String targetURL = TARGET_URL;
   URL url;
   HttpsURLConnection connection = null;
   BufferedReader bufferedReader = null;
   InputStream is = null;

   try {
       //Create connection
       url = new URL(targetURL);
       //Uncomment this in case server demands some unsafe operations
       //System.setProperty("sun.security.ssl.allowUnsafeRenegotiation", "true");
       connection = (HttpsURLConnection) url.openConnection();

       connection.setRequestMethod("POST");
       connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
       connection.setRequestProperty("Content-Language", "en-US");

       SSLSocketFactory sslSocketFactory = getFactory(new File(CERTIFACATE_FILE), CERTIFACATE_PASS, CERTIFACATE_ALIAS);
       connection.setSSLSocketFactory(sslSocketFactory);

       //Process response
       is = connection.getInputStream();

       bufferedReader = new BufferedReader(new InputStreamReader(is));
       String line;
       StringBuffer lines = new StringBuffer();
       while ((line = bufferedReader.readLine()) != null) {
           lines.append(line).append(LINE_BREAKER);
       }
       logger.info("response from " + targetURL + ":" + LINE_BREAKER + lines);

   } catch (Exception e) {
     ...
   }
 }

    private static SSLSocketFactory getFactory(File pKeyFile, String pKeyPassword, String certAlias) throws Exception {
        KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("SunX509");
        KeyStore keyStore = KeyStore.getInstance("JKS");

        InputStream keyInput = new FileInputStream(pKeyFile);
        keyStore.load(keyInput, pKeyPassword.toCharArray());
        keyInput.close();
        keyManagerFactory.init(keyStore, pKeyPassword.toCharArray());

        //Replace the original KeyManagers with the AliasForcingKeyManager
        KeyManager[] kms = keyManagerFactory.getKeyManagers();
        for (int i = 0; i < kms.length; i++) {
            if (kms[i] instanceof X509KeyManager) {
                kms[i] = new AliasForcingKeyManager((X509KeyManager) kms[i], certAlias);
            }
        }

        SSLContext context = SSLContext.getInstance("TLS");
        context.init(kms, null, null);
        return context.getSocketFactory();
    }

    /*
     * This wrapper class overwrites the default behavior of a X509KeyManager and
     * always render a specific certificate whose alias matches that provided in the constructor
     */
    private static class AliasForcingKeyManager implements X509KeyManager {

        X509KeyManager baseKM = null;
        String alias = null;

        public AliasForcingKeyManager(X509KeyManager keyManager, String alias) {
            baseKM = keyManager;
            this.alias = alias;
        }

        /*
         * Always render the specific alias provided in the constructor
         */
        public String chooseClientAlias(String[] keyType, Principal[] issuers, Socket socket) {
            return alias;
        }

        public String chooseServerAlias(String keyType, Principal[] issuers, Socket socket) {
            return baseKM.chooseServerAlias(keyType, issuers, socket);
        }

        public X509Certificate[] getCertificateChain(String alias) {
            return baseKM.getCertificateChain(alias);
        }

        public String[] getClientAliases(String keyType, Principal[] issuers) {
            return baseKM.getClientAliases(keyType, issuers);
        }

        public PrivateKey getPrivateKey(String alias) {
            return baseKM.getPrivateKey(alias);
        }

        public String[] getServerAliases(String keyType, Principal[] issuers) {
            return baseKM.getServerAliases(keyType, issuers);
        }
    }
}

Try to set

-Dsun.security.ssl.allowUnsafeRenegotiation=true

if you get the error message like:

javax.net.ssl.SSLException: HelloRequest followed by an unexpected  handshake message

If anything goes wrong, turn on -Djavax.net.debug=all to debug. Verify the keystore and truststore locations. Verify the presence of the certificates. Here is a sample log file with successful connection: http://java.sun.com/javase/6/docs/technotes/guides/security/jsse/ReadDebug.html

Grails Environment Specific Logging to a File

2010-02-25T12:35:00.000-08:00

I need to log my Grails application across different environments. For example, a Grails embeded Jetty server in DEV environment and a Tomcat server in TEST environment. Different servers have different log directory locations. How to dynamically pick up the right location? The trick is to define the file location (a relative path in my case) as a global variable inside Config.groovy, customized it in the environment blocks, and use the variable location inside log4j closure.

The following is the Config.groovy:


def logDirectory = '.'
environments {
 development {
 }
 test {
     logDirectory = "../logs"    
 }
}

// log4j configuration
log4j = {
 appenders {
     console name:'stdout'
     rollingFile  name:'file', file: logDirectory + '/app.log', threshold: org.apache.log4j.Level.INFO, maxFileSize:"1MB", maxBackupIndex: 10, 'append':true    
 }

 error  'org.codehaus.groovy.grails.web.servlet',  //  controllers
        'org.codehaus.groovy.grails.web.pages', //  GSP
        'org.codehaus.groovy.grails.web.sitemesh', //  layouts
        'org.codehaus.groovy.grails.web.mapping.filter', // URL mapping
        'org.codehaus.groovy.grails.web.mapping', // URL mapping
        'org.codehaus.groovy.grails.commons', // core / classloading
        'org.codehaus.groovy.grails.plugins', // plugins
        'org.codehaus.groovy.grails.orm.hibernate', // hibernate integration
        'org.springframework',
        'org.hibernate',
        'org.codehaus.groovy.grails.scaffolding.view.ScaffoldingViewResolver'

 warn   'org.mortbay.log'

 root {
     info 'stdout', 'file'
     additivity = true
 }
}

It has been tested to work on Grails version is 1.1.1 with Tomcat 6.0.

Grails and Query Into Database View and Multiple Datasources

2009-12-30T06:51:00.000-08:00

I want to bypass the Grails stack to query a database directly. It sounds silly. Anyway the very first reason why I chose Grails is to avoid that hassle. However I have a special case that it demands read-only queries into:

multiple databases
a database view
Netezza, a database with no hibernate dialect

In my opinion, the difficulties of 2) and 3) keep any Object Relational Mapping (ORM) solutions out of the game. Otherwise, the Grails plugin, datasources, will be a good candidate to resolve issue 1).

The following solution is a Groovy-style DAO query:

define the second datasource in grails-app/conf/spring/resources.groovy:


import org.apache.commons.dbcp.BasicDataSource
beans = {
dataSourceNetezza(BasicDataSource) {
 driverClassName = "org.netezza.Driver"
 url = "jdbc:netezza://server:5480/database_name"
 username = "xyz"
 password = "abc"
}
}

Use the dataSourceNetezza defined above in a controller where the Spring application context is available. Note that the dataSourceNetezza is auto-wired by Spring and the names have to match.


import groovy.sql.Sql

class TablesController {

def dataSourceNetezza

def list = {
 if (servletContext.dbList == null)
  servletContext.dbList = new Sql(dataSourceNetezza).rows("SELECT distinct DATABASE_NAME FROM REPOSITORY")
 print servletContext.dbList
}
 ...
}

This approach looks extremely simple with less than 15 lines of code. Keep in mind that, there is no ORM.

Grails and Legacy Database

2009-12-14T10:22:00.001-08:00

Recently I played around Grails with a legacy database. Here are some tips I gathered around the journey:
1. I chose to use SpringSource Tool Suite, an IDE based on Eclipse. I love its feature to bring up Grails command prompt with a convenient short cut of 'ctrl-alt-g'.
2. Google 'GRAG' and use its latest version to reverse engineer your legacy database. GRAG certainly has room for improvement but it does a decent job to create the domain objects and save you a lot of typing. BTW, it's totally fine if you don't want to use GRAG and prefer to manually generate the domain classes.
3. Refine the domain definition. Note that Grails/Hibernate uses 'id' as the default field name for primary keys. You may not wish to change this convention or define another field as existingId. Unfortunately that's just the default behavior of GRAG 1.1.


static mapping = {
 table 'Legacy_Table'
 version false
 id column:'Existing_ID'
} ...

4. If the existing primary key is in the type of a String the domain definition needs additional tweak:


static mapping = {
 table 'Legacy_Table'
 version false
 id column:'String_ID', generator:'assigned'
}
String id

An exception of 'Fail to convert to internal representation' will be thrown if you fail to declare the id as a String

5. If you are careful enough you may find that the generator is defined as 'assigned' instead of the default value. This means the application will generate the primary key. Accordingly the save() operation has to explicitly set the String 'id'. Controller's save operation will look like:


   def save = {
       def legacyTable = new legacyTable(params)
legacyTable.id = params.id
       if(!legacyTable.hasErrors() && legacyTable.save())
       ...
    }

Please let me know if it works. I will be surprised if not.

One of the limitations Grails have is that, the 'version false' statement basically turns off the versioning from Gorm and its underline Hibernate implementation. If two users update the same database entry simultaneously, the system's behavior may not be predictable. In another word, we are counting on the optimistic lock on the data integrity.

Multiple Installation Instances of IBM Websphere Commerce 6.0 Developer

2009-08-05T09:06:00.000-07:00

Websphere Commerce Developer installation is really intensive. It's perfectly fine to have multiple Toolkit setup on the same machine for multiple profiles, for example, two different fix pack versions.

Steps:
1. Install or reuse existing installation of RAD/WAS
2. Install Commerce to a new location
3. Run setup.bat if you don't have a setup.log in ${WCToolkitEE60_ROOT}\logs
4. Run setdbtype to point the Commerce to the right database

That's it.

Spawn a Thread in a Servlet

2008-10-01T14:07:00.000-07:00

When I search this topic on Google, the first impression is that, 'Stop! Just Don't Do It'. Threading code is error prone, not mentioning in a managed container. Additionally there are alternatives like Message Driven Beans or Java Message System, which are specially designed for this purpose. However, you may have a specific situation where threading in Servlet seems to be the only viable solution, for example the JSP page kicks off a long running process at the backend. What can you do? Fortunately it's totally doable if you are careful enough.

First, let's evaluate the risks. Obviously if your servlet is going to spawn multiple threads, this servlet cannot serve many concurrent users at the same time. Otherwise the thread pool will easily dry up and brings the server down to its knees. Concurrent access may cause race conditions and subtle synchronization problems too, especially in a load-balanced or clustered environment. Thread leak may be another risk. If not cleaned up appropriately, each leaked thread will have a reference to its classloader and prevent this classloader from being garbage collected. After enough reload of the application, you may run into out of PermGen space memory errors.

If the risks mentioned above do not apply to your particular situation or didn't scare you off, let's continue. The solution is as the following:
1. Instantiate your thread object as a daemon in your Servlet or ServletContextListener and call yourThread.start();
2. Store it as an attribute in your application context;
3. Return the HttpServletResponse;
4. Clean up the thread when the servlet or the application context is destroyed
5. Don't pass HttpservletRequest/Response/Session objects to the thread

List JNDI Names

2008-08-18T13:10:00.000-07:00

Sometimes we need to list all the JNDI names in an application server for debugging purpose. Other times, we may wish to list available datasources in a dropdown list for end-users to choose from. How to do this? In a nutshell, the JNDI bindings are organized in a tree strucuture. The following code is a simple example to recursively exhaust this binding tree:


Context ctx = (Context)new InitialContext().lookup("java:comp/env");
listContext(ctx, "");

/**
* Recursively exhaust the JNDI tree
*/
private static final void listContext(Context ctx, String indent) {
try {
   NamingEnumeration list = ctx.listBindings("");
   while (list.hasMore()) {
       Binding item = (Binding) list.next();
       String className = item.getClassName();
       String name = item.getName();
       logger.log(Level.INFO, indent + className + " " + name);
       Object o = item.getObject();
       if (o instanceof javax.naming.Context) {
    listContext((Context) o, indent + " ");
       }
   }
} catch (NamingException ex) {
   logger.log(Level.WARNING, "JNDI failure: ", ex);
}

Tag Libraries: JSTL, NetUI, Spring MVC, or Struts Tag Library

2008-08-12T05:09:00.000-07:00

Not occasionally JSP developers need to iterate through collections and evaluate conditions before presenting the data. Tag libraries come in handy to provide a solution. However, there are quite a few Tag Libraries available, for example, the JSTL, NetUI, Spring MVC, and Struts tag libraries. Which one to choose? The quick answer is to use JSTL whenever possible. Use other tag libraries only when absolutely necessary.

Tag Libraries are not much different than any other utility libraries. Different vendors provide different features. JavaServer Pages Standard Tag Library (JSTL) is the most-accepted one and still powerful enough for common usages. If a non-standard tag library is chosen for additional features and convenience, other developers may have to climb a learning curve before being productive.

Additionally JSTL is designed to complement the HTML tags with 'logic control'
capabilities. This is in contrast to NetUI tag libraries, who replaces the HTML tags. Another example is Spring MVC's tag library. It is tightly integrated ONLY with the Spring MVC framework for convenient data binding. Tag libraries competing with HTML tags may create further compatibility issues. Sometimes JSP pages are touched by both developers and UI design artists. The UI design artists do not want to see any specialized tags by all means. Struts Tag Library used to gain a lot of popularity due to the success of the legend Struts framework. Now its document website has comments like, 'Some of the features in this taglib are also available in the JavaServer Pages Standard Tag Library (JSTL). The Struts team encourages the use of the standard tags over the Struts specific tags when possible.'

In a conclusion, JSTL tag library is the standard way to provides common logic processing features, such as to iterate through collections and to evaluate conditions. It complements the HTML tags. It will be the best choice for most common situations. Alternative ones, like NetUI, Spring MVC, or Struts Tag Library, may be chosen for additional functionalities.

Alternative Ways to Define JSP/Servlet Attribute Scopes

2008-08-06T06:23:00.000-07:00

As we know, there are four types of scopes that a JSP/Servlet attribute can use:
Application: The attribute lasts the lifetime of an application instance.
Session: The attribute survives throughout the conversation session.
Request: The attribute will be discarded after the request is serviced.
Page: The attribute is only visible within a particular JSP page.

To keep data integrity, it's important to put attributes into the right scope. Most of the time developers will retrieve attributes from requests and put them into appropriate scopes thereafter. There are alternative ways.

1. Set application scope attributes in a ServletContextListener.
Sometimes "static" information should be loaded once and visible throughout the lifetime of an application. For example,


web.xml
    <listener>
        <listener-class>
            com.denistek.SampleListener
        </listener-class>
    </listener>

Java
public class SampleListener implements ServletContextListener {
    @Override
    public void contextInitialized(ServletContextEvent event) {
     //... get your attributes
        event.getServletContext().setAttribute("attributes", attributes);
    }

    @Override
    public void contextDestroyed(ServletContextEvent event) {
    }
}

2. Set application scope attributes in a Servlet with <load-on-startup> features.
This is very similar to the above approach. For example,


web.xml
    <servlet>
        <servlet-name>SampleServlet</servlet-name>
        <servlet-class>com.denistek.servlet.SampleServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

Java
    @Override
    public void init() throws ServletException {
        super.init();
        //... get your attributes
        getServletContext().setAttribute("attributes", attributes);
    }

3. Use filters to define session attributes.
Sometimes there are multiple entry points where a session attribute can be updated. In this case we'd better use a filter to watch for this kind of update instead of scattering the logic across various JSP's/Servlets. For example,


web.xml
    <filter>
        <filter-name>SampleFilter</filter-name>
        <filter-class>com.denistek.SampleFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>SampleFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

Java
public class SampleFilter implements Filter {

    private SampleAttribute[] attributes = null;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) 
            throws IOException, ServletException{
        HttpServletRequest rq = (HttpServletRequest) request;
        //... populate attributes from request here
        rq.getSession().setAttribute("sampleAttributes", attributes);
        chain.doFilter(request, response);
    }

    @Override
    public void init(FilterConfig config) {
        //... populate attributes from context here
        attributes = config.getServletContext().getAttribute("sampleAttributes");
    }

    @Override
    public void destroy() {
    }
}

Configure Tomcat 6 DataSource using Sql Server 2005

2008-07-28T12:11:00.000-07:00

This is a step-by-step instructions on how to configure Tomcat 6 DataSource using Sql Server 2005. The installation of Tomcat and SqlServer is not covered.
1. Verify that you can login to the SQL Server using 'SQL Server Authentication'. You may wish to change the 'Server Authentication mode' to 'SQL Server and Windows Authentication mode'. You may also wish to check that the particular user's status of Login is 'Enabled'.
2. Verify that 'Local and remote connections' is enabled.
Go to Microsoft SQL Server 2005>Configuration Tools>SQL Server Surface Area Configuration>Remote Connections: Enable TCP/IP
3. Restart the database
4. Download and drop the JDBC driver to tomcat_home/lib. You may find tomcat-dbcp.jar in the directory. This will make the driver available to both the Tomcat internal classes and the web application.
5. Change the context.xml, for example:


<?xml version="1.0" encoding="UTF-8"?>
<context path="/BFS2">
    <Resource name="jdbc/TestDB" auth="Container" type="javax.sql.DataSource"
          maxActive="100" maxIdle="30" maxWait="10000"
          username="sa" password="passw0rd" driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
          url="jdbc:sqlserver://localhost:1433;databaseName=TestOne"  />
</context>

6. Change the web.xml, for example:


<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xsi="http://www.w3.org/2001/XMLSchema-instance" schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <resource-ref>
        <description>DB  Connection</description>
        <res-ref-name>jdbc/TestDB</res-ref-name>
        <res-type>javax.sql.DataSource</res-type>
        <res-auth>Container</res-auth>
    </resource-ref>
</web-app>

7. Write a testing JSP page like this:


<%@   page   contentType="text/html;charset=UTF-8" %>   
<%@   page   import="java.sql.*" %> 
<%@   page   import="javax.sql.*" %> 
<%@   page   import="javax.naming.*" %> 
<HTML>
<HEAD>
<TITLE>JSP example</TITLE>
</HEAD>
<BODY>
  <h1>Hello,test JNDI !  </h1>
  <%
    Context ctx = new InitialContext();  
    Context envctx =  (Context) ctx.lookup("java:comp/env"); 
    DataSource ds =  (DataSource) envctx.lookup("jdbc/TestDB");  
    Connection  conn=ds.getConnection();   
    Statement  st=conn.createStatement();
    String    sql="select * from status";  
    ResultSet    rs=st.executeQuery(sql); 
    while(rs.next())   {
  %>   
  ID:<%=rs.getInt(1) %>  
       Value:<%=rs.getString(2) %> 
       <br>
  <%
   }
  %>   
  Here is just JNDI datasource SQL Server 2005 + tomcat example 
  <%
   rs.close(); 
   st.close();  
   conn.close();   
  %> 
</BODY>
</HTML>

That's it!

5 Minute Introduction to Dependence Injection (DI, IoC)

2008-06-19T07:56:00.000-07:00

Newbies are confused by the words, Dependency Injection (DI) and Inversion of Control (IoC). I was confused. This article is for those who dare to add these hot and fancy words onto their resume but have only 5 minutes bandwidth to learn the details. Without any further delay, this is the explanation.

Minute One: Concept Background
Generally in a Object Oriented Language (OOL) like Java, there are no more than three relationships between two classes: inheritance, reference, and aggregation. Inheritance and polymorphisms are powerful features already built-in in the Java language. However, as to the the 'reference' relationship, it was awkward in Java until the appearance of DI.

Minute Two: Traditional Coding Style
Traditionally, we write code like this when ClassB has a reference relationship with ClassA:


public ClassA implements InterfaceA { ...}
public ClassB {
 InterfaceA a = new ClassA();
}

If we change our mind to plug in another implementation, say ClassAA, then we have to go back to the ClassB's source code to change at least one line of code:
InterfaceA a = new ClassAA();

In another word, ClassB is still strongly coupled with ClassA.

Minute Three: DI Coding Style
DI code looks like this:


public ClassA implements InterfaceA { ...}
public ClassB {
   InterfaceA a;
   public void setA(InterfaceA input) { a = input; }
    ....
}

Note that ClassB does not prescribe whether ClassA or ClassAA will be used prematurely. Only at the run time, the container will render either ClassA or ClassAA to ClassB according to the configuration. Details of the container and the configuration file are intentionally ignored here because they are trivial to the concept understanding. By the way, the new setA() method is needed by the container.

Minute Four: So What?
What is the big deal to change one line of code into another line of configuration with more lines of code and a totally new container?

The gain is that, now the hard-coded relationships are minimized. Only ClassB-InterfaceA and ClassA-InterfaceA relationships are maintained. The ClassB-ClassA relationship disappeared! Finally ClassA and ClassB are totally independent to each other. Each one can have it's own changes without affecting the other party and finally we can easily swap ClassA out without touching ClassB.

Minute Five: Spring Framework
DI and AoP are the two fundamental ideas behind the Spring Framework, with which our Plain Old Java Objects (POJO) can be easily enriched by a complete pack of enterprise level features, such as transaction control, persistence, security and etc. See http://www.springframework.org/ for more details.

In my humble opinion, DI is simply the most important concept after Object Oriented concepts in Java.

Glassfish and Windows Service

2008-05-27T11:49:00.000-07:00

If you want to automatically restart a Glassfish domain on Windows platform, you need to pay attention to some dirty details:
1) Try not to install Glassfish under a directory with space. It will save you some troubles down the road. If you happen not to do so the first hand, be prepared to customize the following script to set up the Windows Service:

C:\windows\system32\sc.exe create domain1 binPath= "C:\Program Files\glassfish-v2ur1\lib\appservService.exe \"\\\"C:\Program Files\glassfish-v2ur1\bin\asadmin.bat\\\" start-domain --user admin domain1\" \"\\\"C:\Program Files\glassfish-v2ur1\bin\asadmin.bat\\\" stop-domain domain1\\\"" start= auto type= own type= interact DisplayName= "Glassfish"

Please pay special attention to those ugly escapes characters.

2) When you log out of Windows, the java.exe will stop while the Windows Service is still running. The JVM shuts itself down upon the user logout event. This "feature" can be disabled by modifying the Glassfish's domains\domain1\config\domain.xml. Open the file and search for jvm-options; add a new line at the bottom of the options for the new -Xrs JVM option following the format in the file; save the file and finally restart the Glassfish Windows Service to activate the change.

JDBC Performance Tuning

2008-05-09T11:28:00.000-07:00

When a JDBC call is slow, you can do a performance tuning on both the Java code side and the database server side. The following is a list of tips.

Java side performance tuning:

Time your database access to have an idea whether the Java code or the database is the culprit.
Make sure you are using the right JDBC driver. There are four types of JDBC drivers. Some types, like JDBC-ODBC bridge, are doomed to be slow.
Be sure to use database connection pool unless your application is entirely single threaded.
Be sure to close database resources, such as Connection, Statement, ResultSet even in situations when Exception's are thrown.
Use PreparedStatement correctly.

Database side performance tuning:

Determine whether indexes are necessary to create.
Update database statistics.
Run Explain Plan or similar tools on expensive queries to avoid full table scan.

These tips sound like vanilla. However they are proven effective ways to kill 80% of your JDBC performance issues.

How to log JDBC Calls to a Stored Procedure

2008-05-08T08:28:00.000-07:00

When debugging nasty problems, JDBC developers want to see logs. Of course we can log on the Java code side. A lot of times it is still not good enough. What if the problem is from the JDBC driver? We want to see what is really going on inside a stored procedure and what exactly do the input parameters or the complicated query statement look like.

One simple approach is to insert the statement as a string into a log table. The following is an example in the syntax of SQL Server:
declare @stmt1 nvarchar(2000)
declare @stmt2 nvarchar(2000)

select @stmt1 = '...'
select @stmt2 = '...'

if ( @input is not null ) begin
select @stmt1 = @stmt1 + '...'
select @stmt2 = @stmt2 + '...';
end

select @stmt1 = @stmt1 + @stmt2
exec sp_executesql @stmt1
insert into LogTable values (@stmt1, CURRENT_TIMESTAMP)

In a conclusion, the trick is to write the statement as a string, insert it into a table before execution.

How to Test a Web Service

2008-05-06T09:07:00.000-07:00

When you develop a Web Service application, either on the client side or on the server side, you need a tester tool. If you are on the client side, before spending too much time on your code, it's good to test whether the Web Services server is stable and have some general idea of the sample requests and responses. If you are on the server side, you definitely need a tester to prove your deliveries. Sometimes developers come up with an in-house GUI or web application tester and end up in a situation where there are more bugs in the tester than the application itself. Fortunately there are free, open-source and stable tools available nowadays. We don't need to reinvent the wheel.

Most of the time, Web Services are nothing more than SOAP/XML over HTTP. In the simplest cases, any tool that can send text over HTTP to an URL can be used to test Web Services. But you will find it a must-have to generate sample SOAP requests from any given WSDL. The famous XMLSpy provides this kind of feature, unfortunately with a hefty fee. The free version of soapUI by Eviware is my favorite.

Log Web Services Incoming Requests in JAX-WS

2008-04-25T08:24:00.001-07:00

For any server side Web Services development, logging the incoming requests is crucial. The following code snippet gives an example. Even though this particular example uses SOAPMessageContext from JAX-WS's SOAPHandler, the approach can be easily modified to fit into other Web Services solutions.

ByteArrayOutputStream buffer = new ByteArrayOutputStream();
soapMessageContext.getMessage().writeTo(buffer);
log.info("SOAP Request:\n" + new String(buffer.toByteArray()));

Note that a lot of times logging comes hand in hand with incoming requests validation. Read this blog for more details.

Continuous Integration

2008-04-25T08:20:00.000-07:00

Agile Methodologies demand frequent deliveries in short cycles. It's desirable to fully automate the mundane process to checkout latest code, build, deploy, test, and send out notification on errors. Continuous Integration (CI) come into the play. There are some other significant benefits to adopt a CI tool:

A fully automated process will improve the quality standard for build processes. You don't need to worry about scenarios when the code works on programerA's environment, but not programerB's.
Code quality will be improved. For a large project with many programmers, it is inevitable to accidentally check in uncompilable code. You can count on the CI tool as a safety net to catch the errors before they pollute the whole team's working environments. To be specifically, the CI tool can be configured to check out code every 15 or 30 minutes, catch the error, and send out notifications accordingly.

You may have vivid memory how frustrated you were when hours of debugging work only pointed to non-reliable build process or non-stable code repository. So you cannot wait to try out a CI tool. Amazingly the efforts needed are trivial. You may only need a hardware, an open source CI tool, and some configuration work. For the hardware, you need a dedicated box to do the CI work exclusively. For the software, there are several major players on the CI arena, for example, Hudson, Cruise Control and Continuum. Java World has a good article to compare them: http://www.javaworld.com/javaworld/jw-11-2006/jw-1101-ci.html

Most configuration work is easy if you are familiar with build scripts like Maven or Ant. It may be a little bit challenging to configure those scripts to be environment-agnostic. You may wish to put all environment-specific variables into a configuration file, which includes, directory locations, database connection strings, external system URL's and etc.

Limitations of Agile Methodologies

2008-04-25T07:29:00.000-07:00

Agile Methodologies are adaptive, iterative and incremental. Their effectiveness is well recognized in the industry. However, there is no silver bullet. The Agile Methodologies have their limitations and demand a close interaction among end users, business analysts, developers and testers in short cycles. It implies a break down of walls among divisions and the formation of a coherent and collaborative team from different groups. The Agile Methodologies work best for co-located teams. In certain circumstances this is simply too expensive or politically impossible. Another consideration point is the team size. When the size of the development team increases, it's agility decreases.

In a conclusion, think twice before adopting Agile Methodologies for large scale projects or distributed development teams.

Agile Methodologies vs Waterfall

2008-04-22T13:06:00.000-07:00

Let's consider the simplified version of a complicated question, how to develop a system composed of ComponentA and ComponentB? ComponentA can be a software sub-system. ComponentA can also be deliverable artifacts like documentation, design, or code. The assumptions are: ComponentB cannot be started until ComponentA is finished; both ComponentA and ComponentB take significant efforts to finish.

The straightforward answer is:

Step One: Finish ComponentA;
Step Two: Finish ComponentB;
Optional Step Three: Assemble ComponentA and ComponentB together.

This is exact the sequential waterfall approach. It mimics what an assembly line has been doing in the mature manufacturing industry over decades of years. However, software is quite different from hardware. By its very nature of flexibility, software development process is much less predictable. Big surprises (or risks) may arise:

The ultimate goal of the project is a moving target;
By any chance, if ComponentA is delayed, all components down the chain will be delayed;
If a working project cannot be delivered finally, ComponentA may be totally useless.

To control the risks, Agile Methodologies, the iterative, adaptive and incremental approach, are preferred. The process looks like:

Step One: Finish (ComponentA-- and ComponentB--); "--" means a simplified version of the system;
Step Two: Finish (ComponentA- and ComponentB-);
Step Three: Finish (ComponentA and ComponentB).

The benefits are listed below:

Be adaptive to changing requirements. End user's feed back can be heard early. To do the right project is more important than to do the project right.
Progress and deliveries are very visible. We don't have to wait until the Step Three to see some workable results.
If by any chance, there are limited budget, time, or resources to finish the Step Three, the results of Step One and Step Two are still acceptable working systems.

For any approaches adaptive in their nature, people will raise the question, how to make the short term targets in line with the long term goals? This question has an assumption that the 'long term goals' are well known. Agile Methodologies help most when the 'long term goal' is unclear or when the process how to achieve the 'long term goal' is unclear. The short term cycles help to clarify things during the iterative process. If both the 'long term goal' and the process how to achieve it are crystal clean and carved into a stone, go ahead using Waterfall with no hesitation.

It's my motto that - Release early, release often, and release working products. However there is no panacea to resolve all software quality and delivery challenges. In another post here, I discussed limitations of Agile Methodologies.

java.lang.NullPointerException when accessing ResultSet from SQL Server

2008-04-21T13:22:00.000-07:00

I used Java JDBC to call a SQL Server Stored Procedure and got a NullPointerException from the last line of the following innocent code below:
stmt.execute();
rs = stmt.getResultSet();
while (rs.next()) ...

The stored procedure looks like:
operation 1
operation 2
select ... from ...

It turns out that 'SET NOCOUNT ON' should be added into the stored procedure before the 'operation 1'.