Performance: Grails vs. Java

Background
I wanted to compare the Grails platform vs. a straight Java platform with a similar stack, to figure out which would provide the best dev and deploy environment for new web projects. Grails has development advantages with its Groovy language and plugin architecture, but Groovy's rampant runtime indirection has an implied cost.

Before I ran into Grails I was a long-time user (and occasional submitter) of the AppFuse framework,  an awesome rapid development platform for Java web-apps.  I was rather pleased to find that Grails has quite a bit in common with the AppFuse stack.  For example:

• Spring IOC-based
• SpringMVC-based (one option among AppFuse's choices)
• Hibernate persistence
• AspectJ AOP
• SiteMesh templating
• SpringSecurity

Conjecture
A comparison of the server portion of the two mentioned platforms would be a decent comparison of performance between a Grails webapp vs. a plain Java webapp.

Objective
My intention is to isolate the server-side processing as much as possible, by reducing client generation to a bare minimum.  Ideally this test would eliminate client page generation altogether and simply invoke server operations via a REST/JSON interface.

I decided not to do that though since it would require significantly more effort, and also introduces I believe a variance in the processing; while AppFuse comes bundled with CXF for this purpose, Grails uses its native DSL on top of SpringMVC plus your choice of JSON processor to produce the same.  While comparing these 2 would be interesting enough, it wasn't my primary objective.

Approach
To reduce variance in the platforms I am simply using SpringMVC to process two kinds of requests:

• 'Create/Add' request
• a nearly empty 'Retrieve/List' request

Initially I had included the list generation, but as the list size grows the DisplayTag functionality in AppFuse begins to take forever to render.  Result of that test posted at the end as an FYI.

Environment
All of the tests were built and run with the following:

• Oracle Java 1.6.0_38 
• started with Java1.7.0_10, but was forced back - read on
• Linux Mint 14
• Grails 2.1.1
• AppFuse 2.x
• Maven 2.2.1 
• MySQL 5.5.29 for debian-linux-gnu
• Apache Tomcat 7.0.32
• Apache JMeter 2.8
• JProfiler7
• Hardware, as reported by Gnome system monitor:
• Intel® Core™ i7-3610QM CPU @ 2.30GHz × 8
• 15.6 GiB

General Test Layout
To produce equivalent tests in both platforms I created 2 POJOs

• Person
• Account

with a simple one-to-one relationship from Person to Account, and 2 functions:

• add new Person and Account
• list all Persons

The test would run 100 users all creating Person/Account objects and retrieving the entire list of Persons.  (Note: Account would be lazily-loaded in each case and hence in the test as posted would not be a factor).


Expectation
I fully expected the AppFuse platform to easily trounce the Grails platform, since Grails is encumbered by DSLs and Groovy, and does a lot of dynamic class decoration for things like installing Hibernate functionality onto domain classes.


Grails Test Creation
To produce the desired test in Grails I used the usual combination of Grails generators:

• grails create-app
• grails create-controller

and 'manually' created 2 domain classes:

class Person {

    String name
    Account account


    static constraints = {
        name(unique:true)
        account(nullable:true)
    }

    String toString() {
        return name
    }

    boolean equals(Object input) {
        return this.name == input.name
    }
}

class Account {

    String accountId
    Float balance
    transient Float amount

    static belongsTo = Person
    static transients = ['amount']

    static constraints = {
        accountId(blank:false)
        amount(blank:true,nullable:true)
    }
}

Then I modified the generated Person list view under grails-app/views/person/list.gsp to only display the total count of Person records.

A zip of the project can be downloaded here.

Java Test Creation
I used AppFuse's project creation maven plugin with the SpringMVC archetype to generate the project.

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes -DarchetypeArtifactId=appfuse-basic-spring-archetype -DarchetypeVersion=2.2.1 -DgroupId=com.uss -DartifactId=txtest3 -DarchetypeRepository=http://oss.sonatype.org/content/repositories/appfuse

then included the full source from the AppFuse framework

mvn appfuse:full-source

and created 2 nearly identical POJOs, 'manually'

@Entity
@Table(name="account")
public class Account extends BaseObject {

    private Long id;
    private String accountId;
    private Float balance;
    private transient Float amount;


    @Id @GeneratedValue(strategy = GenerationType.AUTO)
    public Long getId() {
        return this.id;
    }
    public void setId(Long id) {
        this.id = id;
    }
  
  
    @Column(name="accountId", length=50)
    public String getAccountId() {
        return accountId;
    }

    public void setAccountId(String accountId) {
        this.accountId = accountId;
    }


    @Column(name="balance")
    public Float getBalance() {
        return balance;
    }

    public void setBalance(Float balance) {
        this.balance = balance;
    }


    @Transient
    public Float getAmount() {
        return amount;
    }
  
    @Override
    public String toString() {
        return "AccountId:: " + getAccountId();
    }
    @Override
    public boolean equals(Object o) {
        if (! (o instanceof Account)) {
            return false;
        }
        return this.getAccountId().equals( ((Account)o).getAccountId() );
    }
    @Override
    public int hashCode() {
        return accountId.toCharArray().hashCode() + (id == null ? 0 : id.intValue());
    }
  
    public static void main(String[] args) {
        HashSet s = new HashSet();
        s.add("test");
        s.add("test #2");
      
        System.out.println("Hash contains test: " + s.contains("test"));
        System.out.println("Hash contains test #2: " + s.contains("test #2"));

    }
}

@Entity
@Table(name = "person")
public class Person extends BaseObject {
   
    private Long id;
    private String name;
    private Account account;

    @Id @GeneratedValue(strategy = GenerationType.AUTO)
    public Long getId() {
        return this.id;
    }
    public void setId(Long id) {
        this.id = id;
    }

   
    @Column(name="name", length=50)
    public String getName() {
        return name;
    }
    public void setName(String in) {
        name = in;
    }
   
    @ManyToOne(optional=true,cascade=CascadeType.ALL,fetch=FetchType.LAZY)
    @JoinColumn(name="acct_id", nullable=true, updatable=false)
    public Account getAccount() {
        return account;
    }
    public void setAccount(Account in) {
        account = in;
    }

   

    public String toString() {
        return "id: " + getId() + "\tname: " + getName();
    }

    public boolean equals(Object input) {
        if (! (input instanceof Person)) {
            return false;
        }
        return this.getName().equals( ((Person)input).getName() );
    }
    @Override
    public int hashCode() {
        return name.toCharArray().hashCode() + (id == null ? 0 : id.intValue());
    }
}

and generated the scaffolding

mvn appfuse:gen -Dentity=Account
mvn appfuse:gen -Dentity=Person

Note: This is where I learned that AppFuse requires Java6 due to an issue with the declaration of the Hibernate descriptors.  (TODO:  Citation)

and modified the generated Persons list view src/main/webapp/WEB-INF/pages/persons.jsp  to only report the total count of Person records.

A zip of the project can be downloaded here.



Deployment
Both projects were independently deployed to Tomcat as packaged WARs and pummeled with 10k requests from a JMeter script.

Results
This is a snapshot of the JMeter results* side-by-side. The results I am focusing on are:

• Person List:   19ms avg (Grails) vs. 112ms avg (Java)
• Add Person: 37ms avg (Grails)  vs. 134ms avg (Java)

 also available here.

*Note: The AppFuse result shows 20k+ total samples due to additional requests required to negotiate security; hence the "Person List" and "Add Person" results should be compared directly.

Complete results are here:

Grails app:
JMeter Performance Results
Zipped JProfiler output

Java app:
JMeter Performance Results
Zipped JProfiler output

Note: The Call Tree results for the Java app show an apparent anomaly in that the sub-tree does not seem to sum up to the parent folder:

53.9% - 80,177 ms - 500 inv. org.hibernate.Criteria.list

5% - 5,217 ms - 5,642,324 inv. com.uss.model.Account_$$_javassist_3.getHibernateLazyInitializer

2% - 3,271 ms - 5,642,324 inv. com.uss.model.Account.<init>

7% - 1,081 ms - 5,642,324 inv. org.appfuse.model.BaseObject.<init>

6% - 954 ms - 5,642,324 inv.com.uss.model.Account_$$_javassist_3.setHandler

6% - 938 ms - 5,642,324 inv. com.uss.model.Person.setId

6% - 925 ms - 5,642,324 inv. com.uss.model.Person.setAccount

6% - 874 ms - 5,642,324 inv. com.uss.model.Person.setName

I contacted the producer of JProfiler and was assured this is due to filtering, so not all of the child nodes are actually shown. I'm not very happy with how they decided to indicate that since the end result is that it is misleading, and how many managers will suffer through such an explanation - but worse IMHO is that their stance is that the numbers in the child nodes should not be expected to add up to their parent values. That has not been my experience with profilers over time. Anyway here was the explanation I received:

Thanks for your email. The explanation is this:

"org.hibernate." is not a profiled package in the default session settings (se the
"Filter settings" tab of the session settings).

Only the first call into this package is measured. Then there are a lot of
internal calls in that package and into other unprofiled packages that take some
time. That time is attributed to "org.hibernate.Criteria.list". Deeper in the call
tree some profiled classes are reached, for example in the "com.uss." packages.
Those are shown separately. In general, the summed execution times of the children
in the call do not have to add up to the execution time of the parent.

Conclusion
I scratched my head for a while when I first saw the JMeter results -  then I decided to use a profiler.  That's when I discovered and removed the horrible DisplayTag taglib. AppFuse was much more competitive afterwards.

But I am still trying to figure out where the significant differences lie.  Some known differences are:

•  AF includes the SpringSecurity filter, which I did not remove prior to testing

• however the profiler only shows this requiring < 1% of overall thread processing
• AF and Grails access to the Hibernate layer are similar - both use HibernateTemplate
• AF by default uses Criteria to list objects
• Grails .list() method uses Criteria as well  (see org.codehaus.groovy.grails.orm.hibernate.metaclass.ListPersistentMethod)

But I am left with concluding that Grails offers development benefits plus runtime benefits, and so will likely be my choice of platform. 


Ways to Improve This Test
Some things I'd like to do when I get time:

• disable the AF SpringSecurity filter
• look up a single record and access the Account sub-object
• add a one-to-many relationship