{Snamell(it)}

  • {Snamell(it)}
  • Blog
  • About
  • Location
  • Contact Us

Category Archives: Java Stuff

July 31, 2013 · pti

Separating business and stuff on the JVM

Java Apps and the External Environment

Table of Contents

  • 1. Overview
  • 2. Patterns for External Files
    • 2.1. Standalone Directory
    • 2.2. Resource Loading Files
    • 2.3. Separate Internal Configuration from External Configuration
    • 2.4. Logging and Monitoring
      • 2.4.1. What if the business asks for Special Monitoring
      • 2.4.2. Real-time monitoring and administration
    • 2.5. Complying to OS rules through packaging.
      • 2.5.1. Main Deploy folder
      • 2.5.2. Config files
      • 2.5.3. Data files
      • 2.5.4. Log Files
      • 2.5.5. Dotfiles
    • 2.6. Apps deployed on a runtime platform
  • 3. Conclusion

1 Overview

Java applications, although isolated by the jvm and standard library
from most OS details, need to interact with the environment to be
useful. We need to configure the app and this is often file based,
and we need to store data and logfiles somewhere.

The danger is that this will introduce subtle dependencies to the
underlying operating system. This causes friction when used in other
environments like development and testing.

Alternatively certain OS-es make assumptions on how things are
handled there. Debian has some strong opinions on what should be
where.

2 Patterns for External Files

2.1 Standalone Directory

Most java apps seem to have chosen the standalone directory pattern
as basis of their deployment.

$ tree -d Tools/apache-maven-3.0.4
Tools/apache-maven-3.0.4
├── bin
├── boot
├── conf
└── lib
    └── ext

$  tree -d -L 2 Tools/apache-tomcat-6.0.35
Tools/apache-tomcat-6.0.35
├── bin
├── conf
├── lib
├── logs
├── temp
├── webapps
│   ├── docs
│   ├── examples
│   ├── host-manager
│   ├── manager
│   └── ROOT
└── work

This has many advantages. All references to files can now be done
using relative links based from the root folder. The amount of
assumptions that need to be made about the underlying OS is minimized.

In the case of the apache folks these differences are handled in
platform specific startup scripts, but the app hardly sees anything
of it. Now these startup scripts are very complicated and are reused
and refined over many projects.

For our own projects we do not need to make such complicated startup
scripts, but this is the right place to put this glue between the OS
and the app. If it is buried in the app, it is also complicated, and
impossible for sysadmins to modify when the deployment conditions
change.

Many frameworks support this way of working by exposing the root
directory in a configuration variable, allowing easy configuration
relative to the root.

2.2 Resource Loading Files

Many Java products use resource loading files from the classpath
instead of directly opening files.

This makes it easy to provide sensible defaults in the jar
files. Maven has also special support for this by adding the
resources folder to the classpath before the classes and the jars.
In case of testing the src/test/resources is added before that. On
deployment the ${appname}/conf folder is added before the jars.

By putting the right config file in the right location for default
(src/main/resources), test (src/test/resources), deploy
(${appname}/conf), the app is properly configured without the need
for any smarts in the app.

2.3 Separate Internal Configuration from External Configuration

This is especially for Spring, i.e. it would be suicide to do
otherwise, but it is in fact generally applicable. The point is that
some configuration is intended to be changed by the sysadmins and
some is not.

Writing modular, loosely coupled software is great and good
practice. Glueing stuff together using some form of configuration
file is just as great. Now part of this is real product design and
changing it will make it a different product. This includes how the
core pieces are wired together. This part should be internal and
separated from the external config.

Other configuration is related to details which do not alter the
purpose but fill in changing details. Like ip-addresses, names,
email, database connections, … and other related detail
config options. These we will find in the external config files.

Note that significant parts of the app can be provided by plugging in
components. Of course these components need to be externally
configured too. So these are in external config files.

Import the external files and the internal files in a way that the
external files can override the internal ones.

Copy the default external config files to the ${appname}/conf
folder so the admin needing to manage it can immediately see the
defaults. Also take care to comment it so that the person editing it
does not need to be digging for the manual.

Please keep the configuration files small. The ideal application is a
zero configuration app which auto-detects its settings from existing
resources, not an app where every feature can be tweaked and
customized. Every configuration parameter need to be coded, documented,
deployed, managed, reviewed, adjusted, corrected (usually several
times). So this ends up being very expensive.

External Configuration is poison, use it in medicinal quantities (not
necessarily homeopathic quantities, if it is needed, it is needed).

2.4 Logging and Monitoring

Since both these things are essentially non-functional requirements,
they should be pushed down to the platform and out of the app.

All logging frameworks are essentially pluggable. The collect all the
log messages in a back-end independent way and send them to an actual
logging implementation, an appender, usually writing to a file, but
this could be an email, JMS message, SNMP trap, …

Of course where those messages end up is largely dependent on the
organization supporting the app and should be decided by them. So the
final loggers should be treated as external configurable components.

So the app should not get involved with the details of logging, just
add a default config with some sensible defaults (size-based rotating
log files so the dev machines and test machines do not run out of
disk space) in an external log file. Please add a comprehensive set
of commented log targets so the admins can change easily the
log levels in a granular way to support the app effectively.

Similarly the app may rely that an external monitoring system is
available which monitors the error logs for critical
errors. Document these in the Operations Manual under the monitoring
section.

Also make sure that the app behaves consistent to the protocols it is
using. A website which has an error should return error status 5xx,
referencing an entity in a REST API which does not exist should return
404, … , whatever is the norm here. This makes monitoring with
tools like Nagios a breeze, as no parsing of the page needs to be
done.

2.4.1 What if the business asks for Special Monitoring

Tough question. In principle it is now a functional business
requirement and there should be a story for it. The risk exists that
this requirement might become broken after a config change during
routine maintenance.

The best way to deal with it is to make it part of the application,
but still push it as far down in the framework/libraries as possible.

For example, if the logging framework can be leveraged, then the
internal configuration could include a predefined appender for the
business notifications, separate from the external appenders.

In practice, deal with them on a case by case basis. Maybe you can
talk the business out of it, or rely on Nagios configuration managed
by Ops? Talk to the stakeholders.

2.4.2 Real-time monitoring and administration

Allowing access to internal value, parameters and admin functions
through some standard management framework like JMX is another
interesting pattern often seen.

Implementing this is straightforward and will be exposed by a
plethora of tools providing a UI for the management of this
information so that the code can focus on the business value, not
adding stuff to manage that stuff. Just do not forget to document it,
self documentation is best of course. Also give instructions in the
Ops manual to control access to this functionality.

Some projects notify developers and stakeholders immediately when
exceptions or other things happen. Another great pattern, but try to
push it out of the app using standard features of frameworks like the
logging framework, Camel, …

Copying classes from other projects is definitely not recommended,
this is a library shouting to come out. Refactor it as a separate
module, ask to make it part of the company foundation so it is just
there when needed. Just look and ask around first if this is not a
wheel which was already invented.

2.5 Complying to OS rules through packaging.

The above assumption to store everything under a folder runs against
the grain of the Linux standards. (Although they are actually the Mac
and Windows way of working).

I’ll treat the case for debian based distros here, but the same is
possible for the redhat and other distros.

In short, use symbolic links to move the folders to the locations
where linux is happy and keep them visible in the local folder for
the JVM. Everyone happy.

2.5.1 Main Deploy folder

All read-only stuff, which is the real application stuff, is expected
somewhere beneath /usr (but not /usr/local which is reserved for
locally compiled packages which we never do).

I recommend to create the app home folder in /usr/share/${appname}
and copy all libraries, binaries, scripts, static resources, etc in
it.

2.5.2 Config files

Config files in debian are expected under the /etc folder and the
package manager will automatically flag files deployed there as
config files so this does not need to be done separately (unless you
want to change the defaults of course).

Just move the default config files to /etc/${appname} and create a
symbolic link

${appname}: ln -s /etc/${appname} conf

Well, I guess debhelpers have better tools for this, so use whatever
is usual using the buildtool you use.

2.5.3 Data files

Storing data should end up under /var somewhere. I recommend to use
a folder under */var/lib/${appname} and create folders there which
you link back to the main deploy folder. If you only need 1 data
folder you do not need to create subfolders of course.

2.5.4 Log Files

Log files are expected beneath /var/log.

Create a folder /var/log/${appname} and link this to
${apphome}/logs. Make sure the folder is owned by the user the app
will be running as.

2.5.5 Dotfiles

Now we get in the hard cases. Normally this is only needed for
desktop apps, server apps should never use personal dotfiles. However
this is one of those cases where you never should say never.

For desktop apps, use the java support for dotfiles. This will use
personal dotfiles on Unixy OS-es and the registry on
Windows. Easy-peasy for greenfield apps. Problem solved.

For 3rd party apps or libs, we have to play the hand we’re
dealt. Typical examples are .netrc which is used to store passwords
outside the app. Good practice, but major headache.

For server apps, try to avoid it. Before you know it, you can no
longer do a ‘git clone …; mvn install’ to build it. Keeping build
dependencies down is critical for long term support and easy
onboarding.

In any case they are no deployment issue other that making sure it is
documented and there are some samples available for complicated files.

2.6 Apps deployed on a runtime platform

Many java apps, components, webapps, … are deployed on some kind of
runtime, be it a servlet container, appserver, OSGi container, …

Great. Leverage it. Push all this stuff down into the container, so
you can surf on the work done by the container packager.

For instance the jboss server has a folder …/conf in the instance
being started, which is on the class path. Just dump your external config
files there with cfengine or whatever you use for deploying.

Logfiles are also taken care of as that is a service the container
should be offering. Just document the important categories and
loglevels as usual, the rest in the concern of the container admin.

In general if you deploy on a controlled environment, expect that
your external dependencies are provided by the container. Work with
the container owner to find the sweet spot.

For testing this is no issue as maven will do the right thing in
unit and integration testing.

3 Conclusion

In orded to focus on the value of apps we must be separating business and stuff like datafiles, configfiles, logfiles as far from each other as possible. It is often already difficult enough (read: expensive) to fix bugs without having all that cruft sprinkled through the codebase and essential configs. Most of the requirement posed by the details of connecting the app code and the external stuff fall in the realm of non-functional and should be
moved as much as possible outside of the programmed code and into
frameworks and runtime containers and into the hands of the admins.

The best way to deal with those external dependencies is to push
them away from the app code and ignore them for the rest. With the
guidelines above this can be realized to a large extent in a
straightforward way.

Both configuration code and configuration parameters are poison over
time. Use them in medicinal doses.

Posted in build, Java Stuff, linux | Leave a comment |
June 28, 2011 · pti

Proxy Support for Grails

Not a big thing, actually for me it is.

I am always struggling with proxy settings. It requires a lot of different incantations to be done, every program deals with it differently, support for platform settings is flaky, …

Grails deals with this is a way which pleasantly surprised me :

grails add-proxy <name> –host=<hostname> –port=<portno>

e.g.

grails add-proxy client –host=proxy –port=3128

allows you to create a setting for a proxy and bind it to a name. It also supports username/password.

Switching to the setting involves only

grails set-proxy client

to enable the proxy setting, and

grails clear-proxy

when I get back in a transparent environment.

(for completeness there is a remove-proxy command which is useful to remove those passwords after the need has passed).

I particularly impressed  that this was done in a simple and straightforward without the need fo brain gymnastics trying to remember which arcane curse needs to be put at what location in which file.

Nice.

 

Posted in Java Stuff | Leave a comment |
November 11, 2008 · pti

Adding audit information to entities in the Seam framework.

Since we write a lot of stuff for use in the production side of the business we need to comply with rules which allow effective control that the processes and procedures have been followed. In the automotive industry people take these measures seriously.

One of these requirements we often get is that all configuration data elements must contain information on who created this information and when, and similarly for the last update.

Since the advent of EJB3 and JPA we now have nice POJO entities which clearly show the structure of the data and the last thing we want is to add a bunch of boilerplate code which multiplies the number of code lines with redundant stuff.

This blogpost of Daniel Pfeifer shows how to do this using an entity listener. However, it does not work using Seam since we cannot get the user information through the EJBContext, at least not on JBoss 4.2.3.

After wasting the better part of an afternoon barking up the wrong tree, the answer came to me next morning during breakfast. I have the unhealthy habit of reading PDF-files between bites, in this case chapter 15 of the Seam manual. The security section. (Yes, I know, I hould have done this sooner, you know : “if all else fails, read the manual.”)

Seam provides a nice utility class aptly named org.jboss.seam.security.Identity which provides all we need to get the name of the person logged in the current session. So replacing the implementation of getCallerIdentity() with the following, we now have nice audit trails.

/**
* Get the name of the currently logged in user.
*
* This part contains the framework specific code to get the active username.
* In this case we use the Seam provided Identity.instance() to get at this
* information.
*
* @param t
* @return
*/
private String getCallerIdentity(Traceable t) {
   try {
     // get the identity principal to get information about the current active user
     Principal principal = Identity.instance().getPrincipal();
     if (log.isDebugEnabled())
       log.debug(“AuditLogger.getCallerIdentity returned ” + principal.getName());

     return principal.getName();

   } catch (Exception e) {
     log.error(“Exception received : ” + e.getClass().getName() + “:” + e.getMessage());
     return “unknown”;
   }
}

I provided both the interface and mapped superclass implementations for convenience. All what is now remaining is to extend our configuration POJO’s from the TraceableEntity class (and create an update script, and modify the UI to get at the audit info, …) to make our stuff auditable.

Posted in Java Stuff | Leave a comment |
August 8, 2008 · pti

Java is a First Class Citizen on Ubuntu Hardy Linux

Lately I hear a lot of Java bashing from a very vocal part of the Linux community. I do a lot of Java, I use Linux on my main laptop and I like this just fine. I use daily freemind (Java mindmapper) and openoffice (which is Java enabled).

I was doing this for the past years since Windows XP ate my C-drive for the N-th time (and now it did the same on my gaming rig at home, grrr… ). I keep a ‘Tools’ directory in my home drive and place there the Java tools and libraries I get from the net. Creating a little bash script in ~/bin which calls the startup script from the java tools almost always does the right thing, so I was contented.

Well, I need to do some LDAP hacking. Last times I did that in perl, python, ruby and Java so it was groovy’s turn.

On a whim I type :

pti@pti-laptop:~/workspace/salesforce$ sudo apt-get install groovy

and to my big surprise I get :

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  antlr junit4 libasm2-java libbcel-java libbsf-java libclassworlds-java  libcommons-cli-java
libcommons-collections3-java libcommons-lang-java libcommons-logging-java liblog4j1.2-java
libmockobjects-java libmx4j-java  libregexp-java libservlet2.3-java libxpp3-java libxstream-java
Suggested packages:
  groovy-doc libbcel-java-doc jython rhino libclassworlds-java-doc
  libcommons-collections3-java-doc liblogkit-java libavalon-framework-java
  libgnumail-java
Recommended packages:
  liblog4j1.2-java-gcj
The following NEW packages will be installed:
  antlr groovy junit4 libasm2-java libbcel-java libbsf-java
  libclassworlds-java libcommons-cli-java libcommons-collections3-java
  libcommons-lang-java libcommons-logging-java liblog4j1.2-java
  libmockobjects-java libmx4j-java libregexp-java libservlet2.3-java
  libxpp3-java libxstream-java
0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded.
Need to get 7025kB of archives.
After this operation, 14.7MB of additional disk space will be used.
Do you want to continue [Y/n]?

Ok, this was not what I expected… Of course I type ‘Y’ and let it rip.

Now my curiosity take the upper hand and I try some more :

pti@pti-laptop:~/workspace/salesforce$ sudo apt-get install maven
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Couldn't find package maven

Tough luck. But wait, there are 2 mavens, let’s try :

pti@pti-laptop:~/workspace/salesforce$ sudo apt-get install maven2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  ant libcommons-codec-java libcommons-collections-java
  libcommons-httpclient-java libcommons-net-java libcommons-openpgp-java
  libdoxia-java libganymed-ssh2-java libjdom0-java libjsch-java libjtidy-java
  liblogkit-java liboro-java libplexus-classworlds-java
  libplexus-component-api-java libplexus-container-default-java
  libplexus-interactivity-api-java libplexus-utils-java
  libplexus-velocity-java libslide-webdavclient-java libwagon-java
  libwerken.xpath-java velocity
Suggested packages:
  ant-doc libcommons-httpclient-java-doc libcommons-openpgp-java-doc
  libdoxia-java-doc libjtidy-java-doc libgnumail-java
  libplexus-classworlds-java-doc libplexus-component-api-java-doc
  libplexus-container-default-java-doc libplexus-interactivity-api-java-doc
  libplexus-utils-java-doc libplexus-velocity-java-doc libwagon-java-doc
  velocity-doc
Recommended packages:
  ant-optional ant-gcj
The following NEW packages will be installed:
  ant libcommons-codec-java libcommons-collections-java
  libcommons-httpclient-java libcommons-net-java libcommons-openpgp-java
  libdoxia-java libganymed-ssh2-java libjdom0-java libjsch-java libjtidy-java
  liblogkit-java liboro-java libplexus-classworlds-java
  libplexus-component-api-java libplexus-container-default-java
  libplexus-interactivity-api-java libplexus-utils-java
  libplexus-velocity-java libslide-webdavclient-java libwagon-java 
  libwerken.xpath-java maven2 velocity
0 upgraded, 24 newly installed, 0 to remove and 0 not upgraded.
Need to get 6211kB of archives.
After this operation, 25.2MB of additional disk space will be used.
Do you want to continue [Y/n]? 

Hmmmmm……. This is better…. There is some refence to ant..

Yep, that too…

Jedit, jython, jruby, scala (I did not do LDAP with scala yet…) follow in quick succession…

Well, time to fire up synaptic and search for Java. A seemingly endless list of packages appears.

App Servers :

  • glassfish
  • tomcat (actually I knew that from before)
  • jetty

Dev Tools:

  • Emma
  • Junit
  • Junit4
  • JMock
  • …

A bunch of the commons libraries, …

Half a dozen of jdk’s.

Java bindings for all the windowing toolkits out there : Gtk, Qt-jambi, …

I also finally found the jdbc driver for postgresql which I was frantically looking for a couple of weeks ago and finally downloaded from the postgres site.

There is an old eclipse version, but actually this is a tool which I prefer to live in my Tools directory for updating the plugins.

The only thing I did not find was findbugs. Well there is probably a lot more missing, but my inspiration dried up.

I could clear out half of my tools folders because they are replaced with ubuntu version.

I am now even more contented than I was. I just love to type ‘apt-get install <something>’ when I want something, and now I found more of what I want.

Posted in Java Stuff, Ubuntu | Leave a comment |
July 12, 2008 · pti

Eclipse Ganymede crashes on 64-bit Ubuntu Hardy Herron

In java6 there is a bug in the 64-bit linux version of the jvm which causes eclipse to crash when opening projects with some aditional plugins installed. Both the openjdk as the sun versions are affected.

for more info see :

  • Bug #206620 in openjdk-6 (Ubuntu)
  • Bug#478560: sun-java6-jdk: SIGSEGV when loading Eclipse workspace


In the mean time this has been fixed in openjdk7, but this might take a while before this shows up in the repositories.

In the mean time the jdk5 does not have this problem and works perfectly well with eclipse.

So a quick

sudo aptitude install sun-java5-jdk


installs the needed software. However if the java6 is still installed, chances are it will be preferred over the java5 version of the programs. There are 2 ways to make sure eclipse uses java5

Use java5 for ALL programs.

To change the preference tell the alternatives system to use the java5 version.

pti@pti-laptop:~$ sudo update-alternatives --config java
[sudo] password for pti: 
There are 3 alternatives which provide `java'.
  Selection    Alternative
-----------------------------------------------
          1    /usr/lib/jvm/java-6-sun/jre/bin/java
          2    /usr/lib/jvm/java-1.5.0-sun/jre/bin/java
*+        3    /usr/lib/jvm/java-6-openjdk/jre/bin/java
Press enter to keep the default[*], or type selection number: 2
Using '/usr/lib/jvm/java-1.5.0-sun/jre/bin/java' to provide 'java'.

To choose the alternatives with a GUI it is a good idea to install galternatives.

Tell only eclipse to use java5.


Since the bug only shows up on my system (so far) in eclipse, I prefer the speedier java6 as default. Luckiliy we can tell eclipse to use a specific jvm in the eclipse.ini file.

Open eclipse.ini in your eclipse home with your favorite editor and add the following 2 lines to the beginning of the file

    -vm
    /usr/lib/jvm/java-1.5.0-sun/jre/bin/java

This will tell the eclipse launcher to use java5 to launch eclipse.

Posted in Java Stuff, Ubuntu | Tags: eclipse, ganymede, hardy, Java, Ubuntu | Leave a comment |
June 11, 2008 · pti

OpenProj Locks up with Blank Grey Windows

The symptom is that after starting sometimes there is some blurp dialog and afterwards the Tip of the Day dialog appears. This stays grey and the application accepts no more events. You need to kill it to get out of there.

This happens at the place which is protected by a firewall and has no transparent proxy. At home it worked fine albeit on my Macbook and not on my Ubuntu laptop.

The reason  is that there is some phone home functionality built in and with wireshark I could see the application trying to connect to a webserver. Probably to get the tips of the day. Behind the firewall this did not work and the application is just hanging there.

I suspect that sooner or later it will time out, but I am not that patient. Since this is a regular occurence with java applications I also immediately knew that I had to tell it where the proxy can be found.

In the file ~/.openproj/run.conf file replace the line :

JAVA_OPTS=”-Xms128m -Xmx768m”

with

JAVA_OPTS=”-Dhttp.proxyHost=proxy -Dhttp.proxyPort=3128 -Xms128m -Xmx768m”

This directs the java runtime library to use the proxy http://proxy:3128/.

And voila!

…

Openproj starts immediately and in its full glory?

Posted in Java Stuff, Project Management | Tags: Java, Project Management | Leave a comment |

Pages

  • {Snamell(it)}
  • Blog
  • About
  • Location
  • Contact Us

Archives

  • July 2013
  • March 2012
  • July 2011
  • June 2011
  • November 2008
  • September 2008
  • August 2008
  • July 2008
  • June 2008

Categories

  • build (2)
  • Java Stuff (6)
  • linux (5)
  • MacOSX (1)
  • Personal (1)
    • Huiswerk (1)
    • Kinderen (1)
  • Plone (1)
  • Project Management (2)
  • Ubuntu (6)
  • Uncategorized (3)

WordPress

  • Log in
  • WordPress

Subscribe

  • Entries (RSS)
  • Comments (RSS)
© My Website