J2EEKade Hansson
Within DPIWE, when developers say "architecture," by default what is meant is a persistence and object brokering mechanism. This perhaps arises from the fact that this was the first area of discourse for which DPIWE built reusable components. It is important to recognise that architectures do not end here, however. Indeed, even the first application to make use of the object brokering components, LID, had other architectures of its own for user interface, login, replication and reporting. Some of these have been reused as well.
In a broader software engineering context, architectures are embodied in the ways applications structure their tasks, and not just the reusable code repositories used to achieve that end. Indeed, some structures have to be reworked considerably for each new application, which makes code disposal important. This is as it should be, as each application presents its own unique set of problems.
The category of architectures needed for the delivery of user interfaces is at least as important as the object broker. This was recognised after the LID development and a reusable code resource continues to be implemented as part of new projects like PIVOT, new CREST and WIST. The original LID components are also in use in TapDB, NELMS and STaRS II.
Alongside architectures are methodologies and tools, and these are also very important developer resources, but we shall concentrate less on them in this discussion paper, and more on:
There are three major architectures and many minor ones used for new applications at DPIWE. This document cannot descibe all the minor ones in detail, but it is very important to recognise the real contributions they all make to the many applications DPIWE has built in the last five years.
J2EECurrently J2EE as an object broker architecture (our definition for the first two architectures of this document) is only used in two projects: the new RecFish and STaRS II. As an object broker API definition (i.e. EJB, the Enterprise Java Bean API) and a network communications layer (i.e. Servlets,) it is also leveraged by Mr Architecture. Additionally, all DPIWE applications since STaRS have used the Servlet API in some form.
J2EE is a vast enterprise resource which requires a large piece of software called an application server to support it. DPIWE are keen to continue exploiting such technology in the future, for many more than just two projects, but its worth bearing in mind that J2EE contains many architectures (two of which we just named.) It is, in fact, more of a platform than an architecture. Therefore choosing J2EE does not answer any question: indeed, for all intents and purposes, J2EE was already chosen by DPIWE at least six years ago, maybe earlier.
With the use of Servlets firmly established, the use of EJB and Web Services APIs are the next big steps in the DPIWE architectural evolution. However, as with all architectures which have come before, one must consider the appropriateness of the solutions the technology provides to the problems each application presents. Problems must come first, and solutions second.
In particular, EJB, while recommended by this author, is just one solution for object brokering and persistence among many choices available within J2EE, and it even has choices within itself: such as SessionBeans, TimerDrivenBeans, MessageDrivenBeans and EntityBeans (whose persistence can be container managed or bean managed,) combined with the further choice of local interfaces and remote interfaces. CORBA and raw JDBC are also available in the Java technology stack for persistence and object brokering solutions. Other more advanced (but heavily hyped) options include Java Data Objects (JDOs) and the Hibernate implementation (which are not truly part of the current standard, version 1.4, but may yet be included in a future revision.)
Web Services, like XML before it, is a bit of a buzz phrase in IT at the moment. It is a solution to a very technical problem with a very user-friendly name, but like any technology it is not appropriate to every problem. It solves some communication problems between business applications in an elegant way, and so for applications which require this, it is a good idea to base the whole application around the framework, as was done with the new RecFish and is being done with the STaRS II development. But for client focused applications with greater user interface demands, forcing developers to use a Web Services model in is not an appropriate course, and is likely to be counter-productive: examples are new CREST, PIVOT and WIST.
Clearly J2EE has many important advantages over comparable architectures, or DPIWE would not have chosen it as the next step in the evolution of its applications. Foremost among these is the ability to have ready-trained developers, both contractors and employees, brought into DPIWE to work on our applications, because J2EE is widely recognised in industry and is a key qualification for those who developers who want to construct enterprise-grade applications. This arises because J2EE is an industry standard. Also arising from this status is the field of choice in application server implementations, though it is important to recognise that vendors have opportunities for lock-in awaiting the unwary.
J2EE has a rich and growing set of APIs which is constantly being amended and added to, reflecting the needs of thousands of businesses around the world. This vibrancy is an important advantage also: it means that new solutions are arriving all the time, and that old solutions are either being actively maintained or migration paths are being actively explored.
The flip side to the advantage of having a rich and growing set of APIs is the disadvantage that it is very hard for developers to keep abreast of the state of the art, or to even have a comprehensive knowledge of the entire catalogue of possibilities in their head. Perhaps it is better to be struggling to choose among many options, than to be stuck with a solitary compromise solution, however.
One important realisation DPIWE has yet to make, however, is that J2EE can only ever be what you make it. DPIWE should not accept the default solutions if they do not meet its needs, and DPIWE should be prepared to work as part of the J2EE community to encourage amendments to existing solutions, or even entirely new solutions, to meet our particular enterprise's needs. Open source J2EE implementations and companion projects exist (notably the JBoss Hibernate project and the Apache Jakarta projects,) and DPIWE should be prepared to contribute to these noble efforts where it can.
If developers are trained in J2EE technologies, and their abilities are kept up to date while they continue to develop applications, then the posibilities are endless, and the potential rewards are great. However, an architecture should never become the default, and its enduring concepts or implementations continue unquestioned. At some time in the future, J2EE may yet need to be disposed of in favour of something else. That time seems quite far off, thankfully.
Mr ArchitectureMr Architecture is a reusable code resource. It will always be a required component of Ms Architecture, so it cannot be completely disposed of as long as Ms Architecture applications exist. However, Mr Architecture should not be considered to be the default architecture for any new application. It should only be chosen for lightweight web applications with few batch processing jobs, less than 64 tables and fewer than 1 million records per table, and perhaps to do initial prototyping work on larger applications. This corresponds to the profile of the initial LID and PIVOT projects except for the batch processing caveat, and by far batch processing is the greatest area of concern within those applications.
When Mr Architecture is used to do initial prototyping, migration paths must be considered. But even given this recommendation, it is important to recognise that Mr Architecture is not broken, or even a burden, to applications which fit within the above profile (and indeed, to applications like LID and PIVOT which don't, mainly due to their batch processing demands.) It is J2EE compliant, and it will be maintained for the forseeable future, though the form of this maintenance may change. Currently Mr Architecture is maintained as part of Ms Architecture work. In future, it may be maintained separately as an open source project on SourceForge. This is the recommended final solution: contribution to the community for further reuse and rework, and not disposal.
Mr Architecture applications currently do not handle some loads well. Large transactions are a problem, and most of these are generated by batch processing. There are experimental components in Mr Architecture 2.9x which may address some of the problems for present applications by allowing the garbage collection of responses which are no longer needed by long-lived jobs. This is experimental in two senses: the new components are not complete and there is no guarantee that the batch jobs (supported by application components) themselves don't actually hold references to the objects for longer than is strictly necessary.
It is important to recognise that it is not Mr Architecture alone which is causing these applications to be slow during some loads, but the combination of Mr Architecture, the application in question, and the overall load level of the small (by enterprise standards) box being used. It was recognised in development that large transactions were problematic: one solution which was rejected--smaller transactions--was an application level improvement which the team did not want to implement. The other solution, which has taken three years to materialize, took so long not because of development difficulty, because of DPIWE's policy of not committing developers to preventative maintenance behind the scenes of existing applications, compounded by the slow but sure practice of loading servers until breaking point. "If it ain't broke, don't fix it," only gets you so far.
In terms of server load, you can barely run two instances of an enterprise app on a small (two processor, two gigabyte) box. This is barely the power of two high-end desktop machines these days. Virtual memory is a developer convenience which should not be discounted by those responsible for maintaining servers. Developers are too often berated for consuming "too much" (virtual) memory, and while some optimizations are good and proper, they are rightfully a last resort. Make it walk before you make it run. Unfortunately, developers at DPIWE just aren't given the incentive to make apps even jog; if that is to be our continued policy, management must accept the associated requirement of more powerful hardware or clusters of smaller hardware, supported by high bandwidth networks, are needed. (This latter option would require some rework of Mr Architecture, although a migration to J2EE technologies, partiucularly EJB, by affected applications should also be considered. The migration path for Mr Architecture applications was discussed in the previous paper: A New Architecture.)
The key advantage Mr Architecture has is that it supports full relational power. This is also the biggest impediment to migrating Mr Architecture applications. J2EE has no good equivalent to the layer provided by Mr Architecture for the purpose of generating arbitrary queries and forming EJBs from them.
Mr Architecture has other advantages, like its small footprint and low overhead, but these are often outweighed by a need for scalability to multiple virtual machines (probably on separate machines, but at least on separate processors.) However, having said this, we do not know that any other architecture will be better on this score in the DPIWE environment: if we only run one small box per application, as has been the norm to this point, Mr Architecture is probably still a better choice than full J2EE.
Mr Architecture is being actively maintained, with new code largely supporting Ms Architecture primitives. Existing classes are mostly static, so this is mainly maintenance mode, apart from occassional bursts of activity to implement new features for the many actively developed applications which use it: PIVOT, NELMS, WIST and new CREST.
It is current policy of the Mr Architecture developer to support only the very latest version of Mr Architecture. This is the case because the developer has very limited time for the maintenance activity. Maintaining multiple branches would be impossible without multiple developers, and maintaining even one version is probably excessive demand for a developer expected to mentor so much else within DPIWE.
As stated, Mr Architecture can only be disposed of if Ms Architecture is disposed of. As this is not a recommended course, Mr Architecture must persist in some form.
However, Mr Architecture in its present form cannot be maintained by one DPIWE developer solely on company time, as has been the practice to this point. This problem has been recognised for some time (probably three years,) but there has been no mitigating action. This failure has lead to LID abandoning contemporary versions of Mr Architecture, which places that application in serious jeopardy. NELMS is maintained with similar policies, so that may also be in jeopardy.
The solution proposed is to allow developers operating outside DPIWE access to Mr Architecture, and to allow them to provide patches back to the main DPIWE trunk. If this ideal were achieved, the current developer would be prepared to cooordinate and contribute to this effort outside company time. Worthwhile patches could then be used to improve Mr Architecture within DPIWE. This is the open source model of software development, and has been the planned final mode of development for Mr Architecture for at least three years, and was the motivation behind the Lesser GNU Public License being attached to the codebase. Migration to this model could be completed in less than two months, assuming demands on the developer resource don't increase much further.
The Ms Architecture ExtensionMs Architecture is an ongoing concern for DPIWE. The problem of user interface development has not been conquered. Many applications still roll their own user interface components and pollute those components with business logic, forever locking the user interface of the application in. This is the kind of failure which lead to the redevelopment of applications like TASFOL and VISTAS. Furthermore, exactly this failure was outlined in the predecessor to this document, A New Architecture, and Ms Architecture partially realises the vision documented there. The parts of the vision not realised are: static deployment of user interfaces and heavyweight client user interface mapping for AWT or Swing.
The next obvious step, apart from ongoing refinements to the support for thin client applications, is to build the components supporting the heavyweight client UI mapping.
Ms Architecture has been successfully applied to thin client web applications although there are issues based around the way it is used: it is forced upon developers of lightweight applications, and it is not allowed to go anywhere near the Swing applications. Developers of lightweight applications have little guidance from the sole Ms Architecture developer on its design precepts because he is unavailable to dedicate time to this important mentor role because of increasing demands on his times from other projects, and also from Ms Architecture (and Mr Architecture) maintenance itself.
In this way, Ms Architecture is eating itself. Ms Architecture at one point had at least one developer willing to collaborate with the primary contributor, but unfortunately this partnership was neither deep enough nor long enough to either solidify what exists in Ms Architecture or allow its expansion to the envisioned frontiers.
Ms Architecture also has not been stabilised to an adequate level due to a lack of time for integration or unit testing. Mr Architecture benefited greatly from having an active server at all times during development, and this contributed to its robustness in contentious situations. Ms Architecture has had nowhere near the level of multiuser testing, and so defficiencies in this area cannot be easily mapped.
Ms Architecture is a more ambitious project than Mr Architecture, so early versions were built much more rapidly in order to prove the concept, and both design and implementation compromises were made during this rapid period of growth. These have lead to many growing pains in its (current) middle life which continue today.
The key advantage of Ms Architecture is its potential to make user interface completely independent of both business logic and targeted devices. It is still something which is very well worth persuing. Developers can only come to realise the expressive power of Ms Architecture if there are multiple rendering modes (static versus dynamic) and available target devices (lightweight versus heavyweight.)
To this end, Ms Architecture is completely compatible with J2EE, and actually complements it exceedingly well. J2EE does not have an interface level comparable to Ms Architecture, although it encourages the Model View Controller patern which Ms Architecture adopts, and it ultimately provides many of the components needed in constructing device-dependent (or at least mark-up language dependent) user interfaces.
It was decided at a meeting last year that user interface was the focus of architectural development concerns, with the object broker becoming the responsibility of a J2EE application server. In line with this, the sole Ms Architecture developer has always made Ms Architecture his priority. However, the development work on the application which was supposed to encourage the development of the Swing components, STaRS II, was delayed, and then began with the new object broker mechanism as its focus, and at no point did the Ms Architecture and STaRS projects synergize as envisioned.
Ms Architecture is being actively developed by one developer who cannot spend all of his time on the project, although he does attempt to dedicate most of his effort to it. This is because it appears to hold the most promise for producing efficiencies in DPIWE's application development and redevelopment activities.
Ms Architecture is in active development, and like Mr Architecture, the latest version is the only one the developer can justify time and effort to support.
It is much to early to consider end of life for this architecture, as many actively developed applications will be bound to it for the forseeable future, and additionally this binding is advantageous. Advantageous because if Ms Architecture does acquire new device mappings, these applications will instantly (i.e. with no rework required) be able to migrate to new devices.
However, it would encourage new device mappings if Ms Architecture were open to more developer input. If this cannot come from within DPIWE, an open source approach similar to that suggested for Mr Architecture could work. To this end, licensing under the Lesser GNU Public License would be a good first step. The developer will therefore append the licence to the codebase shortly.
DPIWE InfrastructureThere are levels of infrastructure built upon Mr Architecture and Ms Architecture, and very shortly there is likely to be (if indeed there is not already) some infrastructure built on J2EE, that serves DPIWE applications in a business agnostic way. We will group these architectures together and call them the DPIWE Infrastructure Architectures.
The ones this author can name, having been involved in their packaging and release, if not some minor aspect of their development, are:
At some point the job of maintaining release versions of the Infrastructure packages was given to one developer. (No prizes for guessing which one.) This was expected to cause a bottleneck, although it was agreed that this also might cause some sort of calming side effect, where changes were tested before being tagged as being of release quality. However, while this was a bottleneck, and because the developer (like all DPIWE developers, it must be said) had too much else to do, any opportunity for quality control fell by the wayside as well.
The advantage of having a common Infrastructure library is that it encourages common low-level concepts (promoting application consistency) and promotes code reuse. Also, some of the more DPIWE-centric design decisions can be isolated from other more general architectures like Mr and Ms Architecture.
Everybody is supposed to maintain these architectures as they see fit. This has worked surprisingly well, but only because in practice there is usually only one person who is the current custodian of the package, and most developers have good knowledge of who to go to find out if a change is warranted and how to best effect it.
These packages, or more likely individual classes within these packages, fall out of vogue, and therefore effectively dispose of themselves. It would perhaps help tidiness (and therefore inspire reuse generally) if somebody actively went through and pruned them from time to time, but it is very hard to justify the necessary time.
DPIWE BusinessThere are levels of business logic built upon the infrastructure which maps, in an application agnostic way, common business entities and processes. We will group these components together and call them the DPIWE Business Architectures.
Currently entities are bound to Mr Architecture, and pure logic components are few and far between. Also, the recognition of reusability and subsequent packaging of business-wide application components is usually left much too late or is not carried through at all: witness the workflow library built for new CREST/CLAC, which is still embedded in those applications.
This is the really the whole point of using EJBs: each application should be able to reuse common business logic and entities from any other. The task of the application developer then becomes to assemble the application from prefabricated (and presumably well-tested) components.
Someone really should actively go through applications trawling for new additions to these architectures. Unfortunately, this would be a large task, and would only produce rewards if documentation for the components was good.
The Business architectures really do need to continue to exist, probably in perpetuity, and should be migrated to new infrastructures whenever technological conditions change. DPIWE might even consider migrating them to a new programming language at some point in the distant future! Their disposal should only be considered when the business process they support is excised from the organisation.
Given the decay of Mr Architecture and the Ms Architecture Extension at the hands of DPIWE, broadly-targeted architectural implementations are clearly not well fostered by a corporate parent whose business is not software development as an end itself. That being the case, Dr Architecture clearly is not appropriate to the software development practices of DPIWE, and is hereby reserved as a private enterprise of the author to develop (or indeed not) as he sees fit.
As with all documents of this kind, this investigation is from the outside, looking in, and no company resources (save a screwdriver, some oxygen and electricity[1]) were used to develop this document, A New Architecture[2] or the Dr Architecture concept, nor were they inspired by others within DPIWE. Therefore Dr Architecture is owned by its creator. Similarly, all opinions, errors of fact and general terseness of style expressed in this document characterize the author and not DPIWE or CIT.
[1] Very little electricity, actually, thanks to the use of low power technology from Castle Technology, constructed with components designed with reference to ARM.
[2] Within DPIWE, A New Architecture is available using the following URLs:
http://liddev.dpiwe.tas.gov.au:8081/Docs/Discussion.html (for screen display)
http://liddev.dpiwe.tas.gov.au:8081/Docs/Discussion.pdf (for printing using Acrobat)
http://liddev.dpiwe.tas.gov.au:8081/Docs/Discussion.ps (for printing using PostScript)