DataNucleus

GAE/J and DataNucleus v3 - Part 2

2011-11-05T07:15:00.000-07:00

In the previous post we saw some initial changes to make GAE/J DataNucleus plugin work with the latest version of DataNucleus plugins. In this post we describe some further features of interest to GAE users that they weren't able to use before.

Storage Version
With v2 of the plugin it will, by default, persist using a new "storage version". In v1 of the plugin it persisted no explicit information about relations, and instead relied on doing queries for parent key to find related objects; obviously when all relations were owned then this was valid. In v2 of the plugin it persists a property in the Entity for each relation (containing the Key(s) of the related object(s)), at the owner side always. In the case of unowned relations (see below) it also will persist a property in the Entity at the non-owner side of a bidirectional relation. Obviously all existing data uses v1 of the storage version, but don't let that concern you since the plugin will check for presence of this property, and if not present then fall back to v1 behaviour to get the related objects. As entities are updated the data will be migrated to v2 storage version (a migration tool to do the job in one pass is in the works also).

Unowned Relations

By default in GAE/J all relations are owned meaning that any child objects have the parent object Key as part of their Key, and persisted as part of the same entity-group. This is obviously useful in optimising retrieval of data, but there are times when you simply want your model persisting and not have imposition of ownership. In v2 of the plugin you can have unowned relations, where each object is in its own entity-group. To define a relation like this, see the following example

@PersistenceCapable
public class A
{
    @Persistent(primaryKey="true", valueStrategy=IdGeneratorStrategy.IDENTITY) 
    long id;

    @Unowned
    B b;

    ...
}

@PersistenceCapable
public class B
{
    @Persistent(primaryKey="true", valueStrategy=IdGeneratorStrategy.IDENTITY) 
    long id;

    @Unowned
    @Persistent(mappedBy="b")
    A a;

    String name;

    ...
}

So when we persist an object of type A with related B it will do the following

PUT the A, generating its Key, but without property for B
PUT the B, generating its Key, and with a property referring to the key of A
PUT the A with the property referring to the key of B.

It should be noted that the @Unowned annotation is simply a shortcut for @Extension(vendorName="datanucleus", key="gae.unowned", value="true")

Be aware that if you persist unowned relations in a transaction then you will need to have multi-entity-group transactions enabled, since each object is in its own entity group.

Datastore Identity

With JDO the user has the choice of having their own primary key field (application-identity), or having the identity of the object defined for them (datastore-identity). GAE v1 only allowed application-identity. In v2 of the GAE DN plugin it also allows datastore-identity. To give an example

@PersistenceCapable
@DatastoreIdentity(strategy=IdGeneratorStrategy.IDENTITY)
public class MyClass
{
    ...
}

So with this class it will persist an Entity and its Key will use IDENTITY strategy.

Interface Fields
With v2 of the plugin you can now have fields of interface type (representing a persistable type) and persist them as you would normally do. Refer to the DataNucleus docs for how to do it (paying particular attention to the type of the field in metadata)

That's a brief summary of some of the more noteworthy improvements, and hopefully now GAE/J using JDO (or JPA) is a much more pleasant place to be, and you can refer almost directly to the DataNucleus docs for many more features now. In addition to the above changes, and in fixing various other minor bugs, the code structure has been changed quite a bit so future enhancements ought to be much more rapidly achievable

GAE/J and DataNucleus v3 - Part 1

2011-08-15T08:48:00.000-07:00

Some time ago I wrote a post about GAE/J and how it provides JDO/JPA. It had many limitations and shortcomings. Recently we have had the chance to update their DataNucleus plugin to work with version 3.0. Here are the major changes that users of that plugin will see if they build and use GAE/J DataNucleus plugin from SVN trunk.

JDOQL/JPQL : Support for methods/operators

If a user sets the query extension/hint "datanucleus.query.evaluateInMemory" then the query will be evaluated in-memory. This has an obvious drawback in terms of memory utilisation (if the number of results is large), but the big plus is that it will evaluate almost all JDOQL/JPQL syntax.

JDOQL : Support for input candidate collection

You can now specify the instances that you want to query over using query.setCandidateCollection(...). Means that you have a list of instances and can query which of them match a particular filter criteria.

Primary Key Types

Previously you could only have Long, String or Key. You can now also have long.

Plugin package naming

Now uses com.google.appengine.datanucleus as its package root, hence not using the DataNucleus-owned domain.

JDOQL/JPQL setResultClass
This is now supported for the standard types of result classes, so you no longer need to convert manually the result into your required type.

Value Generation
GAE/J users can now make use of other DataNucleus value generators, such as "uuid", "uuid-hex"

PersistenceManagerFactory
The PersistenceManagerFactory used is now the standard DataNucleus PMF, not any custom GAE variant. To be specific the PersistenceManagerFactoryClass is now org.datanucleus.api.jdo.JDOPersistenceManagerFactory. If you want to have a singleton PMF, simply set the persistence property datanucleus.singletonPMFForName to true. This will then return any existing PMF if present for the requested persistence-unit, or create it if not present.

EntityManagerFactory
The EntityManagerFactory used is now the standard DataNucleus EMF, not any custom GAE variant. To be specific the PersistenceProvider is org.datanucleus.api.jpa.PersistenceProviderImpl. If you want to have a singleton EMF, simply set the persistence property datanucleus.singletonEMFForName to true. This will then return any existing EMF if present for the requested persistence-unit, or create it if not present.

JPA2
By using DataNucleus v3 you now have available all of the changes made in JPA2, so things like Criteria queries, metamodel, etc.

JDO3
By using DataNucleus v3 you now have available all of the changes made in JDO3.0/JDO3.1. This means query timeouts, metadata API, enhancer API, as well as the DataNucleus proposal for Typesafe JDO queries.

Level2 Caching
Level2 Caching is enabled by default, using an internal map-based cache. You can improve this further by setting the persistence property datanucleus.cache.level2.type to "javax.cache" and include datanucleus-cache.jar in your CLASSPATH. This will then cache using GAE Memcache

Non-transactional Persistence
DataNucleus non-transactional behaviour is different now, with any call to pm.makePersistent, pm.deletePersistent, em.persist, em.merge, em.remove being atomic, sent to the datastore immediately. Any updates to fields via setters are still queued.

JPA RetainValues
JPA usage, by default, has datanucleus.RetainValues set to true now. This means that when you commit a transaction the object will retain the values of its fields (previously it migrated to hollow state).

Persistence of other java types
In GAE/J v1 you can only persist fields of the following types : primitive, primitive wrapper, String, Date, Enum, BigDecimal, some com.google.appengine types, as well as Collection types. With v2 you can now persist fields of types Currency, Locale, Timezone, BigInteger, Color, Point, StringBuffer, Jodatime, javax.time, and many more.

Be aware though ... there is more to come

Performance, benchmarking

2011-03-03T04:17:00.000-08:00

Every so often some individual or group decides they're going to "invent" a new benchmark. They ignore all that have been done before (like PolePos) maybe due to NotInventedHereSyndrome, or maybe due to "oh someone who helped write that was related to some datastore therefore it must be dodgy". While it is highly likely that existing benchmarks don't include their particular specific case, they never make any reference to what was there before that people are familiar with. They make minimal efforts at configuring other persistence solutions other than what is their favourite or what they have experience with, and then publish their results with a flourish to websites.

Such benchmarks have been known to draw conclusions that you have a standard performance difference across all types of tests. Anyone who has ever looked at the persistence process would know that this conclusion is flawed since good persistence solutions all provide particular features, and by turning on these features you get beneficial things at the expense of some performance, and so certain operations will be better with one tool than with another. Hibernate has some good features, DataNucleus has some good features. If you enable particular features you get poorer performance in other areas.

The other aspect of their conclusion is that they simply want a headline grabber black and white this is better than that. They seemingly aren't interested in thinking about the different methods employed by the software under test and the particular options. If I was exploring the topic of performance (and its an interesting topic, that can be useful in influencing priorities) I'd want to think about what I was asking the software to do and, bearing in mind that these benchmarks use open source software, and I'd have information available as to how a particular implementation attempts to do something. This could then be termed constructive benchmarking.

Recently we had one which took a flat class, no inheritance or relationships, and persisted it, ... many times ... and then called itself a "Detailed Comparison". While this may be an operation that an application may need to do, the response time for the persist of an object of this form would not differ by any perceptible amount to the end user of that software. If a user is wanting to persist a large number of objects (like in bulk loading), then this would typically be performed in a background task anyway. DataNucleus has never been optimised for such a task (it would logically be better at complex object graphs due to the process it uses for detection of changes), but even with it, is actually configurable to give pretty good performance. As above, it's good at some things and less good at others. Anyone claiming to make a "detailed comparison" would have bothered to try it on a range of cases, to look at the full capabilities, to look at outstanding issues etc.

Here, to give you something to talk about. I just ran a flat class

@Entity
public class A
{
    @Id
    Long id;

    String name;

    public A(long id, String name)
    {
        this.id = id;
        this.name = name;
    }
}

I ran it through the following persistence code

EntityManager em = emf.createEntityManager();
EntityTransaction txn = em.getTransaction();
txn.begin();
for (int i=0;i<10000;i++)
{
    A a = new A(i, "First");
    em.persist(a);
}
txn.commit();

I used the config of the aforementioned (Hibernate) users for running Hibernate, and tuned DataNucleus myself. And the runtimes ? Hibernate (3.6.1.Final) took 8164ms, and DataNucleus (SVN trunk on 3/Mar/2011) took 7227ms (using HSQLDB 1.8.0.4 embedded). So on that case DataNucleus was faster. Is this significant ? Well no, since as I already explained it's a particular case, but demonstrates the principle clearly enough. We can all turn on/off particular options and get some results. Besides which persisting 10000 objects in just 7-8 seconds (on this PC!) is pretty impressive in anyone's book and never a bottleneck in a normal application.

If you are performing something of the nature of a flat object bulk persist you would not use transactions to avoid the overhead, and you would turn off various features that are not of interest - managed relationships, persistence-by-reachability-at-commit, L2 caching even. Then if using an identity generator you would allocate large blocks of values with your metadata. That said, if you are so serious about performance and persisting flat objects to RDBMS then anyone sensible would use either JDBC direct, or a JDBC wrapper like MyBatis or SpringJDBC. This is "right tool for the job". Since the benchmark was provided by a group of Hibernate users (they provide various Hibernate tutorials and nothing for any other persistence implementation) we only have to assume that this must be what they think is the "best tool" - if so why not reproduce your benchmark with well-written JDBC and let us know what you find. I even posed this question about applicability of a flat class on their blog and my comment was deleted. Whether it was deleted by them, or by the blogger system I've no idea, but it was there for an hour and at that point was deleted. Our blog has never lost comments, and it's on the same host system.

One of the (undeleted) comments on their benchmark was from the author of an ODBMS who seems to like to decide publically what I ought to spend my spare time on. Is he a commercial client ? no. Is his software open source ? no. Or free ? not if I want to use it on anything serious. The response to him is simple : let me decide what I spend my time on, and you concentrate on your own software; I don't tell you what to do. It's an ancient custom called "respect" - sadly symptomatic of the attitudes in the IT profession.

A benchmark to use as the basis for choosing what software you use in your own application needs to cover the different persistence operations that you will perform. If you have a web application that creates a few objects, deletes a few, updates a few, etc continually, then the likelihood is that the performance will not impact on you or your end users to one iota. What will impact on you is whether the persistence solution allows you to do what you want to do or whether it has a large number of unfixed bugs that force you to continually implement workarounds or compromises on your design. Why not have a look through the issue tracker of the software and see what types of problems people are having, and how long the issues have existed?

DataNucleus 3.0 development is initially focussing on architecture, since we believe in getting the architecture right first to take the software to the level we think it needs to be. This means that in early milestones (the benchmark referenced above decided to use 3.0M1) we spend time on refactoring etc, and not on performance. This doesn't mean that performance isn't important, just that we feel our users want to be able to perform their tasks first and foremost and then speed things up later. This is the same methodology employed by PostgreSQL to much success, who for years had to listen to "MySQL is faster" comments. Even with that general philosophy anyone using current SVN code would already see some very noticeable speed up in non-transactional persistence performance, and anyone using the MongoDB plugin would also see much more optimised insert performance. These benefits are due to extending the architecture to do some things that we've wanted to do for some time but didn't since we wanted to maintain backwards compatibility and due to resourcing.

Next time you look at some "performance benchmark" we suggest that you bear this in mind. We won't be spending our time analysing their results, or responding to their claims. This is because we'd rather spend it developing this software, rather than "mine is better than yours" negative mentality discussions.

DataNucleus v3 and MongoDB

2011-02-13T02:19:00.000-08:00

There has obviously been a recent shift to look at highly scalable datastores for use in "the cloud". Google wrote a plugin for their own BigTable datastore back in 2009, providing access to some of the features of JDO/JPA. Unfortunately they didn't have the intention of providing a full and fair reflection of those persistence specifications, and so reaction to it was mixed. Some people attempted to argue that APIs like JDO did not match these types of datastores (I see nothing in the API or query language of JDO that led to this conclusion, but anyway) and that using standard APIs on them was inappropriate; they were asked to provide concrete examples of features of these datastores could not be handled adequately by JDO but unfortunately didn't come up with anything.

With DataNucleus v3 we have the opportunity to spend some time on providing good support for these types of datastores, adding support for missing features. A previous blog post documented efforts to upgrade the support for HBase. In this blog post we describe the features in the new plugin for the MongoDB document-based "NoSQL" store.

Features that this plugin currently supports include

Support for single MongoDB instances, and for MongoDB replica sets
Support for application identity (defined by the user), and datastore identity (surrogate field in the JSON document)
Basic persistence/update/delete of objects
Support for persistence of (unembedded) Collections/Maps/arrays by way of storing the identity of any related object(s) in the field.
Persistence of related objects (1-1/N-1) as "flat" embedded, where all fields of the related object are fields of the owner JSON document. This also supports nested related objects (unlimited depth).
Persistence of related objects (1-1/N-1) as nested embedded, where the related object is stored as a nested JSON document within the owner JSON document. This also supports nested related objects (unlimited depth).
Persistence of related collections/arrays (1-N) as nested embedded, where the related objects are stored as an array of nested JSON documents. This also supports nested relations.
Persistence of related maps (1-N) as nested embedded, where the related map is a nested array of map entries with fields "key","value". Supports nested relations.
Persistence of fields as serialised.
Polymorphic queries. If the query requests a class or subclasses then that is what is returned. This implies the execution of any query against the MongoDB collection for each of the possible candidate classes.
Access to the native MongoDB "DB" object via the standard JDO datastore connection accessor
Support for persistence of object version, stored in a separate field in the JSON document and support for optimistic version checking
Support for "identity" value generation using the MongoDB "_id" field value. The only restriction on this is that a field/property using "identity" value generation has to be of type String
Support for "increment" value generation (numeric fields).
Support for SchemaTool creation/deletion of schemas. This supports the document collection for the classes, as well as any indices required (including unique).
JDOQL/JPQL querying, including support for fetch groups, so you can restrict how much data is returned by the query.
Basic JDOQL/JPQL filter clauses (comparison operations) are evaluated in the datastore where possible.
Support for running queries on MongoDB slave instances.
Support for persistence of discriminators, so a user can store multiple classes into the same MongoDB collection, and we use the discriminator to determine the object type being returned.

As you can see, we already provide a very good JDO (and JPA) capability for MongoDB, and the feature list is shown as a matrix here.

Input to this plugin is obviously desired, particularly from people with more intimate knowledge of MongoDB. Source code can be found at SourceForge, and issues are tracked via the NUCMONGODB project in JIRA.

JPA : TCK request and JPA2.1

2011-01-24T06:57:00.001-08:00

It is now almost exactly a year since we submitted a request for access to the JPA2 TCK (JSR0317). We provided everything requested of us by Oracle, and we still haven't received the JPA2 TCK. Just to say Happy Anniversary Oracle on this evidently controversial request. The only possible reasons that I can think of for lack of provision of this are either incompetence, or deliberate prevention of access. Needless to say, we no longer wish to have access to this TCK.

On a related note I was asked whether I'd be participating in the JPA2.1 (JSR0338) expert group. I did think about it, no really! :-) But then I remembered that I spent many hours sending emails chasing Oracle employees to fulfil the request for the JPA2.0 TCK all to no avail. We also have to remember that JPA was born out of politics nothing more. Consequently the answer had to be a NO.

I have every willingness to participate in truly open standards on Java persistence, based on technical reasons. Any such participation would be with open mailing-lists/forums, and an open testing capability so that anyone can validate the compliance of any implementation. JPA, and the JCP process it operates under, is not that; JPA standardises things like "Criteria" that is widely recognised as inelegant (many call it ugly actually) and hard to use. A standard should take something that has become accepted as a good way of doing something (such as QueryDSL, for example) and convert it into something coherent with a clean API (like what we did with JDO Typesafe); it should not invent something and impose it on people. The JPA philosophy is simply wrong in this respect. I wish them good luck with JPA2.1, and DataNucleus will, at some point (typically as soon as the spec is made public), implement what they come up with, but I will not be party to its "design".

DataNucleus v3 and HBase

2011-01-13T10:40:00.000-08:00

DataNucleus AccessPlatform v3 provides an opportunity to bring some of our other datastore plugins closer to the standard of the more mature long-supported datastores (e.g RDBMS). In the case of HBase, the plugin provided with v2.0 offered basic persistence and querying, at best. In v3 this is already much improved.

You can now run SchemaTool against HBase. This operates in either "create" or "delete" modes, and allows you to manage the schema required by your persistable classes.
When a relationship was persisted with v2.0 it simply serialised the related object. This broke JDO/JPA cascade rules. In v3 the column for the relation in the owner stores the identity (or identities when persisting multi-value relations). This also provides correct cascading of persist and update.
When a String field was persisted in v2.0 it was Java serialised, meaning that it was not readable with something like "hbase shell". In v3.0 String/char fields are persisted as the bytes of the field value, hence readable.
With v2.0 we provided value generation using "uuid", "uuid-hex", etc simple generators, but not accessible by default. In v3.0 the default (JDO "native", JPA "auto") is "uuid-hex", and we also provide an "increment" generator (contrib from Peter Rainer).
In v2.0 we only supported "application identity". In v3.0 we also support "datastore identity" (surrogate identity column).
In v2.0 we didn't support storing a version against the object. In v3.0 we do allow this.
You can now embed persistable fields into the table of the owning object, and also nested embedded fields.
Now supports HBase 0.90

Obviously there is still much more that can be done - see the datastore features comparison table. One that will give more performance is to handle more of any query filter in the datastore, rather than just processing all in-memory. Contributions for this and other things are very welcome. But then we aren't even at 3.0 Milestone 1 yet ...

JDO Typesafe .vs. JPA Criteria

2010-11-08T23:24:00.000-08:00

JPA2 introduced its "Criteria" queries, providing an API for typesafe query generation without the need to hardcode field names etc in queries; it built on the approach of Hibernate Criteria. DataNucleus includes a proposal for JDO "Typesafe" queries. It takes a slightly different approach aiming at usability and elegance. In this blog post we compare some queries using the two APIs.

In these examples we have two classes, Inventory and Product, where Inventory has a set of products.

Select of persistable objects with simple filter

JPQL single-string would be

SELECT p FROM Product p WHERE p.name = 'MP3 Extra'

whilst JDOQL single-string would be

SELECT FROM Product WHERE this.name == 'MP3 Extra'

JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(productRoot);
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
List<Product> products = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.filter(cand.name.eq("MP3 Extra")).executeList();

Select of result of attributes of persistable objects

JPQL single-string would be

SELECT p.value, p.manufacturer FROM Product p WHERE p.name = 'MP3 Extra'

JDOQL single-string would be

SELECT this.value, this.manufacturer FROM Product WHERE this.name == 'MP3 Extra'

JPA Criteria would be

CriteriaQuery criteria = builder.createQuery();
Root<Product> productRoot = criteria.from(Product.class);
criteria.multiselect(productRoot.get(Product_.value), 
productRoot.get(Product_.manufacturer);
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
List<Object[]> results = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Object[]> results =
tq.filter(cand.name.eq("MP3 Extra"))
.executeResultList(cand.value, cand.manufacturer);

Select of aggregate of attribute of persistable objects

JPQL single-string would be

SELECT MAX(p.value) FROM Product p WHERE p.name = "MP3 Extra"

JDOQL single-string would be

SELECT MAX(this.value) FROM Product WHERE this.name == "MP3 Extra"

JPA Criteria would be

CriteriaQuery<Integer> criteria = builder.createQuery(Integer.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(builder.max(productRoot.get(Product_.value)));
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
Object result = em.createQuery(criteria).getSingleResult();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
Integer result =
tq.filter(cand.name.eq("MP3 Extra")).executeResultUnique(Integer.class, cand.value.max());

Select of persistable objects with simple filter and parameter

JPQL single-string would be

SELECT p FROM Product p WHERE p.name = :param

whilst JDOQL single-string would be

SELECT FROM Product WHERE this.name == :param

JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(productRoot);
ParameterExpression<String> valueParam = builder.parameter(String.class);
criteria.where(builder.equal(productRoot.get(Product_.name), valueParam));
TypedQuery<Product> query = em.createQuery(criteria);
query.setParameter(valueParam, "MP3 Extra");
List<Product> products = query.getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results =
tq.filter(cand.name.eq(tq.stringParameter("prefix")))
.setParameter("prefix", "MP3 Extra").executeList();

Select of persistable objects with joined filter condition

JPQL single-string would be

SELECT i FROM Inventory i JOIN i.products p WHERE (p.name = 'MP3 Extra')

JDOQL single-string would be

SELECT FROM Inventory WHERE this.products.contains(var) && var.name == "MP3 Extra"

JPA Criteria would be

CriteriaQuery<Inventory> criteria = builder.createQuery(Inventory.class);
Root<Inventory> invRoot = criteria.from(Inventory.class);
criteria.select(invRoot);
Join<Inventory, Product> productJoin = invRoot.join(Inventory_.products);
criteria.where(builder.equal(productJoin.get(Product_.name), "MP3 Extra"));
List<Inventory> inventories = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Inventory> tq = pm.newTypesafeQuery(Inventory.class);
QProduct var = QProduct.variable("var");
QInventory cand = QInventory.candidate();
List<Inventory> results =
tq.filter(cand.products.contains(var).and(var.name.eq("MP3 Extra"))).executeList();

Select of persistable objects with subquery filter

JPQL single-string would be

SELECT p FROM Product p WHERE p.value < (SELECT AVG(q.value) FROM Product q)

JDOQL single-string would be

SELECT FROM Product WHERE this.value < (SELECT AVG(q.value) FROM Product q)

JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root productRoot = criteria.from(Product.class);
criteria.select(productRoot);
Subquery<Double> sub = criteria.subquery(Double.class);
Root<Product> subRoot = sub.from(Product.class);
criteria.where(builder.lt(productRoot.get(Product_.value), 
sub.select(builder.avg(subRoot.get(Product_.value)))));
List<Product> products = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
TypesafeSubquery<Product> tqsub = tq.subquery(Product.class, "q");
QProduct candsub = QProduct.candidate("q");
List<Product> results = 
tq.filter(cand.value.lt(tqsub.selectUnique(candsub.value.avg()))).executeList();

As you have seen, for all typesafe queries JDO Typesafe is more elegant and shorter. It requires no "builder" too. Access of a field of a class is one particular example, typing "cand.value" instead of "productRoot.get(Product_.value)"

JDO Typesafe Queries : Part 3 - Examples

2010-11-02T03:07:00.000-07:00

In two previous blog posts, we introduced the idea of having a typesafe refactor friendly API for queries for JDO, and then described typical expression types necessary to achieve that. In this blog post we take that a step further, particularly now that DataNucleus SVN has a proof of concept query API to fulfil this requirement.

In these examples we have a class Product. Consequently we have a "query" class autogenerated - QProduct. Here is the QProduct class


public class QProduct 
    extends PersistableExpressionImpl<Product> 
    implements PersistableExpression<Product>
{
    public static QProduct candidate() {...} // candidate "this"
    public static QProduct candidate(String name) {...}
    public static QProduct parameter(String name) {...}
    public static QProduct variable(String name) {...}

    public NumericExpression<Long> id;
    public StringExpression name;
    public NumericExpression<Double> value;

    ...
}

So, as you can see we have access to the persistable fields/properties of Product by way of public fields in its query class. Now on to some examples

Filter only
Here we select all Products that have a value less than 40.0


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0));
List<Product> results = tq.executeList();

Filter + Order
Here we select all Products that have a value less than 40.0 and ordering by their name


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc());
List<Product> results = tq.executeList();

Filter + Order + Result
Here we select the product name and product value for all Products that have a value less than 40.0 , ordering by their name


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc());
List<Object[]> results = tq.executeResultList(true, cand.name, cand.value);

Filter with Methods
Here we select all Products that have a value less than 40.0 and name starting with "Wal", and ordering by their name


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0).and(cand.name.startsWith("Wal")));
List<Product> results = tq.executeList();

Filter, using Parameters
Here we select all Products that have a value less than 40.0, using a parameter


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate()
tq.filter(cand.value.lt(tq.doubleParameter("param1"))).setParameter("param1", 40.0);
List<Product> results = tq.executeList();

Order + Range
Here we select Products, ordering by the name and restrict to the first two


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.orderBy(cand.name.asc()).range(0,2);
List<Product> results = tq.executeList();

Filter with Variable
Here we select Inventory (which has a collection of Products) that contain a Product with a particular name


TypesafeQuery<Inventory> tq = pm.newTypesafeQuery(Inventory.class);
QInventory cand = QInventory.candidate();
QProduct var = QProduct.variable("var");
tq.filter(cand.products.contains(var).and(var.name.startsWith("Wal")));
List<Inventory> results = tq.executeList();

Filter with Subquery
Here we select Products with a value less than the average value of any Product


TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
TypesafeSubquery tqsub = tq.subquery(Product.class, "p");
QProduct candsub = QProduct.candidate("p");
tq.filter(cand.value.lt(tqsub.select(candsub.value.avg())));
List<Inventory> results = tq.executeList();

As you can see from the above queries we have no hard-coded class or field names, providing better refactorability. We can obviously make use of all JDOQL supported methods in the above queries, and define parameters and variables just like in the current JDOQL API.

It should be noted that this JDOQL facility is much more elegant and requires less code than the equivalent queries using JPA "Criteria".

Writing data to Excel from Java - the simple way

2010-07-14T11:05:00.000-07:00

On the web you can find some blogs about how to use Apache POI to create an Excel spreadsheet. They typically contain reams of code just to add a simple table and some rows of data. It's always amazing the amount of suffering people put themselves through to perform simple tasks. An example is this blog (there are others) which demonstrates some typical calls you will make using Apache POI to create a simple spreadsheet. Whilst that may represent what Apache POI requires, a user choosing to follow this way is inflicting unnecessary suffering on themselves, really. Read on ...

Here's how we would achieve the same thing as the referenced blog using JDO and DataNucleus. Firstly lets create a class that equates to a row of data in the table.


@PersistenceCapable(table="SampleDataSheet1")
public class SampleData
{
    @PrimaryKey @Column(name="Employer Name")
    String employerName;
    @Column(name="Designation")
    String designation;
    @Column(name="Country")
    String country;

    public SampleData(String e, String d, String c)
    {
       this.employerName = e;
       this.designation = d;
       this.country = c;
    }
}

Now we define our datastore persistence properties (datanucleus.properties)


javax.jdo.PersistenceManagerFactoryClass=
    org.datanucleus.jdo.JDOPersistenceManagerFactory
javax.jdo.option.ConnectionURL=excel:file:test.xls

Finally let's do some persistence


PersistenceManagerFactory pmf = 
    JDOHelper.getPersistenceManagerFactory(“datanucleus.properties”);
PersistenceManager pm = pmf.getPersistenceManager();
Transaction tx = pm.currentTransaction();
tx.begin();
SampleData ds =
    new SampleData(“Intelligent User”, “Software Engineer”, “Bolivia”);
pm.makePersistent(ds);
tx.commit();

and this generates the same output that was shown in the blog, just that the code is now readable, shorter, and uses standardised APIs, never mind the fact that you could conceivably just persist the same data to RDBMS, XML, ODF, ODBMS, Cassandra and many more datastores with a simple change of that datastore URL. Additionally it allows you to focus on objects, which is likely why you chose Java in the first place. Why put yourself through more pain ?

[Needless to say, DataNucleus would also allow you to do the same using JPA]

TJDO has left the building

2010-07-13T11:51:00.001-07:00

As anyone who has been using DataNucleus (and JPOX before it) for some time will know, it all started out (in 2003) as TJDO. This project was forked to provide all of the missing functionality that JDO requires and users of that technology demand (starting with application identity, and all common ORM relationship patterns). Over the last 7 years components have been rewritten one-by-one

Table handling code was upgraded to cater for JDO2 inheritance strategies

MetaData was rewritten to match the JDO2 definition (replacing TJDO's JDO1-style handling)

Transaction handling was rewritten to cater for XA transactions

Mapping code was rewritten to split into java type handling and datastore handling

DataNucleus SVN trunk (v2.2) now removes the last significant component of TJDO ... its JDOQL implementation for RDBMS (known in v2.1 as "Legacy"); it is now totally replaced by the generic-query-expression-tree approach that allows many many additional JDOQL queries to execute. There still remain some TJDO vestiges but nothing of any significance. R.I.P

JDO Typesafe Queries : Part 2 - Expressions

2010-07-07T10:43:00.000-07:00

Continuing on from the previous post, in order to be able to express components of a query in a Java style we need to represent all fields/properties/parameters/variables as expressions. The type of the field/property/parameter/variable determines its expression type. Let's take an example


public class Person
{
    String firstName;
    String lastName;
    int age;
    Person bestFriend;
}

We need to represent these field types with expressions, so let's start with

StringExpression - for Strings

NumericExpression - for int, long, short, double, float, Integer, Long, Short, Double, Float, BigDecimal, BigInteger etc

BooleanExpression - for boolean, Boolean

ByteExpression - for byte, Byte

CharacterExpression - for char, Character

DateExpression - for Date-based types

ObjectExpression - for other Object-based types where we have no specific handling

Ok, so this is all well and good and we can express firstName as a StringExpression, and similarly age is a NumericExpression. So for that we can do


QPerson person = QPerson.person;
Query<Person> q = pm.newTypesafeQuery(person);
Person bob = 
    q.filter(person.firstName.eq("Bob")).executeUnique();

To represent a persistable field (i.e a 1-1 relation, bestFriend in the above example) we have another expression type PersistableExpression.

"Q" classes

Above you see use of metamodel query classes. We refer to them as "Q" classes currently, but the naming is arbitrary for now. So what does a "Q" class look like ?


public class QPerson implements PersistableExpression 
{
    public static final QPerson person = new QPerson("person");

    public final QPerson bestFriend;
    public final NumericExpression age;
    public final StringExpression name;

    ... (implementation of other PersistableExpression methods).
}

So in simple terms, we have a public field for each of the normal fields in Person, but that are of XXXExpression types. So when a user accessed QPerson.person they get the candidate for use in a query. Then they can do "person.name" and this is a StringExpression. The NumericExpressionImpl/StringExpressionImpl are the implementations for the particular provider of this typesafe query (e.g DataNucleus). You also notice above that the bestFriend field is also a QPerson, and hence a PersistableExpression, so we can chain field access as "person.bestFriend.firstName"

Methods of field types

Obviously in JDOQL (and Java), we allow some method calls. This is represented here by adding the supported methods to StringExpression, NumericExpression etc. For example StringExpression has a method toUpperCase() to match what Java allows, so we can upgrade the query example to be


QPerson person = QPerson.person;
Query<Person> q = pm.newTypesafeQuery(person);
Person bob = 
    q.filter(person.firstName.toUpperCase().eq("BOB")).executeUnique();

There are still some more challenging areas of the API to work out, but the above is just to give a further taster and provoke comment.

Please refer to the current javadocs here (Work-in-progress).

JDO Typesafe Refactorable Queries

2010-06-22T07:29:00.000-07:00

One of the primary drawbacks with current JDO queries is that they are string-based, so you are hardcoding field names etc. When you refactor a (model) class, the field names are left unchanged, so the query no longer works. Some efforts have been made to simplify this process by providing a form of type-safe query (called "criteria" queries by some, but the name is misleading since all queries define criteria) - see JPA2 Criteria queries, and also QueryDSL. Of the current offerings the QueryDSL fluent API approach is considered by many to be the most elegant.

JDO needs such a type-safe refactorable query API. However it is important that this does not go down the track followed by JPA Criteria queries (IMHO) since it leads to excessive code just to generate a query, hence the fluent approach make more sense. What we are proposing would be something similar to the following (taken from QueryDSL, but adapted to use more JDOQL-like syntax)

QCustomer customer = QCustomer.customer;
Query<Customer> q = pm.newTypesafeQuery(customer);
Customer bob = q.filter(customer.firstName.eq("Bob"))
     .unique(true).execute();

where the QCustomer class is generated automatically (during compilation via an annotation processor, or optionally via a separate process when not using JDO annotations).

The idea is to provide support for all JDOQL syntax via this API, yet provide type-safe handling and refactorability, introducing this in the DataNucleus AccessPlatform 2.2 timeframe, and finally to standardise something similar into JDO 3.1. Your ideas/comments are welcome.

Persisting multisets (bags) with DataNucleus

2010-06-03T05:36:00.000-07:00

A multiset is a set where you can have a particular object multiple times in the set (a normal java.util Set will remove dups). This is also known by some as a "bag". Google's Collections library has such a container class (Multiset), and fields of this type are now persistable using DataNucleus.

Put the datanucleus-googlecollections plugin in the CLASSPATH

Develop your model using the Multiset java type.

Make sure that your Multiset field uses a join-table when persisting to RDBMS, and that the join-table doesn't have a primary-key defined

So if we have


@PersistenceCapable
public class Sample
{
    @Join(extensions=
      @Extension(vendorName="datanucleus", 
            key="primary-key", value="false"))
    Multiset elements = null;
    ...
}

this will create an RDBMS join table as


CREATE TABLE SAMPLE_ELEMENTS 
(
    ID_OID BIGINT NOT NULL, 
    ID_EID BIGINT NOT NULL
);

i.e no primary key, and so we can have duplicates of the same element in the Multiset.
Thereafter the persistence just performs as normal (for JDO or JPA).

JPA, and the role of compliance

2010-04-16T08:16:00.001-07:00

For some time we've been trying to get our hands on the JPA2 TCK (JSR317). This is a suite of tests that is supposed to define if your JPA implementation is "compliant". JPA (1, 2) are developed in private (secret) by the JCP. You can try to get on their "Expert Group" but since JPA1 did not allow in "just anybody" (and people were hand-picked in those days), there seems little point in participating in that.

To actually get your hands on the "TCK" for any JSR you need to sign some form and send it to Sun/Oracle by snail mail. Since I sent one of these forms in (for the JPA1 TCK) back in 2007 then it really ought not to be necessary one would think. Well that's what we were advised by SUN/Oracle also (after a wait of a month and half). So here we are more than 2 months on and still no TCK. Not even a sniff of one.

Why is this TCK so secret ?
This seems a reasonable question. What is so sensitive in it that it shouldn't be open to everyone ? After all, with an open TCK, anybody would be able to actually test the implementation of their choice and verify if it is indeed compliant. They would also be able to view the testcases and contribute new ones that were missed in the test suite to cover other functionality. Obviously not everyone would want to download it and check people's claims, but then some would and it would hold people's claims up for review, unlike now. Really, why do I have to sign an NDA just to see it ? and why can't I publish the results in full on our website ?

Why does it take so long ?
Now granted SUN is merging into Oracle, but really 2+ months doesn't really seem too efficient. Why is the process manual, snail-mail driven ? After all, this is something electronic at the end of the day, so you would have expected that if people really have to apply for something then there would be a page on which to request it, and it would forward the infos to some team who approve or otherwise. Is this something typical of US organisations in general, or just Oracle?

Why can other implementations claim compliance when we can't even get a chance to verify it?
Really does say a lot about the corporate-driven JCP and politics. The JCP is inefficient, secretive, and doesn't present a level playing field for all.

PS1. When I tried to get the JPA1 TCK back in 2007 it took 3 months of continual emailing before I finally got my hands on it. That was when they weren't merging with anyone else, so seems to be fairly typical of the wasteful process they have.

PS2. JDO obviously has an open TCK, Apache2 licensed, available for all to download and contribute to. You don't need to sign NDA's just to see it.

Update (22nd June 2010) : Now more than 4 and a half months have passed since the original request for the JPA2 TCK. Still nothing.

So do you think Oracle is deliberately delaying this ? or is it just inefficient and disorganised ? Answers on a postcard please. The point that is becoming clearer is that JPA "compliance" is not worth the effort and DataNucleus releases will go forward claiming compliance since the JCP is preventing us from participating in this little game of theirs.

JDOQL2 status

2010-03-04T02:02:00.000-08:00

Some time ago I started a new implementation of JDOQL for RDBMS, to replace the legacy (TJDO) JDOQL implementation. The legacy implementation had problems with generation of SQL for many queries, particularly involving "contains", "!contains" and use of unbound variables. As described here the replacement implementation builds on the generic query compilation used for all other datastores.

With the release of AccessPlatform 2.0.2 "JDOQL2" is ready for use in the vast majority of situations. It fixes various issues with the legacy implementation, successfully generating SQL for many queries that failed with legacy. It still doesn't compile every complicated query perfectly, but it is at least understood what parts it is failing on now, and none of those parts in general work in the legacy implementation.

In AccessPlatform 2.1 "JDOQL2" will be renamed to "JDOQL", and the legacy implementation will become "JDOQL-Legacy" so you will get this new implementation by default, and have to explicitly select the legacy implementation if you wish to use it.

By AccessPlatform 2.2 we intend to remove the legacy JDOQL and just move forwards with this new implementation.

GAE/J and JDO/JPA

2010-01-28T00:08:00.000-08:00

In April 2009 Google announced AppEngine for Java, providing support for JDO and JPA. Since that time people have had the chance to see what they think of it and identify shortcomings. This has lead to many misconceptions of JDO and JPA (though mainly JDO) due to thinking that what Google provide is a full and true representation of these standards; it isn't. This blog entry attempts to correct some of these misconceptions, and to suggest some areas where Google could remedy the situation. All of the following items would help in providing a true reflection of these persistence standards and aid GAE/J users in having (much more) portable applications.

JDO/JPA : Lack of support for many methods and operators in JDOQL/JPQL

GAE/J only supports a relatively small subset of the available methods and operators of JDOQL/JPQL. This is due to the underlying datastore not supporting certain capabilities in its queryability. The problem is that this gives the impression that JDOQL/JPQL are somehow weak. What would make way more sense would be to evaluate all that is evaluatable in the datastore, and then have a flag set while compiling for whether the query contains any feature that the datastore query cannot handle and, if so, run the DataNucleus in-memory query evaluator on the resultant instances. This would provide a transparent interface to JDOQL/JPQL and mean that people don't have to have build some custom queries just to get around GAE/J shortcomings.

JDO : No support for input candidate collection for JDOQL

GAE/J does not support inputting a candidate collection and query over that collection. This is a trivial thing to support, particular since the code necessary to do it was contributed some time ago, yet isn't in the current plugin.

JDO : pm.getExtent doesn't handle subclasses

The current pm.getExtent() implementation in GAE/J doesn't support the subclasses flag. The root cause is that the underlying datastore doesn't support a single query to retrieve a class and its subclasses. The simple solution would be to run "n" queries on the datastore, one for each of the possible subclass types, and merge the results. This would be simple to do and implement, and would provide correct JDO behaviour so users don't see any shortcoming.

JDO/JPA : exposing Google-specific id classes

With GAE/J there is some flexibility on what is allowable as a PK field; Long, String, and "Key". The first two are standard classes and standard JDO. The latter is environment specific. Firstly GAE/J ought to allow short/int/Short/Integer/long for true JDO/JPA operation so that users see no difference. Secondly, the "id" exposed to the user should be a JDO or JPA id and should be portable. When we implemented support for persisting to db4o we didn't expose db4o's id, instead wrapping it internally, so the user has no unexpected classes popping up that prevent their migration elsewhere later on.

JDO/JPA : one entity group per transaction

With GAE/J you have a restriction on the number of "entity groups" that can be enlisted in any transaction. Ok, but why expose this to the user and restrict what they do ? The logical way to do it would be to have multiple "internal" transactions for a JDO/JPA transaction and have each of these for a particular entity group. Since the underlying datastore doesn't provide ACID transactions anyway there is little impact of doing this. It would then mean that you don't impose on users having to split their persistence code apart just to get it to run, and hence mean that it is portable

Unowned relations

This term is being used in GAE/J seemingly where you have a Collection<id> and so no real relation, although the "id" relate to other objects. This is perfectly representable in JDO/JPA as a Collection<long>, or Collection<String> or even Collection<Object>. With one of these "relations" the onus is on the user to manage the relation.

Support for types persistable as String

DataNucleus has, for some time, provided a mechanism for defining how to persist a type as a String (and retrieve its value from the String) - see ObjectStringConverter in DataNucleus "core" code. GAE/J could easily provide support for this in their plugin (if a type is not natively supported then check if there is an ObjectStringConverter and use that) and this would mean that many more Java types are persistable using AppEngine.

Documentation : @Persistent

In the GAE/J docs, every field has @Persistent marked against it. This is totally unnecessary, and you only need @Persistent for a non-standard field type. It leads to people believing that you must specify this to get something persisted, and so when they want to have a field not persisted they just remove the annotation. Please update the docs to reflect the minimal configuration required so we give a fair reflection of JDO and its spec. For example

@PersistenceCapable
public class MyClass
{
@PrimaryKey
Long id;

String name;

double value;

...
}

Package naming "org.datanucleus.*"

This plugin is provided by Google not DataNucleus. It's currently packaged as "org.datanucleus.store.appengine". This leads to people believing that DataNucleus itself is at fault for its shortcomings. This is unacceptable and we own the domains datanucleus.org/datanucleus.com. Please rename your packages ASAP.

Nowhere have we seen any attribute of the GAE/J BigTable datastore that cannot be handled by the JDO or JPA API's. The JDO API (and metadata) in particular was designed as generic, and there is nothing in a "NoSQL" datastore that should cause it any problems with representation. We challenge anyone to define where there is such a problem area and it can then be addressed (there's a JIRA open on the Apache JDO project for just this situation); if you really can come up with a problem area then its in all of our interests to understand it and tackle it.

AccessPlatform 2.0 is released

2010-01-08T03:50:00.000-08:00

a day ahead of schedule! Many internal APIs have changed so if making use of DataNucleus classes be prepared to update your code before using this release. The migration guide can be found here.

As ever, report any issues found in this release in the forum, and via JIRA (with testcase following the DN standard reproducing the issue).

AccessPlatform 2.0 will arrive soon

2010-01-01T12:25:00.000-08:00

... just until the 9th January 2010 to wait. The result of 10 months of restructuring and extensions to DataNucleus, extending the capabilities and simplifying the process of adding on new datastore support for many things.

Auto-generation of primary-key classes

2009-11-19T06:34:00.000-08:00

The JDO spec defines a series of PK classes that will be used when you have a single primary-key field and don't specify the objectidClass. This provides for the majority of situations meaning that the user can omit this time-consuming process of creating a PK class. Obviously this leaves the cases where you have more than 1 primary-key field. There are too many possible combinations of field types to be able to provide a series of PK classes as part of JDO, hence why this has always been an activity best handled by the user.

DataNucleus AccessPlatform 2.0 now has an option on the byte-code enhancer to auto-generate a primary key class when enhancing the class itself; it is turned on by default (you can turn it off using the option "generatePK", setting it to false) and whenever the class has more than 1 PK field and has no objectidClass defined it will generate a PK class file. The generated PK follows all JDO rules on PK classes, implementing Serializable, having public fields with the same names as the class PK fields, String constructor that can process the String created by the toString() method, etc.

Note that this mode of the enhancer doesn't generate the Java file, just the class file, meaning that you don't have the overhead of having to manage the PK class once created - it will be created whenever you enhance the class in question.

Give it a try and see what you find. It's been tested on simple cases only so far.

"JPQL2" becomes the default JPQL implementation for RDBMS

2009-11-06T04:16:00.000-08:00

The rewrite of JPQL for RDBMS, formerly known as "JPQL2" is now the default JPQL implementation in DataNucleus SVN trunk. It passes the JPA TCK completely, and so no reason not to swap over to using it now. So if you want to use the old (legacy) implementation of JPQL for RDBMS you must define the implementation

datanucleus.query.JPQL.implementation=JPQL-Legacy

It's likely that there are some combinations of expressions that still need adding but provides a good basis for JPA querying.

The reason that JDOQL2 is not yet the default is that JDOQL requires support for variables to define joins, whereas JPQL has explicit joins defined in its syntax, hence it was easier to provide JPQL. Hopefully JDOQL2 won't be too far behind. Report any problems with this implementation of JPQL using JIRA for project "NUCRDBMS".

"JPQL2"

2009-04-30T08:50:00.000-07:00

As mentioned earlier DataNucleus querying capabilities are being progressively rewritten, making use of a "generic" query compilation. In the previous post I mentioned a new implementation of JDOQL. This now implements the vast majority of querying capabilities required with one exception ... variables (to be added soon hopefully). In addition, for this RDBMS implementation, you can evaluate queries

Totally in the datastore.
Totally in-memory (you supply the candidate instances, and it queries over those instances).
Retrieve all candidates from the datastore, and evaluate the filter, result etc in memory.

SVN trunk now has a new addition following the same basic idea ... "JPQL2". This implements JPQL using the same underlying SQL API, and using the bulk of the new code added for "JDOQL2". Since JPQL doesn't support variables then this implementation is close to complete (with the exception of bulk update/delete - to be added soon).

One important feature to note is that a user can contribute their own implementation of various methods, or functions by adding their own plugin (just a single method typically). See the docs for an example.

The intention is to make these new implementations the default in the next major release of DataNucleus Access Platform. To try them out add the following persistence properties

datanucleus.query.JDOQL.implementation=JDOQL2
datanucleus.query.JPQL.implementation=JPQL2

"JDOQL2"

2009-02-18T05:35:00.000-08:00

DataNucleus started out as JPOX, which started as TJDO. This had support for RDBMS, and in particular JDOQL querying. Unfortunately the TJDO process for JDOQL was a sequential parse of the query string, generating the SQL for the query. While we extended this a lot during the JPOX lifetime this was always limited by the underlying API available and the process (not being able to easily inspect the query to see if something has AND or OR at the same level, and hence decide if we should do an EXISTS, or an INNER JOIN etc etc). It was always planned to rewrite it at some point, particularly to make the query compilation available to other types of datastores, but also to try to resolve some of the problems.

DataNucleus AccessPlatform 1.1 now has a generic query compilation process (proof of concept written by Erik, and extended by me to come close to complete compilation, with some minor patch contributions from others). The generic query compilation results in expression "trees" for each component of the query; these are easier to analyse and hence to generate any native query from, for the datastore being used. This generic query compilation is what is used by all datastore plugins (with the exception of legacy RDBMS). In addition it includes an "in-memory" query evaluator. This is intended to take in a collection of candidates for the query, and evaluate the filter clause using the data in those instances. This in-memory evaluation works for many things but currently doesn't yet support Collection.contains, Map.containsKey, etc and also variables.

AccessPlatform 1.1 RDBMS plugin also includes code for something referred to as "JDOQL2". This is not a new query language, but instead an alternative implementation of JDOQL for RDBMS datastores. It uses the underlying generic query compilation (like for the other datastores), and has a step converting this into SQL. This new implementation has a few key concepts

Uses a newly written SQL API to generate the statement

SQL statement strings are JDBC-compatible, having parameters only where the user query has them (TJDO SQL didn't follow this)

Users have control over the naming scheme for table aliases in the SQL

Implementation of support for JDOQL "methods" via a plugin-point, so you could write a plugin for Collection.size(), for example, if you wanted.

Support is provided for all aggregate functions, the main String methods, the main Date methods, JDOHelper.getObjectId, Collection.size(), Collection.isEmpty(), Map.size(), Map.isEmpty(), and some others ... but not yet Collection.contains/Map.containsKey/Map.containsValue/Map.get

All queries compiled using JDOQL2 are cached, meaning they are not compiled again for that PMF

With the legacy JDOQL implementation, a query would be compiled at execute (and also at compile if compile() was called). In JDOQL2 it is only compiled once.

JDOQL2 implements the JDO2.3 query timeout/cancel API, so you can, in principle, cancel queries if they were submitted from a different thread and are still running. This would then cancel the underlying JDBC query (assuming the JDBC driver supports it)

Queries will retrieve not just non relation fields (like with legacy JDOQL) but also any 1-1 fields, hence avoiding the "1+N" problem

JDOQL2 is there for you to use and try, just do something like (for RDBMS)


Query q = pm.newQuery("JDOQL2",
   "SELECT FROM mydomain.MyClass WHERE field1 < :param");
q.execute(value);

and see what you find. The idea is that this will replace (legacy) JDOQL for RDBMS sometime in the AccessPlatform 2.0 lifecycle; maybe not if time doesn't allow it.

JDO 2.3 Metadata API

2009-01-27T02:32:00.001-08:00

Defining how classes are to be persisted has always been a subjective part of any persistence process. Some people like XML, some like annotations, and some want it all to be done magically with a sprinkle of stardust. JDO has always allowed configuration using XML. In JDO 2.1 we introduced the ability to use annotations (and a mix of annotations and XML). With JDO 2.3 we're introducing a third way ... via an API. Let's assume we have a class we want to persist

public class Client
{
String name;
...
}

The basic idea is as follows. We want to define the metadata programmatically for our class
PersistenceManagerFactory pmf = JDOHelper.getPersistenceManagerFactory(propsFile);
...
JDOMetadata md = pmf.newMetadata();

So our PMF has created a new metadata object to work with. This corresponds to the top level of a metadata file. We can now add packages and classes as required to define the persistence. So something like this ...
PackageMetadata pmd = md.newPackageMetadata("org.datanucleus");

so we have a package for our class. Now let's define the persistence of the class :-
ClassMetadata cmd = pmd.newClassMetadata("Client");
cmd.setTable("CLIENT").setDetachable(true);
cmd.setIdentityType(IdentityType.DATASTORE);
cmd.setPersistenceModifier(ClassPersistenceModifier.PERSISTENCE_CAPABLE);

and the class needs persistence for the field "name" defining (since we don't want to accept default persistence specifications) :-
FieldMetadata fmd = cmd.newFieldMetadata("name");
fmd.setNullValue(NullValue.DEFAULT).setColumn("client_name");
fmd.setIndexed(true).setUnique(true);

and of course we can define everything else as required, so to persist it into its own table, and to have versioning, etc :-
InheritanceMetadata inhmd = cmd.newInheritanceMetadata();
inhmd.setStrategy(InheritanceStrategy.NEW_TABLE);
DiscriminatorMetadata dmd = inhmd.newDiscriminatorMetadata();
dmd.setColumn("disc").setValue("Client");
dmd.setStrategy(DiscriminatorStrategy.VALUE_MAP).setIndexed(Indexed.TRUE);

VersionMetadata vermd = cmd.newVersionMetadata();
vermd.setStrategy(VersionStrategy.VERSION_NUMBER);
vermd.setColumn("version").setIndexed(Indexed.TRUE);

All that remains to do is enable this metadata in our persistence process
pmf.registerMetadata(md);

Key points :

you get your starting object (JDOMetadata) from the PMF.

you only need to add fields and attributes that are non-defaulted. As with a metadata file, all fields of standard types are by default persistent

you can chain mutator operations. I've split them above for readability, but they could have been on a single line.

once populated you register the start object with the PMF. Thereafter when that class is used in persistence it will use that metadata definition.

If a class is already known to the PMF (has its file-based metadata registered) then you can't redefine it using this API.

The logical next step is to be able to view the metadata defined in XML or annotations. You can do this as follows
ComponentMetadata compmd = pmf.getMetadata("mydomain.MyClass");

and you can now interrogate the metadata of this class (cast it to ClassMetadata or InterfaceMetadata). Please note that it is not supported being able to change metadata that is already registered, just view it.

Final point worth making is that the DataNucleus implementation of these metadata classes will output the (approximate) XML text representation when you call "toString()" on the metadata object, so for example the metadata generation above results in

<jdo>
<package name="org.datanucleus">
<class name="Client" type="datastore" modifier="persistence-capable" table="CLIENT" detachable="true">
<inheritance strategy="new-table">
<discriminator strategy="value-map" column="disc" value="Client" indexed="true">
</discriminator>
</inheritance>
<version strategy="version-number" indexed="true" column="version">
</version>
<field name="name" column="client_name" indexed="true" unique="true">
</field>
</class>
</package>
</jdo>

so you also have an easy way of converting your API generated metadata into XML should you wish to transfer it to a file for later use.

This is now in DataNucleus SVN trunk. You need the latest Apache JDO API jar (also in DataNucleus SVN, and in the nightly build repositories).

Welcome to the DataNucleus

2008-10-04T11:28:00.001-07:00

DataNucleus is a project providing Open Source software for use in the Java language, for persisting objects to datastores.

DataNucleus AccessPlatform 1.0 (Faraday) was released on September 1st 2008, providing JDO/JPA persistence to RDBMS, db4o, XML, Excel, LDAP, NeoDatis and JSON datastores.

DataNucleus AccessPlatform 1.1 (Rutherford) development started soon after and will build on the support in 1.0, aiming at migrating the RDBMS query support to use the new generic query mechanism, as well as moving to a JDK 1.5+ environment.

As with the majority of Open Source software, DataNucleus is freely downloadable. Limited amounts of free support is available by our forum. We do also provide commercial support, and by purchasing this you are keeping the project going.