One of the reasons why I wrote this article is my, rather unusual, encounter with a missing message body writer in a Jersey 1 test application. This problem of mine occurred during collecting of data to get a Jersey 1 performance baseline in order to see where Jersey 2 stands when compared to its predecessor. The problem is described at the end of this article because I think it’s pretty rare, but still not impossible :-), and not many of you will (hopefully) face it. But first I’d like to take a look at more common issues regarding missing providers and what you can do to solve them.
The first thing to try when your application doesn’t behave as it should is to turn on logging (on CONFIG level at least) to get more information from the underlying JAX-RS implementation (in case you use Jersey, take a look at this post). For example, Jersey would print out all registered resources and providers which may show whether there is or isn’t problem in your configuration very quickly.
When you find out that your provider (or resource) is registered and there is still something wrong with your application, try registering LoggingFilter or using Tracing Support. However, if you realize that your application knows nothing about the provider you need then the solution may fall into one of these categories:
Class-path scanning – @Provider (JAX-RS)
If you rely on class-path scanning during deployment of your application make sure the classes are visible (have appropriate constructors) and the providers are annotated with @Provider annotation otherwise they won’t be discovered (example).
No class-path scanning – Application (JAX-RS)
In case you’re configuring your application via Application subclass (or via proprietary mechanism provided by JAX-RS implementation you use, e.g. ResourceConfig) be sure your providers are registered using
- getClasses() – example,
- getSingletons() – example, or
- register(…) (Jersey) – example.
Service Lookup – META-INF/services (JAX-RS)
Some 3rd party libraries (e.g. Jackson 2) may use META-INF/services lookup mechanism to register their providers in JAX-RS runtime. In this case make sure the Jars are really on the class-path. If they are and you still experiencing troubles look at the Note below (for Jersey 2), or My Case at the end.
Note: Jersey 2 doesn’t automatically pick-up providers registered via META-INF/services for quite some time now. If you want to use those providers either register them manually via Application (see above) or add org.glassfish.jersey.ext:jersey-metainf-services extension module to your application. The module enables automatic registration of JAX-RS providers (MBW/MBR/EM) via META-INF/services mechanism.
Uber JAR (JAX-RS)
When an (executable) über jar is your way of deploying/running applications you may encounter an exception like:
SEVERE: com.sun.jersey.api.MessageException: A message body writer for Java class java.lang.String, and Java type class java.lang.String, and MIME media type text/plain was not found
SEVERE: The registered message body writers compatible with the MIME media type
are:
*/* ->
com.sun.jersey.server.impl.template.ViewableMessageBodyWriterWhen you see an exception like this (I mean a provider for String, required by JAX-RS itself, cannot be found) you know there’s something really wrong with your application.
The reason is you’re creating your über jar using mis-configured assembly plugin (e.g. from Maven) that, by default, handles META-INF/services incorrectly. Files in META-INF/services are not CONCATENATED as they should be, they’re overwritten by each other. This means that if you have same services (e.g. MessageBodyWriter) in multiple Jars then only services from one (the last one) Jar are registered.
Solution to this problem is to either use prepared “all-in-one” bundles of JAX-RS implementation of choice or use plugin that concatenates META-INF/services files instead of overwriting them (e.g. maven-shade-plugin).
Note: This problem is uncommon when using Jersey 2.
LC_ALL and LC_CTYPE – My Case
Our performance test-cases are simple applications designed to test features of JAX-RS 2.0 and Jersey 2. Some of them are JAX-RS 1.1 compliant and we decided to run them using Jersey 1 to be able to compare performance of major versions of Jersey. To do so we have a hudson job that starts one test-case per time on a “server” node (via ssh) and it also starts wrk clients on two “client” nodes (also via ssh). No mysteries here.
I selected test-cases to be measured and using this hudson job tried that the Jersey 1 runner was working as expected (on my machine). Then I tried to run the tests on our infrastructure expecting no problems. The first run created 750MB of logs that looked like:
SEVERE: com.sun.jersey.api.MessageException: A message body writer for Java class java.lang.String, and Java type class java.lang.String, and MIME media type text/plain was not found
SEVERE: The registered message body writers compatible with the MIME media type
are:
*/* ->
com.sun.jersey.server.impl.template.ViewableMessageBodyWriterExactly as in the “Uber JAR” case. Except the test-cases wasn’t deployed as one big fat Jar. It worked on my machine, it didn’t on the remote one. I tried to copy binaries built on my machine (they worked, right?) to the remote one and ran the case. It didn’t work. Then I tried to use different versions of JDK (maybe something is broken there). It didn’t work either.
“Awesome”.
I put few debug logs into ServiceFinder (class that loads services/provider from META-INF/services in Jersey) to find out what’s going on. ServiceFinder loads META-INF/services files using Classloader#getResources(String) method. When executed it looked like only one META-INF/service resource file for a particular service was loaded from all Jars present on my class-path. The first that was found. Others were ignored even though they were there as well. And I got no ideas how to solve that.
It worked on my machine. It didn’t work when hudson started the server via ssh on a separate node. And it didn’t work when I tried to start the server on the same node via ssh.
Out of desperation I asked Pavel to try the test-case on his machine. It worked as it had worked on mine. Then he tried to connect to the remote machine and start the test-case there. It worked. Wot?! I tried it again from my machine. And … it didn’t work.
This was interesting. We compared environment variables when connected to the remote machine and there was one difference. Value of LC_CTYPE. My terminal (iTerm2) doesn’t set this variable on remote machine but Pavel’s terminal (OS X Terminal) does. This was it. I spent few hours trying to figure out what the problem was and in the end the remedy to the problem was so stupidly simple. Add the following line to .bashrc on the hudson slave that invokes *nix scripts on remote machines:
export LC_ALL=UTF-8Conclusion: When class loader loads resources stored in Jars present on your class-path the result depend on value of the LC_CTYPE environment variable (*nix boxes). So, when you ssh’d to a remote host make sure you have correctly set LC_ALL or LC_CTYPE because it does matter!
Note: The value of this variable on the remote host is set from the value that is set on the machine you’re trying to connect to the remote host from. Keep this is mind.