Hibernate Traps: @Transactional Integration Tests

Introduction

After a short summer break we're getting back to tracking down the most sneaky and vicious traps that Java developer may stumble upon when using Hibernate. This time we are stretching the meaning of 'Hibernate trap' a little, because things that are described in this article require us to use Spring Framework in conjunction with Hibernate. Hibernate on its own doesn't offer any kind of automated Transaction management. However, given that 9 out of 10 Java devs uses Hibernate with Spring (or at least that's my assumption), I will happily carry on with the title as it is.

Spring Transactions 1-0-1

Let's do a short recap on how Spring handles the @Transactional annotation:

It uses AoP (Aspect-oriented Programming) to detect the methods and classes annotated with @Transactional.

depending on whether we use Spring Aspects or AspectJ underneath, @Transactional will be detected on either Spring Beans with public methods only, or anywhere in the code.

It wraps all eligible methods with a Proxy, that starts the Transaction before the actual Method logic is called, commits it after the Method has finished and rolls it back in case of any Exception thrown from the Method.
When used in Integration Tests, it automatically rolls back the Test Method, after it has finished its work.

It is a very convenient piece of technology that cuts down boilerplate code needed for manual Transactions management. Its convenience however can become a very dangerous trap when we use it in Integration Tests with Hibernate on board. Why?

When the Scope ain't in scope

One of the most fundamental traits of a Database Transaction is it's Scope. Scope of a Transaction is what decides which pieces code fall under which Transaction. This is very important because changing the Scope of Transactions can have a profound impact on how the code behaves. It's especially visible when using Hibernate. Hibernate uses Transactions (and Transactional Entity Manger instances) to perform lazy loading. Let's set an example:

@Entity
@Table(name = "APP_USERS")
public class User {

    @Id
    @Column(name = "USER_ID", nullable = false)
    @GeneratedValue
    private Long id;

    @Column(name = "NAME", nullable = false, unique = true)
    private String name;

    @OneToMany(mappedBy = "user", fetch = FetchType.LAZY, cascade = CascadeType.ALL)
    private Set<Address> addresses = new HashSet<>();
}

When we fetch an instance of the User Entity, the addresses field will not be initialized. It will be an instance of PersistentSet - a Hibernate's proprietary Set implementation that will fetch the list of User's addresses from the database upon first call to any of the Set's methods. It acts as a Lazy Proxy. OK so where's the catch?

Hibernate's Lazy Loading works correctly only if we're in a scope of an active Database Transaction. As soon as we try to lazily load anything after the original Transaction has ended, we will be introduced to much beloved LazyInitializationException.(this may be changed in Hibernate by setting an appropriate property, but is not recommended) So by changing the scope of a Transaction, we may introduce Runtime Exceptions into our logic! Let's follow 2 examples:

Valid Transaction Scope:

        //Transaction has been started 
        transactionTemplate.executeWithoutResult(transactionStatus -> {
            //User is fetched from the Database
            User u = userService.getUserByName(USER_NAME);
            //Lazy-loaded properties are correctly fetched
            u.getAddresses().forEach(this::doSomethingWithAddress);
            //Transaction has been ended
        });

Invalid Transaction Scope:

        //Transaction has been started
        User u = transactionTemplate.execute(transactionStatus -> {
            //User is fetched from the Database
            return userService.getUserByName(USER_NAME);
            //Transaction has been ended
        });
        //Trying to lazy-load properties resulting in LazyInitializationException
        u.getAddresses().forEach(this::doSomethingWithAddress);

This is understandable and easy to follow, because we're explicitly using Spring's TransactionTemplate to manually manage Transaction Scope. This won't be as easy when using @Transactional 'magic'.

Prove it or it didn't happen

I've created an executable example on GitHub, that proves my point. It consists of 2 Test cases. Let's analyze them:

    @Test
    @Transactional
    public void webIntegrationTestWithNotActualProductionTransactionScope() throws Exception {
        //GIVEN: There is a User created
        createNewUser(getNewUser());

        //WHEN: we try to fetch the User by name, including its lazy-loaded properties
        MvcResult createdUserResponse = getUserByName(USER_NAME);

        //THEN: unlike in production, no exception is thrown when trying to access lazy-initialized properties
        assertEquals(200, createdUserResponse.getResponse().getStatus());
        UserDto createdUser = getUserFromResponse(createdUserResponse);
        assertEquals(USER_NAME, createdUser.getName());
        assertEquals(2, createdUser.getAddresses().size());
    }

The Test is annotated with @Transactional, enabling Spring's special magic

We're creating a new Instance of User in a Transaction:

    @Transactional
    @ResponseStatus(HttpStatus.CREATED)
    @PostMapping
    public void createUser(@RequestBody UserDto user) {
        userService.createUser(user);
    }

We're finding the User instance by name and transforming it into a DTO, using its lazy-loaded 'address' property:

    @GetMapping("/{name}")
    public UserDto getUserByName(@PathVariable("name") String name) {
        User user = userService.getUserByName(name).orElseThrow(() -> new RuntimeException("User not Found"));
        return new UserDto(user.getName(), user.getAddresses().stream().map(Address::getAddress).collect(Collectors.toList()));
    }

What would have happened in Production?

Those are 2 separate REST Calls we're testing. In such case the Transaction used to create the User would have been finished by the time the HTTP Response was returned by the Controller. Getting the User by Name would've been executed outside of the original Transaction. Transforming the User Entity into the UserDto would yield LazyInitializationException because we've been trying to lazy-load User's addresses field without a Transaction.

What actually happened in the Integration Test?

Everything worked correctly, the created User was returned by the getUserByName call. There was no Exception thrown whatsoever. We are confident that our code works correctly.

Conclusion?

This is a true disaster. We've created an Integration Test that yields false results, reassuring us that the actually broken code works and is ready to be shipped to Production.

Why is this happening?

When using @Transactional Integration Tests, Hibernate caches all Entities from all Transactions that were performed during the Test. Because User was once known to Hibernate alongside all its adresses (when we were creating the User), Hibernate has cached them and when we called the getUserByName method, Hibernate fetched the addresses Collection from Cache. Even forcing Spring to explicitly create new Transaction during the User's creation by setting

@Transactional(propagation = Propagation.REQUIRES_NEW)

won't help. It seems that adding @Transactional to an Integration Test makes Spring re-use the same Hibernate Session for every Transaction. That's only logical because Spring will want to perform a Rollback on it, after the Test is completed. A very dangerous side-effect is introduced though, as you can see.

Can we fix the Test?

Sure, let's just drop the @Transactional annotation from it and suddenly, our previously working code starts to show how it really works:

    @Test
    public void webIntegrationTestWithActualProductionTransactionScope() throws Exception {
        //GIVEN: There is a User created
        createNewUser(getNewUser());

        //WHEN: we try to fetch the User by name, including its lazy-loaded properties
        MvcResult createdUserResponse = getUserByName(USER_NAME);

        //THEN: just like in production, an exception is thrown when trying to access lazy-initialized properties
        assertEquals(500, createdUserResponse.getResponse().getStatus());
        assertEquals(LazyInitializationException.class, createdUserResponse.getResolvedException().getClass());
        
        //CLEANUP: we can't rely on Spring to roll back our test, we need to cleanup the Test Database manually
        TestDatabaseUtil.resetDatabase(dataSource);
    }

Now our Integration Test shows that in fact we're missing a @Transactional declaration from our getUserByName Controller method:

    @Transactional(readOnly = true)
    @GetMapping("/{name}")
    public UserDto getUserByName(@PathVariable("name") String name) {
        User user = userService.getUserByName(name).orElseThrow(() -> new RuntimeException("User not Found"));
        return new UserDto(user.getName(), user.getAddresses().stream().map(Address::getAddress).collect(Collectors.toList()));
    }

Adding that declaration will fix our production code. But how were we supposed to know that the code is broken, if our Integration Test worked?

Cleaner Cleaner Chicken Dinner

One thing we will need to take care of when not using @Transactional Integration Tests is after-test cleanup. When we drop the Spring's magic, there's no longer any automatic behind the scenes roll back after our Test is finished. We are testing real life Production appliance of our code, and that includes Commits being made into our Test Database.

This is a very small price to pay, considering the gains from writing Tests that actually do their job. Cleaning the Test Database (assuming we're using in-memory H2 database) can be done very easily, issuing couple of clever SQL Statements after each Test.

Conclusion

Misleading Tests are one of the worst thing that can happen in Software Development. Don't be tempted by a bit of convenience and NEVER use @Transactional Integration Tests. They have potential to turn your life into a Friday Evening Production Issue Extravaganza.

Thanks!

Search This Blog