IN5020: Distributed Systems
Master's course on how distributed systems are built: middleware, replication, consensus and peer-to-peer lookup. Three group assignments: a load-balanced Java RMI statistics service with caching and Docker deployment, a replicated bank account kept consistent with a totally ordered broadcast, and a Chord peer-to-peer ring with finger tables and logarithmic lookups.
IN5020: Distributed Systems at the University of Oslo, autumn 2026. The course covers the principles and practice of building distributed systems: fundamental system models, distributed middleware (object-based middleware and component architectures), distributed algorithms for consensus and related problems, and newer challenges from cloud, multimedia and IoT. Each assignment is a group project in its own repository.
The three assignments at a glance
| Assignment | Topic | Status | |
|---|---|---|---|
| A1 | Java RMI International Statistics Service | Middleware, load balancing, caching | Completed |
| A2 | Replicated Bank Account | Replication, totally ordered broadcast | Completed |
| A3 | Chord Peer-to-Peer Lookup | Distributed hash tables, routing | Completed (private repository) |
Assignment 1: Java RMI International Statistics Service
Status: completed. Group of four. My part: the client and the load-balancing proxy, and the overall request flow between them.
A distributed statistics service over a dataset of about 140,000 cities. A client sends queries (“what is the population of Norway?”) through a load-balancing proxy to one of several zone servers, with optional caching on the server side or the client side. Built with plain Java RMI and no external libraries.
System design

The diagram shows one query’s path through the system. Every box inside the dashed frame runs as its own Docker container.
- On startup each zone server registers once with the proxy (
registerServer(...)) and is given a zone number. - For every query the client asks the proxy which server to use (
getServerForZone(zone)). The proxy looks up its zone-to-server map, checks for overload and calculates distance, then returns the server’s host, port, zone and simulated network delay. - The client then calls that server directly over RMI (for example
getPopulationofCountry(...)). The request goes into the server’s FIFO queue and is answered by a single execution thread from the ~140,000-city dataset. - The server sends back a
QueryResultwith the answer and timing data.
The dashed yellow boxes are the optional caches: the client cache (45 entries) is checked before the proxy is contacted at all, and the server cache (150 entries) is checked before the dataset is searched. Zones 2 to 5 are identical copies of zone 1.
Architecture
- Proxy (load balancer): servers register with it on startup and get a zone number. For every query the client asks the proxy which server should handle it.
- Zone servers: each has its own thread-safe request queue drained by a single execution thread, so requests are processed in strict FIFO order.
- Client: starts each query on its own thread every T ms (50 or 20), so queries run concurrently and put realistic load on the system.
Load-balancing logic
The proxy sends a query to the server in the requested zone unless it is overloaded (18 or more waiting requests). Otherwise it picks the least-loaded neighbour zone, breaking ties by clockwise distance, and falls back to the original zone if every server is overloaded. Choosing a zone further away adds simulated network delay (30 ms per step), so the design had to weigh queue time against distance.
To keep load information fresh without slowing clients down, the proxy refreshes a server’s queue length every 18th time it hands that server out, on a background thread, so the bookkeeping call never makes a client wait.
Caching
A shared generic cache supports two eviction strategies, FIFO and least-recently-used, chosen at startup. The server cache holds 150 entries and the client cache 45. With caching on the server, about 35% of queries were served from cache with execution time under 1 ms. With caching on the client, around 530 of 3,157 queries never touched the network at all.
Deployment and testing
- One two-stage Docker image (Maven build, slim JRE runtime) runs as proxy, server or client depending on the command. Docker Compose starts the proxy and five zone servers with health checks, so the client only starts once everything it depends on is ready.
- A test script runs all 10 configurations (naive, server cache and client cache, each with both eviction strategies, at T=50 and T=20) against the full 3,166-query input, and a Python script plots turnaround time and queue length per zone (60 graphs in total).
Assignment 2: Replicated Bank Account
Status: completed. Group of three. My part: the command handling, exactly-once execution, the switch between the naive and correct getSyncedBalance, logging, and the fix that stops a frozen replica from blocking the server.
One bank account that exists as several identical copies at the same time. Each copy (a replica) runs in its own bank server process, possibly on different machines. Users can send commands to any replica: deposit money, add interest, ask for the balance. Wherever a command is given, every replica must end up with exactly the same balance, even while replicas join, leave or crash. Built with plain Java RMI.
All the diagrams below were drawn by our group while working on the assignment, as part of a longer explanation document. Each one is meant to make sense on its own, and the text around it is there to help you get into it quickly. Click any diagram to see it in full size.
At a glance
- What: a replicated bank account. N bank servers each hold a full copy, and one Message Delivery server decides the order of every change.
- Core idea: same start state + same transactions + same order = same end state. The replicas never compare balances. They only have to agree on the order.
- Main mechanisms: a sequencer for total order, ACK before the next message, resend after 2 s, FAIL after 5 s, state transfer for late joiners, and duplicate detection so every transaction runs exactly once.
- Built with: Java 11, Java RMI in both directions, no external libraries.
How to read the diagrams
The same colours are used in every diagram.
| Element | Meaning |
|---|---|
| Blue | the Message Delivery server |
| Green | a bank server (replica) |
| Yellow | data inside a replica: balances, lists, counters |
| Purple | a thread |
| Red | a failure, or the naive version |
| Grey | input, commands and notes |
| Green arrow | bank → server (broadcast, ACK) |
| Blue arrow | server → bank (delivery, state transfer) |
| Grey arrow | membership: join, leave, membershipChanged |
The big picture
This is the most important diagram. Everything else below zooms in on a part of it.
One transaction, five calls
- A command like
deposit NOK 100becomes a transaction in the replica’soutstanding_collection(the inbox). The balance does not change yet. Every 10 seconds the broadcast thread sends the inbox to the server withbroadcastTransactions. - The server gives the list a message id (0, 1, 2, …), queues it, and sends it to every replica in parallel with
receiveTransaction. - Each replica executes the transactions in that order and calls
ack. The server only sends the next message when everyone has answered. join,leaveandmembershipChangedkeep everyone’s member list up to date.- A replica that joins late gets a copy of the state with
updateStatebefore it receives any new messages.
RMI goes both ways here. The bank passes itself as an argument to join, and because it is a UnicastRemoteObject the server gets a stub back to it and can push messages to the bank instead of the bank polling.
Why the order is everything
It is easy to think it is enough that every replica gets every transaction. It is not.
It does not matter which order the server picks, only that everyone uses the same one. That makes the account a replicated state machine, and the server a sequencer: the simplest way to get a totally ordered broadcast.
Life of one transaction
The detail that matters: bank A does not book its own transaction when it is created. It waits, like everyone else, until the transaction comes back from the server. That is the only way A gets it in the same position in the order as B and C.
The delivery loop: ACK, resend and FAIL
Why wait for every ACK before sending the next message?
- Same order everywhere. No replica can get message 8 before everyone has confirmed message 7.
- Nobody falls far behind. The difference between replicas is at most one message.
- A safe point for state transfer. Between two messages all replicas are in exactly the same state, so that is where a new replica can get its copy.
The cost: the group moves at the speed of the slowest replica, and in the worst case stalls for 5 seconds before a dead one is removed. An ACK means “received and executed”, not just “received”. A replica that leaves with exit calls leave() and is removed at once, so a planned leave never waits for the timeout.
Inside a replica
Exactly once. Duplicates can happen in two ways: the broadcast thread resends the whole inbox if the last send has not come back yet, and the server resends a message after 2 s without ACK. Every replica keeps the ids of everything it has executed in executedIds and skips anything it has seen before, but still sends the ACK. Because everyone sees the messages in the same order, everyone skips the same duplicates.
Why not just check executed_list? Because cleanHistory empties it. After a cleanHistory the replica would forget what it had executed, and a duplicate would run twice. executedIds is never cleared.
Membership: startup, late join, leave and crash
- Startup: each replica waits until N members have joined. The first replicas get no state transfer, so if A sent a transaction before C joined, C would miss it forever.
- Late join: the server waits until the current message is ACKed by everyone, takes a snapshot (balances, history,
order_counter,executedIds) from an existing replica, and gives it to the new one before the next message.executedIdshas to be included, otherwise the new replica could execute a duplicate the others skip. - Leave: send the rest of the inbox, call
leave(), exit. - Crash: detected by the missing ACK. Transactions that were still in the crashed replica’s inbox are lost, but the customer never got them confirmed either.
Threads and locks
Without the lock, receiveTransaction could remove a transaction from the inbox while the broadcast thread is copying it. getSyncedBalance waits for its answer without holding the lock, because otherwise receiveTransaction could never get in to deliver it: a deadlock.
Running and testing
What we checked:
- every replica that lived to the end has the same final balance
- every replica has the same transaction at every order number (the late joiners’ lists are the tail of the others’)
- naive and correct mode give different getSyncedBalance answers, and the difference can be explained
- a replica killed in the middle of a run is marked FAIL after 5 s, and the rest carry on
In the final runs all six replicas executed the transactions in the same order and ended on the same balance, in both modes.
Design choices worth defending
| Choice | Why |
|---|---|
executedIds in addition to executed_list | The duplicate check must still work after cleanHistory |
| ACK even when every transaction was a duplicate | Otherwise the server would wait forever, or wrongly declare the bank dead |
| Wait for N replicas before starting | The first replicas get no state transfer, so nobody can send before everyone is in |
| getSyncedBalance does not block the command loop | The only way to observe the naive/correct difference and keep the command file’s timing |
Markers get their own ids (sync-0, sync-1, …) | outstanding_counter only numbers deposit and addInterest, so checkTxStatus N hits the right transaction |
leave() on exit | A planned leave should not wait 5 s and end as FAIL |
| Server name = PID + random suffix | The name is part of every transaction id, and a PID alone is only unique on one machine |
| Timeouts on calls the server makes while holding a lock | A frozen replica must not be able to block the server |
Questions worth being able to answer
- Why not let the banks send to each other directly? Messages could arrive in different orders at different replicas. Without a central server you need a distributed ordering algorithm, for example with Lamport clocks. A sequencer is the simplest way to get total order.
- Isn’t the server a single point of failure? Yes, and a bottleneck, since all traffic goes through it. The assignment does not ask us to solve that. In a real system the sequencer would itself be replicated with a consensus protocol like Raft or Paxos.
- What does sequential consistency mean here? The result is as if all operations ran in one order, where each replica’s own operations appear in the order it issued them. The server makes the single order, and the inbox is sent as a list, so each replica’s own order is kept.
- Is getQuickBalance wrong? No, it is deliberately weaker. It is fast and needs no communication, but can miss transactions that are not ordered yet.
- What if a replica is just slow, not dead? If it does not answer within 5 s it is removed anyway. That is the classic trade-off with timeout-based failure detection: you cannot tell a slow process from a dead one.
- Why send transactions and not the new balance? That is what makes it a replicated state machine, and it gives every replica the same history and order numbers. Sending balances would mean someone has to decide which one wins when two changes happen at once.
- Why are the replicas equal when the timestamps in the logs differ? The timestamps are local: when each replica executed the transaction. What has to match is the order and the order number, and it does.
- Why is the inbox only sent every 10 seconds? The assignment says so. It batches transactions into fewer messages and ACK rounds, at the cost of up to 10 s of extra delay.
Assignment 3: Chord Peer-to-Peer Lookup
Status: completed. Group of three. My part: the repository setup, and correcting the finger table intervals and the lookup so they follow the assignment text exactly and every lookup in the three configurations is correct.
An implementation of the core of the Chord protocol in a provided simulator: hashing nodes onto an m-bit identifier ring, building each node’s finger table, and routing key lookups across the ring in a logarithmic number of hops. Results are measured as the average hop count for networks of 10, 100 and 1,000 nodes. Built with plain Java 11 and shipped as a runnable jar.
At a glance
- What: a Chord ring of N nodes where every key is stored at its successor, the first node at or after the key’s position on the ring.
- Core idea: each node knows only m other nodes through its finger table, but each hop roughly halves the remaining distance to the key, so a lookup takes about log N hops instead of N.
- Our three parts: build the overlay network, build the finger tables, and look up a key.
- Results: average hop counts of 2.8 (10 nodes, m = 10), 3.95 (100 nodes, m = 20) and 5.89 (1,000 nodes, m = 20), with no wrong lookups in any run.
One run of the simulator
The diagram was drawn by our group as part of a longer explanation document, so its labels are in Norwegian. The list below the diagram walks through it in English. Click the diagram to see it in full size.
- Simulator.java reads the node count and m, creates Node 1 … Node N, picks the number of keys and starts the run.
- assignKeys() hashes every “key i” and stores its index at the successor node.
- buildProtocol() hands the network to our protocol and calls steps 3 and 4.
- buildOverlayNetwork() (part 1) places every node on the ring at ch.hash(name), in a sorted TreeMap, with its successor as its single neighbour. A hash collision stops the run with an error.
- buildFingerTable() (part 2) gives each node m rows. Row i has start = (n + 2^(i-1)) mod 2^m, an interval up to the next start, and as successor the first node at or after start. The intervals cover the whole ring exactly once, including intervals that wrap around 0.
- printNetworkToFile() writes the ring and every node’s finger table to
network_N_m_.txt. - doLookUpAndGenerateOutputFiles() calls lookUp(keyIndex) (part 3) for every key. The lookup always starts at the node with the lowest index and follows the finger table row whose interval contains the key, adding each visited node to the route, until it reaches the node that stores the key. Every answer is checked, and one line per key plus the average hop count is written to
configuration_N_m_.txt.
Underneath, ConsistentHashing maps any text to a position on the ring with MD5 mod 2^m, and Network/Node hold each node’s name, index, neighbour, finger table and stored keys.
Repositories
| Assignment | Repository |
|---|---|
| A1: Java RMI statistics service | IN5020-A1-G1-RMI |
| A2: Replicated bank account | IN5020-A2-G1-RBA |
| A3: Chord protocol | IN5020-A3-G1-CP (private while the course runs) |