Performance measurements for inter-process communication

This lab deals with performance. Your work consists of evaluating, measuring and comparing the performance of several versions of programs, all following the the same general design pattern.

High-level system description

We consider a simple application where a producer sends data to a consumer. The data is grouped into "messages" that are sent separately. Each message consists of an array of Employee object instances (of length messageSize) and the number of messages transmitted is called repetitions. Your task is to measure the time it takes to transmit N Employee instances from the producer to the consumer for the different versions of programs and different execution environments. Note: N = messageSize * repetitions. It is assumed that an employee has the following attributes: name, address, age, and salary.

All versions of programs have the following architecture: There is a producer thread and a consumer thread. The producer has a large number of dummy employees. Some of these are placed within the array of the message. Then he "sends" the message to the consumer. When the consumer "receives" a message, it consumes the message (that is, it calculates the total salary of the employees received) and sends an acknowledgement to the consumer. The consumer waits for the reception of the acknowledgement before preparing the next message.

The system also contains a module called Communication. It is this module that provides methods for sending and receiving messages and acknowledgements. The different versions of the program differ in the way the Communication module is implemented. The following versions are foreseen:

1. CommSimple (simple communication) : a single Employee[] data structure is contained in the communication module. The data is directly placed into this data structure by the producer and is read by the consumer directly from the same structure without copying. Two semaphors are used to control the timing: the sent semaphore allows the receiver to consume the message, and the received semaphore allows the producer to prepare the next message content.
2. CommCoded (coded communication) : Note that for all versions, the producer and consumer see the same interface of the Communication module. In this version, the message submitted by the producer is encoded as a text string, in a single Java String. This string is decoded when the consumer receives the message, such that the receiver gets the orginal sequence of employees prepared by the producer. It can be expected that the execution time of this version is slower because of the encoding and decoding work and the associated copying.
3. CommByFile (communication through a shared file) : This version also encodes the sequence of emplyees, but instead of placing the result into a variable of type String, this version writes the string of characters into a file which is closed when it contains a complete message. The same file is read when the consumer receives the message.
4. CommByTCP (communication through the Internet using a TCP connection) : In this version, each encoded message is sent as a character string through a TCP connection to the consumer. There are two Java (main) programs CommByTCP_sender and CommByTCP_receiver. The sender program uses a Socket and establishes first a connection to the receiver program. The latter uses a Socket on which it starts listening. Once the connection is established, the programs function much like the versions 2 and 3. Note, however, that the synchronization between the producer and the consumer (provided by the Communication module), in this version does not use semaphores but message transmission. When the consumer executes the sendAck() method, a simple acknowlegement character sequence will be sent to the producer.
5. Two variants of CommCoded were originally planned as well: CommCoded-JavaSerialization and CommCoded-XML. However the TA did not have enough time to prepare the Java programs for these examples. The purpose of these two variants would have been to compare the performance of XML and Java-Serialization encoding with the straightforward String encoding implemented in version 2 above and used in version 4.

Here are the Java programs. Note that for each program version, there is one version of the Communication module (called CommXXX) and one version of the Java main program (called Producer_Consumer_System). You can change the length of the Employee[] array and the number of messages by changing the constants in the Producer_Consumer_System Class. The producer thread expects at the beginning that the user enters a Carriage Return to begin its operation. It calculates the total time required for its operations.

Note that the classes CommByFile, CommByTCP_sender and CommByTCP_receiver are not given. You should program these classes by reusing parts of the CommCoded class. Here are some suggestions.

Here are some implementation notes prepared by Salman Hooshmand (TA in 2015).

Your tasks

1. Execute the current version of the class Producer_Consumer_System of the program for 10 subsequent executions of the program and note the execution times obtained (using the CommSimple class). Write down the 10 execution times obtained.
• Enter the ten numbers into an Excel sheet and calculate the average measured execution time and the standard deviation of the results obtained.
• What is the expected expected (statistical) error of the average obtained ? (That is, how much do you expect the average to vary when you do again several sets of 10 measurements ? ) - Use the well-known formula from statistics (see Wikipedia - "Standard error of mean" - see also Standard Deviation) to calculate the expected error of the measured average value.
2. Put some other load on the computer - other heavy processing (e.g. displaying a video, opening a large file under Word). Use the Windows utility Performance (under the Administrative Tool control panel) to see what the level of usage of the main computer resources is in the computer idle state, during the execution of the Java program alone, during the execution of the heavy processing alone, and when the Java program is executed during heavy processing. Measure the execution time of our Java program and discuss the obtained results by taking into account the Window performance indications observed. You may also introduce several instances of DummyThreads into the Java main program. - How does the performance vary as you introduce these changes ? - Explain your findings.
3. Use the CommCoded version of the Java program and repeat the measurements of point (1). Calculate the average execution time and the expected error of this value. Compare the execution time with the CommSimple version and try to explain your findings.
• For this purpose, it is sufficient to change some comments lines in the Producer_Consumer_System class.
• You may reduce the number of repetitions by a factor 10.
• Question: Could you think about another approach to writing the coding and decoding routines in Java in order to make the execution of the program more efficient ?
4. Complete the CommByFile version of the Java program and repeat the measurements of point(2). Compare the execution time with the CommCoded version and try to explain your findings. What is the performance of the file reading and writing compared with the performance of the data coding and decoding ?
5. Try different sizes of messages. Keep the same total numbers of employees to be transferred while changing the message size and the number of messages, accordingly. Do the execution time measurement for many values of message size and make a plot of the execution time as a function of number of the messages transferred. Explain your results.
6. Complete the CommByTCP version of the Java program (there are two versions of the communication model, one for the sender who establishes a TCP connection to the receiver, one for the receiver that accepts the connection) and repeat the measurements of point (3) by running the sender and receiver threads in the same JVM.. (You may also execute the sender part and the receiver part on two different computers if you like). Again, measure the execution time and experiment with different message sizes (as under point 6 5). Explain your results. Note: You can find a tutorial about using Sockets in Java at Sun (Oracle). Here is an example of two programs, client and server. The client program makes a TCP connection to a server, and the server returns the text received from the client back to the client.

Documentation to be shown to the TA:

• Your notes concerning your work done for points 1 through 6 above, and the measurements results obtained.
• Summary of your conclusions from this exercise.

Last update: April 1, 2016