Transfer Reinforcement Learning for 6G RAN Slicing

Machine learning techniques have been widely applied for network slicing. However, most existing works do not take advantage of the knowledge transfer capability in ML. We develop a transfer reinforcement learning solution for joint radio and cache resources allocation to serve 5G RAN slicing. We define a hierarchical architecture for the joint resources allocation. We propose two TRL algorithms: Q-value transfer reinforcement learning (QTRL) and action selection transfer reinforcement learning (ASTRL). In the proposed schemes, learner agents can utilize the expert agents' knowledge to improve their performance on target tasks. Selected publications are: 

Selected Publications:

[US22] H. Zhou, M. Erol-Kantarci, H. V. Poor, "Learning from Peers: Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G Network Slicing," https://arxiv.org/abs/2109.07999, Sep. 2021.

Optimization and Machine Learning Solutions for O-RAN Architecture

The Open Radio Access Network (O-RAN) architecture aims to define open interfaces in disaggregated vRANs such that equipment from multiple vendors can be interoperable. Along with that openness, the separation of user and control plane functions leads to improved key performance indicators (KPIs). We focus on contributing several solutions to these KPIs in our recent projects by using several optimization and machine learning techniques.

Selected Publications:

[Globecom21] S. Mollahasani, M. Erol-Kantarci, and R. Wilson, "Dynamic CU-DU Selection for Resource Allocation in O-RAN Using Actor-Critic Learning," in Proc. of IEEE GLOBECOM, 2021.

AI-Enabled Wireless Networks and Ultra-Reliable and Low Latency Communications

Ultra-Reliable and Low Latency Communications (uRLLC) is a key enabler for many use cases of 5G and 6G including autonomous vehicles, smart grid and mobile AR/VR. We use reinforcement learning and deep learning to address allocation of resources in wireless systems such that the complexity of the environment is learnt and optimal decisions are taken by the agents. Our most recent work in the area is listed below.

Selected Publications:

[TWM21] M. Elsayed, Melike Erol-Kantarci, H. Yanikomeroglu, "Transfer Reinforcement Learning for 5G-NR mm-Wave Networks," IEEE Transactions on Wireless Communications, vol. 20, no. 5, pp. 2838-2849, May 2021.

[ToSG20] M. Elsayed, Melike Erol-Kantarci, B. Kantarci, L. Wu, J. Li, "Low-latency Communications for Community Resilience Microgrids: A Reinforcement Learning Approach," in IEEE Transactions on Smart Grid, vol. 11, no. 2, pp. 1091-1099, 2020.

[VTSMag19] M. Elsayed, Melike Erol-Kantarci, "AI-enabled Future Wireless Networks: Challenges, Opportunities and Open Issues," in IEEE Vehicular Technology Magazine, Special Issue on 6G: What is Next?, Volume: 14 , Issue: 3 , pp. 70-77, Sept. 2019.

[ICC20] M. Elsayed, K. Shimotakahara, Melike Erol-Kantarci, "Machine Learning-based Inter-Beam Inter-Cell Interference Mitigation in mmWave," in  IEEE ICC, Dublin, June 2020.

[Globecom19] M. Elsayed, Melike Erol-Kantarci, "Reinforcement Learning-based Joint Power and Resource Allocation for URLLC in 5G," in IEEE Globecom, Waikoloa, HI, USA, December 2019.

Carrier Aggregation in 5G using Machine Learning

5G wireless networks aim to support different services with a variety of Quality of Service (QoS) requirements. To increase the perceived throughput, 5G (also LTE-advanced) employs Carrier Aggregation (CA) to expand the bandwidth by aggregating a number of Component Carriers (CCs) in the same or different frequency bands. However, a CC cannot be activated and deactivated frequently, since it increases the control channel overhead. Moreover, monitoring CCs in CA increases the user energy consumption even if they do not contain data. Thus, efficient CC activation/deactivation schemes have to consider different QoS parameters such as throughput, delay and energy consumption especially for bursty traffic models. We have machine learning solutions to improve CA in 5G.

Selected Publications:

[GC21] M. Elsayed, R. Joda, H. Abou-zeid, R. Atawia, A. B. Sediq, G. Boudreau, M. Erol-Kantarci, Reinforcement Learning Based Energy-Efficient Component Carrier Activation-Deactivation in 5G," IEEE Global Communications Conference, 2021.

[NetL21] R. Joda, M. Elsayed, H. Abou-zeid, R. Atawia, A. B. Sediq, G. Boudreau, M. Erol-Kantarci, Carrier Aggregation With Optimized UE Power Consumption in 5G," IEEE Networking Letters, vol. 3, no. 2, pp. 61-65, June 2021.

[ICC21] R. Joda, M. Elsayed, H. Abou-zeid, R. Atawia, A. B. Sediq, G. Boudreau, M. Erol-Kantarci, QoS-Aware Joint Component Carrier Selection and Resource Allocation for Carrier Aggregation in 5G, IEEE International Conference on Communications (ICC), 2021.

Load Balancing in Wireless Networks using Competitive Multi-Agent Machine Learning

In the context of load balancing, we use machine learning techniques to balance the load among base starions such that QoS of users are improved [MASS21]. We also propose a load balancing approach using novel Multi-Agent Deep Deterministic Policy Gradient with Adaptive Policies (MADDPG-AP) and ranked buffer scheme that considers throughput, resource block utilization and latency in the network [CCNC22]. 

Selected Publications:

[MASS21] P. E. Iturria-Rivera and M. Erol-Kantarci, "QoS-Aware Load Balancing in Wireless Networks using Clipped Double Q-Learning," accepted to IEEE 18th International Conference on Mobile Ad-Hoc and Smart Systems, October 2021.

Multi agent Reinforcement learning for Task Offloading in 5G-MEC Networks

In 5G and 6G networks, the fast growth of applications with immense computational needs, such as streaming video analysis, AR/VR mobile games, navigation of self-driving cars and drones call for higher computational capacity than a smart device can accommodate on board. Therefore, leveraging advanced wireless communications and multi-access edge computing (MEC) technologies to support AI-enabled applications become inevitable. In MEC, mobile devices can offload the computation-intensive tasks to a nearby MEC server and augment their computing capability. Our solution handles the problem of offloading a computation task and resource management in the radio side in the AI-enabled networks, using multi-agent reinforcement learning (MARL) concepts. Related papers are:

Selected Publications:

Reinforcement Learning Techniques for D2D-Enabled Microgrid Communications 

Device-to-device (D2D) mobile communications offer a low latency solution for the exchange of information in microgrids. We have investigated the usage of reinforcement learning techniques to help D2D-enabled grid entities self-allocate their cellular resources to meet the quality of service requirements of mission-critical smart grid applications [IEEE Access 2019]. To help do this, novel simulation techniques were developed to combine the heterogeneous characteristics of power and communication system dynamics [GlobalSIP 2019].

Selected Publications:

[SysJ21] K. Shimotakahara, M. Elsayed, K. Hinzer, Melike Erol-Kantarci, "Mobile Communications-Enabled Smart Grid Co-Simulator System Design," IEEE Systems Journal, vol. 15, no. 2, June 2021.

[GlobalSIP 2019] K. Shimotakahara, M. Elsayed, K. Hinzer and Melike Erol-Kantarci, "Integrated Power and Device-to-Device (D2D) Communications Simulator for Future Power Systems," IEEE Global Conference on Signal and Information Processing (GlobalSIP), Ottawa, ON, Canada, 2019.

[IEEE Access 2019] K. Shimotakahara, M. Elsayed, K. Hinzer and Melike Erol-Kantarci, "High-Reliability Multi-Agent Q-Learning-Based Scheduling for D2D Microgrid Communications," in IEEE Access, vol. 7, pp. 74412-74421, 2019.

Machine Learning-Enabled Energy Trading in Microgrid Communities

Microgrids are anticipated to be a significant ingredient of the smart grid as they simplify the integration of distributed renewable energy generation units, offer reliable service, provide faster restoration capabilities and support sustainable grid operation. In [NETMAG 2011], we have proposed to organize individual microgrids in a smart microgrid network which is a dynamically built platform allowing power transfers among microgrids in order to maximize the utilization of the renewable resources and minimize dependency to the utility grid. In [GBCOMM 2011], we extended this work to include cost-aware energy transactions. The basic idea of interconnected microgrids can be used to enhance resilience of the smart grid [ElectricityJ17]. In [PIMRC19], we extended our prior studies to form microgrid coalitions  to reduce power loss. We used machine learning to address the uncertainty.

Selected Publications:

[Energies21] M. Sadeghi, S. Mollahasani, M. Erol-Kantarci, "Cost-Optimized Microgrid Coalitions Using Bayesian Reinforcement Learning,"� Energies, November 2021.

[PIMRC19] M. Sadeghi, Melike Erol-Kantarci, "Power Loss Minimization in Microgrids Using Bayesian Reinforcement Learning with Coalition Formation," in Proc. of IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Istanbul, Turkey, September 2019.

[ElectricityJ17] L. Wu, J. Li, Melike Erol Kantarci, B. Kantarci, "An Integrated Reconfigurable Control and Self-Organizing Communication Framework for Community Resilience Microgrids," The Electricity Journal, Vol. 30, no. 4, pp. 27-34, May 2017.

[NETMAG 2011] M. Erol-Kantarci, B. Kantarci, H. T. Mouftah, `Reliable Overlay Topology Design for the Smart Microgrid Network," in  IEEE Network Magazine, Special issue on Communication Infrastructures for Smart Grid, September 2011.

[GBCOMM 2011] M. Erol-Kantarci, B. Kantarci, H. T. Mouftah, `Cost-Aware Smart Microgrid Network Design for a Sustainable Smart Grid,` in Proc. of IEEE GLOBECOM - Workshop on Smart Grid Communications and Networks, pp. 1223-1227, Houston, TX, USA, Dec. 5-9, 2011.

Edge Caching and Computing for Mobile AR/VR and Tactile Internet

As a result of increasing popularity of augmented reality and virtual reality (AR/ VR) applications, there are significant efforts to bring AR/VR to mobile users. Parallel to the advances in AR/VR technologies, tactile internet is gaining interest from the research community. Both AR/VR and tactile internet applications require massive computational capability, high communication bandwidth, and ultra-low latency that cannot be provided with the current wireless mobile networks. By 2020, long term evolution (LTE) networks will start to be replaced by fifth generation (5G) networks. Edge caching and mobile edge computing are among the potential 5G technologies that bring content and computing resources close to the users, reducing latency and load on the backhaul. 

Selected Publications:

[Multimedia19]S. Sukhmani, M. Sadeghi, Melike Erol-Kantarci, A. El-Saddik, "Edge Caching and Computing in 5G for Mobile AR/VR and Tactile Internet," in IEEE Multimedia, 2019.

[AdHocNets17] Melike Erol-Kantarci, S. Sukhmani, "Caching and Computing at the Edge for Mobile Augmented Reality and Virtual Reality in 5G," ADHOCNETS, Niagara Falls, 2017.

Energy-Efficient In-Network Content Caching and Streaming for 5G

Video upload and download are among the most popular applications for mobile users. However, those have high bandwidth demand and consume device power rapidly. Given the fact that popular contents are acquired by a large number of users at different times, it is practical to fetch those contents from the content source and store in-network. This is known as in-network caching. In the literature, latency and transport energy minimization aware in-network caching has been studied widely. Uplink power optimization which aims prolonging User Equipment (UE) lifetime have not been considered yet. To address this problem, we cache contents at wireless relays of a Heterogeneous Network (HetNet) and propose two Integer Linear Programming (ILP) models to optimize uplink power. The first ILP model, namely Minimize Uplink Power (minUP). MinUP aims to minimize uplink power when relays and their contents are known. The second ILP-based scheme is called Minimize Uplink Power and Relay Cost (minUPREC). MinUPREC jointly minimizes uplink energy and relay deployment cost. We have shown that minUPREC significantly minimizes the uplink energy incurred by UEs while minUP consumes lower caching.

Selected Publications:

[WCNC15] Melike Erol-Kantarci, "Content Caching in Small Cells with Optimized Uplink and Caching Power, " IEEE Wireless Communications and Networking Conference (WCNC), New Orleans, USA, 2015.

[ICNC15] Melike Erol-Kantarci, "Uplink Power Optimized In-Network Content Caching for HetNets, " International Conference on Computing, Networking and Communications (ICNC) - Workshop on Computing, Networking and Communications (CNC), Anaheim, CA, USA, 2015.

Energy Trading and Electric Vehicles

Plug-In Hybrid Electric Vehicles (PHEVs) have lower operating costs and lower emissions than the conventional vehicles which position them as promising candidates among the future transportation alternatives. On the other hand, they face the risk of endangering power grid stability due to their extremely high power demand. To address these challenges, in [CAMAD 2011], we have proposed an admission control scheme that runs on the distribution substation level using utility provided thresholds to provide access to the charging infrastructure. We extended this work to include differentiated admission decisions in [IEVC 2012]. In [ADJ 2014], energy transactions between delay tolerant loads and PEVs are studied. Our approach provides a convenient supply for loads via PEV batteries while addressing the mobility of vehicles and preserving fairness.

Selected Publications:

[AHJ 2014] Melike Erol-Kantarci, J. H. Sarker, H. T. Mouftah, `A Four-way-handshake Protocol for Energy Forwarding Networks in the Smart Grid`, Elsevier Ad Hoc Networks Journal, vol. 22, pp. 83-92, 2014.

[CAMAD 2011] M. Erol-Kantarci, J. Sarker, H. T. Mouftah, `Analysis of Plug-in Hybrid Electrical Vehicle Admission Control in the Smart Grid, ` International Workshop on Computer-Aided Modeling Analysis and Design of Communication Links and Networks (CAMAD), Kyoto, Japan, June 10-11, 2011.

[IEVC 2012] M. Erol-Kantarci, J. H. Sarker, H. T. Mouftah, `Quality of Service in Plug-in Electric Vehicle Charging Infrastructure, ` in Proc. of IEEE International Electric Vehicle Conference, Greenville, SC, March 4-8, 2012.

[IWCM 2011] M. Erol-Kantarci, H. T. Mouftah, `Management of PHEV Batteries in the Smart Grid: Towards a Cyber-Physical Power Infrastructure,` Workshop on Design, Modeling and Evaluation of Cyber Physical Systems (in IWCMC11), Istanbul, Turkey, July 5-8, 2011.

Battery-Free Sensors and Wireless Energy Transfer

WSNs are immensely deployed in various environments including industrial facilities, smart homes, health systems and other cyber-physical systems. However, conventional battery-powered WSNs have significant maintenance cost as a result of replacing depleted sensor batteries. Battery-free sensor technology offers a solution to the battery replacement and disposal burden by ripping batteries off the sensors, powering sensors with stored energy in super capacitors and topping the super capacitor from ambient energy. Wireless Energy Transfer (WET) promises charging wireless sensor networks, cell phones and on-body medical devices without the need of battery replacement nor plugging in to the mains. Radio Frequency WET (RF-WET) can be used for prolonging UE lifetime in a Heterogeneous wireless network (HetNet).

Selected Publications:

[PIMRC 2014] Melike Erol-Kantarci, H. T. Mouftah, `Challenges of Wireless Power Transfer for Prolonging User Equipment (UE) Lifetime in Wireless Networks,` in Proc. of IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) Workshop on Current Challenges for Wireless Power Transfer, Washington DC, USA, 2014.

[ISCC 2014] Melike Erol-Kantarci, H. T. Mouftah, `Radio-Frequency-Based Wireless Energy Transfer in LTE-A Heterogeneous Networks, ` in Proc. of IEEE Symposium on Computers and Communications (ISCC), Portugal, June 23-26, 2014.

[ISCC 2012] M. Erol-Kantarci, H. T. Mouftah, `'Mission-Aware Placement of RF-based Power Transmitters in Wireless Sensor Networks, ` in Proc. of IEEE Symposium on Computers and Communications (ISCC), Cappadocia, Turkey, July 1-4, 2012.

Sustainable and Reliable WSNs for Smart Grid Monitoring

For a reliable smart grid, accurate, robust monitoring and diagnosis tools are essential. Wireless Sensor Networks (WSNs) are promising candidates for monitoring the smart grid, given their capability to cover large geographic regions at low-cost. On the other hand, limited battery lifetime of the conventional WSNs create a performance bottleneck for the long-lasting smart grid monitoring tasks, especially considering that the sensor nodes may be deployed in hard to reach, harsh environments. In [TOSG 2011, VTC 2010], we have proposed a WSN-based residential demand management scheme which combines the Time Of Use (TOU) tariff from the smart meter, energy generation input from the renewable generators and provides a schedule for controllable appliances. [TOSG 2011] is listed in IEEE ComSoc Best Readings in Smart Grid Communications. This Best Readings is considered to be `a concise list of must-read books and articles` on communication networks and systems in the smart grid. In [AHNJ 2011a], we have investigated the opportunities and challenges of employing Wireless Multimedia Sensor and Actor Networks (WMSAN) in the smart grid. This paper has been listed by Elsevier Ad Hoc Networks journal as fourth most downloaded article of 2011 and ninth most cited article since publication. In [SYSJ 2013], we proposed delay-aware and fairness-aware schemes for prioritized packets, and in [TETC 2013] we extended this work to address further challenges in large-scale cyber physical systems, i.e. to enable prioritized medium access in IEEE 802.15.4 in cluster-tree topologies. In [WICOMM 2012], we have proposed the Sustainable wireless Rechargeable Sensor network (SuReSense) which employs mobile chargers that provide wireless power to multiple sensors from several landmark locations.

Selected Publications:

[SYSJ 2013] I. Al-Anbagi, Melike Erol-Kantarci, H. T. Mouftah, `Priority and Delay Aware Medium Access for Wireless Sensor Networks in the Smart Grid`, IEEE Systems Journal, vol.8, no.2, pp. 608 - 618, 2014.

[TETC 2013] I. Al-Anbagi, Melike Erol-Kantarci, H. T. Mouftah, `A Reliable IEEE 802.15.4 Model for Cyber Physical Power Grid Monitoring Systems,` to appear in IEEE Transactions on Emerging Topics in Computing, 2013.

[TOSG 2011] M. Erol-Kantarci, H. T. Mouftah, `Wireless Sensor Networks for Cost-Efficient Residential Energy Management in the Smart Grid`, IEEE Transactions on Smart Grid, vol.2, no.2, pp.314-325, June 2011.

[VTC 2010]M. Erol-Kantarci, H. T. Mouftah, `TOU-Aware Energy Management and Wireless Sensor Networks for Reducing Peak Load in Smart Grids,` Green Wireless Communications and Networks Workshop in IEEE VTC2010-Fall, Ottawa, ON, Canada, September 6-9, 2010.

[AHNJ 2011a] M. Erol-Kantarci, H. T. Mouftah, `Wireless Multimedia Sensor and Actor Networks for the Next-Generation Power Grid,` Elsevier Ad Hoc Networks Journal, vol.9 no.4, 2011, pp. 542-511.

[WICOMM 2012] M. Erol-Kantarci, H. T. Mouftah, `SuReSense: Sustainable Wireless Rechargeable Sensor Networks for the Smart Grid,` IEEE Wireless Communications, vol.19, no. 3, pp. 30-36, June 2012.

Smart Grid Security and Forensic Science

Smart grid forensic science is a newly flourishing research area that is tightly coupled with cyber and physical security of the smart grid. Post-mortem analysis of a power system after a cyber attack or after a natural disaster generally provides the most accurate comprehension of the real world vulnerabilities of the system and helps to protect the grid against similar attacks in the future as well as avoiding failures under disasters. In [COMMAG 2013], we have introduced the emerging application areas of smart grid forensic science, discussed the challenges and outlined the open issues in the topic. This paper serves as a roadmap for future smart grid forensic studies. This paper has been listed in IEEE ComSoc Technology News (CTN) in March 2013. IEEE CTN each month selects `interesting, timely, and newsworthy papers and articles from IEEE publications`.

Selected Publications:

[COMMAG 2013] M. Erol-Kantarci, H. T. Mouftah, `Smart Grid Forensic Science: Applications, Challenges and Open Issues, ` in IEEE Communications Magazine, January 2013.

Underwater Sensor Networks and Localization

Underwater research aims at providing new opportunities to explore the oceans in order to improve our understanding of the ocean habitat and climate change, develop early warning systems for earthquakes and tsunamis, monitor oil rigs, as well as to enhance the underwater warfare capabilities of the naval forces. Underwater Acoustic Sensor Networks (UASNs) are ideal for the harsh ocean environment where unmanned operation is desired for most of the missions. We proposed several localization algorithms and a mobility model for underwater sensor networks.

Selected Publications:

[WASA 2007] Melike Erol-Kantarci, L. Vieira, M. Gerla, `AUV-Aided Localization for Underwater Sensor Networks,` WASA special track on Underwater Sensor Networks, August 1-3, 2007, Chicago, IL.

[WUWNet 2007] M. Erol-Kantarci, L. Vieira, M. Gerla, `Localization with Dive`N`Rise (DNR) Beacons for Underwater Acoustic Sensor Networks,` The Second ACM International Workshop on UnderWater Networks WUWNet (in conjunction with ACM MobiCom 2007), September 14 2007, Montreal, Quebec, Canada.

[AHNJ 2011b] M. Erol-Kantarci, S. Oktug, L. Vieira, M. Gerla, `Performance Evaluation of Distributed Localization Techniques for Mobile Underwater Acoustic Sensor Networks`, Elsevier Ad Hoc Networks Journal, vol.9 no.1, 2011, pp. 61-72.

[INFOCOM 2008] A. Caruso, F. Paparella, L. Vieira, M. Erol-Kantarci, M. Gerla, `The Meandering Current Mobility Model and its Impact on Underwater Mobile Sensor Networks,` IEEE INFOCOM, 13-19 April 2008, Phoenix, AZ.

[COMMAG 2010] M. Erol-Kantarci, H. T. Mouftah, S. Oktug, `Localization Techniques for Underwater Acoustic Sensor Networks,` IEEE Communications Magazine, vol. 48, no.12, 2010.

[COMMST 2011] M. Erol-Kantarci, H. T. Mouftah, S. Oktug, `A Survey of Architectures and Localization Techniques for Underwater Acoustic Sensor Networks,` accepted for publication, IEEE Communications Surveys and Tutorials, 2011.

[AICT 2009] S. Kuruoglu, M. Erol-Kantarci, S. Oktug, `Localization in Wireless Sensor Networks with Range Measurement Errors,` The Fifth Advanced International Conference on Telecommunications (AICT), May 24-28, 2009, Venice, Italy. (Best Paper Award)

[GLOBECOM 2009] S. Kuruoglu, M. Erol-Kantarci, S. Oktug, `Three Dimensional Localization in Wireless Sensor Networks using the Adapted Multi-Lateration Technique Considering Range Measurement Errors`, Design and Development Forum in IEEE GLOBECOM, Hawaii, 2009.

Internet Traffic Analysis and Self-similarity Modeling

Internet traffic is characterized by its long-range dependence and self-similarity which can lead to particularly large queuing delays and cause congestions to spread over a long time. The degree of self-similarity is measured by the Hurst parameter and several methods are available in the literature for estimating this parameter. In [TOIM 2011], we derived analytical expressions for widely used estimation methods in various domains for time series with periodic anomalies and demonstrated through simulations that periodicity-based anomalies negatively impact the Hurst parameter estimation methods.

Selected Publications:

[LNCS 2006] M. Erol-Kantarci, T. Akgul, S. Oktug, S. Baykut, `On the Use of Principle Component Analysis for the Hurst Parameter Estimation of Long-Range Dependent Network Traffic`, Lecture Notes on Computer Science, Vol.4263, pp. 464-473, 2006.

[TOIM 2011] T. Akgul, S. Baykut, M. Erol-Kantarci, S. Oktug, `Periodicity-based Anomalies in Self-similar Network Traffic Flow Measurements, ` IEEE Transactions on Instrumentation and Measurement, vol.60, no.4, pp.1358-1366, April 2011.