Details of Research Outputs

TitleOSC: An Online Self-Configuring Big Data Framework for Optimization of QoS (TC-2020-02-0128.R1)
Author (Name in English or Pinyin)
Bei, Z.1; Kim, N.S.2; Hwang, K.3; Yu, Z.4
Date Issued2021
Source PublicationIEEE Transactions on Computers
Education discipline科技类
Published range国外学术期刊
[1] K. S. Beyer et al., "Jaql: A scripting language for large scale semistructured data analysis," Proc. VLDB Endowment, vol. 4, pp. 1272-1283, Sep. 2011.
[2] H. Herodotou and S. Babu, "Profiling, what-if analysis, and costbased optimization of MapReduce programs," Proc. VLDB Endowment, vol. 4, no. 11, pp. 1111-1122, 2011.
[3] T. White, Hadoop: The Definitive Guide. Newton, MA, USA: O'Reilly Media, Inc., 2012.
[4] Z. Bei et al., "RFHOC: A random-forest approach to auto-tuning hadoop's configuration," IEEE Trans. Parallel Distrib. Syst., vol. 27, no. 5, pp. 1470-1483, May 2016.
[5] H. Herodotou et al., "Starfish: A self-tuning system for big data analytics," in Proc. Biennial Int. Conf. Innovative Data Syst. Res., 2011, pp. 261-272.
[6] S. Babu, "Towards automatic optimization of MapReduce programs," in Proc. ACM Symp. Cloud Comput., 2010, pp. 137-142.
[7] A. E. Gencer,D. Bindel, E. G. Sirer, and R. van Renesse, "Configuring distributed computations using response surfaces," in Proc. Annu. ACM/IFIP/USENIXMiddleware Conf., 2015, pp. 235-246.
[8] Z. Yu, Z. Bei, and X. Qian, "Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing," in Proc. ACM Conf. Architectural Support Program. Lang. Operating Syst., 2018, pp. 564-577.
[9] G. Liao, K. Datta, and T. L. Willke, "Gunther: Search-based autotuning of MapReduce," in Proc. Eur. Conf. Parallel Process., 2013, pp. 406-419.
[10] H. Du, P. Han, Q. Xiang, and S. Huang, "MonkeyKing: Adaptive parameter tuning on big data platforms with deep reinforcement learning," BigData, vol. 8, no. 4, pp. 270-290, 2020.
[11] T.-Y. Mu, A. AI-Fuqaha, and K. Salah, "Automating the configuration of MapReduce: A reinforcement learning scheme," IEEE Trans. Syst.,Man, Cybern., Syst., vol. 50, no. 11, pp. 4183-4196,Nov. 2020.
[12] M. Li et al., "MRONLINE: MapReduce online performance tuning," in Proc. 23rd Int. Symp. High-Perform. Parallel Distrib. Comput., 2014, pp. 165-176.
[13] H. Herodotou, Y. Chen, and J. Lu, "A survey on automatic parameter tuning for big data processing systems," ACM Comput. Surv., vol. 53, no. 2, pp. 43:1-43:37, Apr. 2020.
[14] S. Memeti, S. Pllana, A. Binotto, J. Kolodziej, and I. Brandic, "Using meta-heuristics and machine learning for software optimization of parallel computing systems: A systematic literature review," Computing, vol. 101, pp. 893-936, Aug. 2019.
[15] V. Ilyukha, "10 best big data tools for 2020," 2020. [Online]. Available:
[16] D. Cheng, J. Rao, Y. Guo, and X. Zhou, "Improving MapReduce performance in heterogeneous environements with adaptive task tuning," in Proc. Annu. ACM/IFIP/USENIX Middleware Conf., 2014, pp. 97-108.
[17] D. Cheng, J. Rao, Y. Guo, C. Jiang, and X. Zhou, "Improving performance of heterogeneous MapReduce clusters with adaptive task tuning," IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 3, pp. 774-786, Mar. 2017.
[18] X. Ding, Y. Liu, and D. Qian, "JellyFish: Online performance tuning with adaptive configuration and elastic container in hadoop yarn," in Proc. IEEE 21st Int. Conf. Parallel Distrib. Syst., 2015, pp. 831-836.
[19] S. Kumar, S. Padakandla, L. Chandrashekar, P. Parihar, K. Gopinath, and S. Bhatnagar, "Scalable performance tuning of hadoop MapReduce: A noisy gradient approach," in Proc. IEEE 10th Int. Conf. Cloud Comput., 2017, pp. 375-382.
[20] N. Yigitbasi, T. Willke, G. Liao, and D. Epema, "Towards machine learning-based auto-tuning ofMapReduce," in Proc. IEEE Int. Symp. Model.Anal. Simul. Comput. Telecommun. Syst., 2013, pp. 11-20.
[21] M. W. ur Rahman, N. S. Islam, X. Lu, D. Shankar, and D. K. Panda, "Performancemodeling forRDMA-enhanced hadoopMapReduce," in Proc. 43rd Int. Conf. Parallel Process., 2014, pp. 50-59.
[22] L. Bao, X. Liu, and W. Chen, "Learning-based automatic parameter tuning for big data analytics frameworks," in Proc. IEEE Int. Conf. Big Data, 2018, pp. 181-190.
[23] D. Wu and A. Gokhale, "A self-tuning system based on application profiling and performance analysis for optimizing hadoop MapReduce cluster configuration," in Proc. 20th Annu. Int. Conf. High Perform. Comput., 2013, pp. 89-98.
[24] G. J. Lee and J. A. B. Fortes, "Hadoop performance self-tuning using a fuzzy-prediction approach," in Proc. 13th IEEE Int. Conf. Auton. Comput., 2016, pp. 55-64.
[25] R. Zhang, M. Li, and D. Hildebrand, "Finding the big data sweet spot: Towards automatically recommending configurations for hadoop clusters on docker containers," in Proc. IEEE Int. Conf. Cloud Eng., 2015, pp. 365-368.
[26] X. Hua, M. C. Huang, and P. Liu, "Hadoop configuration tuning with ensemble modeling and metaheuristic optimization," IEEE Access, vol. 6, pp. 44 161-44 174, Aug. 2018.
[27] J. L. Berral, N. Poggi, D. Carrera, A. Call, R. Reinauer, and D. Green, "ALOJA-ML: A framework for automating characterization and knowledge discovery in hadoop deployments," in Proc. 21th ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2015, pp. 1701-1710.
[28] M. Bilal and M. Canini, "Towards automatic parameter tuning of stream processing systems," in Proc. ACM Symp. Cloud Comput., 2017, pp. 189-200.
[29] M. Li, Z. Liu, X. Shi, and H. Jin, "ATCS: Auto-tuning configurations of big data frameworks based on generative adversarial nets," IEEE Access, vol. 8, pp. 50 485-50 496, Mar. 2020.
[30] Y. Zhu, J. Liu, M. Guo, Y. Bao, and W. Ma, "BestConfig: Tapping the performance potential of systems via automatic configuration tuning," in Proc. ACM Symp. Cloud Comput., 2017, pp. 338-350.
[31] O. Alipourfard, H. H. Liu, J. Chen, S. V. M. Yu, and M. Zhang, "CherryPick: Adaptively unearthing the best cloud configurations for big data analytics," in Proc. USENIX Symp. Netw. Syst. Des. Implementation, 2017, pp. 469-482.
[32] M. W. ur Rahman, N. S. Islam, X. Lu, D. Shankar, and D. K. Panda, "MR-advisor: A comprehensive tuning tool for advising HPC users to accelerate MapReduce applications on supercomputers," in Proc. IEEE 28th Int. Symp. Comput. Archit. High Perform. Comput., 2016, pp. 198-205.
[33] J. Elith, J. Leathwick, and T. Hastie, "A working-guide to boosted regression trees," J. Animal Ecology, vol. 77, pp. 802-813, 2008.
[34] R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee, "Boosting the margin: A new explanation for the effectiveness of voting methods," in Proc. 14th Int. Conf. Mach. Learn., 1997, pp. 1-9.
[35] L. Breiman, "Bagging predictors," Mach. Learn., vol. 24, pp. 123-140, 1996.
[36] R. J. Lewis, "An introduction to classification and regression tree (CART) analysis," in Proc. Annu. Meeting Soc. Academic Emergency Medicine, 2000, pp. 1-14.
[37] Y. Freund and R. Schapire, "A decision-theoretic generalization of online learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, Feb. 1997.
[38] Y. Tao and K. Shivkumar, "A recursive random search algorithm for large-scale network parameter configuration," SIGMETRICS Perform. Eval. Rev., vol. 31, no. 1, pp. 196-205, 2003.
[39] V. Torczon and M. W. Trosset, "From evolutionary operation to parallel direct search: Pattern search algorithms for numerical optimization," Comput. Sci. Statist., vol. 29, pp. 396-401, 1998.
[40] L. Lie, "Heuristic artificial intelligent algorithm for genetic algorithm," Key Eng. Materials, vol. 439, pp. 516-521, 2010.
[41] M. Kumar, M. Husian, N. Upreti, and D. Gupta, "Genetic algorithm: Review and application," Int. J. Inf. Tech Knowl. Manage., vol. 2, no. 2, pp. 451-454, 2010.
[42] C. B. Lucasius and G. Kateman, "Understanding and using genetic algorithms Part 1. Concepts, properties and context," Chemometrics Intell. Lab. Syst., vol. 19, no. 1, pp. 1-33, 1993.
[43] S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, "The HiBench benchmark suite: Characterization of the MapReduce-based data analysis," in Proc. 26th IEEE Int. Conf. Data Eng. Workshops, 2010, pp. 41-51.
[44] F. Ahmad, S. Lee, M. Thottethodi, and T. N. Vijaykumar, "PUMA: Purdue MapReduce benchmark suite," ECE Tech. Rep., Paper 437, 2012. [Online]. Available: ecetr/437
[45] S. S. Gill et al., "Transformative effects of IoT, BlockChain and artifical intelligence on cloud computing: Evolution, vision, trends and open challenges," Internet of Things, vol. 100118, no. 8, pp. 1-26, Sep. 2019.
Citation statistics
Cited Times:1[WOS]   [WOS Record]     [Related Records in WOS]
Document TypeJournal article
CollectionSchool of Data Science
Corresponding AuthorYu, Z.
1.Cloud Computing Department, Alibaba Group, 518860 Hangzhou, Zhejiang, China, (e-mail: [email protected
2.Electrical and Computer Engineering, University of Illinois, Urbana, Illinois, United States, 1111 (e-mail: [email protected
3.Computer Science and Technology, Chinese University of Hong Kong, 26451 Shenzhen, Guangzhou, China, (e-mail: [email protected
4.Research Center for Heterogeneous Intelligent Computer Architecture and Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong, China, (e-mail: [email protected
Recommended Citation
GB/T 7714
Bei, Z.,Kim, N.S.,Hwang, al. OSC: An Online Self-Configuring Big Data Framework for Optimization of QoS (TC-2020-02-0128.R1)[J]. IEEE Transactions on Computers,2021.
APA Bei, Z., Kim, N.S., Hwang, K., & Yu, Z. (2021). OSC: An Online Self-Configuring Big Data Framework for Optimization of QoS (TC-2020-02-0128.R1). IEEE Transactions on Computers.
MLA Bei, Z.,et al."OSC: An Online Self-Configuring Big Data Framework for Optimization of QoS (TC-2020-02-0128.R1)".IEEE Transactions on Computers (2021).
Files in This Item:
There are no files associated with this item.
Related Services
Usage statistics
Google Scholar
Similar articles in Google Scholar
[Bei, Z.]'s Articles
[Kim, N.S.]'s Articles
[Hwang, K.]'s Articles
Baidu academic
Similar articles in Baidu academic
[Bei, Z.]'s Articles
[Kim, N.S.]'s Articles
[Hwang, K.]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Bei, Z.]'s Articles
[Kim, N.S.]'s Articles
[Hwang, K.]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.