I'm doing performance test for HDFS. I'm using HDFS C++ API. I notice that most of the reads take up to 5 microseconds. But some of reads take 500-800 microseconds. What could be the reason, considering all the data is in OS cache? No disk access should be occur.
Why some of HDFS reads are so slow, when all the data is cached in off-heap?
137 Views Asked by user1289 At
0
There are 0 best solutions below
Related Questions in HADOOP
- Can anyoone help me with this problem while trying to install hadoop on ubuntu?
- Hadoop No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster)
- Top-N using Python, MapReduce
- Spark Driver vs MapReduce Driver on YARN
- ERROR: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "maprfs"
- can't write pyspark dataframe to parquet file on windows
- How to optimize writing to a large table in Hive/HDFS using Spark
- Can't replicate block xxx because the block file doesn't exist, or is not accessible
- HDFS too many bad blocks due to "Operation category WRITE is not supported in state standby" - Understanding why datanode can't find Active NameNode
- distcp throws java.io.IOException when copying files
- Hadoop MapReduce WordPairsCount produces inconsistent results
- If my data is not partitioned can that be why I’m getting maxResultSize error for my PySpark job?
- resource manager and nodemanager connectivity issues
- ERROR flume.SinkRunner: Unable to deliver event
- converting varchar(7) to decimal (7,5) in hive
Related Questions in FILESYSTEMS
- Where exactly is the first data sector on a FAT file system?
- `df` command not capturing entire output in perl
- Is it possible to mount a logical volume without wiping the data?
- Speed up search of remote directories
- How to change the directory file system without losing files?
- Flutter SDK: Files Deleted Automatically (e.g., dart.exe), Errors in Android Studio
- How to store metadata for a UTF-8 text file cross-platform?
- fsck error on boot: dev/mapper/ubuntu--vg-ubuntu--lv: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY
- rouble with mounting Python code to FUSE: No response and prolonged processing ---
- mkfs.erofs erofs: failed to lookup selabel
- How to deny user access MacFuse file system by the really path
- Is it faster to read a file on an NVMe using threads?
- list folders containing mp3 files using the Capacitor Filesystem
- How to use xdg-open in bwrap environment to open dir in the unsandboxed filesystem
- How to provide content of (locally) encrypted files to the iOS and macOS system
Related Questions in HDFS
- Can anyoone help me with this problem while trying to install hadoop on ubuntu?
- ERROR: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "maprfs"
- How to optimize writing to a large table in Hive/HDFS using Spark
- Update hadoop hadoop-2.6.5 to haddop 3.x. Operation category WRITE is not supported in state standby
- Copy/Merge multiple HDFS files using Nifi Processor
- HDFS too many bad blocks due to "Operation category WRITE is not supported in state standby" - Understanding why datanode can't find Active NameNode
- distcp throws java.io.IOException when copying files
- ERROR flume.SinkRunner: Unable to deliver event
- Apache flume does not run hadoop 3.1.0 Flume 1.11
- Livy session to submit pyspark from HDFS
- ClickHouse Server Exception: Code: 210.DB::Exception: Fail to read from HDFS:
- Confluent HDFS Sink connector error while connecting HDFS to Hive
- Node Transitioned from NEW to UNHEALTHY and Attempting to remove non-existent node
- Error associated with Azure Datalake Gen2 and Hadoop connection
- How do I directly read files from HDFS using dask?
Related Questions in LIBHDFS
- Pyarrow issue with hdfs connectivity
- Pyarrow unable to load libhdfs on Windows 10
- Not able to connect to Hadoop Server using HadoopFileSystem in pyarrrow
- libhdfs - cannot open shared library libhdfs.so.0.0.0.0
- Pyarrow fs.HadoopFileSytem reports unable to load libhdfs.so
- How do i set the path of libhdfs.so for pyarrow?
- No such file exists while running Hadoop pipes using c++
- WinError 126 Error when connecting to HDFS using hdfs3
- ImportError: libarrow.so.14: cannot open shared object file: No such file or directory | python
- How to write on HDFS using pyarrow
- When libhdfs hdfsRename will return -1
- Segmentation Fault 11 when listing HDFS files
- AppData\Local\Temp\cc59LXDc.o:test1.cpp: undefined reference to `constructNewObjectOfClass(JNIEnv_*, _jthrowable**, char const*, char const*, ...)'
- Why some of HDFS reads are so slow, when all the data is cached in off-heap?
- connecting pyarrow with libhdfs3
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?