Repository navigation
Handle erroneously empty rfile block reads #3694
Description
Describe the bug
When HDFS returns empty results for a rfile block that actually has data, Accumulo does not error or attempt additional validation of the rfile block. This can cause issues with metadata table reads as specific tablet contents fail to return.
Versions (OS, Maven, Java, and others, as appropriate):
- Affected version(s) of this project: 1.10.3
Expected behavior
Accumulo should be able to determine if a rfile content location is actually empty or if data should have been returned.
Activity
dlmarion commentedon Aug 15, 2023
So many questions:
You are saying that HDFS is returning a zero byte file block and the RFile Reader does not fail?
Does the DataNode log any error in this case?
What does
fscksay about the file?Is this a transient issue?
Are other copies of the block correct?
ddanielr commentedon Aug 15, 2023
The RFile Reader does not fail and the HDFS contents are currently speculated to be empty.
Dmesg reports IO errors for the disk that hosted the rfile block.
The DataNode log does report
Slow flushOrSyncoperations for the drive that contained the requested rfile block.Currently unknown
Yes, on subsequent reads, the rfile contents are returned correctly.
Unknown, but is speculated that HDFS did not attempt to query the other block locations.
This issue is an attempt to chase down a transient bug with the AccumuloGC.
For this situation, the metadata table is separated into different locality groups
groupA: loc,~tab,groupB: file,log,srv,future.The tserver that hosted the groupB locality group reported disk errors.
The AccumuloGC queried the metadata table to find file references for deletion candidates.
The metadata table scan returned
~tab:~prentries but failed to return anysrvorfileentries.Because of this, the GC concluded that no file references existed for a collection of deletion candidates and then removed files from HDFS that had valid references on the metadata table.
ctubbsii commentedon Aug 16, 2023
This implies that one measure to try to prevent this from happening again could be to remove the locality groups in the metadata, so the file entries are in the same locality group as the file entries. However, this comes with its own problems: First, you'd have to force a full major compaction on the metadata table to force this to take effect (and that could be costly... and the IO could stress whatever underlying filesystem or disk errors are causing the problem in the first place). Second, this could cause slower metadata performance. Third, there may still be weird behavior where file entries and prev row entries are in different RFile blocks or different HDFS blocks, and there could still be hidden errors when one block is read and not when the other is read... it's not clear to me how that would actually manifest itself, though (because it's not clear to me why HDFS errors are failing to propagate up into our code to begin with).
keith-turner commentedon Aug 17, 2023
I was reviewing some of the 2.1 Rfile code today and I realized this issue happened in 1.10. That made me diff the 1.10 and 2.1 code and I found that the fixes (#3622 and #3623) for #3617 are not in 1.10. Not sure if that matters, but just recording the finding.