HDFS How to Fix Corrupt or Missing Blocks

Blocks with corrupt replicas - These blocks have at least one live, intact replica and at least one corrupt replica.
Missing blocks - These blocks are gone. They have no replicas.

• Run a file system consistency check:

  hdfs fsck hdfs://my-host1:50070/ | egrep -v '^\.+$' | grep -v eplica

• Gather some info and find the node with corrupted data:

  hdfs fsck hdfs://my-host1:50070/my-dir1/my-file1.dat -locations -blocks -files

• Look for issues and repair what you can on nodes with corrupted data. Look for:
  • datanode down
  • missing mount points
  • file system / disk issues
• Restore Corrupt Replicas (not missing)

  • Potentially increase the replication factor if it is low.

    hdfs dfs -setrep 3 hdfs://my-host1:50070/my-dir1/my-file1.dat

  • Potentially wipe the files from the corrupted nodes and let it sync back

• Delete any remaining corrupted files that can’t be fixed.:

  hdfs dfs -rm hdfs://my-host1:50070/my-dir1/my-file1.dat

• Restore any deleted files from backup if available.

You can run this to get statistics missing or corrupt blocks:

hdfs dfsadmin -report

Find the number of missing or corrupted blocks:

hdfs fsck -list-corruptfileblocks

HDFS How to Fix Corrupt or Missing Blocks - Another Method

If you like to live life on the edge you can just go right ahead and let the fsck command delete corrupted blocks for you.

hdfs fsck / -delete

Fix Under Replicated Blocks

You might find it useful to increase the replication factor for any blocks that are under replicated.

• See Here for more info

hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files 

for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ;  hadoop fs -setrep 3 $hdfsfile; done

References

• Handling missing blocks and corrupt blocks in HDFS
• How to fix corrupt blocks in HDFS
• Fix under replicated blocks