Re: Deleted, But Still Full

TudorRaduta

Friday Challenge: The Case of the Ghost File

Happy Friday! Let’s end the week with a real world brain teaser.

This one is not about typing a command. It is about understanding how Linux really handles files under the hood. This scenario shows up in interviews and production more often than people expect.

The Scenario

You receive an urgent alert. The /var partition is 100% full.

You find the culprit fast. A huge 50GB log file:

/var/log/app_debug.log

You delete it immediately:

[root@server ~]# rm /var/log/app_debug.log

You confirm it is gone with ls -l.

But df -h still shows 100% usage. No space was freed.

Your Challenge

Why is the disk still full? Time to solve the mystery.

The Theory: Explain why the space was not released. Think inodes and open file descriptors.
The Investigation: What exact command using lsof would you run to find which process still holds the deleted file?
The Fix: Once you identify the process, what is the cleanest way to release the disk space without rebooting the server?

Bonus: What command should you have used instead of rm to safely clear the log without breaking the running application?

Let’s see who can crack the case of the ghost file.

sa_sachin

This was one of the most challenging issues I faced early in my Linux administration career. Once I understood why it happened and how to fix it, the solution went on to save our critical Hadoop environment countless times.

1.The Theory: Explain why the space was not released. Think inodes and open file descriptors.

2.When you delete a file in Linux using rm, the file’s directory entry is removed, but if a process still has the file open, the data blocks remain allocated until that process closes the file.

This happens because the kernel tracks open file descriptors by inode, not by filename.
So even though ls shows the file is gone, the inode is still in use by the process, and disk space cannot be reclaimed.
The Investigation: What exact command using lsof would you run to find which process still holds the deleted file?

#lsof +L1 /var/log

3.The Fix: Once you identify the process, what is the cleanest way to release the disk space without rebooting the server?

Once we identify the PID (eg 1234), we have two options:
Graceful:

#kill -HUP 1234

Force Ful:
#kill 1234
#echo > /proc/<pid>/fd/<fd_number> # Truncate open file in process

4.Bonus: What command should you have used instead of rm to safely clear the log without breaking the running application?

Instead of rm, we can use truncate or : > file to clear contents without removing the file:
#truncate -s 0 /var/log/app_debug.log
or
# > /var/log/app_debug.log

There is an well documented redhat article for same.https://access.redhat.com/solutions/2316

Ad_astra

1) The disk space was not released because a running process still has a link pointing to the file on disk. Whilst a link to the file was deleted by the rm command, the running process still has a link (via a file descriptor) to a file table in the operating system which then points to the inode on the disk. The process will still be writing to the file on the disk, which has not been deleted.

2) Running the following command will show which processes are still using the 'deleted' file:

lsof | grep /var/log/app_debug.log

The results will mark the log file as (deleted), even though it is still present on disk.

3) Now that you have the process id, the cleanest way to release the disk space would be to restart the application using systemctl restart <name.service>.

If the process is not enabled as a service, then the command kill -15 <process id> will attempt to close the running process in a graceful fashion; restart the process manually after this.

Bonus:

The command truncate -s 0 /var/log/app_debug.log would have cleared the log without breaking the running application.

Chetan_Tiwary_

@Ad_astra right ! you can use lsof +L1 which is to list open files with fewer than 1 link or you can use lsof | grep deleted for deleted entries search.

if your graceful restart is not possible , you can truncate open file in process :

echo > /proc/<pid>/fd/<fd_number>

Bonus : safe approach :

 > /var/log/app_debug.log

https://access.redhat.com/solutions/2316

Ad_astra

A fun challenge.