Git stores everything. It is a version control tool after all. What is once committed will always be available for anyone to checkout. But how does git stores all versions of your files? Does it copy every version? No. It just detects changes and takes note of them. After all, adding a new function to your class is just inserting some lines into a text file. But this doesn’t always help you to save repository space.
If you’ve added a file then deleted it in the next commit, it doesn’t go away. It has to be stored somewhere to allow you to access it in the future if you wish. These kinds of commits might cause your repository to grow and make it uncomfortably large for cloning. “There must be a way to clean this mess!” I hear you say and there is! But first, we have to figure out why your git repository is unnecessarily large.
I want to demonstrate this on an example repository that I will create on my home directory. Then we will collect the data to find out why this repository takes up so much free space yet empty.
cagdas@cagdas-mint:~$ mkdir repo
cagdas@cagdas-mint:~$ cd repo/
cagdas@cagdas-mint:~/repo$ git init
Initialized empty Git repository in /home/cagdas/.git/
cagdas@cagdas-mint:~$ echo "test message" > testfile
cagdas@cagdas-mint:~/repo$ git add testfile
cagdas@cagdas-mint:~/repo$ git commit -am "init"
[master (root-commit) 85f3338] init
1 file changed, 1 insertion(+)
create mode 100644 testfile
cagdas@cagdas-mint:~/repo$ echo "second init" >> testfile
cagdas@cagdas-mint:~/repo$ git commit -am "second commit"
[master 968d90d] second commit
1 file changed, 1 insertion(+)
cagdas@cagdas-mint:~/repo$ dd if=/dev/zero of=3gib-file bs=1073741824 count=3
3+0 records in
3+0 records out
3221225472 bytes (3,2 GB, 3,0 GiB) copied, 3,37279 s, 955 MB/s
cagdas@cagdas-mint:~/repo$ git add 3gib-file
cagdas@cagdas-mint:~/repo$ git commit -am "adds 3gb file"
[master 4ac6eda] adds 3gb file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 3gib-file
cagdas@cagdas-mint:~/repo$ mv ~/video.mp4 .
cagdas@cagdas-mint:~/repo$ git add video.mp4
cagdas@cagdas-mint:~/repo$ git commit -am "adds video"
[master 1fe0acc] adds video
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100755 video.mp4
cagdas@cagdas-mint:~/repo$ rm video.mp4
cagdas@cagdas-mint:~/repo$ git commit -am "removes video"
[master 6779dfc] removes video
1 file changed, 0 insertions(+), 0 deletions(-)
delete mode 100755 video.mp41. Check current directory sizes in your repository
The first thing I’ll recommend is to check if any large and unnecessary file is still in our project. Surprise surprise! It is not! Yeah, remember I’ve deleted it.
cagdas@cagdas-mint:~/repo$ ls -altr
total 16
-rw-r--r-- 1 cagdas cagdas 25 Sep 1 23:36 testfile
drwxr-xr-x 3 cagdas cagdas 4096 Sep 2 00:02 .
drwxr-xr-x 8 cagdas cagdas 4096 Sep 2 00:02 .git
drwxr-xr-x 46 cagdas cagdas 4096 Sep 2 19:19 ..
cagdas@cagdas-mint:~/repo$ du -h
4,0K ./.git/objects/info
8,0K ./.git/objects/39
8,0K ./.git/objects/47
8,0K ./.git/objects/96
8,0K ./.git/objects/4a
8,0K ./.git/objects/8a
4,0G ./.git/objects/pack
8,0K ./.git/objects/b7
8,0K ./.git/objects/75
8,0K ./.git/objects/1f
8,0K ./.git/objects/da
8,0K ./.git/objects/1b
8,0K ./.git/objects/85
8,0K ./.git/objects/67
4,0G ./.git/objects
8,0K ./.git/info
8,0K ./.git/logs/refs/heads
12K ./.git/logs/refs
20K ./.git/logs
4,0K ./.git/refs/tags
8,0K ./.git/refs/heads
16K ./.git/refs
52K ./.git/hooks
4,0K ./.git/branches
4,0G ./.git
4,0G .So as you can see we have a huge .git directory while the only file I have in my project is just a small text file called "testfile". Why is that? Because I’ve moved committed and deleted several files to and from my repository. I want to try and find out which commit is causing my "git/object/pack” file to grow so large. To do that we first need to find that large pack file in our git repository.
2. Find large pack files in git repository
cagdas@cagdas-mint:~/repo$ cd .git/objects/pack/
cagdas@cagdas-mint:~/repo/.git/objects/pack$ ls -altr
total 4170984
-r--r--r-- 1 cagdas cagdas 3130927 Sep 1 23:42 pack-ded0f1878c9d35bce7a53a0083af2bca3975f563.pack
-r--r--r-- 1 cagdas cagdas 1100 Sep 1 23:42 pack-ded0f1878c9d35bce7a53a0083af2bca3975f563.idx
-r--r--r-- 1 cagdas cagdas 4267932401 Sep 2 00:01 pack-bbd317c8ab9b282182a18017e49e5692769557b7.pack
-r--r--r-- 1 cagdas cagdas 1100 Sep 2 00:01 pack-bbd317c8ab9b282182a18017e49e5692769557b7.idx
drwxr-xr-x 2 cagdas cagdas 4096 Sep 2 00:01 .
drwxr-xr-x 16 cagdas cagdas 4096 Sep 2 00:02 ..3. View the commits in a pack file
With its large size compared to other pack files, It seems obvious our friend pack-bbd317 is hiding something large. To see what a pack file holds we can use a command called verify-pack with --verbose flag. So If we call verify-pack on pack-bbd317 we can see which commits are in it.
cagdas@cagdas-mint:~/repo/.git/objects/pack$ git verify-pack -v pack-bbd317c8ab9b282182a18017e49e5692769557b7.pack
7cde50ff361fb1388f9c0e22759965de1a5c96bc blob 4284456062 4267932369 12
non delta: 1 object
pack-bbd317c8ab9b282182a18017e49e5692769557b7.pack: okThe format of this output as it was mention in the documentation, is like this. So we can see our commits size on the third column.
SHA-1 type size size-in-packfile offset-in-packfile
4. Sort commits by size
Because my git repository is fairly new and doesn’t have too many commits in it, verify-pack is showing only one commit in my pack file. But If you have an old repository this list will be very, very long. In these kinds of situations, it is a bit harder to sort lines by size so you need to work on that data a bit. I’ve personally found it easier to replace spaces with commas and saving it as a CSV file in order to sort it in Calc or Excel. You can use [^\S\r\n]+ regex to replace all with sed or any text editor that supports regex matching.
5. View changes in a commit
After this point, you can locate and view the changes in that exact commit to figure out what happened. Sometimes you just forget to add a file to .gitignore and commit it without noticing or someone who is not familiar with the project commits something without knowing it is unnecessary. In these kinds of times, we need to figure out what happened in the past and try to clean it. Yes, you can clean it. But that’s a topic for another day. 🙂