Custom Source Control

I, like a lot of developers these days, really enjoy working with Git. It just makes sense to me. However, not every shop has bought in. Some use Subversion, Perforce, Mercurial, and/or Fossil to name a few. That is just to mention a few, as there are even more that I have not worked with. Git though, I have used enough to appreciate. While working with other SCM’s I’ve faced issues where I find myself wishing for one (or many) of Git’s features.

Working with Perforce the other day, I started editing a file. It was an unfamiliar codebase and I just wanted to get an idea of the flow of the application. When I hit save however I get a ‘readonly’ message. So I could do p4 open , or ignore the message and save anyway. Now, I may have to do this several times and its more than likely that none of the edits I made have saved to code. This is quite a hindrance. So I continue ignoring the readonly message and keep saving files and deleting the changes.

The thing I like about Git is that it stays out of your way for the most part, until it’s time to commit. So I can edit to my heart’s content and do a git status to see if I left anything in the code that shouldn’t be there, and quickly.

That being said, I wanted to build a little tool to abstract some of the differences of other SCM’s away. Things like a git status amongst others. The problem however, is that I have a hard time finding the time to build such a tool. Also, I haven’t really figured out all of the functionality I would like this tool to have.

Then I came across a video “Source Control Made Easy” by Jim Weirich, a man well known in the Ruby community, who recently passed away. I liked his teaching style and feel I’ve learned a lot from his talks. One of my personal favorites is on testing called “Roman Numerals Kata”. I didn’t know him Jim, but it seems like he would have been a fun person to be friends with.

“Source Control Made Easy”, is kind of a talk about Git, but not directly. Or at least it doesn’t seem that way at first. The following is part of the description for this video:

“In this 49-minute screencast, Jim Weirich takes you on a journey of how you might design and build a source control system from scratch. Along the way you’ll gain a deeper understanding of the first principles behind systems like Git, so things begin to make more sense.”

I highly recommend this video to anyone interested in learning about not only Git, but understanding the principles of source control in general. Also, note that the great people over at the Pragmatic Programmers are donating 100% of the purchase price to Jim’s family.

So, as a tribute to Jim Weirich, I decided to take a shot at implementing the source control system he talks about in ruby and to incorporate some test driven development as well.

Initial Stories:

I’ll start by writing out some quick user stories.

Initialize a new repository As a user I want the ability to initialize a repository So that I can begin adding snapshots to it

Create snapshots of my work As a user I want to create snapshots So that I can save snapshots of my work

Checkout previous snapshots As a user I want to checkout a previous snapshot So that I can fix issues or get back to a working state

With those stories in place to drive my development, I wanted to take a few minutes up front to think about or pseudo code a quick possible implementation. I don’t really have much in terms of expectations here. I just want to get some of the ideas into focus.

initialize

calling initialize from within a directory should: create new hidden directory .esc create new sqlite database to store what? create HEAD file which contains hash (manifest filename) of the latest snapshot (empty initially) (maybe a db entry)



snapshot

create metadata file ( metadata ) which will contain the manifest hash, snapshot author (name, email), timestamp, comments, current head (parent to this snapshot)

) which will contain the manifest hash, snapshot author (name, email), timestamp, comments, current head (parent to this snapshot) create manifest file ( manifest ) which will contain hashes of the files in the snapshot, maps the hashes to original filenames and directories

) which will contain hashes of the files in the snapshot, maps the hashes to original filenames and directories get a list of all files with paths in the working directory

iterate through the list, calculate the file hash

search the repository for a file with the calculated hash if found, just add the hash and the filename with pathname to the manifest file if not found check if we need to create a directory for the file (create hash directories: a..z maybe) copy the file to the repository directory add the hash and the original filename with pathname to the manifest file

calculate the hash of the manifest file rename it from manifest to hash

to hash calculate the hash of the metadata file and rename it from metadata to hash update metadata file with snapshot hash

to hash update metadata file with snapshot hash update HEAD to point to this snapshot (metadata filename)

checkout(version number)

check .esc for the metadata file (version number/hash) if metadata file is not found, fail and inform the user if found, get the manifest hash print the metadata info out to the console open the manifest scan the file line by line, get the actual path/filename for an entry and see if it exists in the working directory if it doesn’t, just copy the file changing the hash to the filename and placing it in the correct directory if it does, calculate the hash for the file in the working directory if the hash is the same, don’t do anything with that file if the hash is different, overwrite the existing file with the file from the repository



One more thing before we get on with the actual coding. I am trying to keep this simple. That being said, I may not adhere to any strict standards or practices. I will try to point them out as I go. This will free me up to:

Write code as fast as possible since it’s been hard enough to find the time to write these days.

Refactoring would be a great exercise for any reader who would like to continue this project. Ideally, I will do a follow up post where I refactor. I want to code this almost raw, and think about things as I go almost like the Roman Numeral Kata.

Initialize

…but before we actually start writing the application code, let’s get our initial testing in place. Create a directory, and call it whatever you want. I am calling mine: custom_source_control .

mkdir custom_source_control cd custom_source_control

Now open your favorite editor, create a new file named custom_source_control.rb and add the following to it.

#!/usr/bin/env ruby require 'minitest/autorun' describe CustomSourceControl do before do @csc = CustomSourceControl . new end describe 'when a repository is initialized' do it 'must create a new hidden directory named .esc' do @csc . repository_exists? . must_equal true end end end

So if you aren’t familiar with minitest you should continue reading the post. If you like what you’ve seen checkout the README over on github. Basically, we just described the first thing we’d like to test. The DSL’s that people write using Ruby are great and this almost reads like English (or some weird robot form of it). Let’s just look at the words between the quotes:

‘when a repository is initialized’, ‘must create a new hidden directory named .esc’

Another thing to point out is the before block. We can pretty much infer that the before block will run before any of our tests. Then there is that require 'minitest/autorun' thing at the top. That just makes the tests run when we execute the ruby file. Let’s make the ruby file executable, and execute it.

chmod u+x custom_source_control.rb ./custom_source_control.rb

(test)

./custom_source_control.rb:5:in `<main>': uninitialized constant CustomSourceControl (NameError)

Here we gave the script the ability to be executed as a command. It ran the script… and failed, kind of.

Actually, this is ruby telling us that we don’t have a constant CustomSourceControl in our script, but were acting as if we did. CustomSourceControl is going to be our class. So we’ll need to add it. We are going to write our actual implementation code above our tests and everything else, but below the shebang. Wait, what’s a ‘shebang’? It’s that little #!/usr/bin/env ruby at the top of our file. Remember the chmod u+x custom_source_control.rb we just did. Well chmod u+x custom_source_control.rb tells the operating system the file is executable, and #!/usr/bin/env ruby tells it to use ruby to execute it.

#!/usr/bin/env ruby class CustomSourceControl end require 'minitest/autorun' ...

Note the use of ... , do not type it in the editor. This is just me saying “more text may come before or after”.

(retest)

E Finished tests in 0.000628s, 1592.3567 tests/s, 0.0000 assertions/s. 1) Error: CustomSourceControl::when a repository is initialized#test_0001_must create a new hidden directory named .esc: NoMethodError: undefined method `repository_exists?' for #<CustomSourceControl:0x007fe974157038> ./custom_source_control.rb:15:in `block (3 levels) in <main>'

We now get an error (denoted by the E ) that the CustomSourceControl class doesn’t have a repository_exists? method. We will just add that method and retest:

class CustomSourceControl ... def repository_exists? end ... end

(retest)

F Finished tests in 0.020405s, 49.0076 tests/s, 49.0076 assertions/s. 1) Failure: CustomSourceControl::when a repository is initialized#test_0001_must create a new hidden directory named .esc [./custom_source_control.rb:17]: Expected: true Actual: nil

We are getting an actual failure (denoted by the F ) now. I mean, where do we get off expecting true to be returned from the repository_exists? method. We haven’t even implemented it, of course it’s going to return nil…

So let’s implement it.

class CustomSourceControl ... def repository_exists? true end ... end

(retest)

. Finished tests in 0.000590s, 1694.9153 tests/s, 1694.9153 assertions/s.

…and we’re passing (denoted by the . )!

Seriously? We passed? Yeah… and I want you to know that I realize just returning true doesn’t mean that the repository actually exists. I know this for a few reasons, but let’s take the most straightforward way and prove this.

ls -la

ls -la if you don’t know, or couldn’t tell, just lists all the files in a directory. We need the a in order to see all files, including the hidden ones. On *nix files/directories that begin with a . are hidden.

drwxr-xr-x 4 mweppler _guest 136 Mar 7 07:05 . drwxr-xr-x 29 mweppler _guest 986 Mar 7 07:05 .. -rwxr--r-- 1 mweppler _guest 377 Mar 7 07:04 custom_source_control.rb

Nope, no .esc directory here… So what did that prove? The testing stuff? Why do it? We are taking a very systematic and pragmatic approach here. This testing stuff is good for a few reasons. We will see this come to light a bit later. For now though, please just accept it.

With this little bit of testing so far, we’ve really just exercised our minds as well as minitest . As our code grows and we move on to other projects, we will likely forget all the intricacies of what we wrote. Our tests here should give us a bit of confidence, even this early on in our development cycle. Also, our code is small and easy to manually test, so we know minitest is doing its job.

Let’s really implement the repository_exists? method now.

class CustomSourceControl ... def repository_exists? Dir . exists? '.esc' end ... end

(retest)

F Finished tests in 0.018134s, 55.1450 tests/s, 55.1450 assertions/s. 1) Failure: CustomSourceControl::when a repository is initialized#test_0001_must create a new hidden directory named .esc [./custom_source_control.rb:18]: Expected: true Actual: false

…and were failing again :(, that rush I got from passing never quite lasts long. Keep calm and carry on. It’s good that we’re failing, we should be failing. We have yet to create our .esc directory. Now I don’t know about you, but I want to get to passing again. I got that itch now…

class CustomSourceControl ... def repository_exists? Dir . mkdir '.esc' Dir . exists? '.esc' end ... end

(retest)

. Finished tests in 0.000720s, 1388.8889 tests/s, 1388.8889 assertions/s.

There is nothing crazy going on here, just calling the mkdir (make directory) method on the Dir class. But guess what, we’re passing again. If we check the file system, we see our newly added directory.

drwxr-xr-x 5 mweppler _guest 170 Mar 7 07:15 . drwxr-xr-x 29 mweppler _guest 986 Mar 7 07:05 .. drwxr-xr-x 2 mweppler _guest 68 Mar 7 07:15 .esc -rwxr--r-- 1 mweppler _guest 412 Mar 7 07:15 custom_source_control.rb

Lets move on to our next test.

describe 'when a repository is initialized' do ... it 'must create an empty HEAD file' do @csc . head_exists? . must_equal true @csc . head_contents . empty? . must_equal true end end

(test)

EE Finished tests in 0.000708s, 2824.8588 tests/s, 0.0000 assertions/s. 1) Error: CustomSourceControl::when a repository is initialized#test_0002_must create an empty HEAD file: NoMethodError: undefined method `head_exists?' for #<CustomSourceControl:0x007fad59a170f8> ./custom_source_control.rb:23:in `block (3 levels) in <main>' 2) Error: CustomSourceControl::when a repository is initialized#test_0001_must create a new hidden directory named .esc: Errno::EEXIST: File exists - .esc ./custom_source_control.rb:5:in `mkdir' ./custom_source_control.rb:5:in `repository_exists?' ./custom_source_control.rb:19:in `block (3 levels) in <main>'

What? Why do we have 2 errors? We were just passing. If we look at our initial test we can see that the .esc file already exists. We have to clean up after ourselves like good TDD citizens. First let’s manually remove the existing .esc directory. Then let’s add some code to our tests that will clean up after each test is run.

rmdir .esc

require 'fileutils' class CustomSourceControl ... end ... describe CustomSourceControl do after do FileUtils . rm_rf '.esc' if Dir . exists? '.esc' end ... end

We added the require 'fileutils' right above our class CustomSourceControl statement, and after our after block inside our main describe block above our before block. Was that confusing? If so, you can double check your work with the project I have hosted on github.

(retest)

.E Finished tests in 0.001229s, 1627.3393 tests/s, 813.6697 assertions/s. 1) Error: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized): NoMethodError: undefined method `head_exists?' for #<CustomSourceControl:0x007feac2805818> custom_source_control.rb:29:in `block (3 levels) in <main>'

Rerun our tests and this time our initial test is passing again. We can now deal with the error we are seeing. I don’t know about you, but I am finding this process very helpful. It’s an iterative approach, one that you likely do anyway, just without the testing.

So, if history is any indicator of things to come (and we read the error message), we know the next step is to create the head_exists? method in our CustomSourceControl class.

class CustomSourceControl ... def head_exists? end ... end

(retest)

F. Finished tests in 0.011323s, 176.6316 tests/s, 176.6316 assertions/s. 1) Failure: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized) [custom_source_control.rb:31]: Expected: true Actual: nil

We could take the same approach we took in our previous test and return true to start, then test, fail, refactor, etc… This would be the right approach, but I’ll leave that for you to do on your own. Get comfortable with the process and messages.

Once you’ve run through the exercise, after a few iterations, you should have a method similar to the repository_exists? method except instead of creating a directory, we’re going to create a file.

class CustomSourceControl ... def head_exists? File . new ( File . join ( '.esc' , 'HEAD' ), 'w' ) File . exists? File . join ( '.esc' , 'HEAD' ) end ... end

(retest)

E. Finished tests in 0.001215s, 1646.0905 tests/s, 823.0453 assertions/s. 1) Error: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized): Errno::ENOENT: No such file or directory - .esc/HEAD custom_source_control.rb:7:in `initialize' custom_source_control.rb:7:in `new' custom_source_control.rb:7:in `head_exists?' custom_source_control.rb:34:in `block (3 levels) in <main>'

…but wait! Why is this you ask! Well we only create .esc when we call repository_exists? , and then after each test, it is removed if it exists. So .esc doesn’t exist anymore.

Let’s think about our story for a second. What we really care about, from a high level, is repository initialization. So lets skip this test and refactor a bit.

describe 'when a repository is initialized' do ... it 'must create an empty HEAD file' do skip @csc . head_exists? . must_equal true @csc . head_contents . empty? . must_equal true end end

(retest)

.S Finished tests in 0.029118s, 68.6860 tests/s, 34.3430 assertions/s.

That S tells us that we’re skipping a test. Update the CustomSourceControl class with the following:

... class CustomSourceControl def head_exists? File . exists? File . join ( '.esc' , 'HEAD' ) end def initialize_repository Dir . mkdir '.esc' File . new ( File . join ( '.esc' , 'HEAD' ), 'w' ) end def repository_exists? Dir . exists? '.esc' end end ...

We’ve created an initialize_repository method. We’ve moved both the .esc directory creation and the HEAD file creation out of the xxx_exists? methods and into initialize_repository . This all makes sense. The xxx_exists? should only be responsible for checking that something actually exists, not creating anything. initialize_repository on the other hand, its purpose is to handle the tasks involved in initializing the repository. One of those tasks is creation of the repository structure.

If we run this now what do you think will happen?

(retest)

FS Finished tests in 0.008715s, 229.4894 tests/s, 114.7447 assertions/s. 1) Failure: test_0001_must create a new hidden directory named .esc(CustomSourceControl::when a repository is initialized) [custom_source_control.rb:33]: Expected: true Actual: false

Well, we fail since we haven’t actually called the initialize_repository method anywhere in our code. Is that what you guessed? So where should we call the initialize_repository method? If you guessed ‘in our before block’, you win.

... before do @csc = CustomSourceControl . new @csc . initialize_repository end ...

(retest)

S. Finished tests in 0.000897s, 2229.6544 tests/s, 1114.8272 assertions/s.

Great, we’re passing again. Let’s remove the skip statement from the test and rerun.

(retest)

E. Finished tests in 0.002892s, 691.5629 tests/s, 691.5629 assertions/s. 1) Error: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized): NoMethodError: undefined method `head_contents' for #<CustomSourceControl:0x007f970d009618> custom_source_control.rb:39:in `block (3 levels) in <main>'

It looks like we’re missing that head_contents method. We should add that.

class CustomSourceControl ... def head_contents File . open ( '.esc/HEAD' , 'r' ) { | f | f . read } end ... end

(retest)

.. Finished tests in 0.001290s, 1550.3876 tests/s, 2325.5814 assertions/s.

You did it! You can now initialize a new repository! Let’s move on to our next story…

Snapshot

Create snapshots of my work As a user I want to create snapshots So that I can save snapshots of my work

If you recall from our brief pseudo code, we’ve already kind of mapped out a few steps. If you don’t recall that, try again, try harder, or just reread that part above.

Let’s update our before block to include a new snapshot method.

before do @csc = CustomSourceControl . new @csc . initialize_repository @csc . snapshot end

(retest)

EE Finished tests in 0.002971s, 673.1740 tests/s, 0.0000 assertions/s. 1) Error: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized): NoMethodError: undefined method `snapshot' for #<CustomSourceControl:0x007fd35a985078> custom_source_control.rb:34:in `block (2 levels) in <main>' 2) Error: test_0001_must create a new hidden directory named .esc(CustomSourceControl::when a repository is initialized): NoMethodError: undefined method `snapshot' for #<CustomSourceControl:0x007fd35a98bce8> custom_source_control.rb:34:in `block (2 levels) in <main>'

As you might have guessed, it failed since we haven’t actually created a snapshot method. You also probably guessed that, that is going to be our next step. And you’d be right.

class CustomSourceControl ... def snapshot end ... end

Deep breath…

(retest)

.. Finished tests in 0.001580s, 1265.8228 tests/s, 1898.7342 assertions/s.

…and we’re passing. Now, let’s create a new describe block for our snapshot story and test the existence of a metadata file. The test will be inside our main describe CustomSourceControl do block, outside and below the describe 'when a repository is initialized' do .

... describe CustomSourceControl do ... describe 'when a repository is initialized' do ... end describe 'when we take a snapshot' do it 'must create a metadata file' do @csc . metadata_exists? . must_equal true end end end

Which will undoubtedly fail…

(test)

..E Finished tests in 0.002115s, 1418.4397 tests/s, 1418.4397 assertions/s. 1) Error: test_0001_must create a metadata file(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `metadata_exists?' for #<CustomSourceControl:0x007fb122a36760> custom_source_control.rb:53:in `block (3 levels) in <main>'

Now, let’s make it pass.

class CustomSourceControl ... def metadata_exists? File . exists? File . join ( '.esc' , '__metadata__' ) end ... def snapshot File . open ( File . join ( '.esc' , '__metadata__' ), 'w' ) end end

First, we are adding the necessary metadata_exists? method and checking that a file .esc/__metadata__ actually exists. It won’t, and has to be created as part of the snapshot so we add that code to our snapshot method.

(retest)

... Finished tests in 0.001947s, 1540.8320 tests/s, 2054.4427 assertions/s.

Here is a homework assignment:

Follow the same process for the manifest file test/creation. Make sure you follow the process as you go.

It should look something like this:

(our test)

describe 'when we take a snapshot' do ... it 'must create a manifest file' do @csc . manifest_exists? . must_equal true end end

…and this:

(method to test existence)

class CustomSourceControl ... def manifest_exists? File . exists? File . join ( '.esc' , '__manifest__' ) end ... end

…and last but not least:

(actually create the file)

class CustomSourceControl ... def snapshot File . new ( File . join ( '.esc' , '__metadata__' ), 'w' ) File . new ( File . join ( '.esc' , '__manifest__' ), 'w' ) end ... end

(retest)

.... Finished tests in 0.002804s, 1426.5335 tests/s, 1783.1669 assertions/s.

Our next step is to get a list of files in the current working directory.

describe 'when we take a snapshot' do ... it 'gets a list of files in the current working directory' do @csc . cwd_files . must_equal [ 'custom_source_control.rb' ] end end

class CustomSourceControl ... def cwd_files all_files_wildcard = File . join '**' , '*' Dir . glob ( all_files_wildcard ) end ... end

Notice how I skipped running the test and went straight to implementation? Well I didn’t actually skip the testing part. I just didn’t write it here. Keep this in mind. You should be testing as often as possible. Get familiar with the messages and try to understand what they are telling you is wrong.

(retest)

..... Finished tests in 0.003872s, 1291.3223 tests/s, 1549.5868 assertions/s.

We moved pretty quickly in that last cycle of: write a test, watch it fail, write code to make it pass. The last part in that cycle which I have not done (for the most part) is refactor. It’s called Red, Green, Refactor. Refactoring is an important part of the cycle and I normally wouldn’t skip over it. I am doing so here however to get you familiar with the other parts of the cycle with the intention that we will revisit and refactor in another post. I mentioned this before, but want to reiterate the point here.

Let’s create our SHA1 file hashes.

(write a test)

describe 'when we take a snapshot' do ... it 'creates a file hash for all files in the current working directory' do hashes = { 'custom_source_control.rb' => '1527a36a8246ad0c07b9d5478c7374d3d576752d' } @csc . cwd_hashes . must_equal hashes end end

(watch it fail)

(test)

.....E Finished tests in 0.006143s, 976.7215 tests/s, 976.7215 assertions/s. 1) Error: test_0004_creates a file hash for all files in the current working directory(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `cwd_hashes' for #<CustomSourceControl:0x007fbafa8158a0> custom_source_control.rb:84:in `block (3 levels) in <main>'

(write code to make it pass)

require 'openssl' ... class CustomSourceControl ... def cwd_hashes sha1 = OpenSSL :: Digest :: SHA1 . new hashes = {} cwd_files . each do | file | hashes [ file ] = sha1 . hexdigest ( File . read ( file )) sha1 . reset end hashes end ... end

What did we just do here? First, we are adding openssl , which provides the methods necessary to hash files. In the cwd_hashes method, we’re creating an instance of OpenSSL::Digest::SHA1 and later using the hexdigest it provides to hash files in the current working directory.

(retest)

..F... Finished tests in 0.020276s, 295.9164 tests/s, 345.2357 assertions/s. 1) Failure: test_0004_creates a file hash for all files in the current working directory(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:95]: --- expected +++ actual @@ -1 +1 @@ -{"custom_source_control.rb"=>"1527a36a8246ad0c07b9d5478c7374d3d576752d"} +{"custom_source_control.rb"=>"0b85c90f96ef4d8f0937338760be827d68f93469"}

We’re getting the wrong hash, and that’s because were actually updating the very file we’re hashing/testing. We can never (at least I can’t think of a cleaver way) pass like this. We need to account for this and work around it. What we’re going to do is create two files manually, sha1sum them and remove all but those two files from the hash returned from the cwd_hashes method.

Here I am going to use the cat command and paste the text in. You can use any means you’re comfortable with to create the files and add the test to them. It is important however that you add the empty newline by hitting the enter key at the end of the sentence. I am also assuming that you have shasum or sha1sum installed. If you don’t you can safely skip over that command.

cat > test_file_1.txt

Copy and paste in the following:

this is test_file_1.text

Then type ctrl+c to quit. Run shasum or sha1sum to get the files hash.

sha1sum test_file_1.txt bb4d8995cfa843effc83d6ddcea1a8351c09497f test_file_1.txt

Repeat the process for the second test file.

cat > test_file_2.txt

this is test_file_2.text

sha1sum test_file_2.txt 5d3140359919315ea06e3755cdc81860e9d7c556 test_file_2.txt

Now let’s update our test.

... it 'creates a file hash for all files in the current working directory' do actual_hashes = { 'test_file_1.txt' => 'bb4d8995cfa843effc83d6ddcea1a8351c09497f' , 'test_file_2.txt' => '5d3140359919315ea06e3755cdc81860e9d7c556' } expected_hashes = @csc . cwd_hashes . keep_if { | key , value | key == 'test_file_1.txt' || key == 'test_file_2.txt' } expected_hashes . must_equal actual_hashes end ...

(retest)

..F... Finished tests in 0.019896s, 301.5682 tests/s, 351.8295 assertions/s. 1) Failure: test_0003_gets a list of files in the current working directory(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:88]: --- expected +++ actual @@ -1 +1 @@ -["custom_source_control.rb"] +["custom_source_control.rb", "test_file_1.txt", "test_file_2.txt"]

This gets us passing the test in question, but now we’re failing a previous test. If we inspect the message we see that the newly introduced files are causing gets a list of files in the current working directory test to fail. We can simply add the new file names to the array of actual file names.

... it 'gets a list of files in the current working directory' do @csc . cwd_files . must_equal [ 'custom_source_control.rb' , 'test_file_1.txt' , 'test_file_2.txt' ] end ...

(retest)

...... Finished tests in 0.004923s, 1218.7690 tests/s, 1421.8972 assertions/s.

Now we need our list of files in the repository.

describe 'when we take a snapshot' do ... it 'gets a list of files in the current working directory' do @csc . repository_file_list . must_equal [ '__manifest__' , '__metadata__' , 'HEAD' ] end end

(test)

......E Finished tests in 0.008568s, 816.9935 tests/s, 816.9935 assertions/s. 1) Error: test_0005_gets a list of files in the current working directory(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `repository_file_list' for #<CustomSourceControl:0x007fd440844718>

class CustomSourceControl ... def repository_file_list all_files_wildcard = File . join '.esc' , '*' Dir . glob ( all_files_wildcard ). map { | pathname | File . basename pathname } end ... end

(retest)

....... Finished tests in 0.004527s, 1546.2779 tests/s, 1767.1747 assertions/s.

Ok, now that we have our list of existing files in the repository. We can compare what is new, with what already exists and return both of those lists.

describe 'when we take a snapshot' do ... it 'returns a list of new and existing files' do deltas = @csc . deltas deltas [ :new ]. keep_if { | key , value | key == 'test_file_1.txt' || key == 'test_file_2.txt' } deltas [ :new ]. must_equal [ 'test_file_1.txt' , 'test_file_2.txt' ] deltas [ :existing ]. must_equal [] end end

(test)

...E.... Finished tests in 0.008715s, 917.9575 tests/s, 917.9575 assertions/s. 1) Error: test_0006_returns a list of new and existing files(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `deltas' for #<CustomSourceControl:0x007f8c2a83e790>

Let’s add that deltas method.

class CustomSourceControl ... def deltas new , existing = [], [] cwd_hashes . each do | key , value | if repository_file_list . include? key existing << key else new << key end end { :new => new , :existing => existing } end ... end

Here, we’re going through our current working directory hashes and checking if any exist in the repository. If they do, we add them to the existing array. If they do not, we add them to the new array. Then we create a hash with the keys :new & :existing , add the arrays, and return that hash.

(retest)

........ Finished tests in 0.005304s, 1508.2956 tests/s, 1885.3695 assertions/s.

I think the next step should be to add the files to the manifest, then based off the manifest copy and hash the files added to the snapshot.

describe 'when we take a snapshot' do ... it 'adds entries to the manifest file' do expected_content = %Q{5d3140359919315ea06e3755cdc81860e9d7c556 => test_file_2.txt (new)

bb4d8995cfa843effc83d6ddcea1a8351c09497f => test_file_1.txt (new)} @csc . write_manifest @csc . manifest_contents . gsub! ( /.*? => custom_source_control\.rb \(new\)

?/ , '' ). chomp . must_equal expected_content end end

(test)

.......E. Finished tests in 0.009311s, 966.5986 tests/s, 1073.9985 assertions/s. 1) Error: test_0007_adds entries to the manifest file(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `write_manifest' for #<CustomSourceControl:0x007fb13084c888>

class CustomSourceControl ... def hash_for_file ( file = nil ) sha1 = OpenSSL :: Digest :: SHA1 . new sha1 . hexdigest ( File . read ( file )) end ... def write_manifest file_deltas = deltas file_list = [] file_deltas [ :new ]. each { | filename | file_list << " #{ hash_for_file filename } => #{ filename } (new)" } file_deltas [ :existing ]. each { | filename | file_list << " #{ hash_for_file filename } => #{ filename } (existing)" } File . open ( File . join ( '.esc' , '__manifest__' ), 'w' ) do | file | file_list . sort! . each { | entry | file . puts entry } end end ... end

Here we’re getting the deltas and writing them to the manifest file. We also added a helper method hash_for_file to return the hash of any file we pass in. I can see this coming in handy.

(retest)

........E Finished tests in 0.010894s, 826.1428 tests/s, 917.9365 assertions/s. 1) Error: test_0007_adds entries to the manifest file(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `manifest_contents' for #<CustomSourceControl:0x007f91e08542f0> custom_source_control.rb:146:in `block (3 levels) in <main>'

We’re going to need to read that manifest file back out, so let’s add that method.

class CustomSourceControl ... def manifest_contents ( manifest = nil ) manifest ||= '__manifest__' File . open ( ".esc/ #{ manifest } " , 'r' ) { | f | f . read } end ... end

(retest)

......... Finished tests in 0.006487s, 1387.3902 tests/s, 1695.6991 assertions/s.

If we take another look at this test we see @csc.write_manifest method call. Really this should be happening in the snapshot method itself. So let’s make that call in snapshot and remove it from the test.

class CustomSourceControl ... def snapshot File . new ( File . join ( '.esc' , '__metadata__' ), 'w' ) File . new ( File . join ( '.esc' , '__manifest__' ), 'w' ) write_manifest end ... end

(retest)

......... Finished tests in 0.014500s, 620.6897 tests/s, 758.6207 assertions/s.

Next we need to copy the files in the manifest to the repository directory

describe 'when we take a snapshot' do ... it 'copies files listed in the manifest to the repository' do @csc . copy_manifest_files_to_repository @csc . verify_manifest ( '__manifest__' ). must_equal true end end

(test)

..E....... Finished tests in 0.015868s, 630.1991 tests/s, 693.2191 assertions/s. 1) Error: test_0008_copies files listed in the manifest to the repository(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `copy_manifest_files_to_repository' for #<CustomSourceControl:0x007ff26103d120> custom_source_control.rb:155:in `block (3 levels) in <main>'

We’re introducing a few methods here so let’s take our time with this.

class CustomSourceControl ... def copy_manifest_files_to_repository end ... end

(retest)

..E....... Finished tests in 0.015557s, 642.7975 tests/s, 707.0772 assertions/s. 1) Error: test_0008_copies files listed in the manifest to the repository(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `verify_manifest' for #<CustomSourceControl:0x007fca9b03d1b8> custom_source_control.rb:159:in `block (3 levels) in <main>'

Add the verify_manifest helper method.

class CustomSourceControl ... def verify_manifest ( manifest = nil ) end ... end

(retest)

.........F Finished tests in 0.024386s, 410.0714 tests/s, 492.0856 assertions/s. 1) Failure: test_0008_copies files listed in the manifest to the repository(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:162]: Expected: true Actual: nil

Now that we’re failing, let’s start implementing the verify_manifest method.

class CustomSourceControl ... def verify_manifest ( manifest = nil ) manifest ||= '__manifest__' File . open ( File . join ( '.esc' , manifest )) do | file | repo_files = repository_file_list () file . readlines . each do | entry | return false unless repo_files . include? entry [ 0 ... 40 ] end end true end ... end

Here we’re reading the __manifest__ file, and for each entry we get the 40 character hash ( entry[0...40] ) and checking the repo_files array (file names) for it.

(retest)

....F..... Finished tests in 0.030663s, 326.1259 tests/s, 391.3511 assertions/s. 1) Failure: test_0008_copies files listed in the manifest to the repository(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:170]: Expected: true Actual: false

This time we’re returning false , and it makes sense since we’re not actually copying the files just yet. So let’s work on the implementing the copy_manifest_files_to_repository method.

class CustomSourceControl ... def copy_entry_to_repository ( manifest_entry ) FileUtils . cp ( manifest_entry [ :pathname ], File . join ( '.esc' , manifest_entry [ :hash ]), { :preserve => true }) end def copy_manifest_files_to_repository entries = [] File . open ( File . join ( '.esc' , '__manifest__' ), 'r' ) do | file | file . readlines . each do | entry | entry =~ /(.*?) => (.*?) \(new\)

?/ entries << { :hash => $1 , :pathname => $2 } if $1 end end entries . each do | entry | copy_entry_to_repository entry end end ... end

There is quite a bit going here. First, we open the __manifest__ file and break down each entry (line). What’s the deal with =~ , and what are $1 and $2 ? =~ is the match operator in ruby. It will match the variable on the left (string or regular expression) to a regular expression on the right. It returns nil if a match is not found, and the position of the match if found. Also if there is a match the $1 , $2 , …, $9 will represent the capture blocks (whatever is enclosed in the () ). That is how we break down the entry into a hash and pathname . For the actual copying we created a helper method copy_entry_to_repository .

(retest)

.......... Finished tests in 0.016832s, 594.1065 tests/s, 712.9278 assertions/s.

That was fun, wasn’t it? Let’s take the same approach and add the copy_manifest_files_to_repository call to the snapshot method. This will allow us to remove it from the test as well. Make sure you’re test still passes before moving on.

…it didn’t pass did it? We’re you able to figure out why? Did you attempt to fix it? Here is what I did. Based on the failure:

........F. Finished tests in 0.039832s, 251.0544 tests/s, 301.2653 assertions/s. 1) Failure: test_0005_gets a list of files in the current working directory(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:168]: --- expected +++ actual @@ -1 +1 @@ -["__manifest__", "__metadata__", "HEAD"] +["3b0930f8589a4eb37a1dbb9cbf355391781c2bba", "5d3140359919315ea06e3755cdc81860e9d7c556", "__manifest__", "__metadata__", "bb4d8995cfa843effc83d6ddcea1a8351c09497f", "HEAD"]

I went right to the gets a list of files in the current working directory test and saw that we’re only accounting for the HEAD file, which should always be in an initialized repository, and then the current working __manifest__ & __metadata__ files. This isn’t the case anymore since our snapshot method is doing more at this point. So what we really want is to make sure that at the point of this test at least those files exist. The must_include assertion provided by minitest is perfect for this.

Let’s update our gets a list of files in the current working directory test to the following:

describe 'when we take a snapshot' do ... it 'gets a list of files in the current working directory' do [ '__manifest__' , '__metadata__' , 'HEAD' ]. each do | filename | @csc . repository_file_list . must_include filename end end ... end

(retest)

.......... Finished tests in 0.026117s, 382.8924 tests/s, 650.9170 assertions/s.

Now we have to calculate the hash of the manifest file and rename it to the hash.

describe 'when we take a snapshot' do ... it 'calculates the hash of the manifest file and renames it to the hash' do manifest_hash = @csc . hash_for_file File . join ( '.esc' , '__manifest__' ) @csc . hash_and_copy_manifest @csc . repository_file_exists? ( manifest_hash ). must_equal true end end

(test)

.......E... Finished tests in 0.026070s, 421.9409 tests/s, 652.0905 assertions/s. 1) Error: test_0009_calculates the hash of the manifest file and renames it to the hash(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `hash_and_copy_manifest' for #<CustomSourceControl:0x007ffa4a0a7020>

class CustomSourceControl ... def hash_and_copy_manifest manifest_file = File . join ( '.esc' , '__manifest__' ) hash = hash_for_file ( manifest_file ) FileUtils . cp ( manifest_file , File . join ( '.esc' , hash )) hash end ... def repository_file_exists? ( filename = nil ) File . exists? File . join ( '.esc' , filename ) end ... end

We added another helper method repository_file_exists? . It simply takes a file name and checks the repository for existence of the filename.

(retest)

........... Finished tests in 0.039280s, 280.0407 tests/s, 458.2485 assertions/s.

Now that we’re passing, let’s add the hash_and_copy_manifest method to the snapshot method and remove @csc.hash_and_copy_manifest from the test. Make sure you’re passing and move on.

We’re almost there. Next, we have to update the metadata file with the necessary info, then hash it.

describe 'when we take a snapshot' do ... it 'adds the snapshot info to the metadata file, calculates its file hash, and renames it to the hash' do manifest_hash = @csc . hash_for_file File . join ( '.esc' , '__manifest__' ) @csc . write_metadata manifest_hash metadata_hash = @csc . hash_for_file File . join ( '.esc' , '__metadata__' ) @csc . hash_and_copy_metadata @csc . repository_file_exists? ( metadata_hash ). must_equal true end end

(test)

..........E. Finished tests in 0.033031s, 363.2951 tests/s, 544.9426 assertions/s. 1) Error: test_0010_adds the snapshot info to the metadata file, calculates its file hash, and renames it to the hash(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `write_metadata' for #<CustomSourceControl:0x007fc77c022f60>

class CustomSourceControl ... def hash_and_copy_metadata metadata_file = File . join ( '.esc' , '__metadata__' ) hash = hash_for_file ( metadata_file ) FileUtils . cp ( metadata_file , File . join ( '.esc' , hash )) hash end ... def write_metadata ( manifest_hash = nil ) File . open ( File . join ( '.esc' , '__metadata__' ), 'w' ) do | file | file . puts "Snapshot Manifest: #{ manifest_hash } " file . puts "Snapshot Parent: #{ ( head_contents . empty? ) ? 'root' : head_contents } " file . puts "Snapshot Taken: #{ Time . now } " end end ... end

(retest)

............ Finished tests in 0.034964s, 343.2102 tests/s, 543.4161 assertions/s.

Let’s clean up our test like we’ve done before. The @csc.write_metadata manifest_hash & @csc.hash_and_copy_metadata calls will happen in the snapshot methods so let’s delete them.

... it 'adds the snapshot info to the metadata file, calculates its file hash, and renames it to the hash' do manifest_hash = @csc . hash_for_file File . join ( '.esc' , '__manifest__' ) metadata_hash = @csc . hash_for_file File . join ( '.esc' , '__metadata__' ) @csc . repository_file_exists? ( metadata_hash ). must_equal true end ...

(test)

..F......... Finished tests in 0.076299s, 157.2760 tests/s, 249.0203 assertions/s. 1) Failure: test_0010_adds the snapshot info to the metadata file, calculates its file hash, and renames it to the hash(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:224]: Expected: true Actual: false

Now that we’re failing lets add the necessary calls to the snapshot method.

... def snapshot File . new ( File . join ( '.esc' , '__metadata__' ), 'w' ) File . new ( File . join ( '.esc' , '__manifest__' ), 'w' ) write_manifest copy_manifest_files_to_repository manifest_hash = hash_and_copy_manifest write_metadata manifest_hash hash_and_copy_metadata end ...

(retest)

............ Finished tests in 0.079316s, 151.2936 tests/s, 239.5481 assertions/s.

The last step for our snapshot story is to update HEAD to point to this snapshot (metadata filename)

describe 'when we take a snapshot' do ... it 'updates HEAD to the latest snapshot' do metadata_hash = @csc . hash_for_file File . join ( '.esc' , '__metadata__' ) @csc . update_head metadata_hash @csc . head_contents . must_equal metadata_hash end end

(test)

......E...... Finished tests in 0.042301s, 307.3213 tests/s, 449.1620 assertions/s. 1) Error: test_0011_updates HEAD to the latest snapshot(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `update_head' for #<CustomSourceControl:0x007ff5908eea58> custom_source_control.rb:231:in `block (3 levels) in <main>'

class CustomSourceControl ... def update_head ( metadata_hash = nil ) File . open ( File . join ( '.esc' , 'HEAD' ), 'w' ) do | file | file . write metadata_hash end end ... end

(retest)

............. Finished tests in 0.040271s, 322.8129 tests/s, 496.6353 assertions/s.

Now refactor the update_head out of the test.

it 'updates HEAD to the latest snapshot' do metadata_hash = @csc . hash_for_file File . join ( '.esc' , '__metadata__' ) @csc . head_contents . must_equal metadata_hash end

(retest)

..F.......... Finished tests in 0.164686s, 78.9381 tests/s, 121.4432 assertions/s. 1) Failure: test_0011_updates HEAD to the latest snapshot(CustomSourceControl::when we take a snapshot) [custom_source_control.rb:237]: --- expected +++ actual @@ -1 +1 @@ -"7a294d8483262c77b5d762a5325efb920e9948b6" +""

def snapshot File . new ( File . join ( '.esc' , '__metadata__' ), 'w' ) File . new ( File . join ( '.esc' , '__manifest__' ), 'w' ) write_manifest copy_manifest_files_to_repository manifest_hash = hash_and_copy_manifest write_metadata manifest_hash metadata_hash = hash_and_copy_metadata update_head metadata_hash end

(retest)

.F........... Finished tests in 0.048889s, 265.9085 tests/s, 409.0900 assertions/s. 1) Failure: test_0002_must create an empty HEAD file(CustomSourceControl::when a repository is initialized) [custom_source_control.rb:177]: Expected: true Actual: false

We expected the previous because we removed the update_head call and didn’t add it to snapshot . Then, we added the update_head to the snapshot method, but since that file is not empty anymore we’re failing our must create an empty HEAD file test. It looks like we’re going to have to refactor a bit more.

Let’s refactor the before block. We know all of our tests depend on @csc to be an instance of CustomSourceControl and they all need an initialized repository. The thing is our when a repository is initialized tests don’t require a snapshot . So let’s move that out and into a before block inside the when we take a snapshot tests.

describe CustomSourceControl do ... before do @csc = CustomSourceControl . new @csc . initialize_repository end ... describe 'when we take a snapshot' do before do @csc . snapshot end ... end end

(retest)

............. Finished tests in 0.040347s, 322.2049 tests/s, 495.6998 assertions/s.

…and our snapshot story is complete! On to our final story!

Checkout

Checkout previous snapshots As a user I want to checkout a previous snapshot So that I can fix issues or get back to a working state

At some point we’re going to need a way to list all the snapshots csc knows about. One way to do this would be to get the HEAD snapshot then recursively scan through the metadata files and their parents all the way up to root, then just list them out. This might end up in a log subcommand. For now, I am trying to keep the functionality really basic. I am going to manually build up the repository with 2 snapshots, then pick the first snapshot to checkout.

To keep this testable, we’ll do this with a before block for this set of tests.

describe CustomSourceControl do ... describe 'when we checkout a previous snapshot' do before do @csc . snapshot end end end

So that’s going to create the first snapshot. I am going to use the pry gem to suspend the test so that I can manually inspect the .esc directory. If you use it, make sure to type quit when you’re finished inspecting things.

describe 'when we checkout a previous snapshot' do ... it 'suspends the test so we can inspect the .esc directory' do require 'pry' ; binding . pry skip end end

There is a way you can accomplish this without having to install a gem. Add a call to ruby’s sleep method with a time of something long enough for you to carry the tasks out for yourself. That would look like this:

describe 'when we checkout a previous snapshot' do ... it 'suspends the test so we can inspect the .esc directory' do sleep 60 * 60 skip end end

* Be sure to clean up after yourself by deleting the suspends the test so we can inspect the .esc directory test when you are finished.

Getting the hashes:

ls -ltr .esc

-rw-r--r-- 1 mattweppler staff 25 Mar 8 11:01 bb4d8995cfa843effc83d6ddcea1a8351c09497f -rw-r--r-- 1 mattweppler staff 25 Mar 8 11:02 5d3140359919315ea06e3755cdc81860e9d7c556 -rwxr-xr-x 1 mattweppler staff 7351 Mar 8 14:43 2c114f49e4935865fb00903dba1df2ba70283748 -rw-r--r-- 1 mattweppler staff 129 Mar 8 14:44 b696627a90073c3d3870a30ac5c8140f853a6b3e -rw-r--r-- 1 mattweppler staff 129 Mar 8 14:44 __metadata__ -rw-r--r-- 1 mattweppler staff 207 Mar 8 14:44 __manifest__ -rw-r--r-- 1 mattweppler staff 40 Mar 8 14:44 HEAD -rw-r--r-- 1 mattweppler staff 207 Mar 8 14:44 87b17efdc68c9c1d806c4bd05ce70d9baacd22bf

So if we inspect HEAD we see the metadata file hash.

cat .esc/HEAD

b696627a90073c3d3870a30ac5c8140f853a6b3e

We then take that file hash, which is the metadata file and inspect that:

cat .esc/b696627a90073c3d3870a30ac5c8140f853a6b3e

Snapshot Manifest: 87b17efdc68c9c1d806c4bd05ce70d9baacd22bf Snapshot Parent: root Snapshot Taken: 2014-03-08 14:44:10 -0800

This shows that this is the first snapshot as denoted by the Snapshot Parent: root . So let’s take a look at the manifest next Snapshot Manifest: 87b17efdc68c9c1d806c4bd05ce70d9baacd22bf

cat .esc/87b17efdc68c9c1d806c4bd05ce70d9baacd22bf

2c114f49e4935865fb00903dba1df2ba70283748 => custom_source_control.rb (new) 5d3140359919315ea06e3755cdc81860e9d7c556 => test_file_2.txt (new) bb4d8995cfa843effc83d6ddcea1a8351c09497f => test_file_1.txt (new)

You can quit pry now. If you used the sleep method do these tasks and it still hasn’t woke up and finished just hit ctrl+c to kill the tests. Let’s add a new file test_file_3.txt and update an existing one test_file_2.txt :

... before do @csc . snapshot # make some edits File . open ( 'test_file_3.txt' , 'w' ) do | file | file . write "this is test_file_3.text

" end File . open ( 'test_file_2.txt' , 'a' ) do | file | file . write "this is an update to test_file_2.text

" end # take our second snapshot @csc . snapshot end ...

We’re also going to want to clean up after ourselves again:

... after do File . open ( 'test_file_2.txt' , 'w' ) do | file | file . write "this is test_file_2.text

" end File . delete ( 'test_file_3.txt' ) end ...

Ok let’s run the test again and this time make note of the hashes. For me, HEAD is 35d91f744401d8d4828c65bd65029dc07119d5a7 . The metadata file ( 35d91f744401d8d4828c65bd65029dc07119d5a7 ) shows:

Snapshot Manifest: 14c86c6a758c197367d417b32d57433446fa63e4 Snapshot Parent: 36e0583c25d5e8107538afa345122e9529b9d6fd Snapshot Taken: 2014-03-08 14:50:32 -0800

So let’s take the parent metadata 36e0583c25d5e8107538afa345122e9529b9d6fd and take a look:

Snapshot Manifest: 1b1ab1bef308608786e9a1ae2e30e370dd032939 Snapshot Parent: root Snapshot Taken: 2014-03-08 14:50:32 -0800

Yep, 1b1ab1bef308608786e9a1ae2e30e370dd032939 that’s the one we want. Just to be sure, I ran through this process a few more times. I noticed that 2 of the hashes were changing, while the remaining files stayed the same. So I opened one of the files where the file hash had changed. I spotted the issue right away… The timestamp! Since the timestamp changed each time I ran it I was not getting a consistent set of hashes. In the spirit of keeping it simple, I am just going to change the timestamp to a constant value ‘2014-03-07 23:59:59 -0800’. This may seem hacky, and it is. :)

class CustomSourceControl ... def write_metadata ( manifest_hash = nil ) File . open ( File . join ( '.esc' , '__metadata__' ), 'w' ) do | file | file . puts "Snapshot Manifest: #{ manifest_hash } " file . puts "Snapshot Parent: #{ ( head_contents . empty? ) ? 'root' : head_contents } " file . puts "Snapshot Taken: #{ '2014-03-07 23:59:59 -0800' || Time . now } " end end ... end

This time, our consistent hash is 3b9158d6cd90b07811496330d873d8a71651cd8b .

Snapshot Manifest: fba7362f34266e6d491c48396f936a8ed4bc5c72 Snapshot Parent: root Snapshot Taken: 2014-03-03 23:59:59 -0800

We can remove the suspends the test so we can inspect the .esc directory test.

... it 'copies files from the manifest into the current working directory' do @csc . checkout '3b9158d6cd90b07811496330d873d8a71651cd8b' restored_hash = @csc . hash_for_file 'test_file_2.txt' restored_hash . must_equal '5d3140359919315ea06e3755cdc81860e9d7c556' end ...

You know the drill by now.

(retest)

..E........... Finished tests in 0.052594s, 266.1901 tests/s, 380.2715 assertions/s. 1) Error: test_0001_copies files from the manifest into the current working directory(CustomSourceControl::when we checkout a previous snapshot): NoMethodError: undefined method `checkout' for #<CustomSourceControl:0x007feb59034bb0>

class CustomSourceControl ... def checkout ( snapshot = nil ) end ... end

(retest)

..F........... Finished tests in 0.056402s, 248.2181 tests/s, 372.3272 assertions/s. 1) Failure: test_0001_copies files from the manifest into the current working directory(CustomSourceControl::when we checkout a previous snapshot) [custom_source_control.rb:274]: --- expected +++ actual @@ -1 +1 @@ -"5d3140359919315ea06e3755cdc81860e9d7c556" +"89cdd7bfa8c31d8a23a61d6b695b762c7f588bee"

class CustomSourceControl ... def checkout ( snapshot = nil ) manifest_hash = '' File . open ( File . join ( '.esc' , snapshot ), 'r' ) do | file | file . readlines . each do | entry | manifest_hash = $1 if entry =~ /Snapshot Manifest: (\w{40})/ end end entries = [] File . open ( File . join ( '.esc' , manifest_hash ), 'r' ) do | file | file . readlines . each do | entry | entry =~ /(.*?) => (.*?) \(new\)

?/ entries << { :hash => $1 , :pathname => $2 } if $1 end end entries . each do | entry | copy_entry_to_working_directory entry end end ... def copy_entry_to_working_directory ( entry ) FileUtils . cp ( File . join ( '.esc' , entry [ :hash ]), entry [ :pathname ], { :preserve => true }) end ... end

Here we’re reading the metadata file and getting manifest hash. Then we’re reading the manifest file and breaking down the entries again, this time calling copy_entry_to_working_directory method to copy the files from the repository to the current working directory.

(retest)

..E........... Finished tests in 0.041946s, 333.7625 tests/s, 476.8035 assertions/s. 1) Error: test_0001_copies files from the manifest into the current working directory(CustomSourceControl::when we checkout a previous snapshot): Errno::ENOENT: No such file or directory - .esc/3b9158d6cd90b07811496330d873d8a71651cd8b

Ugh… another issue with the hashing. Not so much the hashing actually, but the fact that we’re editing the very file we’re trying to code/test custom_source_control.rb . This test can never pass. So what are, our options? Well, the first that comes to mind is to just run the script from another directory. We can do this by adding the directory we’re working in to our path. This actually would’ve solved an issue we faced earlier as well. However, I tried to avoid it to keep things simple.

First, let’s skip the current failing test.

(retest)

..S........... Finished tests in 0.115535s, 121.1754 tests/s, 173.1077 assertions/s.

Ok, we’re passing again so we can restructure a bit. Let’s get the current working directory.

pwd

/Users/mattweppler/developer/projects/custom_source_control

Now let’s add it to our path so we can execute it from a different directory. We can even bring our command a little closer to the command Jim Weirich mentions: csc , by creating a symlink. Create a new directory (it can even be within the current directory), I am calling it test_dir . Then, let’s move the test files into the test_dir and change into that directory.

export PATH=$ PATH:/Users/mattweppler/developer/projects/custom_source_control ln -s $ HOME/developer/projects/custom_source_control/custom_source_control.rb $HOME /developer/projects/custom_source_control/csc mkdir test_dir mv test_file_* test_dir cd test_dir

Some of our tests will fail since I did a few hackish things here and there. Again, I was trying to cut down on the amount of possible new concepts. Oh well… Rerun the tests and let’s see what we get.

csc

(retest)

..S....F.E.... Finished tests in 0.178928s, 78.2438 tests/s, 106.1880 assertions/s. 1) Failure: test_0003_gets_a_list_of_files_in_the_current_working_directory(CustomSourceControl::when we take a snapshot) [/Users/mattweppler/developer/projects/custom_source_control/csc:218]: --- expected +++ actual @@ -1 +1 @@ -["custom_source_control.rb", "test_file_1.txt", "test_file_2.txt"] +["test_file_1.txt", "test_file_2.txt"] 2) Error: test_0007_adds_entries_to_the_manifest_file(CustomSourceControl::when we take a snapshot): NoMethodError: undefined method `chomp' for nil:NilClass /Users/mattweppler/developer/projects/custom_source_control/csc:245:in `block (3 levels) in <main>'

Ok so what are we working with here. Well we no longer need to account for the custom_source_control.rb file. Let’s go update that. So this:

... it 'gets a list of files in the current working directory' do @csc . cwd_files . must_equal [ 'custom_source_control.rb' , 'test_file_1.txt' , 'test_file_2.txt' ] end ...

becomes this:

... it 'gets a list of files in the current working directory' do @csc . cwd_files . must_equal [ 'test_file_1.txt' , 'test_file_2.txt' ] end ...

We can also remove the keep_if ’s since we were trying to guard against anything but our control files ( test_file_1.txt , test_file_2.txt ). So this:

... it 'creates a file hash for all files in the current working directory' do actual_hashes = { 'test_file_1.txt' => 'bb4d8995cfa843effc83d6ddcea1a8351c09497f' , 'test_file_2.txt' => '5d3140359919315ea06e3755cdc81860e9d7c556' } expected_hashes = @csc . cwd_hashes . keep_if { | key , value | key == 'test_file_1.txt' || key == 'test_file_2.txt' } expected_hashes . must_equal actual_hashes end ...

becomes this:

... it 'creates a file hash for all files in the current working directory' do actual_hashes = { 'test_file_1.txt' => 'bb4d8995cfa843effc83d6ddcea1a8351c09497f' , 'test_file_2.txt' => '5d3140359919315ea06e3755cdc81860e9d7c556' } @csc . cwd_hashes . must_equal actual_hashes end ...

and this:

... it 'returns a list of new and existing files' do deltas = @csc . deltas deltas [ :new ]. keep_if { | key , value | key == 'test_file_1.txt' || key == 'test_file_2.txt' } deltas [ :new ]. must_equal [ 'test_file_1.txt' , 'test_file_2.txt' ] deltas [ :existing ]. must_equal [] end ...

becomes this:

... it 'returns a list of new and existing files' do deltas = @csc . deltas deltas [ :new ]. must_equal [ 'test_file_1.txt' , 'test_file_2.txt' ] deltas [ :existing ]. must_equal [] end ...

Lastly this:

... it 'adds entries to the manifest file' do expected_content = %Q{5d3140359919315ea06e3755cdc81860e9d7c556 => test_file_2.txt (new)

bb4d8995cfa843effc83d6ddcea1a8351c09497f => test_file_1.txt (new)} @csc . manifest_contents . gsub! ( /.*? => custom_source_control\.rb \(new\)

?/ , '' ). chomp . must_equal expected_content end ...

becomes this:

... it 'adds entries to the manifest file' do expected_content = %Q{5d3140359919315ea06e3755cdc81860e9d7c556 => test_file_2.txt (new)

bb4d8995cfa843effc83d6ddcea1a8351c09497f => test_file_1.txt (new)} manifest_contents = @csc . manifest_contents if manifest_contents =~ /.*? => custom_source_control\.(?:md|rb) \(new\)

?/ manifest_contents . gsub! ( /.*? => custom_source_control\.(?:md|rb) \(new\)

?/ , '' ). chomp . must_equal expected_content else manifest_contents . chomp . must_equal expected_content end end ...

(retest)

..S........... Finished tests in 0.032450s, 431.4330 tests/s, 616.3328 assertions/s.

…and we’re back to passing! Let’s remove that skip statement and continue working on that last test. If you recall, we need to suspend the test long enough so that we can go through the metadata files and get our root snapshot.

For me, that hash is 485ac882b4e89e929584acdfed522499f0a45464 . With that let’s update the test and run it.

... it 'copies files from the manifest into the current working directory' do @csc . checkout '485ac882b4e89e929584acdfed522499f0a45464' restored_hash = @csc . hash_for_file 'test_file_2.txt' restored_hash . must_equal '5d3140359919315ea06e3755cdc81860e9d7c556' end ...

For the win…

(retest)

.............. Finished tests in 0.040760s, 343.4740 tests/s, 515.2110 assertions/s.

We… are… passing! Good job! I really enjoyed writing this post. I hope this was helpful for you. Just a few last things before you go.

How do I use this thing now that it’s built?

Well, while we have the methods to handle some of the functionality, we haven’t added the ability to pass arguments on the command line. You can add something very simple like the following code:

First change the require 'minitest/autorun' to require 'minitest/spec' and add the following to the bottom of the file.

... def main unless [ 'initialize' , 'snapshot' , 'checkout' ]. include? ARGV [ 0 ] puts " #{ ARGV [ 0 ] } is not a subcommand." exit 1 end csc = CustomSourceControl . new case ARGV [ 0 ] when 'initialize' csc . initialize_repository when 'snapshot' csc . snapshot when 'checkout' if ARGV [ 1 ] csc . checkout ARGV [ 1 ] else puts "'checkout' subcommand takes a second argument, SHA1 of the metadata file to checkout." exit 1 end end end main if __FILE__ == $0

You would then be able to call it from the command line like this:

csc initialize

tree . ├── .esc/ │ └── HEAD ├── test_file_1.txt └── test_file_2.txt

csc snapshot

tree . ├── .esc/ │ ├── 485ac882b4e89e929584acdfed522499f0a45464 │ ├── 5d3140359919315ea06e3755cdc81860e9d7c556 │ ├── 74aec4d0ab199369fc8fe3fd38a9e1459678b2ea │ ├── HEAD │ ├── __manifest__ │ ├── __metadata__ │ └── bb4d8995cfa843effc83d6ddcea1a8351c09497f ├── test_file_1.txt └── test_file_2.txt

cat > test_file_3.txt

this is test_file_3.text

cat > > test_file_2.txt

this is an update to test_file_2.text

cat test_file_2.txt

this is test_file_2.text this is an update to test_file_2.text

tree

. ├── .esc/ │ ├── 485ac882b4e89e929584acdfed522499f0a45464 │ ├── 5d3140359919315ea06e3755cdc81860e9d7c556 │ ├── 74aec4d0ab199369fc8fe3fd38a9e1459678b2ea │ ├── HEAD │ ├── __manifest__ │ ├── __metadata__ │ └── bb4d8995cfa843effc83d6ddcea1a8351c09497f ├── test_file_1.txt ├── test_file_2.txt └── test_file_3.txt

csc snapshot

tree

. ├── .esc/ │ ├── 485ac882b4e89e929584acdfed522499f0a45464 │ ├── 5d3140359919315ea06e3755cdc81860e9d7c556 │ ├── 6ace300838a9818ec987a0e483b7a3ae598afe7f │ ├── 74aec4d0ab199369fc8fe3fd38a9e1459678b2ea │ ├── 89cdd7bfa8c31d8a23a61d6b695b762c7f588bee │ ├── HEAD │ ├── __manifest__ │ ├── __metadata__ │ ├── bb4d8995cfa843effc83d6ddcea1a8351c09497f │ ├── bdb94f80a11a03f2f739faa297b3dc219df93e0c │ └── e9e0ddd6a9d8f998c41fb83978942e1021f21cac ├── test_file_1.txt ├── test_file_2.txt └── test_file_3.txt

cat test_file_2.txt

this is test_file_2.text this is an update to test_file_2.text

cat .esc/HEAD

e9e0ddd6a9d8f998c41fb83978942e1021f21cac

cat .esc/e9e0ddd6a9d8f998c41fb83978942e1021f21cac

Snapshot Manifest: bdb94f80a11a03f2f739faa297b3dc219df93e0c Snapshot Parent: 485ac882b4e89e929584acdfed522499f0a45464 Snapshot Taken: 2014-03-07 23:59:59 -0800

csc checkout 485ac882b4e89e929584acdfed522499f0a45464

cat test_file_2.txt

this is test_file_2.text

You should notice the timestamp still shows 2014-03-07 23:59:59 -0800 . You can remove that line of code, but the tests will fail again.

. You can remove that line of code, but the tests will fail again. We don’t really clean up after ourselves so that functionality needs to be added.

Getting the checkout hash is also a manual process so that csc log functionality we talked about would come in handy.

functionality we talked about would come in handy. We are not handling any types of errors mind you

…so its not quite production ready.

What do I do next?

Some of the things I’d like to address in a future post include:

Separating the tests from the actual implementation code.

DRY’ing out our code. Many times I have had to fight the urge to do it in this post. I really wanted this information to be approachable by anyone though, so I didn’t use any gems, even minitest/given which was created by Jim Weirich.

which was created by Jim Weirich. Testing for more edge cases, and fixing any bugs we find.

Adding code coverage.

Adding the ability to handle command line arguments with OptionParser .

. Adding tests and functionality to diff checkins.

Adding tests and functionality to list the history (metadata file hashes from head all the way back to root)

Possibly turning this into a gem.

Lastly I’d like to thank a few people for helping with this post. Austin Puri, thanks for running through this as a developer and giving some great feedback. Devon Mahnken, for catching a lot of spelling and English grammar mistakes. After all your corrections, for the first time I think my father was right about me being a robot. Really appreciate the help guys!

You can double check your work with the project I have hosted on github

You may have some questions that this didn’t quite answer. Feel free to email me or leave a comment.