Sunday, 23 May 2010

Visualize perforce change log with gource

Having seen amazing videos of gource I was keen to vizualize the changelog of the codebase I work on. The only trouble was that gource does not understand perforce directly. Fortunately, gource can read text files where each line is a change to one file in the format "timestamp|author|filename|action|colour".

Perforce logs produced by `p4 describe -s` command have the following format:
Change 106 by max@max_perforce on 2008/10/20 16:17:31
mt makefiles

Affected files ...

... //depot/infra/main/src/infra/Makefile#3 edit
... //depot/infra/main/src/infra/Makefile.mt#3 delete

Which should be converted for gource to:
1224515851|max|M|//depot/infra/main/src/infra/Makefile|
1224515851|max|D|//depot/infra/main/src/infra/Makefile.mt|
It is a two-stage process to convert perforce logs to gource logs. The first stage is to extract perforce logs into a file, because it takes a while and you may like to make several passes over it. It can be done with a couple of lines of bash:
[max@truth etc]$ p4 changes -m 1 | awk '{ print $2 }'
17146
[max@truth etc]$ for ((i=1; i<17146; ++i)); do p4 describe -s $i >> p4.log; done
The second stage is to filter and convert perforce changelog into gsource log. Here is how to convert changes only in //depot/infra/main with the python script I wrote:
[max@truth etc]$ ./p4-gource.py -p //depot/infra/main -o main.gource p4.log
Now gource can be used to vizualize the changelog:
[max@truth etc]$ gource --highlight-all-users main.gource
Enjoy!

Download p4-gource.py.

14 comments:

  1. Hi Max,

    What is the final output of these scripts? We use perforce too, so i am interested to see if your script would give us something cool.

    Cheers,
    Ivan

    ReplyDelete
  2. Hi Ivan,

    Scripts input is perforce changelog. The output is the changelog in gource format. The gource changelog is input to gource, which renders an animation visualizing the history of your depot. Have you seen the videos mentioned in the post?

    ReplyDelete
  3. Good post - Helped out a lot. One thing that caused me to stumble (given my lack of Python experience) is that the blog strips out the group name in the regex, so when I simply cut and paste the python code it didn't work.

    ReplyDelete
  4. Thanks. And yes, I am struggling with formatting of this. It is switching between HTML and WYSIWYG mode in the online editor that breaks my formatting. I should probably try using a decent HTML editor...

    ReplyDelete
  5. Maxim, you need to add the perforce 'purge' command, I just mapped it to the gource delete 'D' command.

    p4_action_to_gource = {
    "add" : "A"
    , "edit" : "M"
    , "integrate" : "M"
    , "branch" : "A"
    , "delete" : "D"
    , "purge" : "D"
    }

    ReplyDelete
  6. At line 75 the assignment "file=gource_file" is not right, is it? How should it be?

    ReplyDelete
  7. @Kostas: it is python-3 style print function. http://docs.python.org/library/functions.html#print

    ReplyDelete
  8. Thanks a LOT for this script! It really helped me out!

    Cheers!

    ReplyDelete
  9. Fantastic! Two years later and this post is still incredibly useful. Thanks a lot, and job well done!

    ReplyDelete
  10. I use git-p4 to get the same result, just import the part of the depot you want and gource will work out of the box

    ReplyDelete
  11. This was very cool, thanks a lot for this! :)

    ReplyDelete
  12. Hi there- I am trying to get in touch- I saw you had run into an SSL bug with FTPSSL .. Did you solve it for Python 2.7, and how?
    http://bugs.python.org/issue8108

    Thanks, I'm working on an open-source Dropbox clone that uses FTP:
    https://code.google.com/p/iqbox-ftp/

    ReplyDelete
  13. Hi Max,
    Trying to use your script resulted in a datestamp error for me.
    Debugging it the first regex never matched because our usernames and therefore the use a dot in their names eg. "m.yegorushin" - since your regex uses a \w+ the dot didn't match - changing it to \S+ fixed that problem for me.

    ReplyDelete