Sam Panzer | 1 Aug 03:34 2012
Picon

AST Matchers tutorial

Hello, again!


As suggested by a few people, I have been writing up my experiences starting development on Clang, and specifically working with Clang-based refactoring tools. The repository where my loop converter will live doesn't exist yet, so I will use the tutorial as a chance to start offering patches for review (they can be rebased to other locations later).

My intention was to begin assuming zero Clang experience and work towards building a slightly simplified version of the loop converter through a series of fully-explained patches to Clang to a unit-tested functional tool. Because the Getting Started page is aimed at people who want to use Clang rather than develop for it, I began with cloning clang+llvm from the git mirrors and preparing a cmake+ninja build.

What I have so far is currently available as a Google Doc here: https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
If enough people think that the tutorial is useful, I can eventually convert it to a nicer format (i.e. HTML with prettier code formatting). I expect to have made mistakes, especially in the section about obtaining and building Clang :)

Thoughts?
-Sam
Attachment (loop-converter-tutorial.patch): application/octet-stream, 47 KiB
_______________________________________________
cfe-dev mailing list
cfe-dev@...
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Sean Silva | 1 Aug 05:09 2012
Picon

Re: AST Matchers tutorial

Thank you for writing this up! :-)

A couple things:

Since you're assuming zero Clang experience, I think you can leave out
the git-svn stuff (since AFAIK that is only relevant for people with
commit access). A simpler way to just checkout LLVM/Clang is

$ git clone http://llvm.org/git/llvm.git
$ cd source/tools
$ git clone http://llvm.org/git/clang.git

The git-svn checkout stuff that you are doing looks kind of fearsome.
It's easier for a newcomer to just use a pure git workflow.

Also, you may want to touch on the scenario that a person already has
CMake installed. E.g., that it should be a certain version, needs to
be compiled with certain flags, etc.

> Ninja dependency resolver.

I would call this "Ninja build tool".

You should probably link to the LLVM commandline library docs
<http://llvm.org/docs/CommandLine.html> in the relevant part of the
document.

> the loop’s index variable is compared against N - 1

generally aren't half-open loops comparing against N, and not N-1? e.g.
for (int I = 0, N = 10; I != N; ++I)

I think that just a little bit more explanation about the use of
FoldingSetNodeID in areSameExpr() would be good. (just explain roughly
what it does, links to docs, etc)

As far as your patch. My gut seems inclined to have this inside the
examples/ subdirectory, since the code is really just example code to
accompany the tutorial.

That's all for now. Thank you so much for the extensive docs!

--Sean.

On Tue, Jul 31, 2012 at 6:34 PM, Sam Panzer <panzer@...> wrote:
> Hello, again!
>
> As suggested by a few people, I have been writing up my experiences starting
> development on Clang, and specifically working with Clang-based refactoring
> tools. The repository where my loop converter will live doesn't exist yet,
> so I will use the tutorial as a chance to start offering patches for review
> (they can be rebased to other locations later).
>
> My intention was to begin assuming zero Clang experience and work towards
> building a slightly simplified version of the loop converter through a
> series of fully-explained patches to Clang to a unit-tested functional tool.
> Because the Getting Started page is aimed at people who want to use Clang
> rather than develop for it, I began with cloning clang+llvm from the git
> mirrors and preparing a cmake+ninja build.
>
> What I have so far is currently available as a Google Doc here:
> https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
> If enough people think that the tutorial is useful, I can eventually convert
> it to a nicer format (i.e. HTML with prettier code formatting). I expect to
> have made mistakes, especially in the section about obtaining and building
> Clang :)
>
> Thoughts?
> -Sam
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev@...
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
Sam Panzer | 1 Aug 20:07 2012
Picon

Re: AST Matchers tutorial

I've enabled comments on the Google Doc to help avoid polluting the mailing list too much :)


On Tue, Jul 31, 2012 at 8:09 PM, Sean Silva <silvas-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> wrote:

Thank you for writing this up! :-)

A couple things:

Since you're assuming zero Clang experience, I think you can leave out
the git-svn stuff (since AFAIK that is only relevant for people with
commit access). A simpler way to just checkout LLVM/Clang is

$ git clone http://llvm.org/git/llvm.git
$ cd source/tools
$ git clone http://llvm.org/git/clang.git

The git-svn checkout stuff that you are doing looks kind of fearsome.
It's easier for a newcomer to just use a pure git workflow.

Good point! I meant to ask about that.
 

Also, you may want to touch on the scenario that a person already has
CMake installed. E.g., that it should be a certain version, needs to
be compiled with certain flags, etc.

On another discussion thread you mentioned that CMake needed to be compiled with an extra flag. I don't happen to know the details beyond what is written, since I set up my environment once, two months ago :)
Would someone be able to list what's necessary and when?
 

> Ninja dependency resolver.

I would call this "Ninja build tool".

Done.
 
You should probably link to the LLVM commandline library docs
<http://llvm.org/docs/CommandLine.html> in the relevant part of the
document.


Link added.
 
> the loop’s index variable is compared against N - 1

generally aren't half-open loops comparing against N, and not N-1? e.g.
for (int I = 0, N = 10; I != N; ++I)


Yes, that's correct and actually what the code implements.
 

I think that just a little bit more explanation about the use of
FoldingSetNodeID in areSameExpr() would be good. (just explain roughly
what it does, links to docs, etc)

I added a few links, along with what I understood from the FoldingSetNodeID and Stmt::Profile() documentation.
 

As far as your patch. My gut seems inclined to have this inside the
examples/ subdirectory, since the code is really just example code to
accompany the tutorial.

Actually, this really is intended as the start of the loop migration tool I've been working on. I didn't just want to dump 3000 lines of code and tests in one patch, though this set is far more fine-grained than I would use for future patches. That aside, it would make sense for the tutorial to live in examples/.
 

That's all for now. Thank you so much for the extensive docs!

--Sean.

On Tue, Jul 31, 2012 at 6:34 PM, Sam Panzer <panzer-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> Hello, again!
>
> As suggested by a few people, I have been writing up my experiences starting
> development on Clang, and specifically working with Clang-based refactoring
> tools. The repository where my loop converter will live doesn't exist yet,
> so I will use the tutorial as a chance to start offering patches for review
> (they can be rebased to other locations later).
>
> My intention was to begin assuming zero Clang experience and work towards
> building a slightly simplified version of the loop converter through a
> series of fully-explained patches to Clang to a unit-tested functional tool.
> Because the Getting Started page is aimed at people who want to use Clang
> rather than develop for it, I began with cloning clang+llvm from the git
> mirrors and preparing a cmake+ninja build.
>
> What I have so far is currently available as a Google Doc here:
> https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
> If enough people think that the tutorial is useful, I can eventually convert
> it to a nicer format (i.e. HTML with prettier code formatting). I expect to
> have made mistakes, especially in the section about obtaining and building
> Clang :)
>
> Thoughts?
> -Sam
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev-Tmj1lob9twqVc3sceRu5cw@public.gmane.org
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

_______________________________________________
cfe-dev mailing list
cfe-dev@...
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Sean Silva | 2 Aug 05:11 2012
Picon

Re: AST Matchers tutorial

> On another discussion thread you mentioned that CMake needed to be compiled
> with an extra flag. I don't happen to know the details beyond what is
> written, since I set up my environment once, two months ago :)
> Would someone be able to list what's necessary and when?

I believe that you need CMake >=2.8.8 for Ninja, and currently need to
build CMake with -DCMAKE_ENABLE_NINJA on Mac (although I think the
bleeding edge version doesn't need this). I still haven't found a
time/place to actually follow all the steps in the tutorial (I've only
read it), but once I do (it might be a while unfortunately), I'll tell
you if anything didn't go well.

> Actually, this really is intended as the start of the loop migration tool
> I've been working on. I didn't just want to dump 3000 lines of code and
> tests in one patch, though this set is far more fine-grained than I would
> use for future patches. That aside, it would make sense for the tutorial to
> live in examples/.

I think that it is perfectly fine to have the code for the tutorial be
in examples/, while the "real tool" would be somewhere else.

--Sean Silva

On Wed, Aug 1, 2012 at 11:07 AM, Sam Panzer <panzer@...> wrote:
> I've enabled comments on the Google Doc to help avoid polluting the mailing
> list too much :)
>
> On Tue, Jul 31, 2012 at 8:09 PM, Sean Silva <silvas@...> wrote:
>>
>>
>> Thank you for writing this up! :-)
>>
>> A couple things:
>>
>> Since you're assuming zero Clang experience, I think you can leave out
>> the git-svn stuff (since AFAIK that is only relevant for people with
>> commit access). A simpler way to just checkout LLVM/Clang is
>>
>> $ git clone http://llvm.org/git/llvm.git
>> $ cd source/tools
>> $ git clone http://llvm.org/git/clang.git
>>
>> The git-svn checkout stuff that you are doing looks kind of fearsome.
>> It's easier for a newcomer to just use a pure git workflow.
>
>
> Good point! I meant to ask about that.
>
>>
>>
>> Also, you may want to touch on the scenario that a person already has
>> CMake installed. E.g., that it should be a certain version, needs to
>> be compiled with certain flags, etc.
>
>
> On another discussion thread you mentioned that CMake needed to be compiled
> with an extra flag. I don't happen to know the details beyond what is
> written, since I set up my environment once, two months ago :)
> Would someone be able to list what's necessary and when?
>
>>
>>
>> > Ninja dependency resolver.
>>
>> I would call this "Ninja build tool".
>
>
> Done.
>
>>
>> You should probably link to the LLVM commandline library docs
>> <http://llvm.org/docs/CommandLine.html> in the relevant part of the
>> document.
>>
>
> Link added.
>
>>
>> > the loop’s index variable is compared against N - 1
>>
>> generally aren't half-open loops comparing against N, and not N-1? e.g.
>> for (int I = 0, N = 10; I != N; ++I)
>>
>
> Yes, that's correct and actually what the code implements.
>
>>
>>
>> I think that just a little bit more explanation about the use of
>> FoldingSetNodeID in areSameExpr() would be good. (just explain roughly
>> what it does, links to docs, etc)
>
>
> I added a few links, along with what I understood from the FoldingSetNodeID
> and Stmt::Profile() documentation.
>
>>
>>
>> As far as your patch. My gut seems inclined to have this inside the
>> examples/ subdirectory, since the code is really just example code to
>> accompany the tutorial.
>
>
> Actually, this really is intended as the start of the loop migration tool
> I've been working on. I didn't just want to dump 3000 lines of code and
> tests in one patch, though this set is far more fine-grained than I would
> use for future patches. That aside, it would make sense for the tutorial to
> live in examples/.
>
>>
>>
>> That's all for now. Thank you so much for the extensive docs!
>>
>> --Sean.
>>
>> On Tue, Jul 31, 2012 at 6:34 PM, Sam Panzer <panzer@...> wrote:
>> > Hello, again!
>> >
>> > As suggested by a few people, I have been writing up my experiences
>> > starting
>> > development on Clang, and specifically working with Clang-based
>> > refactoring
>> > tools. The repository where my loop converter will live doesn't exist
>> > yet,
>> > so I will use the tutorial as a chance to start offering patches for
>> > review
>> > (they can be rebased to other locations later).
>> >
>> > My intention was to begin assuming zero Clang experience and work
>> > towards
>> > building a slightly simplified version of the loop converter through a
>> > series of fully-explained patches to Clang to a unit-tested functional
>> > tool.
>> > Because the Getting Started page is aimed at people who want to use
>> > Clang
>> > rather than develop for it, I began with cloning clang+llvm from the git
>> > mirrors and preparing a cmake+ninja build.
>> >
>> > What I have so far is currently available as a Google Doc here:
>> >
>> > https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
>> > If enough people think that the tutorial is useful, I can eventually
>> > convert
>> > it to a nicer format (i.e. HTML with prettier code formatting). I expect
>> > to
>> > have made mistakes, especially in the section about obtaining and
>> > building
>> > Clang :)
>> >
>> > Thoughts?
>> > -Sam
>> >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > cfe-dev@...
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>
>
Sean Silva | 1 Aug 05:14 2012
Picon

Re: AST Matchers tutorial

Thank you for writing this up! :)

A couple things:

Since you're assuming zero Clang experience, I think you can leave out
the git-svn stuff (since AFAIK that is only relevant for people with
commit access). A simpler way to just checkout LLVM/Clang is

$ git clone http://llvm.org/git/llvm.git
$ cd source/tools
$ git clone http://llvm.org/git/clang.git

The git-svn checkout stuff that you are doing looks kind of fearsome.
It's easier for a newcomer to just use pure git workflow.

Also, you may want to touch on the scenario that a person already has
CMake installed. E.g., that it should be a certain version, needs to
be compiled with certain flags, etc.

> Ninja dependency resolver.

I would call this "Ninja build tool".

You should probably link to the LLVM commandline library docs
<http://llvm.org/docs/CommandLine.html> in the relevant part of the
document.

> the loop’s index variable is compared against N - 1

generally aren't half-open loops comparing against N, and not N-1? e.g.
for (int I = 0, N = 10; I != N; ++I)

I think that just a little bit more explanation about the use of
FoldingSetNodeID in areSameExpr() would be good. (just explain roughly
what it does, links to docs, etc)

On Tue, Jul 31, 2012 at 6:34 PM, Sam Panzer <panzer@...> wrote:
> Hello, again!
>
> As suggested by a few people, I have been writing up my experiences starting
> development on Clang, and specifically working with Clang-based refactoring
> tools. The repository where my loop converter will live doesn't exist yet,
> so I will use the tutorial as a chance to start offering patches for review
> (they can be rebased to other locations later).
>
> My intention was to begin assuming zero Clang experience and work towards
> building a slightly simplified version of the loop converter through a
> series of fully-explained patches to Clang to a unit-tested functional tool.
> Because the Getting Started page is aimed at people who want to use Clang
> rather than develop for it, I began with cloning clang+llvm from the git
> mirrors and preparing a cmake+ninja build.
>
> What I have so far is currently available as a Google Doc here:
> https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
> If enough people think that the tutorial is useful, I can eventually convert
> it to a nicer format (i.e. HTML with prettier code formatting). I expect to
> have made mistakes, especially in the section about obtaining and building
> Clang :)
>
> Thoughts?
> -Sam
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev@...
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
Manuel Klimek | 1 Aug 10:39 2012
Picon

Re: AST Matchers tutorial

Nice! I *love* it. I have some stuff in the pipeline that will be
pretty orthogonal to what you wrote, but I also have to simply admit
defeat to your writing :)

On Wed, Aug 1, 2012 at 3:34 AM, Sam Panzer <panzer@...> wrote:
> Hello, again!
>
> As suggested by a few people, I have been writing up my experiences starting
> development on Clang, and specifically working with Clang-based refactoring
> tools. The repository where my loop converter will live doesn't exist yet,
> so I will use the tutorial as a chance to start offering patches for review
> (they can be rebased to other locations later).
>
> My intention was to begin assuming zero Clang experience and work towards
> building a slightly simplified version of the loop converter through a
> series of fully-explained patches to Clang to a unit-tested functional tool.
> Because the Getting Started page is aimed at people who want to use Clang
> rather than develop for it, I began with cloning clang+llvm from the git
> mirrors and preparing a cmake+ninja build.
>
> What I have so far is currently available as a Google Doc here:
> https://docs.google.com/document/d/1oTkVLhCdRJUEH1_LDaQdXqe8-aOqT5GLDL9e4MhoFF8/edit
> If enough people think that the tutorial is useful, I can eventually convert
> it to a nicer format (i.e. HTML with prettier code formatting). I expect to
> have made mistakes, especially in the section about obtaining and building
> Clang :)
>
> Thoughts?
> -Sam
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev@...
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

Gmane