Python Extensions In C? - Programmers Heaven

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Categories

Welcome to the new platform of Programmer's Heaven! We apologize for the inconvenience caused, if you visited us from a broken link of the previous version. The main reason to move to a new platform is to provide more effective and collaborative experience to you all. Please feel free to experience the new platform and use its exciting features. Contact us for any issue that you need to get clarified. We are more than happy to help you.

Python Extensions In C?

JonathanJonathan Posts: 2,914Member
Hi,

I'm working on a cross-platform audio engine [1], and have got language bindings for a number of languages now. While Python isn't a language I use, I know it's got some degree of popularity and it'd be nice to open up the functionality of my engine to Python folks, if they're interested in using it.

Now, excuse me while I waffle about Perl. :-) In Perl, I can use Perl XS to write an extension in C. It sorts out all the makefiles, provides a tool to convert my XS module into pure C that will compile natively, and then prepares a standard Perl module that loads up my C functions and lets me just call them. Then I can drop a nice OO wrapper around them and it's all done.

Before I go down the C line, can Python do non-blocking bi-directional pipe communication in a cross-platform compatible way? Assuming not, I imagine I need to take the same stance as I did with Perl and do a C extension. Has anyone here done that before? (Does anyone here want to do it again? ;-)) I guess I should go RTFM, but any advice as to how easy/hard this is to do with Python and general guidance would be appreciated.

Thanks,

Jonathan

[1] http://amamp.sourceforge.net/

###
for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
(tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
/(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

Comments

  • infidelinfidel Posts: 2,900Member
    : Hi,
    :
    : I'm working on a cross-platform audio engine [1], and have got language bindings for a number of languages now. While Python isn't a language I use, I know it's got some degree of popularity and it'd be nice to open up the functionality of my engine to Python folks, if they're interested in using it.
    :
    : Now, excuse me while I waffle about Perl. :-) In Perl, I can use Perl XS to write an extension in C. It sorts out all the makefiles, provides a tool to convert my XS module into pure C that will compile natively, and then prepares a standard Perl module that loads up my C functions and lets me just call them. Then I can drop a nice OO wrapper around them and it's all done.
    :
    : Before I go down the C line, can Python do non-blocking bi-directional pipe communication in a cross-platform compatible way? Assuming not, I imagine I need to take the same stance as I did with Perl and do a C extension. Has anyone here done that before? (Does anyone here want to do it again? ;-)) I guess I should go RTFM, but any advice as to how easy/hard this is to do with Python and general guidance would be appreciated.

    Boy, you're just jumping in feet first, aren't you? You likely won't find much help here. At least not from me. I've never ventured into the C extension stuff. I know that Python can do non-blocking bi-directional socket communication (asyncore, Twisted) but I don't know about pipes.

    Here is the Extending/Embedding document: http://docs.python.org/ext/ext.html

    Here is the Python/C API reference: http://docs.python.org/api/api.html

    Hope that helps, though I'm sure you found them already on your own :-)


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

  • JonathanJonathan Posts: 2,914Member
    : Boy, you're just jumping in feet first, aren't you? You likely won't
    : find much help here. At least not from me. I've never ventured into
    : the C extension stuff.
    I guessed that's the way it would be, but figured it was worth a try.

    : I know that Python can do non-blocking bi-directional socket
    : communication (asyncore, Twisted) but I don't know about pipes.
    :
    Perl can probably handle the non-blocking bi-di sockets stuff too, probably because that's slightly more standardised between platforms to my knowledge (but don't quote me on it - my network programming knowledge sucks). I found plenty of stuff about non-blocking sockets in the MSDN, but not pipes. Took me a while to work that one out...

    : Here is the Extending/Embedding document: http://docs.python.org/ext/ext.html
    :
    : Here is the Python/C API reference: http://docs.python.org/api/api.html
    :
    : Hope that helps, though I'm sure you found them already on your own :-)
    :
    Aye, but thanks for pointing me at them anyway. Guess I'll have a crack at it when I have time.

    I just dream of times where Parrot is everywhere, and the module I write in Perl can be used by Python and PHP folks as well as folks using other languages that Parrot supports.

    Jonathan

    ###
    for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
    (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
    /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

  • infidelinfidel Posts: 2,900Member
    : Perl can probably handle the non-blocking bi-di sockets stuff too, probably because that's slightly more standardised between platforms to my knowledge (but don't quote me on it - my network programming knowledge sucks). I found plenty of stuff about non-blocking sockets in the MSDN, but not pipes. Took me a while to work that one out...

    I've had some trouble using the asyncore module on Windows, but it's an interesting concept nonetheless.

    : I just dream of times where Parrot is everywhere, and the module I write in Perl can be used by Python and PHP folks as well as folks using other languages that Parrot supports.

    Been a long time since "Leaping Kakapo", anything interesting happening over there?


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

  • infidelinfidel Posts: 2,900Member
    : : I know that Python can do non-blocking bi-directional socket
    : : communication (asyncore, Twisted) but I don't know about pipes.
    : :
    : Perl can probably handle the non-blocking bi-di sockets stuff too, probably because that's slightly more standardised between platforms to my knowledge (but don't quote me on it - my network programming knowledge sucks). I found plenty of stuff about non-blocking sockets in the MSDN, but not pipes. Took me a while to work that one out...

    Based on a quick Google search and consulting the Python manuals, it appears that the select() function in the select module (which asyncore uses with sockets) will also handle pipes, but not on Windows.


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

  • JonathanJonathan Posts: 2,914Member
    : : Perl can probably handle the non-blocking bi-di sockets stuff too, probably because that's slightly more standardised between platforms to my knowledge (but don't quote me on it - my network programming knowledge sucks). I found plenty of stuff about non-blocking sockets in the MSDN, but not pipes. Took me a while to work that one out...
    :
    : I've had some trouble using the asyncore module on Windows, but it's
    : an interesting concept nonetheless.
    :
    Windows can do AIO, but as per usual it's just done differently to the way your average POSIX platform does it.

    : : I just dream of times where Parrot is everywhere, and the module I write in Perl can be used by Python and PHP folks as well as folks using other languages that Parrot supports.
    :
    : Been a long time since "Leaping Kakapo", anything interesting
    : happening over there?
    :
    Plenty of odds and ends going on in the work towards the next release, which I believe will happen in about a months time. I've only managed a measely three patches so far this release. Stuff of note:-
    * Loads of discussion about strings, unicode and all that ungliness. Some stuff here got implemented. ICU got chucked for a range of reasons.
    * Discussion about the way namespaces will work.
    * Garbage Collection API being cleared up for pluggable Garbage collectors.
    * JIT on some new platforms.
    * Improvements to register allocation stuff in IMCC.
    * Base behaviour decisions
    * Interfaces discussed - but maybe not implemented.
    * New exec opcode (the Win32 implementation by yours truly... ;-))
    * Loads of bugfixes.
    * After the work on the Piethon, Parrot is now capable of running quite a lot of Python bytecode.

    Dan Sugalski also made this very scary observation:-

    --
    1) We're going to have MMD for functions soon
    2) Function invocation and return continuation invocation's
    essentially identical
    3) Therefore returning from a sub/method can do MMD return based on
    the return values
    --

    I guess this is an upshot of using a continuation calling scheme...

    Jonathan

    ###
    for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
    (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
    /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

  • JonathanJonathan Posts: 2,914Member
    : Based on a quick Google search and consulting the Python manuals, it appears that the select() function in the select module (which asyncore uses with sockets) will also handle pipes, but not on Windows.
    :
    Well, I have a bunch of code that does it on Windows and POSIX OSes, so I'll probably just go along with the original plan for a C extension. :-)

    Jonathan

    ###
    for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
    (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
    /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

  • infidelinfidel Posts: 2,900Member
    : : Based on a quick Google search and consulting the Python manuals, it appears that the select() function in the select module (which asyncore uses with sockets) will also handle pipes, but not on Windows.
    : :
    : Well, I have a bunch of code that does it on Windows and POSIX OSes, so I'll probably just go along with the original plan for a C extension. :-)

    I look forward to it. Let me know if I can help in some small way. I don't know C that well, though.


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

  • infidelinfidel Posts: 2,900Member
    : Windows can do AIO, but as per usual it's just done differently to the way your average POSIX platform does it.

    From what I've read, the WinSock sockets aren't true file descriptors (or something to that effect). Therefore the WinSock library provides its own functions for the AIO. When implementing the select module for python, I guess since sockets were the most common application for which the select() function would be used they decided to use the WinSock functions instead of the File/Pipe functions.

    : Plenty of odds and ends going on in the work towards the next release, which I believe will happen in about a months time. I've only managed a measely three patches so far this release. Stuff of note:-
    : * Loads of discussion about strings, unicode and all that ungliness. Some stuff here got implemented. ICU got chucked for a range of reasons.

    What is ICU?

    : * Improvements to register allocation stuff in IMCC.

    Don't know what that means.

    : Dan Sugalski also made this very scary observation:-
    :
    : --
    : 1) We're going to have MMD for functions soon
    : 2) Function invocation and return continuation invocation's
    : essentially identical
    : 3) Therefore returning from a sub/method can do MMD return based on
    : the return values

    Not sure what MMD is or why it's scary.

    : I guess this is an upshot of using a continuation calling scheme...

    If you say so :-)


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

  • JonathanJonathan Posts: 2,914Member
    : : Windows can do AIO, but as per usual it's just done differently to the way your average POSIX platform does it.
    :
    : From what I've read, the WinSock sockets aren't true file descriptors (or something to that effect). Therefore the WinSock library provides its own functions for the AIO. When implementing the select module for python, I guess since sockets were the most common application for which the select() function would be used they decided to use the WinSock functions instead of the File/Pipe functions.
    :
    File APIs can hanlde AIO too, from what I remember. ReadFileEx and WriteFileEx support it, for sure. You use the same APIs to read/write pipes, but I don't know if they are handled by the AIO stuff and if they are I seem to rememeber it was only available from NT and onwards, and I wanted to support 9x OSes too.

    : : Plenty of odds and ends going on in the work towards the next release, which I believe will happen in about a months time. I've only managed a measely three patches so far this release. Stuff of note:-
    : : * Loads of discussion about strings, unicode and all that ungliness. Some stuff here got implemented. ICU got chucked for a range of reasons.
    :
    : What is ICU?
    :
    International Components for Unicode. Basically, a unicode implementation. However, it had build issues on a lot of platforms Parrot was to run on, and the opinion seemed to be that it did way too much platform specific stuff for what essentially should be a "character twiddling library".

    : : * Improvements to register allocation stuff in IMCC.
    :
    : Don't know what that means.
    :
    The Parrot Virtual Machine has at its center something a bit like a CPU, but done in software - I guess we could call it a software CPU. Like a hardware processor has registers, so does Parrot. Unlike in hardware, Parrot has 4 types of register (integer, number, string and PMC[1]) and 32 of each of those.

    When you're writing a bit of software that's non-trivial, you'll run into the problem that you have more "variables" than you do registers. Thefore you need to do register spilling (saving registers elsewhere). Register allocation is basically the process of working out the best way to do this.

    When you write in PASM (Parrot assembly) you have to take care of this yourself, but IMC (Intermediate Code) removes this responsibility from the programmer (and the compiler writer targetting Parrot). IMCC (the intermediate code compiler) thus has to do all of the register allocation stuff. Extremely big blocks of code can make this process exceptionally slow, because the algorithms are non-linear (quadratic order, to my knowledge). Some improvements have been made to the algorithms, but there's still work to be done. I think the big thing to do is to get the algorithm to realise when it's trying to hard and just resort to a simple, unintelligent way of doing register allocation that at least gets the job done in a reasonable amount of time, even if the code is less efficient when executed.

    : : Dan Sugalski also made this very scary observation:-
    : :
    : : --
    : : 1) We're going to have MMD for functions soon
    : : 2) Function invocation and return continuation invocation's
    : : essentially identical
    : : 3) Therefore returning from a sub/method can do MMD return based on
    : : the return values
    :
    : Not sure what MMD is or why it's scary.
    :
    Multi-method dispatch. Sorry, I should use less acronyms. It's basically the idea that imagine I did the following (in a pseudo-language (that accidentally looks very much like C++ (which I don't know (damm, this is starting to look like Lisp ;-))))):-

    void myMethod(int a) { ... }
    void myMethod(double a) { ... }
    void myMethod(char *a, int b) { ... }

    Then when I called the function myMethod, MMD would look at the available method signatures (parameter type list) and find the one that matched what was being passed most closely and then call that method.

    : : I guess this is an upshot of using a continuation calling scheme...
    :
    : If you say so :-)
    :
    Continuations are strange enough alone without MMD stuff. A continuation is basically a bit like a closure that also closes over the control flow. What the heck does that mean? Well, it took me a while to figure that one out too. :-) When you take a continuation, you are saving the current lexical state (e.g. all lexical variables - meaning no global variables are involved here) and the call stack at that point. This means that when you invoke the continuation, you essentially end up back where you were where you started with the same lexical state. However, the global context may have changed. For example, the data held in an object you had a lexical reference too may have changed, but you'll still have the reference. Global variables won't be the same.

    When you call a function, you need to save some state, from simple things like a return address to all kinds of other stuff. The problem that was realised was that if you wanted to add other stuff to be saved in the future, then you'd probably have to change the calling conventions - a big pain. A continuation is good at saving state and a standard way of doing it, however. So a continuation passing scheme was used.

    Basically, you take a continuation, called the return continuation, and pass it to the thing you're calling. When you want to return from the function, you simply invoke the return continuation, and you're back where you were (of course, internally Parrot must be careful you're sent back after the function call itself happened, otherwise you'd end up in an infinite loop - I don't know the gory details of how that bit is handled).

    I hope this kind of puts into context what Dan was talking about. I'm not sure I fully understand the real possibilities of this yet, but at least it's not as mind-numbing as serializable continuations, which Parrot will apparently have too.

    Jonathan

    [1] PMC = Parrot Magic Cookie. It's basically for more complex data types, e.g. arrays, objects, classes, etc. Like objects, they have a vtable.

    ###
    for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
    (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
    /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

  • infidelinfidel Posts: 2,900Member
    : Before I go down the C line, can Python do non-blocking bi-directional pipe communication in a cross-platform compatible way? Assuming not, I imagine I need to take the same stance as I did with Perl and do a C extension. Has anyone here done that before? (Does anyone here want to do it again? ;-)) I guess I should go RTFM, but any advice as to how easy/hard this is to do with Python and general guidance would be appreciated.

    I've been thinking about the non-blocking pipe question and I've been experimenting with some of David Mertz's (a python god) examples on using generators to simulate coroutines and "weightless threads". If you used functions like PeekNamedPipe (available in the win32 extensions for python) to check a pipe before trying to read from it, you could maybe use something like this as a sort of "event loop" to do other things while your pipes are busy doing their own thing:

    [code]
    import sys, time

    quitflag = False

    class ThreadPool:
    """Enhanced threads list as class
    threads = ThreadPool()
    threads.append(threadfunc) # not generator object
    if threads.query(num) <<has some property>>:
    threads.remove(num)
    """
    def __init__(self):
    self.threadlist = []
    self.threaddict = {}
    self.avail = 1
    def __iter__(self):
    return iter(self.threadlist)
    def __getitem__(self, n):
    return self.threadlist[n]
    def append(self, threadfunc, docstring=None):
    # Argument is the generator func, not the gen object
    # Every threadfunc should contain a docstring
    docstring = docstring or threadfunc.__doc__
    self.threaddict[self.avail] = (docstring, threadfunc())
    self.avail += 1
    self.threadlist = [p[1] for p in self.threaddict.values()]
    return self.avail-1 # return the threadID
    def remove(self, threadID):
    del self.threaddict[threadID]
    self.threadlist = [p[1] for p in self.threaddict.values()]
    def query(self, threadID):
    #Information on thread, if it exists (otherwise None)
    return self.threaddict.get(threadID,[None])[0]

    threads = ThreadPool()

    def something():
    "do something and another thing in two separate next() calls"
    while True:
    print 'something',
    yield None
    print 'another thing',
    yield None

    def something_else():
    "do something else"
    while True:
    print 'something else',
    yield None

    def counter():
    "increments a counter and sets the 'quit' flag"
    n = 1
    while True:
    print n
    n = n + 1
    global quitflag
    if n > 100: quitflag = True
    yield None

    def quitter():
    "stops the scheduler by raising StopIteration"
    while True:
    global quitflag
    if quitflag: raise StopIteration
    yield None

    def scheduler():
    while True:
    try:
    for thread in threads: thread.next()
    except StopIteration:
    break

    def main():
    global threads
    threads.append(quitter)
    threads.append(something)
    threads.append(quitter)
    threads.append(something_else)
    threads.append(quitter)
    threads.append(counter)
    threads.append(quitter)
    starttime = time.clock()
    scheduler()
    print 'TOTAL TIME: ', time.clock() - starttime

    if __name__ == '__main__': main()
    [/code]

    Let me know if you need clarification on any of this. It's a pretty trivial example, but seems very powerful without getting into C coding.


    [size=5][italic][blue][RED]i[/RED]nfidel[/blue][/italic][/size]

    [code]
    $ select * from users where clue > 0
    no rows returned
    [/code]

Sign In or Register to comment.