Pygtk and garbage collecting

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Philip Van Hoof
This PyGTK bug is going to be important for OLPC as it might slow down
object destruction.

http://bugzilla.gnome.org/show_bug.cgi?id=320428

For example in the tinymail-python-test.py*, you will find a
gc.collect(). This is to speed up the destruction of objects in the
situation described in Bug #320428

https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-test/tinymail-python-test.py

In case of tinymail the GtkTreeModel that holds a reference on all the
headers of the current folder, is such an object.

model = MsgHeaderListModel ()
treeview.set_model (model)

newmodel = MsgHeaderListModel ()
treeview.set_model (newmodel)

Will not immediately mark "model" for garbage collection (when model
goes out of scope). Adding a gc.collect(), however, will.

That model instance, in case of tinymail, holds a reference to the
headers of your folder. This is where most of the memory tinymail
consumes is located (the summary information). Therefore it's a small
disaster on a device with few memory resources, that Pythons garbage
collector is slow at detecting this.

Applying the patch of Bug #320428 or performing gc.collect() in your
Python code fixes this.

Note that tinymail isn't going to be the only case.


--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be

Reply | Threaded
Open this post in threaded view
|

Re: [OLPC-devel] Pygtk and garbage collecting

Johan Dahlin-2
Philip Van Hoof wrote:
> This PyGTK bug is going to be important for OLPC as it might slow down
> object destruction.
>
> http://bugzilla.gnome.org/show_bug.cgi?id=320428

This is true for all GObject subclasses. GdkPixbufs is another problematic
case, since they're usually quite large.

Without the patch they appear to leak and will not be freed immediately.

The patch could use some wider testing, large applications are known to run
under it, but there may still be some corner cases which are not solved.

I think we're going to apply it early on in the next PyGObject/PyGTK release
cycle (after GNOME 2.16).

Johan

Reply | Threaded
Open this post in threaded view
|

Re: [pygtk] Re: [OLPC-devel] Pygtk and garbage collecting

John Finlay
Johan Dahlin wrote:

> Philip Van Hoof wrote:
>  
>> This PyGTK bug is going to be important for OLPC as it might slow down
>> object destruction.
>>
>> http://bugzilla.gnome.org/show_bug.cgi?id=320428
>>    
>
> This is true for all GObject subclasses. GdkPixbufs is another problematic
> case, since they're usually quite large.
>
> Without the patch they appear to leak and will not be freed immediately.
>
> The patch could use some wider testing, large applications are known to run
> under it, but there may still be some corner cases which are not solved.
>
> I think we're going to apply it early on in the next PyGObject/PyGTK release
> cycle (after GNOME 2.16).
>
>  
Does changing the gc thresholds help as a workaround in the meantime?
For example gc.set_threshold(10,2,2) to really get collection to run
frequently.

John
Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Federico Mena Quintero
In reply to this post by Philip Van Hoof
On Mon, 2006-07-31 at 17:32 +0200, Philip Van Hoof wrote:

> In case of tinymail the GtkTreeModel that holds a reference on all the
> headers of the current folder, is such an object.
>
> model = MsgHeaderListModel ()
> treeview.set_model (model)
>
> newmodel = MsgHeaderListModel ()
> treeview.set_model (newmodel)
>
> Will not immediately mark "model" for garbage collection (when model
> goes out of scope). Adding a gc.collect(), however, will.

The usual idiom  is to have a way to explicitly dispose an object which
holds a reference to a non-GC resource.  In C# you do

  obj = new SomeObject ();
  treeview.set_model (obj);

  newobj = new SomeObject ();
  treeview.set_model (newobj);

  obj.Dispose ();

or use syntactic sugar:

  using (Gdk.Pixbuf pixbuf = new Gdk.Pixbuf ("foo.jpg")) {
      do_something_with_pixbuf (pixbuf);
  } /* exiting the "using" block disposes the pixbuf */

This is pretty useful for managed objects which maintain non-managed
data (file descriptors, memory buffers, database connections, etc.).

Does Python let you do something like this?

  Federico

Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Robert Staudinger
On 8/2/06, Federico Mena Quintero <[hidden email]> wrote:

[...]

> The usual idiom  is to have a way to explicitly dispose an object which
> holds a reference to a non-GC resource.  In C# you do
>
>   obj = new SomeObject ();
>   treeview.set_model (obj);
>
>   newobj = new SomeObject ();
>   treeview.set_model (newobj);
>
>   obj.Dispose ();
>
> or use syntactic sugar:
>
>   using (Gdk.Pixbuf pixbuf = new Gdk.Pixbuf ("foo.jpg")) {
>       do_something_with_pixbuf (pixbuf);
>   } /* exiting the "using" block disposes the pixbuf */
>
> This is pretty useful for managed objects which maintain non-managed
> data (file descriptors, memory buffers, database connections, etc.).
>
> Does Python let you do something like this?

It should be possible to export g_object_unref() through the pygobject API.
That would of course be not pythonic at all and not trigger collection
if any other (uncollected) python objects are still holding
references.
(/me waits for pygtk devs to point out more flaws)

- Rob
Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Ivan Krstić-2
In reply to this post by Federico Mena Quintero
Federico Mena Quintero wrote:
> This is pretty useful for managed objects which maintain non-managed
> data (file descriptors, memory buffers, database connections, etc.).
>
> Does Python let you do something like this?

Yes, using the 'with' statement that's new in Python 2.5, which we'll
have on the laptop:

 http://docs.python.org/dev/whatsnew/pep-343.html

--
Ivan Krstic <[hidden email]> | GPG: 0x147C722D
Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Federico Mena Quintero
On Tue, 2006-08-08 at 12:29 -0400, Ivan Krstic wrote:
> Federico Mena Quintero wrote:
> > This is pretty useful for managed objects which maintain non-managed
> > data (file descriptors, memory buffers, database connections, etc.).
> >
> > Does Python let you do something like this?
>
> Yes, using the 'with' statement that's new in Python 2.5, which we'll
> have on the laptop:

Oooh, excellent.  And it looks trivial to implement.

  Federico

Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Robert Staudinger
On 8/8/06, Federico Mena Quintero <[hidden email]> wrote:

> On Tue, 2006-08-08 at 12:29 -0400, Ivan Krstic wrote:
> > Federico Mena Quintero wrote:
> > > This is pretty useful for managed objects which maintain non-managed
> > > data (file descriptors, memory buffers, database connections, etc.).
> > >
> > > Does Python let you do something like this?
> >
> > Yes, using the 'with' statement that's new in Python 2.5, which we'll
> > have on the laptop:
>
> Oooh, excellent.  And it looks trivial to implement.

After a quick glance over the PEP I doubt that the resources will be
actively deallocated rather than just be out of scope and left for the
gc to pick up though. Looks like a way to explicitely free the wrapped
data is still needed if the patch mentioned by Johan is not deemed
sufficient.

- Rob
Reply | Threaded
Open this post in threaded view
|

Pygtk and garbage collecting

Ivan Krstić-2
Robert Staudinger wrote:
> After a quick glance over the PEP I doubt that the resources will be
> actively deallocated

Context managers need to be written for smart deallocation. They will
already be present around the Python primitives where it makes sense
(file objects, locks and such).

--
Ivan Krstic <[hidden email]> | GPG: 0x147C722D
Reply | Threaded
Open this post in threaded view
|

[OLPC-devel] Re: Pygtk and garbage collecting

Christopher Blizzard
In reply to this post by Ivan Krstić-2
Ivan Krstic wrote:

> Federico Mena Quintero wrote:
>> This is pretty useful for managed objects which maintain non-managed
>> data (file descriptors, memory buffers, database connections, etc.).
>>
>> Does Python let you do something like this?
>
> Yes, using the 'with' statement that's new in Python 2.5, which we'll
> have on the laptop:
>
>  http://docs.python.org/dev/whatsnew/pep-343.html
>

I don't think we're going to move to python 2.5 on the laptop.  2.4 is
what's well integrated at the moment and I don't think we want to try to
keep more than one python around.

--Chris
Reply | Threaded
Open this post in threaded view
|

[OLPC-devel] Re: Pygtk and garbage collecting

Ivan Krstić-2
Christopher Blizzard wrote:
> I don't think we're going to move to python 2.5 on the laptop.

For reasons explained on IRC -- an important memory management patch in
2.5, and the fact that 2.5 will have been the stable version for about
half a year by the time we ship -- I don't see shipping 2.4 as an
option. That's not even mentioning the actual bunch of very useful new
features in 2.5.

Given that Python is our main development platform on the machine, we
just can't afford to be lazy about it. I care little about the versions
of the rest of the stack, but I don't want to compromise on Python; if
you don't want to do the 2.5 migration, please let me know now so I can
schedule time to do it myself. Having to deal with that would be
unfortunate, though, as there are far more pressing things for me to
work on.

--
Ivan Krstic <[hidden email]> | GPG: 0x147C722D

Reply | Threaded
Open this post in threaded view
|

[OLPC-devel] Re: Pygtk and garbage collecting

Christopher Blizzard
Ivan Krstic wrote:
> Christopher Blizzard wrote:
>> I don't think we're going to move to python 2.5 on the laptop.
>
> For reasons explained on IRC -- an important memory management patch in
> 2.5, and the fact that 2.5 will have been the stable version for about
> half a year by the time we ship -- I don't see shipping 2.4 as an
> option. That's not even mentioning the actual bunch of very useful new
> features in 2.5.

It might be 6 months from the time that we ship, but it's hasn't been 6
months today and it hasn't been integrated into Fedora.  And there's no
way that 6 months is enough time to let this stuff stabilize.  We're
still finding bugs against the current python, let alone a new and
relatively untested code base.

The memory management patch sounds nice, but I don't think we've done
enough testing to know if the pain of upgrading and maintaining our own
python code base (releases, bugfixing and security fixes) is worth the
reported benefit of a patch.  Especially since we don't even know that
we have a problem today.

>
> Given that Python is our main development platform on the machine, we
> just can't afford to be lazy about it. I care little about the versions
> of the rest of the stack, but I don't want to compromise on Python; if
> you don't want to do the 2.5 migration, please let me know now so I can
> schedule time to do it myself. Having to deal with that would be
> unfortunate, though, as there are far more pressing things for me to
> work on.
>

I _certainly_ don't think that we're being lazy.  But I do think we're
picking the battles that we want to fight.  And if you want to spend you
time on that, I guess that's fine.  But I would rather you were spending
your time on higher impact things, like making the wiki stuff work.

--Chris
Reply | Threaded
Open this post in threaded view
|

[OLPC-devel] Re: Pygtk and garbage collecting

Ivan Krstić-2
Christopher Blizzard wrote:
> still finding bugs against the current python, let alone a new and
> relatively untested code base.

It's neither new nor untested; it's a relatively conservative
incremental improvement from 2.4, and has gone through three betas to
chase down potential problems. As I mentioned earlier, there have only
been three point releases of 2.4, which is a reassuring track record.

> The memory management patch sounds nice [...]
> Especially since we don't even know that we have a problem today.

We certainly do have a problem; we will have a persistent Python server
process running the entire time the laptop is powered on. In 2.4, this
process won't release memory back to the kernel. Ever. On a 128MB
system, this upgrades the importance of Evan's patch from 'nice' to
'critical'.

Details at http://evanjones.ca/python-memory.html.

> And if you want to spend you
> time on that, I guess that's fine.

I don't want to spend my time on distro work. However, I don't see
having 2.5 as an option but as a necessity, which means I'll spend time
on it if I can't convince you to. The alternative is to backport Evan's
patch to 2.4, which again means maintaining our own Python.

--
Ivan Krstic <[hidden email]> | GPG: 0x147C722D