Tag Archives: Smalltalk

Oz object spaces in Esug 2013

Hi!

This little post is to add a summary of what I’ll be showing in the upcoming ESUG conference. I’ll present a paper about my work in the workshop and present the recovery tools of Oz in the innovation awards :).

For the paper

I’ll present the ideas and implementation of Oz object spaces. Especially the metacircular problems it solves, and a bit on the vision we (I and my supervisors have on this line of work). If you want to read the paper, you can go to:

http://rmod.lille.inria.fr/archives/papers/Poli13a-IWST13-ObjectSpacesVirtualization.pdf.

For the lazy, I paste here the abstract:

Reflective architectures are a powerful solution for code browsing, debugging or in-language process handling. However, these reflective architectures show some limitations in edge cases of self-modification and self-monitoring. Modifying the modifier process or monitoring the monitor process in a reflective system alters the system itself, leading to the impossibility to perform some of those tasks properly. In this paper we analyze the problems of reflective architectures in the context of image based object-oriented languages and solve them by providing a first-class representation of an image: a virtualized image.
We present Oz, our virtual image solution. In Oz, a virtual image is represented by an object space. Through an object space, an image can manipulate the internal structure and control the execution of other images. An Oz object space allows one to introspect and modify execution information such as processes, contexts, existing classes and objects. We show how Oz solves the edge cases of reflective architectures by adding a third participant, and thus, removing the selfmodification and self-observation constraints.

For the innovation awards

I’ve prepared a demo and with that I recorded a little video showing it. You can see it in:

Also, we have a sexy logo!

Oz logo

Oz logo, created by my friend Ximena Fernandez
Copyright (C) 2013 Ximena Fernandez

I asked my friend Ximena to make a logo for my projects, even pushing her with the deadline of ESUG 2013 (sorry xime ;), and she created this one, that I really like.

So, thanks again Xime!

Now, waiting for next week @ ESUG, to taste the ESUG beers ;).
Enjoy!
Guille

Customizing ZeroConf scripts

As you may already know, ZeroConf scripts are bash scripts that ease the installation of a Pharo environment. A funny thing about these ZeroConf scripts is that they are seen as bash scripts by a bash terminal, and as simple and minimal html pages by a web browser. These scripts are extensively used to simplify the configuration of Pharo CI jobs. They allow you to easily download many versions of the Pharo image and VM.

As I’m working for my Phd, and have a custom version of my virtual machine and image, and also want to make use of the advantages our CI server provides, I wanted to build my own ZeroConf scripts specialized for my needs. I also heard recently on the pharo mailing list that there was some work on customizing ZeroConf scripts[1] for Moose[2]. So I wanted to do it as well for my project :).

[1] http://get.moosetechnology.org/
[2] http://www.moosetechnology.org/

Downloading the ZeroConf package

The ZeroConf scripts are generated automatically by the ZeroConf Pharo package. You can find this package in [2]. To download the current version, you just have to execute the following piece of code in a workspace:

Gofer it
	smalltalkhubUser: 'Pharo' project: 'ZeroConf';
	package: 'ZeroConf';
	loadVersion: '1.0'

This code snippet will install in your image the ZeroConf package for the 1.0 version, containing the script generator, and some tests that are not currently working :).
[2] http://www.smalltalkhub.com/#!/~Pharo/ZeroConf

Getting what’s inside ZeroConf

The ZeroConf package is pretty small and simple. There is an abstract class AbstractZeroConfBashScript implementing most of the script generation and bash writing utils. Its subclasses will implement the concrete script generation. Current implementation includes three main classes below the AbstractZeroConfBashScript hierarchy, implementing a composite pattern:

  • ZeroConfImageScript: Generates scripts in charge of downloading image files.
  • ZeroConfVMScript: Generates scripts in charge of downloading Virtual Machine files (and source files).
  • ZeroConfCombinedScript: Generates scripts that combine several scripts. It will point to its combined scripts and downloading it means to download them all.

zeroconf

Customizing our ZeroConf scripts

As you can see in the picture, in order to customize the ZeroConf scripts, you have to create your own subclasses and overriding the correct hooks.

Customizing an image script

A custom image script is defined by a subclass of ZeroConfImageScript. ZeroConfImageScript already defines some image common behavior, such as the release number, which we will use in our script.

ZeroConfImageScript subclass: #OzZeroConfImageScript
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'OurZeroConf'

Then we override the methods that tell some information about the files to download:

  • OzZeroConfImageScript>>imageName: The name of the image and changes files
  • OzZeroConfImageScript>>imageUrl: The url of the image zip, instance of ZnUrl
  • OzZeroConfImageScript>>defaultBaseName: The baseName of the script
  • OzZeroConfImageScript class>>baseUrl: The base url where the scripts are found, instance of ZnUrl
  • OzZeroConfImageScript class>>fileBaseUrl: The base url where the files the scripts will download are found, instance of ZnUrl

I implemented them in my class as follows

OzZeroConfImageScript>>imageName
	^'Oz'

OzZeroConfImageScript>>imageUrl
	^ self fileBaseUrl / 'image' / self release asString / self imageFileName / 'download'

OzZeroConfImageScript>>imageFileName
	^'Oz-image-', self release asString , '.zip'

OzZeroConfImageScript>>defaultBasename
	^ self imageName, self release

OzZeroConfImageScript class>>fileBaseUrl
	^ 'https://sourceforge.net/projects/ozobjectspaces/files' asZnUrl

OzZeroConfImageScript class>>baseUrl
	^ self fileBaseUrl / 'get'

I also extended my script so it generates a custom html title and uses my combining script when combining:

OzZeroConfImageScript>>htmlTitle
	^ self imageName, ' Zeroconf Script'

OzZeroConfImageScript>>defaultCombiningScript
	^ OzZeroConfCombinedScript

Finally, I created a convenience method for creating a script corresponding to the 1.0 version of my custom image.

OzZeroConfImageScript class>>oz10
	^self new
		release: '1.0';
		yourself

Now you can try generating your script in a workspace,

OzZeroConfImageScript oz10 generate

and see the generated results in your working directory!

Customizing a vm script

A custom vm script is defined by a subclass of ZeroConfVMScript. ZeroConfVMScript defines, as its image friend, some vm common behavior, such as the release number and virtual machine type(i.e., if it is a jitted vm or not), which we will use in our script.

ZeroConfVMScript subclass: #OzZeroConfVMScript
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'OurZeroConf'

Then we override the methods that tell some information about the files to download:

  • OzZeroConfVMScript>>binaryName: The name of the vm binary name
  • OzZeroConfVMScript>>binaryNameLinux: The name of the vm binary name in linux, which tends to be different
  • OzZeroConfVMScript>>vmUrl: The url of the vm zip, instance of ZnUrl
  • OzZeroConfVMScript>>defaultBaseName: The baseName of the script
  • OzZeroConfVMScript class>>baseUrl: The base url where the scripts are found, instance of ZnUrl
  • OzZeroConfVMScript class>>fileBaseUrl: The base url where the files the scripts will download are found, instance of ZnUrl

I implemented them in my class as follows

OzZeroConfVMScript>>binaryName
	^'ozstack'

OzZeroConfVMScript>>binaryNameLinux
	^self binaryName

OzZeroConfVMScript>>vmUrl
	^self fileBaseUrl asString, '/vm/', self release asString,'/', self vmFileName, '/download'

OzZeroConfVMScript>>vmFileName
	^'OzVm-${OS}-', self release asString , '.zip'

OzZeroConfVMScript class>>fileBaseUrl
	^ 'https://sourceforge.net/projects/ozobjectspaces/files' asZnUrl

OzZeroConfVMScript class>>baseUrl
	^ self fileBaseUrl / 'get'

I also extended my script so it uses my combining script when combining:

OzZeroConfVMScript>>defaultCombiningScript
	^ OzZeroConfCombinedScript

Finally, I created a convenience method for creating a script corresponding to the 1.0 version of my custom vm.

OzZeroConfVMScript class>>ozvm10
	^self new
		type: 'oz';
		release: '1.0';
		yourself

Now you can try generating your script in a workspace,

OzZeroConfVMScript ozvm10 generate

and see the generated results in your working directory!

Customizing a combined script

A combined script is the one we use to combine several scripts. It is defined by a subclass of ZeroConfCombinedScript.

ZeroConfCombinedScript subclass: #OzZeroConfCombinedScript
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'OurZeroConf'

Then we override the methods that tell some information about the files to download:

  • OzZeroConfCombinedScript class>>baseUrl: The base url where the scripts are found, instance of ZnUrl
  • OzZeroConfCombinedScript class>>fileBaseUrl: The base url where the files the scripts will download are found, instance of ZnUrl

I implemented them in my class as follows

OzZeroConfCombinedScript class>>fileBaseUrl
	^ 'https://sourceforge.net/projects/ozobjectspaces/files' asZnUrl

OzZeroConfCombinedScript class>>baseUrl
	^ self fileBaseUrl / 'get'

As you can see, my methods baseUrl and fileBaseUrl are always the same in all scripts. I extracted them into another class later, but keep the code here as is for clarity.

I also extended my script so it uses my combining script when combining and the html title:

OzZeroConfCombinedScript>>defaultCombiningScript
	^ OzZeroConfCombinedScript

OzZeroConfCombinedScript>>htmlTitle
	^ self scripts first htmlTitle

Integrating everything and automating generation

As I already showed you, every script understands the message #generate to generate itself. However, we may want to generate many scripts, and combine them. The ZeroConf infrastructure already provides for that the ZeroConfCommandLineHandler. The ZeroConfCommandLineHandler is a command line handler that knows which are the scripts we want to generate, combines them appropriately and generates them. So we will subclass from ZeroConfCommandLineHandler and specialize it to fulfill our needs.

ZeroConfCommandLineHandler subclass: #OzZeroConfCommandLineHandler
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'OzZeroConf'

Once we have it, we configure it as a command line handler specifying its command name and description:

OzZeroConfCommandLineHandler class>>commandName
	^ 'ozzeroconf'

OzZeroConfCommandLineHandler class>>description
	^ 'Generate Oz zeroconf bash scripts'

And finally we specialize it to tell it about our scripts:

OzZeroConfCommandLineHandler>>defaultScript
	^ self defaultImage, self defaultVM

OzZeroConfCommandLineHandler>>defaultImage
	^ OzZeroConfImageScript oz10

OzZeroConfCommandLineHandler>>defaultVM
	^ OzZeroConfVMScript ozvm10

OzZeroConfCommandLineHandler>>imageScripts
	^ { 
		OzZeroConfImageScript oz10.
	}

OzZeroConfCommandLineHandler>>vmScripts
	^ { 
		OzZeroConfVMScript ozvm10
	}

OzZeroConfCommandLineHandler>>indexScriptExamplesHtml
	^ ''

Now we have our command line handler, we can test it and make it generate our scripts:

OzZeroConfCommandLineHandler new
	commandLine: CommandLineArguments new;
	generateScripts

Finally, if you have all this code in your image, you can just activate it through the command line thanks to the command line handler!

./pharo PharoMyZeroConf.image ozzeroconf

And look at the results. Upload your files, archive them, and use them :).

Guille

The Bootstrap Chronicles – The end of a phase

As some of you already know, the GSOC project edition 2012 is coming to an end.  And along with it, the bootstrap project reaches a checkpoint. This post covers is the news since the last chapter, and discusses about the future steps. In a next post I’ll document  the details of the project deliverables.

Where we where?

The first product of this project was a renewal of an image serializer: the SystemTracer. It takes an object graph and serializes it into the image format a vm works with. The System tracer was refactored reifying the memory object formats and updated to write Cog images.

The second step was to work on bootstrapping. And it was successful.  Hazelnut, the bootstrap process tool, is able to build a smalltalk image from a description. To ensure the quality and health of these newly created images I set up jenkins jobs loading different packages on top of them, and running tests over them.

What happened since last time: Declarative kernel descriptions

A kernel definition has two main parts:

  • the code and definitions of the entities of that kernel
  • a definition on how to build the basic model of that kernel and how to initialize it finally

The first one can take the form of source files like in https://github.com/guillep/PharoKernel, which is the source code Hazelnut actually uses to bootstrap Pharo images. The second part of the kernel definition contains some imperative parts, and by now they are declared in simple pharo classes you download from Monticello.

So now, the bootstrap loads the kernel definition from those source files, generates the bootstrapped environment, and serializes it into a new image file.

The future of bootstrapping pharo

Since our goal is to bootstrap pharo to support it’s modularity and  evolution, there are some keypoints to attack in the near future:

  • getting the pharo sourcecode in sync with this bootstrap representation
  • choosing the really important parts for a kernel. What should be and what should not in those source files? Where do we package what’s not going kernel?
  • building pharo from bootstrapped images.

Even, when looking at the upcoming pharo changes like first class slots and class layouts, or the new Tanker package manager, the bootstrap will need for sure some updates.

Conclusions

I hope this project makes pharo grow and get better! We can now generate images with the source code defined statically in source files, so for the GSOC program the scope has been fulfilled.

See you in a next post documenting the project!

Salut!

Smalltalk behind the scenes: the meta model

Have you ever evaluated this pieces of code in Pharo?

 ProtoObject superclass.
 ProtoObject class superclass.
 Metaclass class class = Metaclass.

Wait, WTF? How is that ProtoObject superclass is nil? Wait again, and the one of it’s superclass is Class? Metaclass class is an instance of Metaclass? Hey, that’s kind of the chiken and the egg problem, which one was first?

You know that when you create a class, you specify a superclass for it.  This superclass will specify some other properties and the VM will use it to perform the method lookup.

Also, probably you already know that when a class is created in Smalltalk, a metaclass is created for it implicitly. That metaclass describes the class side behavior: class side methods, class instance variables…

Funny thing about this implicit metamodel, is that a second class hierarchy is built in parallel to the original class hierarchy.

Now, if you think about this, you can understand why the method lookup works also in the class side methods, and they are not static like in Java or C# :).

You can have a look at the following invariants:

aClass superclass class = aClass class superclass.
aClass class class = Metaclass.

Which of course have it’s exceptions. The method lookup ends when it reaches a class whose superclass is nil.  And the class side objects also behave like a Class, because they finally inherit from Class. HA! But then the metaclass hierarchy re-enters the non-metaclass hierarchy. Thinking of this in an operational way is kind of meta confusing, isn’t it?

But this is not the motivation of this post. The motivation is this: Are we really coupled to that meta model?  How can I create my own?

If you remember from my post on vm limitations I learned during the bootstap, the vm only expects 3 things from a class:

  • that it’s first instance variable is it’s superclass.
  • that it’s second instance variable is it’s method dictionary.
  • that it’s third instance variable is it’s format.

Any object respecting that contract can be treated like a class by the VM.  Then you can think on creating your own metaclass loop, kind of independent from the original one…

classFormat := ...
metaclassFormat := ...

"This metaclass defines how our metaclass instances will be.  It is only here to define the first metaclass format, and it will be discarded"
metaclassClass := Metaclass new.
metaclassClass
superclass: Class
methodDictionary: (MethodDictionary new)
         format: classFormat.

metaclass := metaclassClass basicNew.
metaclass instVarAt: 1 put: Metaclass.
metaclass instVarAt: 2 put: MethodDictionary new.
metaclass instVarAt: 3 put: metaclassFormat.

metaclassClass := metaclass basicNew.
metaclassClass instVarAt: 1 put: Metaclass class.
metaclassClass instVarAt: 2 put: MethodDictionary new.
metaclassClass instVarAt: 3 put: classFormat.

metaclassClass adoptInstance: metaclass.

Once you have a metaclass, instantiate it to create your class, and instanciate it to create your little object! That’s crafting Smalltalk using Smalltalk. Well, that is bootstrapping the meta model :).
The only ugly thing is that in order to create a new meta model with different instance variables, you have to create a transient class in the middle, because the VM does not like to have objects with a format X, whose class defines a format Y… So the hack just solves the format problem :).

Now you can think about simpler stuff like a class instance of itself, subclass of nil. Or more complex one :).

You can change it, I told you. Now it’s up to you how to use it…

The Bootstrap Chronicles Chapter 3 – It’s Alive!

So now that you now a bit what the bootstrap is about and what are some of the problems to face.  I’ll show you some solutions and progress for real.

How does the bootstrap implementation work

To bootstrap a Smalltalk environment, you need to create a new environment, with it’s classes and objects, and initialize some state in them. I could have done that in C, using mallocs and initializing everything by hand using plain memory :). But doing it in smalltalk is easier: you have late binding, polymorphism, closures…  Even, once you have done your first steps in the bootstrap, you can send messages to your objects. THAT is nice.

So, that is the way we cho0se to go: A new environment is created (a guest) into the current environment (the host).  The guest will have it’s own classes and objects.

What about the special objects of the vm?  We share them with the host environment, because if not, out new image will not be able to run… Afterwards, when this new image is written into an image file, we will swap references to point to our own new special objects array.

Here is a picture of how it looks like:

How the host and guest are related

The Current Version

I’ve been through several versions of the image with different capabilities, sizes, correctness. You have to know, everything you forgot to initialize, or initialize in a wrong order, or if you took an extra object you do not need, you will have a not running image, or one that carries all the objects in the host also…

Another thing is that current version takes a sample of the objects in the host to build the guest. I’m already working on starting from scratch + source code, but that is the future and I like enjoying the present :).

So, how do you load the current code?

Gofer it
    url: 'http://www.smalltalkhub.com/mc/Guille/Seed/main';
    package: 'ConfigurationOfHazelnut';
    load.

(ConfigurationOfHazelnut project version: '1.3') load.

Also, as the code is very sensitive on what you do,  it is also sensitive on what image you’re running it on.  If you play with it in a wrong/different image, you will have different/unexpected results.  So, I suggest you to use the same image as me for testing it: Latest Pharo 2.0.  In particular, I’ve tested it on versions 20133 and 20134.

Do not scare when loading the configuration on Pharo 2.0 for the first time It will raise an error. It is an issue related with unzipping old mcz in Pharo (http://code.google.com/p/pharo/issues/detail?id=6054) which will fortunately fixed soon. Just close the debugger, try again, and get the project working.

And, How do I try these weird stuff?

With Nicolas Petton, we have written some examples in the class named HazelBuilderExamples.

To run them, you can try the following scripts:

"Writes an image which when opened prints a spaceTally on fileok.txt"
HazelBuilderExample new buildImageWithSpaceTally

"Writes an image which when opened prints a report on all the packaging/initialization deficiencies of the image on fileok.txt"
HazelBuilderExample new buildImageWithBrokenReferencesReport

"Writes an image which when opened prints a report on all the packaging/initialization deficiencies of the image on fileok.txt. The report is written in the xml format Jenkins Junit plugin likes."
HazelBuilderExample new buildImageWithBrokenReferencesReportForJenkins

After evaluating this code, you’ll have some bootstrapped image and changes files. Open that file with your CogVM and wait until it closes.  Then have a look at the fileok.txt file :).

Of couse you can look at the code in the examples, and try to build your own. Email me if you have ideas to improve this :).

There are also probably some problems with file overwritting that came with some latest Pharo changes with the file management.  Please, if you notice this, just remove the bootstrapped* files from the folder where your image is and try again.

Hey! How is this thingy useful?

Well, if you’ve had a look at the examples, the three examples I’ve shown you are very useful:

  • The first one tells you how the space is distributed in the new image. You can use this knowledge to attack space problems if you want an even smaller image.
  • The second one is used to detect bugs in the pharo Packaging or in the initialization process: You have not initialized some classes, or the have been initialized but the initialization code does not initialize all the variables.
  • The third one is the same as the second, but adapted to have this kind of CI integration: http://car.mines-douai.fr/ci/job/Seed%20Broken%20References%20Report/

Nice huh? Now we can use jenkins to validate the core of Pharo is well initialized and well packaged when that list becomes empty.

What’s next?

  • Each one of those things in the list should be tracked as issues.
  • continue working on the bootstrap from sourcecode.
  • Making the fuel seed work so we can install fuel packages on our little image.

Keep u updated!

Hasta Luego!

Guille