Tag Archives: Image

Recover a broken image with Oz

While hacking the VM the other day, I went into a very special situation: I got an image that crashed on startup. And I had stuff without commit!! So I started the crusade to recover it.

I know that some time ago, people like Sean De Nigris had the same problem and ended up hacking the VM to solve their problem. You can have an idea of what exactly happened by reading the original mailing list post.

But doing that required debugging the vm at the bytecode level (that is, debugging each bytecode internally) until I reach my point and solve my problem.

The concrete situation

The last thing I executed in my broken image was the following two statements in one doit:

Smalltalk snapshot: true andQuit: true.
someObject doSomethingThatCrashesTheVM.

So when my image was started, it executed all the startup code, returned to the context of the code above, and tried to send the doSomethingThatCrashesTheVM message to someObject, causing a crash.

I knew that Oz could do it, so I went into it.

The recovery environment

I needed to put my image in a place where I could fix it. And that place is an object space, which I am implementing in Oz. The idea is to put the broken image inside another image with an object space, so we can manipulate it freely, fix it, and restart it. With the Oz library and VM, creating the object space is as easy as:

objectSpace := Pharo20 loadFrom: 'broken.image'.

The diagnostic

Once we have the objectSpace object, we need to understand how to solve our problem. So I wondered around by getting the scheduler of the loaded image, and looking at the active process.

ps := (objectSpace specialObjectsArray at: 4) value asSchedulerMirror.
activeProcess := ps activeProcess.

And once we have the active process, we can have a look at its context and the method that is being executed. We can follow the sender chain until we reach the context with the problem:

cm := objectSpace
          fromRemoteCompiledMethod: activeProcess context sender method.
cm decompile ==> ' DoIt
	Smalltalk snapshot: true andQuit: true.
	^ someObject doSomethingThatCrashesTheVM.

Now that we are there, we have to fix it.

The medicine

There are many ways to manipulate the contexts and and processes to solve that. But there is also one that is really easy and simple.

We only need to change the someObject reference to nil. That way, we avoid the crash in the startup and obtain a debugger instead with a:

“Undefined Object does not understand doSomethingThatCrashesTheVM”

Nice! So, in order to do that, I needed to understand a bit how the code was compiled. I did it by sending a couple of messages to the compiled method, to understand its structure. Finally, I got the following conclusion: someObject was the first literal in the compiled method. And I fixed it:

association := cm literalAt: 4.
association instVarAt: 2 put: objectSpace nilObject.

I restarted the image in the object space (with the so far ad-hoc method):

[[ objectSpace giveChanceToRun. true ] whileTrue: [ ]] forkAt: 80.

So you make a loop and in each iteration you give the object space a window of time to run. And after that the broken image will appear before you (if you fixed it reasonably well) :).
Then, I saved it with another name, restored manually the changes file (because my object spaces solution does not handle yet :)) and voilá. I had my image back again, with all my objects there. I and could commit :).

The Bootstrap Chronicles Chapter 3 – It’s Alive!

So now that you now a bit what the bootstrap is about and what are some of the problems to face.  I’ll show you some solutions and progress for real.

How does the bootstrap implementation work

To bootstrap a Smalltalk environment, you need to create a new environment, with it’s classes and objects, and initialize some state in them. I could have done that in C, using mallocs and initializing everything by hand using plain memory :). But doing it in smalltalk is easier: you have late binding, polymorphism, closures…  Even, once you have done your first steps in the bootstrap, you can send messages to your objects. THAT is nice.

So, that is the way we cho0se to go: A new environment is created (a guest) into the current environment (the host).  The guest will have it’s own classes and objects.

What about the special objects of the vm?  We share them with the host environment, because if not, out new image will not be able to run… Afterwards, when this new image is written into an image file, we will swap references to point to our own new special objects array.

Here is a picture of how it looks like:

How the host and guest are related

The Current Version

I’ve been through several versions of the image with different capabilities, sizes, correctness. You have to know, everything you forgot to initialize, or initialize in a wrong order, or if you took an extra object you do not need, you will have a not running image, or one that carries all the objects in the host also…

Another thing is that current version takes a sample of the objects in the host to build the guest. I’m already working on starting from scratch + source code, but that is the future and I like enjoying the present :).

So, how do you load the current code?

Gofer it
    url: 'http://www.smalltalkhub.com/mc/Guille/Seed/main';
    package: 'ConfigurationOfHazelnut';
    load.

(ConfigurationOfHazelnut project version: '1.3') load.

Also, as the code is very sensitive on what you do,  it is also sensitive on what image you’re running it on.  If you play with it in a wrong/different image, you will have different/unexpected results.  So, I suggest you to use the same image as me for testing it: Latest Pharo 2.0.  In particular, I’ve tested it on versions 20133 and 20134.

Do not scare when loading the configuration on Pharo 2.0 for the first time It will raise an error. It is an issue related with unzipping old mcz in Pharo (http://code.google.com/p/pharo/issues/detail?id=6054) which will fortunately fixed soon. Just close the debugger, try again, and get the project working.

And, How do I try these weird stuff?

With Nicolas Petton, we have written some examples in the class named HazelBuilderExamples.

To run them, you can try the following scripts:

"Writes an image which when opened prints a spaceTally on fileok.txt"
HazelBuilderExample new buildImageWithSpaceTally

"Writes an image which when opened prints a report on all the packaging/initialization deficiencies of the image on fileok.txt"
HazelBuilderExample new buildImageWithBrokenReferencesReport

"Writes an image which when opened prints a report on all the packaging/initialization deficiencies of the image on fileok.txt. The report is written in the xml format Jenkins Junit plugin likes."
HazelBuilderExample new buildImageWithBrokenReferencesReportForJenkins

After evaluating this code, you’ll have some bootstrapped image and changes files. Open that file with your CogVM and wait until it closes.  Then have a look at the fileok.txt file :).

Of couse you can look at the code in the examples, and try to build your own. Email me if you have ideas to improve this :).

There are also probably some problems with file overwritting that came with some latest Pharo changes with the file management.  Please, if you notice this, just remove the bootstrapped* files from the folder where your image is and try again.

Hey! How is this thingy useful?

Well, if you’ve had a look at the examples, the three examples I’ve shown you are very useful:

  • The first one tells you how the space is distributed in the new image. You can use this knowledge to attack space problems if you want an even smaller image.
  • The second one is used to detect bugs in the pharo Packaging or in the initialization process: You have not initialized some classes, or the have been initialized but the initialization code does not initialize all the variables.
  • The third one is the same as the second, but adapted to have this kind of CI integration: http://car.mines-douai.fr/ci/job/Seed%20Broken%20References%20Report/

Nice huh? Now we can use jenkins to validate the core of Pharo is well initialized and well packaged when that list becomes empty.

What’s next?

  • Each one of those things in the list should be tracked as issues.
  • continue working on the bootstrap from sourcecode.
  • Making the fuel seed work so we can install fuel packages on our little image.

Keep u updated!

Hasta Luego!

Guille

The Bootstrap Chronicles Chapter 2 – Do not mess with the VM

Everything you want in life has a price connected to it. There’s a price to pay if you want to make things better, a price to pay just for leaving things as they are, a price for everything.

Harry Browne

And I found myself trying to bootstrap for real. But of course it was not going to be easy. I had to pay the Iron price.

The VM we use plays a very important role in the day to day development. It is the one in charge of defining the method lookup, garbage collection, some platform dependent code, some optimizations. And as it does some nice things for us, it also puts restrictions on what we do. Have you ever heard about the special objects array? The compact classes array? Primitives? We are going to talk a bit about them and other secrets, and how they bother in the bootstrap process :).

The Special Objects Array

The special objects array is an array shared between the VM and the image. This array points to some objects that are important or interesting to the VM, you can have a look at it inspecting the following expression in Pharo:

Smalltalk specialObjectsArray

If you have an overview, you will see some things like the Processor instance, the Array, Smalltalk, SmallInteger, Float, Compiled method, Semaphore classes, some Semaphore instances…

What does the vm do with them? It for example introduces hard validations against concrete classes -yeap, like checking if an instance’s class is the same as the object in it’s slot 20, which BTW is Character…

Ok…

Doing those validations through messages could be too expensive in terms of speed. If you want to be fast, you have to pay some price. If you want to have a tiny mermory footprint, you have to pay some price. There are side effects for decisions in general…

So, some may wonder, Why is this array so important for the bootstrap? Imagine I want to have a new Array class, Class class, Character class, and a new CompiledMethod class.  What should happen if the VM does not recognize them as I would like? CogVM only recognizes one special objects array.

The solution? Hack and cheat.  You choose, you can cheat on the VM side, or in the image side.  Each has a price to pay.  But today is not the day for telling you how I cheated :).

Now, look at the field 29 of the special objects array. It is another array, …

The Compact Classes Array

U remember about compact objects?  If not, you can refresh in here: https://playingwithobjects.wordpress.com/2012/05/30/understanding-object-formats-in-cogvm/.

In two words, compact objects do not have an extra header for the class pointer: they have some bits in it’s base header which is an index into this nice compact classes array, where it’s class is. This mechanism is normally used for classes with tons of instances, saving 1 header for each object. Complexity against space.

Here again, we have the same problem.  Even worse, having this guy here means that if I have my nice Array’ class, which is also compact, and it’s compact class index points to the original Array class, the method lookup will end up in the original class instead of mines :(.

The solution? Hack, and cheat.

The Primitives

Now think what happens when my bootstrap classes use primitives methods. It’s nice because the vm returns me the objects it wants :). It’s actually a sinptom of the last two points. But it is good to know that primitives can give you headaches…

Other problems? Of course…

Literals: I can’t change so easily SmallInteger’s class because it is an inmediate object for example. The same happens with the other numbers, or strings, or blocks.  They all give you headaches. Even if I could make it work with the VM, I should change the compiler to use a different set of classes…

Vm magic assumptions: Like class instances’ first three instance variables are superclass, method dictionary and format.  In that order. Try changing the order :).  Or doing something like:

myA := A new.
A become: 'hello'.
myA crashTheVM.

And there are some other like this. So far I know LinkedList, ProcessScheduler, and Class. Find your own Waldo!

You already know what the solution is, do you?

HACK AND CHEAT

Yeah, this is what the bootstrap is really about. And learning hardcore stuff too :). Of course I can’t tell you every detail because this post will be larger than anyone would care to read.

So, I’ll keep you updated, I have now to continue paying the iron price.

$33 ¥0µ £473r!

9µ1££3

Bootstrapping: finding the missing link

A few months ago I got involved in some crazy project: bootstrapping Pharo. I took some existing code, played with it, hacked it, modified it, understood it. Now I think I have some idea of what is a bootstrap and what are it’s advantages. I’ll try to give a brief introduction to the project: what is it about, advantages, an overview of the current military secret results, and an insight of what is to come.

I recommend you to have look at my last post (the image dilemma) before reading.

This project is one of the ESUG projects supported this year by the google summer of code program.

What is a Bootstrap

The encyclopedic definition: A Bootstrap of system is a process that can generate the smallest subset of that system that may be used to reproduce the complete one.

I mean, you have an explicit process that can generate a the minimal version of your system.

Ok, easier: You kick yourself to get impulse and start from a better place :).

Then, bootstrapping software systems or languages normally means that you will somehow enhance the original process that created the system.

Some examples to clarify:

  • When your computer is turned on, some one has to bootstrap the little program able to load other programs :).
  • Generating a development environment with very basic tools will improve your work a lot (when you use your development environment).
  • C compiler is written in C. That means that It somehow compiles itself. Of course if you do not like assembly much, reading the C implementation is a great improvement.
  • Pharo implements traits and uses them in the core of the system to empower the design.
  • A big part of Pharo’s VM is written in smalltalk!

The need of bootstrapping Pharo

Did you know the image you download from the pharo website is the same one that comes from years ago? I mean, not the exactly same one, but a binary copy :). The fact is that years ago (yeah, ancient history) god created the first image and it started evolving, one little change after the other, to the Pharo we know today.

Now, as in evolution, we found in Pharo missing links.  We do not know how some object became the way it is. Or how it was initialized.  The code is simple nowhere, it’s a missing link. Also, as years passed, our ancestor became chaotic. It grew in many different uncontrolled and unordered ways. Since the Pharo Project started, one if it’s goals was cleaning this mess, but re-modularising and cleaning the system is a hard, long, and bothersome process.

The outputs and advantages of Bootstrapping Pharo will be:

  • getting tools to detect problems: bad dependencies, unexistent initializations, code that really do not work but was never executed before.
  • This initialization process will be explicit and open.
  • We will be able to start the next Pharo from scratch, and since we will be able to change this explicit process, our next generation Pharo will be cleaner and fancier. It will be able to acquire easily new features: namespaces/modules, security, remote tools, mirrors.
  • But also, since it will allow people to create a custom system, researchers will have an invaluable tool to fulfil their own purposes.  They will be able to experiment

Current status of the project

The project has already had a first output, which is the image writer.  The image writer is a little tool that traces a graph from an SmalltalkImage object, and deploys that graph into a .image file.  I’ll talk about this sub project in a future post.

The rest of the code is a military secret yet. Ok, I can give it to you, but you have to be responsible if it blows up on your face :).

The results of the project so far are:

  • It can create an Smalltalk image living inside another image.
  • This inner/guest image can be written in a new .image file.
  • With this approach a small kernel of 1.1MB has been reached.
  • SpaceTally runs and prints reports to understand how the space is distributed among objects.
  • A tool to detect every uninitialized class variable/class instance variable and references to unexistant globals was developed.

Soon we will have all these public on Pharo Jenkins server.

Next Steps

  • Jenkins jobs :)
  • Taking jenkins feedback to speed up the cleaning process
  • Remodularizaton to get even a smaller kernel
  • Maybe some little experiment to bootstrap MicroSqueak and learn from it
  • Bootstrap from source code.

I’ll try to keep you updated often.  BTW, any ideas, critics or suggestions are welcome. I’m not a gurú, I’m just learning, as everyone :).

Saludos!

The Image Dilemma

Without self knowledge, without understanding the working and functions of his machine, man cannot be free, he cannot govern himself and he will always remain a slave.
Goerge Gurdjieff

Many people I’ve talked to think Smalltalk is weirdy because it has an image.  Funny thing is most people that thinks that, do it because they feel it’s different.  Or they do Java.  Of course, Java and Smalltalk are different :). Okok, joke, I’m not interested in flamewars. Seriously what puzzles me is that normally people’s opinion on technology is based on feelings instead of concrete rational arguments.

So, let’s think a bit on the pros and cons of using an image and not using it.  Really. Only after understanding a bit you can decide whatever you want to use. That’s what the quote at the start is there for :).

Also, this post will be one of the corner stones to explain the bootstrap gsoc project basics later ;).

Non Image Based Development Environments

Ok, we all know these. It’s the kind of environment we use when we code in C, Java, Python, Ruby, JavaScript and many other.

We have source files which will be somehow interpreted on runtime or traduced to machine code in an executable file.  Naively explained, the interpreted ones normally depend on another program normally called interpreter or VM and the other ones depend only the first time on a compiler. So far, nothing strange.

Image Based Development Environments

Image based environments are the main topic of this post.  But before talking about it’s pros and cons, we have to define what is an image. And to avoid preconceptions, let’s name them snapshots. A snapshot is a photo taken of a system in a moment of time x, from which we can rebuild the whole system again.

If you have ever used VirtualBox or VMWare solutions you know what I am talking about: you have, for example, a linux mint virtual machine, which can be opened with your virtual box software.  If you save the state of your linux mint with virtual box, the next time you open it, it will wake up in the same state you left it. It will not start everything from scratch unless you tell it explicitly. And if you quit without saving, the next time will open without your changes.

It’s a very simple concept:  you are treating a whole running system (a linux machine in this case) as any of your other data files.

So that’s how most of the smalltalk systems work: you have a running smalltalk system, and when you wake it up, it will start from where it was left. Even if you was in the middle of a computation, this computation will continue running.

Comparison

System Configuration

How heavy or reproducible is our system configuration?

non image based

When a program start it normally does some basic initialization, configuration loading, setup stuff…  Every time our program does this same basic stuff.  It’s a repetitive task, but it’s done by the machine, so I’m ok with it.

Even more, one of the advantages of loading the application again from scratch is that you are testing your building process. And having a working artifact after ensures that you will be able to rebuild it in the future following the same steps.

But have you ever wondered how much time takes a java class loader to load the classes you will use?  And how much time takes Hibernate or Spring or JBoss to read your configuration, and provide an usable environment?  And how about to compile a large C/C# application? Ok, those times are machine time. You can go to take a coffee every time you restart your tomcat, or change a source file with many dependencies. That time is not even important in a deployed product, since it is delayed to the user or webserver which may run it once every hours/days. But it is important some times.  For example, when you are developing software. Software development is a highly demanding task which requires concentration. Taking a coffee every 15 minutes makes you lose concentration. And you can’t avoid checking/compiling/testing your code for long periods of times because handling lots of changes only in your head it’s a really hard task.

image based

When an snapshot is taken, the whole objects and computations that were occupying some memory in your program will be saved in a file in the state they were.  Object references are not broken. Then, when you load your smalltalk snapshot again, the objects take place in memory, and continue working as they were.  Normally there is not heavy initialization on image startup other than reallocating resources from outside the environment (files, sockets are OS resources and become invalid for sure when the system is halted).  You don’t have to configure almost none of your program neither, because it is already configured!

Now, since you don’t have to reinitialize your configurations from scratch every time, maybe you initialized it, then accidentally deleted the method that performed the configuration. And then you are screwed, because if you want to start from scratch you will have pieces of your program lost that will make it impossible.  This is why we may consider an image based system and evolving environment.  Because the system is changed by mutations that may get lost or untracked.

The Development process

How is our development process altered by the environment we use?

non image based

  1. Write code in a file
  2. Compile or similar or however you want to call it :).
  3. Run the whole program from scratch
  4. Test
  5. Go to 1

This process is slooow.  Specially because of steps 2/3.  For every change you do, no matter if it is long of little, you have to rebuild everything and lost time with it, and reinitialize, and reconfigure…  Even if you only wanted to replace a 2 by a 3, or fix a typo in a string.

However, there exist currently some tools that implement what they call hot deployment which is nothing else that exploiting some dynamic features that let a program change while it’s running.  But hey!  That’s what we do with an image based approach: we change the program while it’s running.

Anyway, those extra steps makes you gain reproducibility by making some parts of the process a bit more explicit.  At the cost of extra bureaucracy.

image based

  1. Open your image
  2. Write code
  3. Evaluate/Accept
  4. Test
  5. Go to 2

Here the main difference is that the modification of our program is made of little deltas during execution.  We replace/change/create methods while our whole program is running.  We create classes, objects, modify class hierarchies while our whole program is running.  We alter our objects state while our whole program is running. This really improves the development process, since the time you have to wait to receive feedback from execution or testing is almost none. And there is less overhead, less context switch in your head.

Now the problem is that to reproduce our program we have to track all the little changes we’ve done.  And understand them, and discard the ones that really overlap o do not make sense.  In the Smalltalk System Pharo we have a .changes file that works as a log of the changes done.  This changes file aims to reduce the impact of this dynamic feature disadvantages.

IDE’s Features

Have you ever used an IDE with refactoring support, syntax highlighting, nice code browsing capabilities?  What do our tools need in order to provide us all those gourgeous features?

IDE are meta-programs. They are programs that help us manipulate programs: modify them, understand them, query them.  And to do that they can do it by:

  • directly modifying source code without a model behind.  Painful and Hard.  I would never try to do it that way :).
  • modeling the concepts of the language they will manipulate to make it easier.

non image based

A program’s meta model (a model representing program entities such as classes or variables instead of domain entities like bank accounts and clients) is often generated when the program runs if you have reflective capabilities in the manipulated language. But IDEs should work while the program is not running, or even if it does not provide reflection. So, how does this approach build this kind of tools?

They have an alternative model and an alternative parser/compiler which generates this model from sourcecode.  A lot of extra work is needed for information that is already there. The problem is that information is lost in text files with source code instead of being stored in a more malleable format.

image based

Our image based approach can contain all the meta information in the snapshot, alive, as first class objects.  This means that we do not have to create a duplicated meta-model, nor an extra parser/compiler.  We can use the ones that are alive in the image.  We make use of the model created by the language, what we can’t do in the non image based approach.

Files or no files

Some times you feel comfortable modifying files, some times not.

non image based

You have files with your source code. You can use them to execute/compile your program, you can store them in svn/git/bazar repositories, you can diff/grep/more/less/find over them just using a terminal.

I think this is the main pro over the image based approach. There are plenty of tools working on plain text files you can use if your source code lives in this kind of storage.

But try to write a program implementing an extract method refactoring over a piece of text.  You have to write a very complex and large program analyzing classes, methods, scopes…

In two words: It’s really cool to use existing tools.  But it is not that nice to write your own.

image based

As the opposite of the not having an image, we can’t just use our nice text manipulation tools on our image file.  We have to rebuild them on our system, or externalize the source code in files to use them.  And then re-insert the feedback in the system.

But there are other tasks-like refactorings and meta programming in general- that become very simple just by the fact of manipulating objects instead of text.

In the Pharo project there are plenty of existing and arising tools aiming us to interact with the outer world, just like:

Conclusion

As I’ve shown you, these two approaches have their good and bad stuff.  None of them is the silver bullet, and we should be aware of that to be better developers.

I hope you are a little more free now ;).

さようなら!