ObjectTravel. A tool to traverse object references

ObjectTravel is a tool to deeply traverse "native" references of objects through instance variables and "array contents".

     Metacello new
         baseline: 'ObjectTravel';
         repository: 'github://dionisiydk/ObjectTravel';
         load.

Usage is quite simple: just create traveler instance on your object and send #referencesDo:

      traveler := ObjectTraveler on: (1@2 corner: 3@4).

      traveler referencesDo: [:eachRef | eachRef logCr].

It will print on transcript:

      (1@2)

      (3@4)

      1

      2

      3

      4

There is convenience method to collect all references:

      traveler collectReferences. "==> an OrderedCollection((1@2) (3@4) 1 2 3 4))

Traveler enumerates each reference in breadth-first direction and visit every reference only once:

      point := 1@2.

      traveler := ObjectTraveler on: {point . #middle. point}.

      traveler collectReferences "==> an OrderedCollection((1@2) #middle (1@2) 1 2)"

"point" is referenced two times. But traveler will go deeply only once. So circles are supported:

      array := Array new: 2.

      array at: 1 put: 1@3.

      array at: 2 put: array.

      traveler := ObjectTraveler on: array.

      traveler collectReferences asArray = { 1@3. array. 1. 3 } "==> true"

You could control traveler to skip particular objects:

      rect := 1@2 corner: 3@4.

      traveler := ObjectTraveler on: rect.

      traveler skip: rect origin.

      traveler collectReferences "==> an OrderedCollection((1@2) (3@4) 3 4)"

(it can be done during enumeration too).

Also you could apply predicate for objects which you want traverse:

      rect :=
      traveler := ObjectTraveler on: {1@2 corner: 3@4. 5@6}. 

      traveler traverseOnly: [:each | each isKindOf: Point]. 

      traveler collectReferences "==> an OrderedCollection((1@2) corner: (3@4) (5@6) 5 6)"

There are another convenient methods:

      traveler countReferences

It will count all references in object graph.

      traveler copyObject

It will make deep copy of original object which is similar to #veryDeepCopy: cyclic references are supported. 

Also traveler could find all paths to given object in object graph:

      root :=  {#one -> #target. {#emptyArray}. {#two -> #target}}.

      traveler := ObjectTraveler on: root.

      traveler findAllPathsTo: #target "==>

           {#one -> #target. root}

            {#two -> #target. {#two -> #target}. root} "

During enumeration traveler could replace visited references:

      rect := 1@2 corner: 3@4.

      traveler := ObjectTraveler on: rect.

      traveler referencesDo: [:each | 

           each isInteger ifTrue: [traveler replaceCurrentReferenceWith: each + 1]].

      rect "==> 2@3 corner: 4@5"

It would be fun project to connect ObjectTraveler to XStreams to make any object "streamable":

      (1@2 corner: 3@4) traveling reading count: [:each | each isInteger].

      (1@2 corner: 3@4) traveling reading get; get; read: 2; rest.

      (1@2 corner: 3@4) traveling writing write: 100@500

Update: project was moved to github

Major Seamless update

I glad to publish new version of Seamless (0.8.2) together with new doc.
It could be loaded by:

       Metacello new
           baseline: 'Seamless';
           repository: 'github://dionisiydk/Seamless';
           load

It works in Pharo 5 and 6. As usually feedback is welcome.

It is complete redesign of original version with the goal to make it more flexible, reliable and simple. (original version was created by Nikolaos Papoulias).

New version introduces new project Basys as foundation for new design.
Basys implements an abstract layer for networks which require bidirectional communication between clients and servers. (details here).
Seamless implements Basys network to organize transparent communication between distributed objects.

To use Seamless SeamlessNetwork should be created on client and server:

       network := SeamlessNetwork new.

To accept connections server should be started:

       network startServerOn: 40422.

Clients could connect to server and retrieve remote environment:

       remotePeer := network remotePeerAt: (TCPAddress ip: #[127 0 0 1] port: 40422).
       remoteSmalltalk := remotePeer remoteEnvironment.

       "or short version"
       remoteSmalltalk := network environmentAt: (TCPAddress localAt: 40422)

remoteSmalltalk here is proxy which delegates any received message to remote object. Remote messages are executed on server which return result back to client. Result can be returned as another proxy or as copy which contains another proxies.

In example result is reference to remote Smalltalk instance. It can access globals from remote environment:

       remoteTranscript := remoteSmalltalk at: #Transcript.
       remoteTranscript open; show: 'remote message'; cr

It will open transcript on server and print text on it.

Arguments of remote message are transferred to server with same logic as message result transferred to client. On server arguments can include proxies and server can send messages to them:

       remoteTranscript print: #(1 2 3 4 5)

Here array will be passed to server as reference. Then on server transcript will interact with it to print it. And as result client will receive messages from server.

Concrete transfer strategy is depends on transferred object. It is specified in method #seamlessDefaultTransferStrategy:

       Object>>seamlessDefaultTransferStrategy
            ^SeamlessTransferStrategy defaultByReference

       Number>>seamlessDefaultTransferStrategy
            ^SeamlessTransferStrategy defaultByValue

Default strategy could be overridden on network level:

       network transferByValue: (Instance of: Point).
       network transferByReference: (Identical to: #specificSymbol)

It allows to tune network for specific application to optimize communication between distributed objects. There are few other transfer strategies which allow minimize remote communication. Most interesting allows properties caching. It will transfer object reference together with specified properties. Following example will configure network to transfer class reference together with name:

       network transferByReference: (Kind to: Class) withCacheFor: #(name)

And then proxy for remote class will return #name from local cache instead of real remote call.

Previously announced project RemoteDebuggingTools uses all these feature to minimize communication between debugger and remote process. It allows to reduce number of remote messages for opening debugger from 800 to 13.

Now another important feature: block evaluation on remote peers:

       remotePeer evaluate: [1 + 2]. "==>3"

Given block is transferred to remote side and evaluated. Result is returned to client. As in other cases it could be proxy or normal object.

Block could use globals. On remote side they will be local globals of this remote environment. Following example will show notification on remote image:

       remotePeer evaluate: [Object inform: 'message from remote image'].

Temps and workspace variables can be used too. They will be transferred according to own strategies:

       | result |
       result := OrderedCollection new.
       remotePeer evaluate: [100 to: 500 do: [:i | result add: i factorial ]].

Non local return is also supported in regular Smalltalk semantics:

       remotePeer evaluate: [1 to: 10 do: [:i | i>5 ifTrue: [^i] ] ]. "==>6"

Also block could be evaluated asynchronously without waiting any result:

       | result |
       result := OrderedCollection new.
       remotePeer evaluateAsync: [result add: 1000 factorial]. "it will not wait result"

Seamless provides integration with GT tools. Remote proxies could be inspected to explore remote objects state with ability to execute remote scripts (doIt, printIt). It is shown on remote debugging demo.

To analyse remote communication Seamless implements special tool SeamlessLogger. It is explained in doc.  Here is little example for following code:

       remoteRect := remotePeer evaluate: [0@0 corner: 2@3].
       remoteRect area. "==>6".
       localRect := 0@0 corner: 5@2.
       remoteRect evaluate: [remoteRect area + localRect area] "==>16"

Statistics will show number of messages, receivers and bytes which was transferred over network in dimension of receiver class or message selector:

At the end I must say about two important issues which will be covered in next versions:

1) No garbage collection

SeamlessNetwork keeps all objects which was transferred by reference. They will never be cleaned while network is live. 

Now It could be done manually by evaluating "network destroy". It will clean all object caches and close all connections. It could be not safe because remote peers could use this objects. Seamless tries to handle it properly with clear errors in such cases. 

In future unused distributed objects will be cleaned automatically.

2) No security: no authorization and encryption. Now for security purpose you need external tools like VPN

Update: project was moved to github

Remote debugging tools is ready

I glad to release first version of RemoteDebuggingTools project. It allows explore and debug remote images.
Here is demo with debugging Seaside application:

On server side you need to install Server group of project:

Metacello new
        smalltalkhubUser: 'Pharo' project: 'RemoteDebuggingTools';
        configuration: 'RemoteDebuggingTools';
        version: #stable;
        load: 'Server'.

Then server should be started on port where client image could connect:

RemoteUIManager registerOnPort: 40423.

Also image could be started with running server from command line:

./pharo PharoWithServer.image debuggerServer --port=40423

On client side Client group should be installed

Metacello new
        smalltalkhubUser: 'Pharo' project: 'RemoteDebuggingTools';
        configuration: 'RemoteDebuggingTools';
        version: #stable;
        load: 'Client'.

Then RemoteDebugger could connect to target image:

debugger := RemoteDebugger connectTo: (TCPAddress ip: #[127 0 0 1] port: 40423).

It registers a RemoteDebugger as debugger tool on remote image. Any error there will open debugger on client image with remote process stack.

With connected debugger instance you can execute scripts:

debugger evaluateAsync: [ [1/0] fork ].
debugger evaluate: [ 1 + 2 ] "==> 3".
debugger evaluate: [ 0@0 corner: 2@3 ] "==> aSeamlessProxy on remote rectangle".

Async evaluation not returns any result of block. It just transfer block to remote side and not waits any result.
Normal evaluation will wait until block competes on remote side. And result will be returned. There is nice integration with GTTools to inspect remote objects.

Blocks could reference local objects like temps or workspace variables. You can write scripts like:

result := OrderedCollection new.
debugger evaluateAsync: [100 to: 500 do: [ :i | result add: i factorial] ].

Non local return is also supported. Following expression works same way as local block evaluation:

debugger evaluate: [ 100 to: 500 do: [ :i | i > 200 ifTrue: [^i] ]].

This project (with scripting facilities) are based on new version of Seamless which will be announced soon next time. It misses very important feature now: distributed garbage collection. When you finish your debugging session you need manually disconnect your debugger. It will clean all garbage:

debugger disconnect.

Other restrictions which will be covered in future:

• debugging expressions from debugger is not working (debugIt from context menu)
• code browse and navigation are not working. Nautilus and MessagesBrowser should be able to work with remote environment.
• remote stack items are printed with prefix SeamlessProxy. It should be improved
• printIt inside debugger/inspector should return remote printString instead of aSeamlessProxy.  
• no security: no authorization and encryption. Now for security purpose you need external tools like VPN

Update. This project is outdated. Use PharmIDE instead which includes full toolset for development: remote browser, remote playground, remote inspector and remote debugger.

Magic with Pharo Reflectivity

Last days I play with idea of compile time evaluation without special language support.
For example in Dolphin Smalltalk there is special syntax for this:

##(100 factorial)

Factorial will be evaluated at method compilation time and result will be saved as literal. And this precomputed value will be used during method execution.

In Pharo we can do this without special syntax. Thank's to powerful reflection system which called Reflectivity. It allows reflection on the level of AST nodes with different kind of transformations.
With Reflectivity we could substitute particular expression with result when method will be executed first time. It is a bit different than compile time evaluation. But after first method execution it will be same: precomputed value of expression will be stored and used as regular literal.

And now magic:

100 factorial asMethodConst

I was quite impressed when I realize that this approach could work. First I tried to use blocks but it was completely not needed:

Object>>asMethodConst
| constNode link |
constNode := thisContext sender sourceNodeExecuted.
link := MetaLink new
metaObject: self;
control: #instead.
constNode link: link.
^self

#asMethodConst will be executed on result of expression. Trick here is that constNode is "message send" ast-node where receiver is target expression. Metalink with control #instead will substitute it as whole and receiver will not be executed anymore.

Wondering how little code is required. It will be available in Pharo soon (18768).