The Rich Engineering Heritage Behind Dependency Injection

Andrew McVeigh takes us on a tour of the rich heritage behind dependency injection, what it represents, and tells us why its here to stay.

Java, the OLPC, and community responsibility

The "One Laptop Per Child" project has a great device ready to ship, but there's no Java on there. Let's think about working together to put Java on OLPC!

Replies: 9 - Pages: 1
Threads: [ Previous | Next ]

[CDO] Question about LRU cache and tx.commit()

At 1:08 PM on Aug 4, 2008, Cédric Brun wrote:

Hi Eike and Simmon (or anybody familiar with CDO internals),

I'm still digging in CDO, and I wrote yet another dummy testcase just filling a model with thousands of instances using a fake backend which does not save anything at all (the addRevision do nothing so I have no references on the java objects). Of course I'm not even trying to browse the model on the repository

My loop was something like this :

int nb_loops = 10000;
for (int i = 0; i MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}

I was surprised to see that my test ended up with an OutOfMemory error

Digging with the *great* memory analyzer I've seen that all the revisions were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering, shouldn't this state be cleared after a commit ?

I changed my my test to create a new transaction and close it on each loop, but it did not helped as I guess the resolver is linked to the session instance.

So I'm wondering, when should the CDORevisionResolverImpl be cleared ? And more especially the "revisions" hashmap ?

Just in case you're wondering I'm using the ganymede CDO.

Thanks,

Cédric

9 replies so far ( Post your own)

Reply 1. At 2:07 PM on Aug 4, 2008, Eike Stepper wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Hi Cédric,

Comments below...

Cédric Brun schrieb:
> Hi Eike and Simmon (or anybody familiar with CDO internals),
>
> I'm still digging in CDO, and I wrote yet another dummy testcase just filling a model with thousands of instances using a fake backend which does not save anything at all (the addRevision do nothing so I have no references on the java objects). Of course I'm not even trying to browse the model on the repository ;)
>
> My loop was something like this :
>
> int nb_loops = 10000;
> for (int i = 0; i > MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
> newF.setName("" + i);
> root.getMembers().add(newF);
> tx.commit();
> }
>
> I was surprised to see that my test ended up with an OutOfMemory error
>
> Digging with the *great* memory analyzer I've seen that all the revisions were kept by CDO through the CDORevisionResolverImpl
> Setting the LRU cache to 0 did not helped and then I am wondering, shouldn't this state be cleared after a commit ?
>
Setting the LRU cache to 0 is a bad idea since it disables eviction
completely. I've added respective JavaDoc. If you feel that -1 is a
better value please open a Bugzilla.
Please try to set the capacity to a value greater than zero but small
enough that you don't run into OutOfMemory.
The revisedLRUCapacity can be much smaller than the currentLRUCapacity
if you don't use audit views.

> I changed my my test to create a new transaction and close it on each loop, but it did not helped as I guess the resolver is linked to the session instance.
>
> So I'm wondering, when should the CDORevisionResolverImpl be cleared ? And more especially the "revisions" hashmap ?
>
Why should it be cleared without need? It's a cache to prevent from
continuous server requests or database queries for missing revisions.
You might ask why I didn't choose a memory sensitive cache
implementation. This cache is the most prominent cache in the server
side repository. With a memory sensitive implementation it would not be
possible to influence which elements to select on eviction. In other
words, no eviction policy would be possible, thereby degrading the cache
hit rate.

Cheers
/Eike

Reply 2. At 3:24 PM on Aug 4, 2008, Simon McDuff wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Which value is your LRU set ? (Client and server)

I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.

Can you send us a complete test (or even better a testcase) ? We will know
exactly your settings you are using.

By the way, do not forget that you need to configure 2 cache.... client
cache and the server cache.
Are your client and server the same java process ? How much memory you JAVA
can use ?(maximum)

"Cédric Brun" a écrit dans le message de news:
g77d26$rs8$1 at build dot eclipse dot org...
> Hi Eike and Simmon (or anybody familiar with CDO internals),
>
> I'm still digging in CDO, and I wrote yet another dummy testcase just
> filling a model with thousands of instances using a fake backend which
> does not save anything at all (the addRevision do nothing so I have no
> references on the java objects). Of course I'm not even trying to browse
> the model on the repository ;)
>
> My loop was something like this :
>
> int nb_loops = 10000;
> for (int i = 0; i > MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
> newF.setName("" + i);
> root.getMembers().add(newF);
> tx.commit();
> }
>
> I was surprised to see that my test ended up with an OutOfMemory error
>
> Digging with the *great* memory analyzer I've seen that all the revisions
> were kept by CDO through the CDORevisionResolverImpl
> Setting the LRU cache to 0 did not helped and then I am wondering,
> shouldn't this state be cleared after a commit ?
>
> I changed my my test to create a new transaction and close it on each
> loop, but it did not helped as I guess the resolver is linked to the
> session instance.
>
> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
> And more especially the "revisions" hashmap ?
>
> Just in case you're wondering I'm using the ganymede CDO.
>
> Thanks,
>
> Cédric

Reply 3. At 3:45 PM on Aug 4, 2008, Eike Stepper wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Simon McDuff schrieb:

Which value is your LRU set ? (Client and server)

I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.

Can you send us a complete test (or even better a testcase) ? We will know 
exactly your settings you are using.

By the way, do not forget that you need to configure 2 cache.... client 
cache and the server cache.
Are your client and server the same java process ?

Just a side note: Currently even if both client and server are running in the same JVM process they would not share the same revision manager instance.

Cheers
/Eike

How much memory you JAVA 
can use ?(maximum)



"Cédric Brun" 
<cedric dot brun at obeo dot fr>
 a écrit dans le message de news: 
g77d26$rs8$1 at build dot eclipse dot org
...

Hi Eike and Simmon (or anybody familiar with CDO internals),

I'm still digging in CDO, and I wrote yet another dummy testcase just 
filling a model with thousands of instances using a fake backend which 
does not save anything at all (the addRevision do nothing so I have no 
references on the java objects). Of course I'm not even trying to browse 
the model on the repository 

My loop was something like this :

int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}

I was surprised to see that my test ended up with an OutOfMemory error

Digging with the *great* memory analyzer I've seen that all the revisions 
were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering, 
shouldn't this state be cleared after a commit ?

I changed my my test to create a new transaction and close it on each 
loop, but it did not helped as I guess the resolver is linked to the 
session instance.

So I'm wondering, when should the CDORevisionResolverImpl be cleared ? 
And more especially the "revisions" hashmap ?

Just in case you're wondering I'm using the ganymede CDO.

Thanks,

Cédric

<!---->

Reply 4. At 3:53 PM on Aug 4, 2008, Simon McDuff wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Exactly!!

I was asking the question about server and the
client to determine at which speed the memory should fill
up.

If they are in the same java process the memory
will fill up twice as faster exactly because we have 2 revisionmanager
in the same process.

If they are not, we need to determine which (client
or the server) have the exception.

The only things I can see so far is
the maximum memory use for the JAVA process is less than the number of
objects the cache(both of them) can hold.

Simon

"Eike Stepper" < href="mailto:stepper at sympedia dot de">stepper at sympedia dot de > a écrit dans le
message de news: href="mailto:g77m8g$ngq$1 at build dot eclipse dot org">g77m8g$ngq$1 at build dot eclipse dot org ...

style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">Simon
McDuff schrieb:

Which value is your LRU set ? (Client and server)


I know in my case I was able to load 51 millions objects...

OutOfmemory are always tricky.


Can you send us a complete test (or even better a testcase) ? We will know 

exactly your settings you are using.


By the way, do not forget that you need to configure 2 cache.... client 

cache and the server cache.

Are your client and server the same java process ?

Just a
side note: Currently even if both client and server are running in the same
JVM process they would not share the same revision manager
instance.

Cheers
/Eike

How much memory you JAVA 

can use ?(maximum)




"Cédric Brun" 
<cedric dot brun at obeo dot fr>
 a écrit dans le message de news: 

g77d26$rs8$1 at build dot eclipse dot org
...

Hi Eike and Simmon (or anybody familiar with CDO internals),


I'm still digging in CDO, and I wrote yet another dummy testcase just 

filling a model with thousands of instances using a fake backend which 

does not save anything at all (the addRevision do nothing so I have no 

references on the java objects). Of course I'm not even trying to browse 

the model on the repository 


My loop was something like this :


int nb_loops = 10000;

for (int i = 0; i < nb_loops; i++) {

MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();

newF.setName("" + i);

root.getMembers().add(newF);

tx.commit();

}


I was surprised to see that my test ended up with an OutOfMemory error


Digging with the *great* memory analyzer I've seen that all the revisions 

were kept by CDO through the CDORevisionResolverImpl

Setting the LRU cache to 0 did not helped and then I am wondering, 

shouldn't this state be cleared after a commit ?


I changed my my test to create a new transaction and close it on each 

loop, but it did not helped as I guess the resolver is linked to the 

session instance.


So I'm wondering, when should the CDORevisionResolverImpl be cleared ? 

And more especially the "revisions" hashmap ?


Just in case you're wondering I'm using the ganymede CDO.


Thanks,


Cédric

<!---->

Reply 5. At 9:05 AM on Aug 5, 2008, Cédric Brun wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Hi Simon,

> Which value is your LRU set ? (Client and server)
>
> I know in my case I was able to load 51 millions objects...
> OutOfmemory are always tricky.
>
> Can you send us a complete test (or even better a testcase) ? We will know
> exactly your settings you are using.
>
> By the way, do not forget that you need to configure 2 cache.... client
> cache and the server cache.

That's a usefull hint, I was indeed only configuring the client cache, configuring the server one seems way better (the test runs longer), but I still ends up with an OutOfMemory.

My caches are set to 100/50 for current/revised.

> Are your client and server the same java process ? How much memory you
> JAVA can use ?(maximum)

Client and Server are sharing the same process and I volontary limited heap space to 256m. My goal is to see wether CDO have leaks or not.

When I get to the OOM, the dump tells me I have 16 000 instances of LRURevisionHolder kept in the client revision manager. Digging a step further it appears that the session activation create a new RevisionManager with a 0 as a current/revised capacity (and then create non-evicting LRU).

This creation happens during the session creation which happens during a :
CDOSession session = (CDOSession) IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.

So I added a postProcessor which sets the current/revised LRU size and it's now working flawlessly

though I don't know if I've done it the way I should.

Thanks for your help, it was really valuable, as usual

Cédric
>
>
> "C?dric Brun" a ?crit dans le message de news:
> g77d26$rs8$1 at build dot eclipse dot org...
>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>
>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>> filling a model with thousands of instances using a fake backend which
>> does not save anything at all (the addRevision do nothing so I have no
>> references on the java objects). Of course I'm not even trying to browse
>> the model on the repository ;)
>>
>> My loop was something like this :
>>
>> int nb_loops = 10000;
>> for (int i = 0; i >> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>> newF.setName("" + i);
>> root.getMembers().add(newF);
>> tx.commit();
>> }
>>
>> I was surprised to see that my test ended up with an OutOfMemory error
>>
>> Digging with the *great* memory analyzer I've seen that all the revisions
>> were kept by CDO through the CDORevisionResolverImpl
>> Setting the LRU cache to 0 did not helped and then I am wondering,
>> shouldn't this state be cleared after a commit ?
>>
>> I changed my my test to create a new transaction and close it on each
>> loop, but it did not helped as I guess the resolver is linked to the
>> session instance.
>>
>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>> And more especially the "revisions" hashmap ?
>>
>> Just in case you're wondering I'm using the ganymede CDO.
>>
>> Thanks,
>>
>> C?dric

Reply 6. At 10:23 AM on Aug 5, 2008, Eike Stepper wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Hi Cédric,

Yes, currently an IElementProcessor is the only chance to configure the internals of the CDOSessionImpl (here: revisionManager) in an IManagedContainer. Could you please file a Bugzilla so that I can add respective properties to the CDOSessionConfiguration?

Cheers
/Eike

Cédric Brun schrieb:

Hi Simon,

Which value is your LRU set ? (Client and server)

I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.

Can you send us a complete test (or even better a testcase) ? We will know
exactly your settings you are using.

By the way, do not forget that you need to configure 2 cache.... client
cache and the server cache.

<!---->
That's a usefull hint, I was indeed only configuring the client cache, configuring the server one seems way better (the test runs longer), but I still ends up with an OutOfMemory.

My caches are set to 100/50 for current/revised.

  </pre>
  <blockquote type="cite">
    <pre wrap="">Are your client and server the same java process ? How much memory you
JAVA can use ?(maximum)
    </pre>
  </blockquote>
  <pre wrap=""><!---->

Client and Server are sharing the same process and I volontary limited heap space to 256m. My goal is to see wether CDO have leaks or not. 

When I get to the OOM, the dump tells me I have 16 000 instances of LRURevisionHolder kept in the client revision manager. Digging a step further it appears that the session activation create a new RevisionManager with a 0 as a current/revised capacity (and then create  non-evicting LRU).

This creation happens during the session creation which happens during a :
CDOSession session = (CDOSession) IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.

So I added a postProcessor which sets the current/revised LRU size and it's now working flawlessly  though I don't know if I've done it the way I should.

Thanks for your help, it was really valuable, as usual 

Cédric

"C?dric Brun" 
<cedric dot brun at obeo dot fr>
 a ?crit dans le message de news:
g77d26$rs8$1 at build dot eclipse dot org
...

Hi Eike and Simmon (or anybody familiar with CDO internals),

I'm still digging in CDO, and I wrote yet another dummy testcase just
filling a model with thousands of instances using a fake backend which
does not save anything at all (the addRevision do nothing so I have no
references on the java objects). Of course I'm not even trying to browse
the model on the repository 

My loop was something like this :

int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}

I was surprised to see that my test ended up with an OutOfMemory error

Digging with the *great* memory analyzer I've seen that all the revisions
were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,
shouldn't this state be cleared after a commit ?

I changed my my test to create a new transaction and close it on each
loop, but it did not helped as I guess the resolver is linked to the
session instance.

So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
And more especially the "revisions" hashmap ?

Just in case you're wondering I'm using the ganymede CDO.

Thanks,

C?dric

<!---->

Reply 7. At 5:46 AM on Aug 6, 2008, Cédric Brun wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Hi Eike,

Bug is tracked : https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279

Since I change the settings with the revised/current LRU, I'm now experiencing the following issue :

java.lang.IllegalStateException: Can not retrieve origin revision for org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl@5e34ce
at org.eclipse.emf.cdo.internal.server.Transaction.computeDirtyObjects(Transaction.java:290)
at org.eclipse.emf.cdo.internal.server.Transaction.commit(Transaction.java:179)
at org.eclipse.emf.cdo.internal.server.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:109)
at org.eclipse.net4j.signal.IndicationWithResponse.execute(IndicationWithResponse.java:46)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

I'm using a MEMStore for test purposes.

Do you have a hint about what's going wrong ?

Eike Stepper wrote:

> Hi Cédric,
>
> Yes, currently an IElementProcessor is the only chance to configure the
> internals of the CDOSessionImpl (here: revisionManager) in an
> IManagedContainer. Could you please file a Bugzilla so that I can add
> respective properties to the CDOSessionConfiguration?
>
> Cheers
> /Eike
>
>
>
> Cédric Brun schrieb:
>> Hi Simon,
>>
>>
>>> Which value is your LRU set ? (Client and server)
>>>
>>> I know in my case I was able to load 51 millions objects...
>>> OutOfmemory are always tricky.
>>>
>>> Can you send us a complete test (or even better a testcase) ? We will
>>> know exactly your settings you are using.
>>>
>>> By the way, do not forget that you need to configure 2 cache.... client
>>> cache and the server cache.
>>>
>>
>> That's a usefull hint, I was indeed only configuring the client cache,
>> configuring the server one seems way better (the test runs longer), but I
>> still ends up with an OutOfMemory.
>>
>> My caches are set to 100/50 for current/revised.
>>
>>
>>> Are your client and server the same java process ? How much memory you
>>> JAVA can use ?(maximum)
>>>
>>
>> Client and Server are sharing the same process and I volontary limited
>> heap space to 256m. My goal is to see wether CDO have leaks or not.
>>
>> When I get to the OOM, the dump tells me I have 16 000 instances of
>> LRURevisionHolder kept in the client revision manager. Digging a step
>> further it appears that the session activation create a new
>> RevisionManager with a 0 as a current/revised capacity (and then create
>> non-evicting LRU).
>>
>> This creation happens during the session creation which happens during a
>> : CDOSession session = (CDOSession)
>> IPluginContainer.INSTANCE.getElement(productGroup, type, description);
>> instruction.
>>
>> So I added a postProcessor which sets the current/revised LRU size and
>> it's now working flawlessly

though I don't know if I've done it the
>> way I should.
>>
>> Thanks for your help, it was really valuable, as usual :)
>>
>> Cédric
>>
>>> "C?dric Brun" a ?crit dans le message de news:
>>> g77d26$rs8$1 at build dot eclipse dot org...
>>>
>>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>>
>>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>>> filling a model with thousands of instances using a fake backend which
>>>> does not save anything at all (the addRevision do nothing so I have no
>>>> references on the java objects). Of course I'm not even trying to
>>>> browse the model on the repository ;)
>>>>
>>>> My loop was something like this :
>>>>
>>>> int nb_loops = 10000;
>>>> for (int i = 0; i >>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>>> newF.setName("" + i);
>>>> root.getMembers().add(newF);
>>>> tx.commit();
>>>> }
>>>>
>>>> I was surprised to see that my test ended up with an OutOfMemory error
>>>>
>>>> Digging with the *great* memory analyzer I've seen that all the
>>>> revisions were kept by CDO through the CDORevisionResolverImpl
>>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>>> shouldn't this state be cleared after a commit ?
>>>>
>>>> I changed my my test to create a new transaction and close it on each
>>>> loop, but it did not helped as I guess the resolver is linked to the
>>>> session instance.
>>>>
>>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
>>>> And more especially the "revisions" hashmap ?
>>>>
>>>> Just in case you're wondering I'm using the ganymede CDO.
>>>>
>>>> Thanks,
>>>>
>>>> C?dric
>>>>
>>
>>

Reply 8. At 5:56 AM on Aug 6, 2008, Eike Stepper wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Cédric Brun schrieb:

Hi Eike,

Bug is tracked :  
https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279


Since I change the settings with the revised/current LRU, I'm now experiencing the following issue :

java.lang.IllegalStateException: Can not retrieve origin revision for org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl@5e34ce
	at org.eclipse.emf.cdo.internal.server.Transaction.computeDirtyObjects(Transaction.java:290)
	at org.eclipse.emf.cdo.internal.server.Transaction.commit(Transaction.java:179)
	at org.eclipse.emf.cdo.internal.server.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:109)
	at org.eclipse.net4j.signal.IndicationWithResponse.execute(IndicationWithResponse.java:46)
	at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
	at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

I'm using a MEMStore for test purposes.

I guess it is either a bug in the MEMStore implementation or it is a misconfiguration of PROP_SUPPORTING_REVISION_DELTAS (which then should have a better exception).
The best would be to file a Bugzilla and attach your server config and the whole console trace of the server.

Cheers
/Eike

Do you have a hint about what's going wrong ?

Eike Stepper wrote:

Hi Cédric,

Yes, currently an IElementProcessor is the only chance to configure the
internals of the CDOSessionImpl (here: revisionManager) in an
IManagedContainer. Could you please file a Bugzilla so that I can add
respective properties to the CDOSessionConfiguration?

Cheers
/Eike



Cédric Brun schrieb:

Hi Simon,

Which value is your LRU set ? (Client and server)

I know in my case I was able to load 51 millions objects...
OutOfmemory are always tricky.

Can you send us a complete test (or even better a testcase) ? We will
know exactly your settings you are using.

By the way, do not forget that you need to configure 2 cache.... client
cache and the server cache.

That's a usefull hint, I was indeed only configuring the client cache,
configuring the server one seems way better (the test runs longer), but I
still ends up with an OutOfMemory.

My caches are set to 100/50 for current/revised.

Are your client and server the same java process ? How much memory you
JAVA can use ?(maximum)

Client and Server are sharing the same process and I volontary limited
heap space to 256m. My goal is to see wether CDO have leaks or not.

When I get to the OOM, the dump tells me I have 16 000 instances of
LRURevisionHolder kept in the client revision manager. Digging a step
further it appears that the session activation create a new
RevisionManager with a 0 as a current/revised capacity (and then create 
non-evicting LRU).

This creation happens during the session creation which happens during a
: CDOSession session = (CDOSession)
IPluginContainer.INSTANCE.getElement(productGroup, type, description);
instruction.

So I added a postProcessor which sets the current/revised LRU size and
it's now working flawlessly  though I don't know if I've done it the
way I should.

Thanks for your help, it was really valuable, as usual 

Cédric

"C?dric Brun" 
<cedric dot brun at obeo dot fr>
 a ?crit dans le message de news:
g77d26$rs8$1 at build dot eclipse dot org
...

Hi Eike and Simmon (or anybody familiar with CDO internals),

I'm still digging in CDO, and I wrote yet another dummy testcase just
filling a model with thousands of instances using a fake backend which
does not save anything at all (the addRevision do nothing so I have no
references on the java objects). Of course I'm not even trying to
browse the model on the repository 

My loop was something like this :

int nb_loops = 10000;
for (int i = 0; i < nb_loops; i++) {
MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
newF.setName("" + i);
root.getMembers().add(newF);
tx.commit();
}

I was surprised to see that my test ended up with an OutOfMemory error

Digging with the *great* memory analyzer I've seen that all the
revisions were kept by CDO through the CDORevisionResolverImpl
Setting the LRU cache to 0 did not helped and then I am wondering,
shouldn't this state be cleared after a commit ?

I changed my my test to create a new transaction and close it on each
loop, but it did not helped as I guess the resolver is linked to the
session instance.

So I'm wondering, when should the CDORevisionResolverImpl be cleared ?
And more especially the "revisions" hashmap ?

Just in case you're wondering I'm using the ganymede CDO.

Thanks,

C?dric

<!---->

Reply 9. At 6:13 AM on Aug 6, 2008, Cédric Brun wrote:

Re: [CDO] Question about LRU cache and tx.commit()

Woah , you are so quick ! Did Ed gave you some kind of doping substance ?
Thanks for being so quick

I was exactly doing that, I tracked a new bug and added a sample project with associated junit testcase.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=243282

Cédric

Eike Stepper wrote:

> Cédric Brun schrieb:
>> Hi Eike,
>>
>> Bug is tracked : https://bugs.eclipse.org/bugs/show_bug.cgi?id=243279
>>
>> Since I change the settings with the revised/current LRU, I'm now
>> experiencing the following issue :
>>
>> java.lang.IllegalStateException: Can not retrieve origin revision for
>> org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl@5e34ce
>> at
>> org.eclipse.emf.cdo.internal.server.Transaction.computeDirtyObjects(Transaction.java:290)
>> at
>> org.eclipse.emf.cdo.internal.server.Transaction.commit(Transaction.java:179)
>> at
>> org.eclipse.emf.cdo.internal.server.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:109)
>> at
>> org.eclipse.net4j.signal.IndicationWithResponse.execute(IndicationWithResponse.java:46)
>> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143) at
>> org.eclipse.net4j.signal.Signal.run(Signal.java:124) at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> I'm using a MEMStore for test purposes.
>>
> I guess it is either a bug in the MEMStore implementation or it is a
> misconfiguration of PROP_SUPPORTING_REVISION_DELTAS (which then should
> have a better exception).
> The best would be to file a Bugzilla and attach your server config and
> the whole console trace of the server.
>
> Cheers
> /Eike
>
>> Do you have a hint about what's going wrong ?
>>
>> Eike Stepper wrote:
>>
>>
>>> Hi Cédric,
>>>
>>> Yes, currently an IElementProcessor is the only chance to configure the
>>> internals of the CDOSessionImpl (here: revisionManager) in an
>>> IManagedContainer. Could you please file a Bugzilla so that I can add
>>> respective properties to the CDOSessionConfiguration?
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>>
>>> Cédric Brun schrieb:
>>>
>>>> Hi Simon,
>>>>
>>>>
>>>>
>>>>> Which value is your LRU set ? (Client and server)
>>>>>
>>>>> I know in my case I was able to load 51 millions objects...
>>>>> OutOfmemory are always tricky.
>>>>>
>>>>> Can you send us a complete test (or even better a testcase) ? We will
>>>>> know exactly your settings you are using.
>>>>>
>>>>> By the way, do not forget that you need to configure 2 cache....
>>>>> client cache and the server cache.
>>>>>
>>>>>
>>>> That's a usefull hint, I was indeed only configuring the client cache,
>>>> configuring the server one seems way better (the test runs longer), but
>>>> I still ends up with an OutOfMemory.
>>>>
>>>> My caches are set to 100/50 for current/revised.
>>>>
>>>>
>>>>
>>>>> Are your client and server the same java process ? How much memory you
>>>>> JAVA can use ?(maximum)
>>>>>
>>>>>
>>>> Client and Server are sharing the same process and I volontary limited
>>>> heap space to 256m. My goal is to see wether CDO have leaks or not.
>>>>
>>>> When I get to the OOM, the dump tells me I have 16 000 instances of
>>>> LRURevisionHolder kept in the client revision manager. Digging a step
>>>> further it appears that the session activation create a new
>>>> RevisionManager with a 0 as a current/revised capacity (and then create
>>>> non-evicting LRU).
>>>>
>>>> This creation happens during the session creation which happens during
>>>> a
>>>> : CDOSession session = (CDOSession)
>>>> IPluginContainer.INSTANCE.getElement(productGroup, type, description);
>>>> instruction.
>>>>
>>>> So I added a postProcessor which sets the current/revised LRU size and
>>>> it's now working flawlessly

though I don't know if I've done it the
>>>> way I should.
>>>>
>>>> Thanks for your help, it was really valuable, as usual :)
>>>>
>>>> Cédric
>>>>
>>>>
>>>>> "C?dric Brun" a ?crit dans le message de news:
>>>>> g77d26$rs8$1 at build dot eclipse dot org...
>>>>>
>>>>>
>>>>>> Hi Eike and Simmon (or anybody familiar with CDO internals),
>>>>>>
>>>>>> I'm still digging in CDO, and I wrote yet another dummy testcase just
>>>>>> filling a model with thousands of instances using a fake backend
>>>>>> which does not save anything at all (the addRevision do nothing so I
>>>>>> have no references on the java objects). Of course I'm not even
>>>>>> trying to browse the model on the repository ;)
>>>>>>
>>>>>> My loop was something like this :
>>>>>>
>>>>>> int nb_loops = 10000;
>>>>>> for (int i = 0; i >>>>>> MyEClass newF = MyPackageFactory.eINSTANCE.createMyEClass();
>>>>>> newF.setName("" + i);
>>>>>> root.getMembers().add(newF);
>>>>>> tx.commit();
>>>>>> }
>>>>>>
>>>>>> I was surprised to see that my test ended up with an OutOfMemory
>>>>>> error
>>>>>>
>>>>>> Digging with the *great* memory analyzer I've seen that all the
>>>>>> revisions were kept by CDO through the CDORevisionResolverImpl
>>>>>> Setting the LRU cache to 0 did not helped and then I am wondering,
>>>>>> shouldn't this state be cleared after a commit ?
>>>>>>
>>>>>> I changed my my test to create a new transaction and close it on each
>>>>>> loop, but it did not helped as I guess the resolver is linked to the
>>>>>> session instance.
>>>>>>
>>>>>> So I'm wondering, when should the CDORevisionResolverImpl be cleared
>>>>>> ? And more especially the "revisions" hashmap ?
>>>>>>
>>>>>> Just in case you're wondering I'm using the ganymede CDO.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> C?dric
>>>>>>
>>>>>>
>>>>
>>
>>

Replies: 9 - Pages: 1 Threads: [ Previous | Next ]

Latest @

The Rich Engineering Heritage Behind Dependency Injection

Java, the OLPC, and community responsibility

[CDO] Question about LRU cache and tx.commit()

At 1:08 PM on Aug 4, 2008, Cédric Brun wrote:

9 replies so far ( Post your own)

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()

Re: [CDO] Question about LRU cache and tx.commit()