I have been working on a VP project file (.vpp) regarding my custom plug-in to Visual Paradigm and I have run into one problem. My plugin requires to iterate through all of the models, diagrams, diagram elements in a project. This is okay up until my project starts to incorporate too many class instances at once at which point the plug-in cannot allocate the memory needed to issue the method. My computer has the memory and I even followed the instructions to increase the java heap size and it worked until my file included too many model elements. I’ve had this problem while reverse engineering a jar as well. This has been a increasing problem with the growth of my file.
The method that creates this problem is ApplicationManager.instance().getProjectManager().getProject().allLevelModelElementIterator(). The java runtime used cannot allocate more than 1.3 gigs of memory at which point the program will come to a halt. I was wondering if this method could somehow be optimized or a new method be created that can get all model element id’s in a project and write them to a variable and/or text file. From that point I can use the getModelElementById(String ID) method to return each individual IModelElement object into memory and out of memory once the information from that class has been obtained.
If there is a known solution and/or workaround that has been discovered I would also be interested in that as well.
This problem is not specific related to plug-ins. In fact, it is a normal phenomenon. Model elements will not be loaded into memory until you access it. Even when you call ApplicationManager.instance().getProjectManager().getProject().allLevelModelElementIterator(), the model elements will not be loaded into memory until you call iterator.next(). Therefore, getting a list of IDs and load the model elements one by one manually will have the same effect as using the iterator.
I think the better way to solve this problem is adjust the way you search for the model elements. I.e. you lookup the elements based on specific type. This can help to load up less elements into memory. Hope this can help.
If this is the case, is there a method that can provide a list of all model types that are available in the current VP Project, or even just a overall list of every possible model element type? If so, what method can provide this and which method can be utilized to return a subset of model elements based on a model element type? Unfortunately I have no choice regarding the limitation of the model elements I need as I need to iterate through them all to properly reference a current version of the file vs an old version of the file. Fortunately the model elements do not need to stay in memory while I iterate through both files. Is there any way to utilize the garbage collector within java to help clean up unused objects/ meta data? I have attempted to keep as many objects out of memory as possible. I have noticed that even after I have reversed a jar file, generate code, etc that my memory has not dropped to what it was before this operation. I currently have to:
Reverse a jar file (usually takes a while).
Wait until the reverse completes.
Save (depending on the memory being used in VP this can take quite a while.
Exit VP
Re-open VP
and magically the memory being used has dropped.
I do understand that there probably is some caching/meta data stored for the model elements to help with continuous re-use of Model Elements, but most of the time I will not need these model elements directly after a reversing of a jar file, Git commit using my custom plugin, etc.
From what I have observed: Visual Paradigm will work well with Small - Medium-ish sized projects. Although this will usually be the case for most projects, I think it would be beneficial to store all of this background data locally in a encrypted file and free up memory for cases where there are bigger projects. The performance would drop as you would have more I/O action due to reading/writing from disk but I would prefer this as opposed to Visual Paradigm not responding after the java heap size has been breached. Even if there were an option to choose between embedded memory (fast) use or use of local storage (not fast) for temporary data would be highly beneficial as a customer.
Java is a great language, but it does have limitations, unfortunately this being one. I have found temporary workarounds where I copy selected components to a new VP project, do the work needed and transfer them back to the original project. I have doubts using this method as I know more than likely I am most likely losing some data when I use this method.
You can call public IModelElement[] toModelElementArray(String modelType); to retrieve model elements in specific type, and the types can be found at IModelElementFactory.MODEL_TYPE_XXX
About your procedures (reverse a jar, save and restart), this is a normal behavior. The reverse engineering process will create lots of model elements, thus consumed memory. Once you restarted the application, the memory will not be consumed until you access the model.
You can call System.GC in Open API to force garbage collection.
Our design capable to handle large scale projects. In most situation, you won’t need to access all model elements in your project (especially in large project), so the performance of the application will still be good even working on a large project. But if you need to access all model elements in a large project, then maybe no application can handle efficiently.
I have been restructuring my plugin so I don’t mind having to change the logic in how I retrieve info for each model elements. As I mentioned previously I do need every Model Element to be loaded in order to get the information needed to properly perform the functionality for my plugin. The big thing that is beneficial for me is that I do not need for these model elements to remain in memory once I retrieve the information needed. The ideal situation would be to load 100 model elements into memory, iterate through each model element in the collection and grab the info needed and use it accordingly, call System.GC and repeat until there are no model elements left. Is there any chance of optimizing a Model Element get method to have similar functionality.
About what you had previously mentioned:
About your procedures (reverse a jar, save and restart), this is a normal behavior. The reverse engineering process will create lots of model elements, thus consumed memory. Once you restarted the application, the memory will not be consumed until you access the model.
I had a rough idea this was most likely the case. Is there a need to have these Model Elements residing in memory once this operation is complete? Do these need to be in memory for info needed while saving a VPP file? Otherwise there is a good chance that most of these model elements may never be accessed during the same VP session and could be consuming memory while it could be distributed more efficiently.
I notices that you mainly retrieve the model for reading it’s property, but and not modifying it. How about we provide a method for you to load a model in read only mode, which can be dispose after using. Do you think this can help?
That would help tremendously. I also use the same type of logic on diagrams and diagram elements. If that same logic can be applied to all 3 I will be one happy developer. If this logic is just applied to model elements I will still greatly benefit from this as I believe the majority of my memory is consumed when iterating through model elements.
I would like to let you know we have deployed a new build which support retrieve element as read-only, and disposable way. Please run the VP Suite Update to update the software to latest patch (20100908u or later). Details about update to latest patch can be found at
With this build, you can retrieve elements in the following way
IProject lProject = ApplicationManager.instance().getProjectManager().getProject();
// getting models by transientAllLevelModelElementIterator(...)
Iterator lIter = lProject.transientAllLevelModelElementIterator();
while (lIter.hasNext()) {
IModelElement lModel = (IModelElement) lIter.next();
// reading model...
lModel.getName();
// after read, dispose it (otherwise, the memory CANNOT be released)
lModel.disposeTransientModel();
}
Feel free to contact me if you require any further information.
I’ll give it a try later today. It may be a day or so before I implement the read only version of the elements (depending on how tedious the changes are). I’ll let you know if I see any changes in performance. Thanks for your help regarding this matter. If this works, I should be able to implement my plugin to larger scale VP projects which would be highly beneficial.
I implemented this logic for the Model Element (Sorry it took me a while to implement). I have been utilizing other api’s which required some changes and testing to code in some of their latest updates. Will this new method soon be available for the Diagram and Diagram Element class? I just tried this on a smaller vpp file and it worked fine. I still haven’t seen the performance on one of my bigger files. I will try using my biggest vpp file which has caused me problems in the past. If this works than I may not need this method in the Diagram and Diagram Element classes.
