TeeChart direct access to data

[webadm]

Hi All,

Time ago, back to 2001 I was working in an application with C++ Builder that required frequent plot updates of a lot of points in a TimePlot. All I tried to make it faster was not fast enough, and then I took a look at the sources and I've found that I could make virtual one method (I can't recall whitch one) and then In C++ Builder I could inherit from a class and develop a proxy class that was allowing to TChart access to my internal storage structures without need to Add the points.

I'm now back to a similar application where plots change frequently. Lot of time is spent just clearing and adding points to get rid of the old view and create the new view. All my data it is already in memory and it will be way more efficient if I could do something similar this time, because the improvement in performance could be huge in my particular user case. As all this clear Series points and add points could be avoided, like I did time ago.

By 2001 TChart sources were way smaller and easier to understand in a reasonable amount of time. 17 years later TeeChart has grown quite a bit and it is not longer so easy to follow the source code in a reasonable amount of time. I would like to know if it is still feasible such a design and where I would need to look in the sources.

Thanks
Javier

[Yeray]

Hello Javier,

We've added series and tools to the component but the basics are roughly the same. I imagine you are working with a TFastLineSeries, so I'll try to explain it's basics.
As all the series, this series has XValues and YValues properties, which are instances of the TChartValueList class; this class has the Values property, which is the array of doubles.
The first thing the DrawValue(ValueIndex) method does is to call CalcPosition(ValueIndex, out X, out Y) to get the X and Y pixels that correspond to the ValueIndex currently being drawn.
Since there are several places where the XValues and YValues arrays are used, the best approach would probably be to modify their "getters" to get your values from memory, instead of taking the values from the Value array:

Code: Select all

Function TChartValueList.GetValue(ValueIndex:Integer):TChartValue;
begin
  {$IFDEF TEEARRAY}
  {$IFOPT R+}
  if (ValueIndex<0) or (ValueIndex>(Count-1)) then
     Raise ChartException.CreateFmt(SListIndexError,[ValueIndex]);
  {$ENDIF}
  result:=Value[ValueIndex]{$IFDEF TEEMULTIPLIER}*FMultiplier{$ENDIF};
  {$ELSE}
  result:=PChartValue(FList[ValueIndex])^{$IFDEF TEEMULTIPLIER}*FMultiplier{$ENDIF};
  {$ENDIF}
end;

You'll also need to maintain the Count property, as it is being used for several calculations. I guess setting XValues.Count would do the trick.

If you need further assistance, don't hesitate to let us know.

[webadm]

>> Hello Javier,

Hi Yeray! Thanks for your reply

>> I imagine you are working with a TFastLineSeries

Not only. I use all sort of non DB series. Does what you said about TChartValueList applies to all series?

>> As all the series, this series has XValues and YValues properties, which are instances of the TChartValueList class; this class has the Values property, which is the array of doubles.

Let's say that TChartValueList is a sort of container. Would it be possible to go a litte futher and allow to tell a series to use a different implementation of the container?

>> The first thing the DrawValue(ValueIndex) method does is to call CalcPosition(ValueIndex, out X, out Y) to get the X and Y pixels that correspond to the ValueIndex currently being drawn.
>> Since there are several places where the XValues and YValues arrays are used, the best approach would probably be to modify their "getters" to get your values from memory, instead of taking the values from the Value array:

I will take now a look at the code and see.

>> You'll also need to maintain the Count property, as it is being used for several calculations. I guess setting XValues.Count would do the trick.

Of course. I will need to maintain the integrity of the container.

>> If you need further assistance, don't hesitate to let us know.

Thanks.

[Yeray]

Hello,

Not only. I use all sort of non DB series. Does what you said about TChartValueList applies to all series?

Yes, all the series inherit from TeeChartSeries, and there's where the XValues and YValues properties are declared as TChartValueList. Some series add extra arrays, like the 3D series, which add the ZValues TChartValueList.

Let's say that TChartValueList is a sort of container. Would it be possible to go a litte futher and allow to tell a series to use a different implementation of the container?

You'd need to modify the sources but I guess it should be possible.

You may want to arrange an example showing how you have the data loaded so we can think on it and try to suggest some options.

[webadm]

>> >>Not only. I use all sort of non DB series. Does what you said about TChartValueList applies to all series?
>> Yes, all the series inherit from TeeChartSeries, and there's where the XValues and YValues properties are declared as TChartValueList. Some series add extra arrays, like the 3D series, which add the ZValues TChartValueList.

Ok.

>> >> Let's say that TChartValueList is a sort of container. Would it be possible to go a litte futher and allow to tell a series to use a different implementation of the container? [/quote]
>> You'd need to modify the sources but I guess it should be possible.
Good.

>> You may want to arrange an example showing how you have the data loaded so we can think on it and try to suggest some options

My current class is a Container that simulates to be an Array of a custom type. My custom made class that is similar to a Variant. That is a class that can store a different types. I only try to draw values of type double.

In my application the container gets data loaded once in memory but then we we applied different conditions over the data that needs to to be drawn.

Currently we delete the Serie data and we add only the data that meets the new condition. What I would like to have is a way to tell a serie to use a different implementation of the TChartListValues. That class would act as a intermediate between the Series and my application. It will be initialized in a way that can query the in memory data plus the current filter and it will provide a new virtual view of the "Array of doubles" that TChartLitstValue provide and ask either the serie or the to redraw.

In order to achieve that I imagine that It would needed:
1) Tp Identify the interface part of TChartListValue list to be able to design a base class that all implementations can inherited from.
2) Then make TChartListValue inherit from that class and to provide the custom implementation so all works as now
3) Modify TChartSeries so the new object is used instead

Is that correct? Design alternatives you might think are easier, quicker to develop, better or any combination ?

It would be ideal if I could keep the changes on delphi to the minimum and implement the specializations of the TChartListValue in C++ Builder.

The purpose of all of this is to avoid have to copy clear and add the data to the series everytime the view(Filter) changes. It does not only takes time, it also duplicates memory and I do have a lot of channels.

side question. Also The project group "TeeChart for VCL.groupproj" it does not find any project In the project group. What project I need to open to navigate through the sources easier?

Thanks,
Javier Cuevas

[Yeray]

Hello Javier,

I've been looking at how to provide this option to directly get the values without duplicating an array that has already been populated, and the best way I can think on would be using pointers.
I think the best would be to separate this option with a new "DEFINE" (ie "TEEPOINTER") because using regular "ifs" could affect performance.
Concretely, the change I'm thinking on would be to create a new pointer at TChartValueList, next to the Value property:

Code: Select all

    {$IFDEF TEEARRAY}
    Value    : TChartValues;
    {$ENDIF}
    {$IFDEF TEEPOINTER}
    PValue   : ^TChartValues;
    {$ENDIF}

Then, we should modify every call to Value to something like this:

Code: Select all

var tmpVal : {$IFDEF TEEARRAY}TChartValue{$ELSE}PChartValue{$ENDIF};
//...
    {$IFDEF TEEARRAY}
    tmpVal:=Value[t];
    {$ENDIF}
    {$IFDEF TEEPOINTER}
    tmpVal:=PValue^[t];
    {$ENDIF}

This option would be available for SourceCode customers, and defining the "TEEPOINTER" variable would allow you to assign a pointer to a

Code: Select all

  SetLength(X,Num);
  SetLength(Y,Num);

  X[0]:=0;
  Y[0]:=Random(10000);
  for t:=1 to Num-1 do
  begin
    X[t]:=t;
    Y[t]:=Y[t-1]+Random(101)-50;
  end;

  With Series1.XValues do
  begin
    PValue:=@X;
    Count:=Num;
    Modified:=True;
  end;

  With Series1.YValues do
  begin
    PValue:=@Y;
    Count:=Num;
    Modified:=True;
  end;

After that, when a change arrives to the X and Y arrays, you only should repaint the series, and probably to set the Count and Modified flag again:

Code: Select all

  With Series1.XValues do
  begin
    Count:=Num;
    Modified:=True;
  end;

  With Series1.YValues do
  begin
    Count:=Num;
    Modified:=True;
  end;

  Chart1.UndoZoom;

  Series1.Repaint;

However, this new functionality would imply a considerable amount of changes in the component, and should be validated before being implemented.

Also, before following that route, we'd need to be sure the already built-in features don't fit your needs.
I understand directly accessing the points already in memory sounds as the most efficient way, but in the majority of cases we don't need all the data loaded into the series as we usually only want to show a part of the data. Also, even when our customers want to plot all their data, most performance problems are solved avoiding to paint repeated points (DrawAllPoints property of the TFastLineSeries), or reducing the number of points in the screen (TDownSamplingFunction).

[webadm]

Yeray,

The pointer does not do the trick to my problem because I still I have to have a copy of the data in a array of doubles somewhere that is consecutive in memory. Every time the visualization condition changes.

I cannot use the TFastLineSeries because I use features that are not available on the TFastLineSeries.

For the time plots, I agree that I could do an optimization just calculating a cache of data with just with the number of pixels and display the max and min for the data that goes in that pixel.

For the XY and other plots I think I need to display all the points.

So I still think that I will be better at having a TChartValueList base class where the current TChartValueList inherits from and be able to create my own implementation of it and allow the series to use an Instance I do assign to them or something like that.

Reducing the points to be painted for sure will help. Do you have an example of TDownSamplingFunction I can look?

Does this make sense?

[Yeray]

The pointer does not do the trick to my problem because I still I have to have a copy of the data in a array of doubles somewhere that is consecutive in memory. Every time the visualization condition changes.

I'm not sure to understand how do you get the data. Could you please put some light here so we can better understand the environment?
A simple example would also make things clearer.

I cannot use the TFastLineSeries because I use features that are not available on the TFastLineSeries.

What features do you use that are not available on the TFastLineSeries?

For the time plots, I agree that I could do an optimization just calculating a cache of data with just with the number of pixels and display the max and min for the data that goes in that pixel.

This uses to notably improve the performance when having many points.

For the XY and other plots I think I need to display all the points.

We could try to implement a TFastPointSeries with some features similar to the ones in the TFastLineSeries.

So I still think that I will be better at having a TChartValueList base class where the current TChartValueList inherits from and be able to create my own implementation of it and allow the series to use an Instance I do assign to them or something like that.

That seems reasonable to me. I would explore the other possibilities before going for this, assuming this option would probably be the most expensive in terms of development time.

Reducing the points to be painted for sure will help. Do you have an example of TDownSamplingFunction I can look?

Yes, you can look at the "Reducing number of points" example at the "Features Demo". Sources here:

Tee9new_2018-07-11_09-42-06.png (177.92 KiB) Viewed 21865 times

[webadm]

Yeray,

I will take a look a the example and see if that could be a solution for my problem.

When I did it years ago I only declared virtual a function on the TChart sources and I was able to do the customization on my side and was quick. But I don't have the sources anymore and I don't remember what I did. That's the reason I started this thread.

It seems that changing TChart on the way it is best for me is more work that I anticipated. I will explore the other options and come to you if I cannot get a way to fix the issue.

Thanks