Add missing @ symbol

Medium user page names are prepended with an @ symbol in the URL.

Thanks!

:+1:


[FEATURE] Ability to add multiple NICs ( same vlan network) to a virtual machine

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

No need to filter vlan networks that have already been used
Describe alternatives you've considered

Additional context

validated on master-head


[WIP] learning rate logging V2

Implements the same functionality from #131 using only minor modifications to elegy.Optimizer.

[x] Add lr_schedule and steps_per_epoch to Optimizer.
[x] Implement Optimizer.get_effective_learning_rate
[ ] Copy logging code from #131

@alexander-g Here is a proposal that is a bit simpler, closer to what I mentioned in #124. What do you think?
@charlielito should we log the learning rate automatically if available or should we create a Callback?

The API would be like this:
model = Model(
    module=MyModule(),
    optimizer=elegy.Optimizer(
        optax.adam(1e-3),
        lr_schedule=lambda step, epoch: my_schedule(step, epoch),
        steps_per_epoch=100,
    )
)

Just to clarify, the Model.__init__.optimizer parameter now is of type tp.Union[elegy.Optimizer, optax.GradientTransformation, None] so you can still pass optax object directly, but to enable this additional functionality the idea is to pass the elegy.Optimizer.
An alternative strategy could be to have Model accept the lr_schedule and steps_per_epoch parameters and forward them to Optimizer, I am good with both.

@cgarciae My main issue with this approach is that only the learning rate multiplier is logged not the actual learning rate (here 1e-3). This is not unimportant for debugging and especially documentation purposes, when doing hyperparameter search or comparing multiple training runs.
That was the main reason for wrapping optax in #131

In my previous example I actually meant:
model = Model(
    module=MyModule(),
    optimizer=elegy.Optimizer(
        optax.adam(1.0), # set this to 1 so lr_schedule has more meaning.
        lr_schedule=lambda step, epoch: my_schedule(step, epoch),
        steps_per_epoch=100,
    )
)

I think we can get away with documenting this behaviour in the Optimizer class that you should set all other learning_rate args to 1.0. I think this solution will be easier to maintain even if we cannot enforce certain things.
Its a shame that the learning_rate parameter is not passed to ScaleState but instead its just captured by a closure, if it were we could've just inspect all the ScaleState objects. Maybe we can send a PR to optax? The change is minimal but would give us more power.

Should we move lr_schedule and steps_per_epoch to the Model's constructor or add more documentation to Model pointing to the direct use of Optimizer?

Both is fine.
But what would be nice if elegy.Optimizer could accept a variable amount of optax.GradientTransformations which it chains. Simply for the user to write a line less. Something like:
elegy.Optimizer(
        optax.scale(1/LOSS_SCALE),
        optax.additive_weight_decay(1e-4),
        optax.sgd(0.1, momentum=1e-4),
        lr_schedule=lambda step, epoch: my_schedule(step, epoch),
        steps_per_epoch=100,
    )

We could definitely accept *args and do the chaining.
Just to clarify, are we interpreting the use of scale as change in the learning rate? Learning rate updates are implemented as
optax.scale(-learning_rate)

by most Optax optimizers, this makes the definition of a learning rate a bit fuzzy since
optax.chain(
    optax.sdg(0.5),
    optax.scale(0.5),
)

is the same as
optax.sdg(0.25)

I would simply use the learning_rate parameter of the functions defined in optax/_src/alias.py .
If the user wants to apply more scaling that's their own issue.

Codecov Report

Merging #135 (27618b6) into master (78777ff) will increase coverage by 0.06%.
The diff coverage is 98.14%.

@@            Coverage Diff             @@
##           master     #135      +/-   ##
==========================================
+ Coverage   79.25%   79.31%   +0.06%
==========================================
  Files         110      110
  Lines        5143     5163      +20
==========================================
+ Hits         4076     4095      +19
- Misses       1067     1068       +1

Impacted Files
Coverage Δ

elegy/model/model_base.py
75.64% <95.00%> (+1.33%)
:arrow_up:

elegy/model/__init__.py
100.00% <100.00%> (ø)

elegy/model/model_test.py
100.00% <100.00%> (ø)

elegy/data/data_adapter.py
57.14% <0.00%> (-6.28%)
:arrow_down:

elegy/data/generator_adapter.py
41.30% <0.00%> (-4.70%)
:arrow_down:

elegy/data/data_handler.py
71.83% <0.00%> (-1.51%)
:arrow_down:

elegy/metrics/metric.py
68.18% <0.00%> (-1.39%)
:arrow_down:

elegy/losses/loss.py
78.18% <0.00%> (-0.77%)
:arrow_down:

elegy/data/list_adapter.py
89.74% <0.00%> (-0.74%)
:arrow_down:

elegy/data/array_adapter.py
90.00% <0.00%> (-0.33%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78777ff...27618b6. Read the comment docs.

@cgarciae  just to answer you I think it would be nice if there is a lr schedule, the learning rate should be logged by default, since you want to monitor that


Changes in the troubleshoot:'oauth:resource-server

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Updated Description

Reference:
https://www.asahp.org/what-is/#:~:text=Allied health may be defined as those health,professions that are distinct from medicine and nursing.

Don't change english. Waste of time. Focus on non-english Only.


refactor(robot-server): Utilize unsubscribe flags for dynamic topics

Closes EXEC-305

Overview
This PR adds unsubscribe flags for dynamic notification topics, e.g. "/runs/:runId".
Now that the dust has settled around what our push notification can (and can’t) do, it’s clear that the burden of unsubscribing should be on the robot-server, specifically for dynamic topics.
In other words, it’s ok for the app to connect to a broker and remain subscribed to topics that will never change, ex. “/maintenance_runs/current_run” (and be subscribed to those topics as long as the app is connected). Dynamic topics, such as “/runs/:runId”, should tell the app to unsubscribe however, as requiring the broker to stores these topics unnecessarily increases memory load on the robot server during long-lived sessions.
This model is much easier to reason about as opposed to the component interest model that we use now. Bug surface is reduced, too.

Test Plan

Start up postman and start a protocol run using the desktop app's dev mode. You'll notice in the app-shell logs a lot of subscribe requests to a /run/:runId. Make note of that runId and in postman, subscribe to /runs/.
After the run concludes, start another protocol run. Verify that the previous run (the run you're subscribed to in Postman), receives a message containing unsubscribe: true.

Risk assessment
low

Will merge alongside https://github.com/Opentrons/opentrons/pull/14633


Wrap source in C++ guards

The source is lacking #ifdef guards required for C++ projects to include the C based source into their project.

#ifdef __cplusplus
extern "C" {
#endif

... original content of headerfile goes here.

#ifdef __cplusplus
}
#endif


Update puppetlabs/apt dependency

The puppetlabs/apt version 5 adds fixes for Ubuntu 18.04.

I added a fix for the CI, as the build error was caused by a newer winrm-fs version. I can remove the fix or change it if this is not the correct approach.

Only time to actually do so. 😉
Merged, thanks!


Download k samples of each class

Right now if you wanted to download k samples of each class (malignant and benign) you would have to manually download the malignants first
python download_archive.py --num-images k --filter malignant

And then in a separate directory download the benigns
python download_archive.py --num-images k --filter benign

Otherwise you'd overwrite some of the images.
It would be nice if the script was able to do this in one go.

I quickly realized you can achieve this by doing something like
python download_archive.py --num-images k --filter malignant
python download_archive.py --num-images k --filter benign --offset k

So maybe you can add this to the documentation? I'm sure it's a fairly common use case, because it's very important that your classes are balanced in classification problems.

Turns out the above does not work because of a similar problem in #22.
Some of the samples of the first downloaded class will be overwritten by the second. I had to download them to separate folders and carefully rename the sample's filenames so as to effectively merge the data.

I think that currently the names that the script gives the files are the same names they have in the website. Therefore different files should have different names, even via separate script runs.
Can you show an example for a scenario where two different files are given the same name?


Add :status header in the case of trailer-only failure reply

This resolves #10172

Test failures:
Basic Tests Mac - #10138


Strip text from HTML body - best practice

I'm using the Norconex HTTP Collector to crawl HTML files and send certain meta-fields and the content text to a Solr server.
What I now want to do is to only send text from the content to the Solr server which is between certain HTML comments and ignore the rest. My HTML code looks like this:

  ...

     Some text to be extracted

   ...
     Ignore this text
   ...

     Some other text to be extracted

  ...

As you can see there can be multiple start-end chunks in one file to be extracted.
I already solved this by using multiple StripBetweenTransformer in the pre parse handler:
<preParseHandlers>

  <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true" sourceCharset="ISO-8859-1" >
    <stripBetween>
      <start><![CDATA[]]></start>
      <end><![CDATA[]]></end>
    </stripBetween>
  </transformer>

  <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true" sourceCharset="ISO-8859-1" >
    <stripBetween>
      <start><![CDATA[]]></start>
      <end><![CDATA[]]></end>
    </stripBetween>
  </transformer>

  <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true" sourceCharset="ISO-8859-1" >
    <stripBetween>
      <start><![CDATA[]]></start>
      <end><![CDATA[]]></end>
    </stripBetween>
  </transformer>

<preParseHandlers>

But I think this is very unhandy, especially because I also have to define a sourceCharset. I have to do this as the HTML files also include links to other document types (like PDF). In case I don't define a sourceCharset I get a NullPointerException error while parsing these documents:
java.lang.NullPointerException: charsetName

Is there a more elegant way to achieve this?
I've tried to use the TextBetweenTagger in the post parse handler but I couldn't figure out how to get the text of the HTML body, modify it with this tagger and send it to the Solr server as content.

The NPE is obviously wrong, so I am marking it as a bug.  You should not have to specify the charset unless you know you have charset issues with your crawling.
All the "taggers" are to manipulate metadata only, without modifying the content.   To manipulate the content, you have to rely on "transformers".  The one that I think is more suited to achieve what you are after is ReplaceTransformer.
That being said, you can keep using TextBetweenTagger and store the results in a new field metadata field (the name attribute).   Let's say you call that new field "myContentField", then when using the Solr Committer, you can overwrite the default by specifying which field to use for the source content instead of the actual document body.
<committer class="com.norconex.committer.solr.SolrCommitter">
     ...
     <sourceContentField keep="false">myContentField</sourceContentField>
     ...
</committer>

Please report if that does not work for you.

Hi Pascal,
Thank you very much for your fast response! :)
Please note that I also get the same NPE when parsing documents with the TextBetweenTagger without using the sourceCharset parameter. It seems that this is the same problem like in the StripBetweenTagger.
So what you suggest is to use the TextBetweenTagger as pre parse handler and store the parsed text into a new field (e.g. contentNew), correct? I've changed my config like this:
...
<preParseHandlers>
  <tagger class="com.norconex.importer.handler.tagger.impl.TextBetweenTagger" inclusive="false" sourceCharset="ISO-8859-1" >
      <textBetween name="contentNew">
          <start><![CDATA[]]></start>
          <end><![CDATA[]]></end>
      </textBetween>
  </tagger>
</preParseHandlers>
...
<committer class="com.norconex.committer.solr.SolrCommitter">
  ...
  <sourceContentField keep="false">contentNew</sourceContentField>
  <targetContentField>text</targetContentField>
  ...
</committer>

This works fine as it now commits only the content between my start-end tags to Solr. Unfortunately I now lose the content of the non-HTML-files (like PDFs) as they don't have these tags included of course.
Is there a way to only apply the TextBetweenTagger to HTML files and keep the content of all other files as it is?
Otherwise my inital solution might not be as unhandy as I thought, provided I don't have to state the encoding anymore in a future release. ;)

Can you provide a copy of your config or the URL of the page causing the NullPointerException with StripBetweenTransformer ?  I cannot reproduce it.

I'm sorry but I can't give you the URL of the page as it is in our intranet and not available for visitors from outside.
But here is the content of my config xml file:
<?xml version="1.0" encoding="UTF-8"?>

<httpcollector id="Minimum Config HTTP Collector">

  <progressDir>${workdir}/progress</progressDir>
  <logsDir>${workdir}/logs</logsDir>

  <crawlers>
    <crawler id="Norconex Minimum Test Page">

      <startURLs stayOnDomain="true" stayOnPort="true" stayOnProtocol="true">
        <url>http://localhost/test/index.html</url>
      </startURLs>

      <numThreads>3</numThreads>
      <orphansStrategy>DELETE</orphansStrategy>
      <workDir>${workdir}</workDir>
      <maxDepth>2</maxDepth>
      <sitemapResolverFactory ignore="true" />
      <delay default="5000" />

      <importer>

        <parseErrorsSaveDir>${workdir}/importer/parseErrors</parseErrorsSaveDir>

        <preParseHandlers>

          <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true">
            <stripBetween>
              <start><![CDATA[]]></start>
              <end><![CDATA[]]></end>
            </stripBetween>
          </transformer>

          <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true">
            <stripBetween>
              <start><![CDATA[]]></start>
              <end><![CDATA[]]></end>
            </stripBetween>
          </transformer>

          <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true">
            <stripBetween>
              <start><![CDATA[]]></start>
              <end><![CDATA[]]></end>
            </stripBetween>
          </transformer>

        </preParseHandlers>

        <postParseHandlers>

          <tagger class="com.norconex.importer.handler.tagger.impl.CharacterCaseTagger">
            <characterCase fieldName="keywords" type="lower" applyTo="field" />
          </tagger>

          <tagger class="com.norconex.importer.handler.tagger.impl.KeepOnlyTagger">
            <fields>id,document.reference,document.contentType,title,keywords,area,subarea,Last-Modified,Last-Save-Date</fields>
          </tagger>

          <tagger class="com.norconex.importer.handler.tagger.impl.RenameTagger">
            <rename fromField="Last-Modified" toField="last_modified" overwrite="true" />
            <rename fromField="Last-Save-Date" toField="last_saved" overwrite="true" />
            <rename fromField="document.contentType" toField="contentType" overwrite="true" />
          </tagger>

        </postParseHandlers>

      </importer>

      <committer class="com.norconex.committer.core.impl.FileSystemCommitter">
        <directory>${workdir}/crawledFiles</directory>
      </committer>

    </crawler>
  </crawlers>

</httpcollector>

When parsing a simple PDF file it returns the following stack trace:
Norconex Minimum Test Page: 2016-03-21 15:50:18 ERROR - Norconex Minimum Test Page: Could not process document: http://localhost/test/docs/test.pdf (charsetName)
java.lang.NullPointerException: charsetName
	at java.io.InputStreamReader.<init>(Unknown Source)
	at com.norconex.importer.handler.transformer.AbstractCharStreamTransformer.transformApplicableDocument(AbstractCharStreamTransformer.java:106)
	at com.norconex.importer.handler.transformer.AbstractDocumentTransformer.transformDocument(AbstractDocumentTransformer.java:56)
	at com.norconex.importer.Importer.transformDocument(Importer.java:552)
	at com.norconex.importer.Importer.executeHandlers(Importer.java:352)
	at com.norconex.importer.Importer.importDocument(Importer.java:309)
	at com.norconex.importer.Importer.doImportDocument(Importer.java:271)
	at com.norconex.importer.Importer.importDocument(Importer.java:195)
	at com.norconex.collector.core.pipeline.importer.ImportModuleStage.execute(ImportModuleStage.java:35)
	at com.norconex.collector.core.pipeline.importer.ImportModuleStage.execute(ImportModuleStage.java:26)
	at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
	at com.norconex.collector.http.crawler.HttpCrawler.executeImporterPipeline(HttpCrawler.java:298)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:487)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:377)
	at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:723)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

If I use sourceCharset="ISO-8859-1" everything works fine.

Haaa... I see what causes this.  When using "pre-parse handlers", you have to be careful to account for files that are not text-based. In your case, since you want to deal with HTML files only, I suggest you restrict to just those files using the restricTo tag.  But I did found a real bug with it dropping <stripBetween> tags you have, sharing the same <start> tag as another one.  This is fixed in an Importer snapshot release I just made.
All in all, this should work for you now (a bit more elegant, and only dealing with HTML):
    <preParseHandlers>
      <transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" inclusive="true">
        <stripBetween>
          <start><![CDATA[]]></start>
          <end><![CDATA[]]></end>
        </stripBetween>
        <stripBetween>
          <start><![CDATA[]]></start>
          <end><![CDATA[]]></end>
        </stripBetween>
        <stripBetween>
          <start><![CDATA[]]></start>
          <end><![CDATA[]]></end>
        </stripBetween>
        <restrictTo field="document.contentType">text/html</restrictTo>
      </transformer>

    </preParseHandlers>

Please give it a try by replacing the norconex-importerXXX.jar you have under lib with the snapshot found in the link above and confirm.  If all works fine for you I will make it an actual Importer release with the fix.

FYI, I just updated the importer snapshot release to now log a warning if the character encoding cannot be detected when using character-based transformers.  With the importer module, if a charset cannot be detected, it more than likely means you are dealing with a binary file.  The warning will highlight this, and suggest options, making it easier to troubleshoot.

Hi Pascal,
I've changed my pre parse handler config according to your proposal and replaced the norconex-importerXXX.jar with the one from the snapshot. Works perfectly and exactly how I need it. :)
I also checked your warning message by removing the restrictTo tag and it also works fine.
I guess this ticket can be marked as closed now. Thank you very much for your great support!

Thanks for confirming!  Importer 2.5.1 has just been formally released.


Contact Admins emails do not respect new lines

Contact admins with a message that has new lines, for example:

email renders:

:+1:


Update PersonTest

Trying to increase CodeCov for Person back to 100%

Codecov Report

Merging #55 (c46fcfc) into test-vacancy (586b6e4) will increase coverage by 0.63%.
The diff coverage is 100.00%.

@@                Coverage Diff                 @@
##             test-vacancy      #55      +/-   ##
==================================================
+ Coverage           71.30%   71.93%   +0.63%
- Complexity            492      495       +3
==================================================
  Files                  85       85
  Lines                1516     1518       +2
  Branches              163      164       +1
==================================================
+ Hits                 1081     1092      +11
+ Misses                394      385       -9
  Partials               41       41

Impacted Files
Coverage Δ

...c/main/java/seedu/address/model/person/Person.java
100.00% <100.00%> (+17.30%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 586b6e4...c46fcfc. Read the comment docs.


Unable to add urls with query string

If I try to shorten any url with a query string I get the following error: The url format is invalid.

Thank you for opening this issue!
Mmh... I can't replicate this.
Validation errors were solved in the commit Updated Regex to allow ?, _ - and =
Maybe you need to update UrlHum from the repo?

I'm running on latest, and fixed it by removeing the trailing $ from this rule in app/Http/Requests/ShortUrl.php
- 'url' => 'required|max:500|regex:/^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\_\-\?\=\/\w \.-]*)*\/?$/',
+ 'url' => 'required|max:500|regex:/^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\_\-\?\=\/\w \.-]*)*\/?/',

You should allow pretty much any character in the query string, stuff like | and & are definitely missing.
A better approach would be to use filter_var($url, FILTER_VALIDATE_URL)

With my push GitHub automatically closed the issue, sorry.
In the end I decided to use default Laravel url validation, which uses FILTER_VALIDATE_URL anyway!
Thank you for your issue. It was very helpful.
Going to close this again 😄


Cutting small pieces of data from very large data sets.

Hello Everyone,
I am in trying to perform a calculation that requires me to access a piece of MITgcm output from a very large output dataset.
An example data set is -
1680 mds files of W velocity, where each file is 1.2GB ~ 2TB of data.
Each file has 2000X2000X76 grid points (X,Y,Z).
I want to perform a calculation that requires me to load a section of this data,  with (800X800) points in space, and 1680 time points. The size of this little piece is 4GB only.
However, when I try to load this data set the memory (RAM) usage goes through the roof (max out RAM usage, which is 1TB!), causing the notebook to die (kernel death, rather than an error).
I am wondering if there is a workaround to this issue?
Is there a way to trigger xmitgcm to be a bit more conservative on the memory usage, maybe at the loss of efficiency or something when the datasets are getting so large?

@dhruvbalwada, can you provide a reproducible code sample of what you are trying to do?

`data_dir = '/swot/SUM05/dbalwada/channel_beta/01km/free_slip_waves_hourly/'
npick = 1 # pick up every npick'th file as the original out is very dense (12 hours)
dt = int(3600/40)
T1 = int(712800)
T2 = int(864000+dt)
iters = range(T1, T2, dt*npick)
ds_free_slip = open_mdsdataset(data_dir,  iters=iters,
delta_t=40, prefix=['W']
,ignore_unknown_vars=True, geometry='cartesian')
ds_free_slip`

I am going to edit your comment to enable proper syntax highlighting.

I'm so sorry, I accidentally hit delete and now I lost your comment

Ok, got it back

data_dir = '/swot/SUM05/dbalwada/channel_beta/01km/free_slip_waves_hourly/'

npick = 1 # pick up every npick'th file as the original out is very dense (12 hours)

dt = int(3600/40)
T1 = int(712800)
T2 = int(864000+dt)
iters = range(T1, T2, dt*npick)

ds_free_slip = open_mdsdataset(data_dir,  iters=iters,
                             delta_t=40, prefix=['W']
                     ,ignore_unknown_vars=True, geometry='cartesian')
ds_free_slip

the output is
<xarray.Dataset>
Dimensions:  (XC: 2000, XG: 2000, YC: 2000, YG: 2000, Z: 76, Zl: 76, Zp1: 77, Zu: 76, time: 1681)
Coordinates:
  * XG       (XG) >f4 0.0 1000.0 2000.0 3000.0 4000.0 5000.0 6000.0 7000.0 ...
  * XC       (XC) >f4 500.0 1500.0 2500.0 3500.0 4500.0 5500.0 6500.0 7500.0 ...
  * YG       (YG) >f4 0.0 1000.0 2000.0 3000.0 4000.0 5000.0 6000.0 7000.0 ...
  * YC       (YC) >f4 500.0 1500.0 2500.0 3500.0 4500.0 5500.0 6500.0 7500.0 ...
  * Z        (Z) >f4 -0.5 -1.57 -2.79 -4.185 -5.78 -7.595 -9.66 -12.01 ...
  * Zl       (Zl) >f4 0.0 -1.0 -2.14 -3.44 -4.93 -6.63 -8.56 -10.76 -13.26 ...
  * Zu       (Zu) >f4 -1.0 -2.14 -3.44 -4.93 -6.63 -8.56 -10.76 -13.26 -16.1 ...
  * Zp1      (Zp1) >f4 0.0 -1.0 -2.14 -3.44 -4.93 -6.63 -8.56 -10.76 -13.26 ...
    dyC      (YG, XC) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    rAz      (YG, XG) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    Depth    (YC, XC) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    rAs      (YG, XC) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    dxC      (YC, XG) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    rAw      (YC, XG) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    dyG      (YC, XG) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    dxG      (YG, XC) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    rA       (YC, XC) >f4 dask.array<shape=(2000, 2000), chunksize=(2000, 2000)>
    PHrefF   (Zp1) >f4 dask.array<shape=(77,), chunksize=(77,)>
    PHrefC   (Z) >f4 dask.array<shape=(76,), chunksize=(76,)>
    drC      (Zp1) >f4 dask.array<shape=(77,), chunksize=(77,)>
    drF      (Z) >f4 dask.array<shape=(76,), chunksize=(76,)>
    hFacC    (Z, YC, XC) >f4 dask.array<shape=(76, 2000, 2000), chunksize=(76, 2000, 2000)>
    hFacW    (Z, YC, XG) >f4 dask.array<shape=(76, 2000, 2000), chunksize=(76, 2000, 2000)>
    hFacS    (Z, YG, XC) >f4 dask.array<shape=(76, 2000, 2000), chunksize=(76, 2000, 2000)>
    iter     (time) int64 dask.array<shape=(1681,), chunksize=(1,)>
  * time     (time) int64 28512000 28515600 28519200 28522800 28526400 ...
Data variables:
    W        (time, Zl, YC, XC) float32 dask.array<shape=(1681, 76, 2000, 2000), chunksize=(1, 76, 2000, 2000)>

ok, so the chunk size shows that chunking did work. I had some optimism that #60 would help fix this, but I don't think so.

I want to perform a calculation that requires me to load a section of this data, with (800X800) points in space, and 1680 time points. The size of this little piece is 4GB only.
However, when I try to load this data set the memory (RAM) usage goes through the roof (max out RAM usage, which is 1TB!), causing the notebook to die (kernel death, rather than an error).

Can you post the code you are using to do these things?

This brings to mind pydata/xarray#1396

Can you also post the output of xarray.show_versions()?

The data size :
ds_free_slip.nbytes/(1024**3)

give 1907.24 GB.
This is the data selection process
ds_piece = ds_free_slip.W.sel(XC=slice(0,800e3), YC=slice(600e3,1400e3)).sel(Zl=-150., method='nearest')
ds_piece

which gives
<xarray.DataArray 'W' (time: 1681, YC: 800, XC: 800)>
dask.array<shape=(1681, 800, 800), dtype=float32, chunksize=(1, 800, 800)>
Coordinates:
  * XC       (XC) >f4 500.0 1500.0 2500.0 3500.0 4500.0 5500.0 6500.0 7500.0 ...
  * YC       (YC) >f4 600500.0 601500.0 602500.0 603500.0 604500.0 605500.0 ...
    Zl       >f4 -146.88
    Depth    (YC, XC) >f4 dask.array<shape=(800, 800), chunksize=(800, 800)>
    rA       (YC, XC) >f4 dask.array<shape=(800, 800), chunksize=(800, 800)>
    iter     (time) int64 dask.array<shape=(1681,), chunksize=(1,)>
  * time     (time) int64 28512000 28515600 28519200 28522800 28526400 ...
Attributes:
    long_name:      Vertical Component of Velocity
    units:          m s-1
    standard_name:  sea_water_z_velocity

And size of this little piece is
ds_piece.nbytes/(1024**3)

which is 4GB.

Here is a piece of the dask graph for ds_piece

And here it is for ds_free_slip.W.data.visualize()

From these, the issue appears to be clear: the whole variable is being loaded before slicing (i.e. calls to getitem) takes place.

@dhruvbalwada I believe I have tracked down part of the problem to the somewhat obscure dask bug referenced above.

I confirmed that your calculation works much better (doesn't blow up; goes fast) if you call open_mdsdataset with endian='='. Of course, this screws up all your data by interpreting it as the wrong datatype. (You have to use .isel instead of .sel since the coordinates will be garbage.) But you can verify for yourself that it works. This means that the problem is with how dask handles big-endian datatypes.
Hopefully this can be fixed upstream in dask.

Thanks for looking into this @rabernat and providing a temporary solution. I really appreciate this.

I'm not sure it's a solution at all! Your data is useless with the wrong
datatype! But we have identified the problem.
On Tue, Mar 6, 2018 at 10:35 AM, Dhruv Balwada<EMAIL_ADDRESS>wrote:

Thanks for looking into this @rabernat https://github.com/rabernat and
providing a temporary solution. I really appreciate this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xgcm/xmitgcm/issues/73#issuecomment-370821387, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJFJtloB82XEgFssTKe_IEHXMwk_ec3ks5tbqzDgaJpZM4Sdrx3
.

Good, the problem is not solved then. I guess then your remark about using isel instead of sel is also pointless, as I will just select a piece out of data that is trash, which does seem to be the case.

With the wrong dtype, all your data is garbage.
As a temporary workaround, however, It might be possible to change the dtype after you have selected it. You can do this by creating a view of the data:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.view.html

I guess then your remark about using isel instead of sel is also pointless, as I will just select a piece out of data that is trash

It is not "pointless" insofar as it demonstrates the fundamental underlying issue. It is not useful for making an actual scientific calculation.

To clarify what I mean, I think you can get the correct raw data back by doing ds_piece.data.view(dtype='>f4'). This will work as long as you haven't done any computations upstream but are just selecting data.

This should be fixed by https://github.com/dask/dask/pull/3301. Can we check?

I checked the effects of the update to dask on loading MITgcm output and the test data that you had created on the issue #3248 in dask.
The test data with different choices of endianness now takes the same time to load.
Data from MITgcm loaded with and without the option endian='=' takes the same time to load.


Node got over 100% uptime

Node ID 3010
Farm ID 1994

https://github.com/threefoldtech/tfgrid_dashboard/blob/development/src/portal/lib/nodes.ts#L20-L28
Not sure what is going on here, but what a user would expect is the uptime in the current minting period, i.e. current uptime (since period start) / time in period (current time - period start )

Receipts are generated as a result of running the minting code, which by definition happens only after the minting, and thus after the period. So you can't get the receipt from the API to calculate the current uptime because there is none. What you need to do is calculate the uptime by getting the current uptime reports (from graphql is easiest).
Here is a rough guide how to do that:
First you want to figure out the start time of the current period. There is some rust code which does this here. With the start timestamp, you can do a query on graph.grid.tf/graphql for uptimeEvents, filtered on nodeID and timestamp_gte the period start. You'll also want to order them based on timestamp, descending or ascending is not really important, whatever is easiest to work with.
Technically there are additional checks based on the timestamps of the data points, but for this visualization they are not really important (the minting does these to detect and punish misbehaving nodes). What we want to do is calculate the confirmed downtime of nodes. In this scenario, confirmed means the node has been rebooted, and we can tell from the uptime that this happened. Specifically, we are looking for data points where either: uptime of the second point is lower than the uptime of the first point (uptime is always increasing as the node is powered on)(this assumed uptime events are sorted in ascending order based on timestamp), or uptime of the second point is higher than the first point uptime, but lower than the difference between the second point timestamp and the first point timestamp (this can happen if a node boots, sends uptime, then powers off again, and some time later powers on again, e.g. a reboot loop). In either of these cases, downtime is calculated as (difference between timstamp - point2 uptime).
This then leaves 2 special cases: the start of the period and the time since the last uptime. For the start of the period, you compare the first uptime event with the period start, so you essentially insert a fake data point with timetamp: period start and uptime: 0 at the start of the data. For the time since the last uptime: a node only sends an uptime report every 40 mins (as it currently stands). We want to allow some headroom, so we can check that the last uptime report was less than an hour ago. So essentially: if the last uptime report was less than an hour ago, the node is considered up. If it is older, the node is considered down since this moment (adding timestamp_now - timestamp of last report as downtime).
With these calculations we now have a total downtime (and thus total uptime in this period). For the visualization, there are some options:

Show the % of uptime over the time which actually passed (i.e. uptime / (uptime + downtime). If a node goes down, uptime percentage will go down as well. If a node which was previously down is online, uptime percentage will increase again. Nodes which are up all the time show 100% uptime and nodes which are down all the time show 0% uptime
Show the % of accrued uptime in the period (i.e. uptime / period_duration). Node uptime starts at 0% every period, and increases until it reaches 100% for nodes which are always online. This might be confusing for people since all nodes will start of at 0% every period, so probably not ideal.
Show the max reachable uptime % (i.e. (period_duration - downtime) / period_duration). Nodes which are always online will have 100% over the period. Nodes which are offline for some time will have a lower percentage, and this will stay the same once they are online again. nodes which are offline for the whole period will go from 100% to 0% over the period. This means all nodes start at 100%, including dead nodes, which might be confusing for people.
...

Personally I'd go with the first option as it is probably the most natural and intuitive for people. Other than that, these are mostly similar, though there are subtle nuances to each of them. Whichever is shown should also be explained by a tooltip, so people understand what they are looking at.

hey, @LeeSmet just a question.  Why do you divide by the STANDARD_PERIOD_DURATION in the offset variable (here)[https://github.com/threefoldtech/minting_v3/blob/master/src/period.rs#L25-L36] and then multiply the offset by it again in the start variable? It just seems useless to be even dividing by the STANDARD_PERIOD_DURATION in the first place when you are going to remove it by multiplying by it.

hey @LeeSmet where can I find the uptimeEvent on graphql? Graphql doesn't seem to have that filter, please clarify

uptimeEvents are a type, similar to node, not a filter on node. You can find them at the bottom of the list. There is indeed no use for the receipts API, as that only has receipts from past periods.


Investigate performance of decimal math / aggregation

What is the problem the feature request solves?
SQL
select sum(
    ss_wholesale_cost+
    ss_list_price+
    ss_sales_price+
    ss_ext_discount_amt+
    ss_ext_sales_price+
    ss_ext_wholesale_cost+
    ss_ext_list_price+
    ss_ext_tax+
    ss_coupon_amt+
    ss_net_paid+
    ss_net_paid_inc_tax+
    ss_net_profit
)
from store_sales;

Query time in seconds with Comet disabled:
[24.197060108184814, 22.000279426574707, 22.047854900360107, 22.033849954605103, 21.86830496788025]

With Comet enabled:
[32.81851553916931, 30.068572998046875, 31.110345125198364, 29.976839780807495, 29.35631513595581]

I do not see a slow down if I add all the integer columns in the table, so this seems specific to decimal.
Part of the issue here may be the creation of all of the intermediate result vectors.
Describe the potential solution
No response
Additional context
No response

microbenchmark results @ sf=1
AMD Ryzen 9 7950X3D 16-Core Processor
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_decimals                                   809            872          67          3.6         281.1       1.0X
add_many_decimals                                   770            788          16          3.7         267.5       1.1X
add_many_decimals: Comet (Scan)                     930            952          38          3.1         323.0       0.9X
add_many_decimals: Comet (Scan, Exec)              2021           2030          12          1.4         701.9       0.4X

Benchmark runs @ sf=100 suggest that reading decimal from parquet could potentially be a performance issue.
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_decimals                                 20502          20648         208         14.0          71.2       1.0X
add_many_decimals                                 20498          20544          65         14.1          71.2       1.0X
add_many_decimals: Comet (Scan)                   28143          28161          26         10.2          97.7       0.7X
add_many_decimals: Comet (Scan, Exec)             19323          19497         246         14.9          67.1       1.1X

TPCDS Micro Benchmarks:                           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
agg_sum_decimals_no_grouping                              10552          10583          44         27.3          36.6       1.0X
agg_sum_decimals_no_grouping                              10406          10450          61         27.7          36.1       1.0X
agg_sum_decimals_no_grouping: Comet (Scan)                46013          46278         375          6.3         159.8       0.2X
agg_sum_decimals_no_grouping: Comet (Scan, Exec)          13840          13956         164         20.8          48.1       0.8X

33 iterations of sf=1
Scan only without sum(decimals)

Scan with Spark sum(decimals)

So somehow once Sum is applied, Comet scan slows down...

I looks like once Comet scan is enabled GangWoker uses more time.
Comet scan + Spark sum

Pure Spark

33 iterations of sf=1
Pure scan without doing sum(decimals)

Scan with doing Spark sum(decimals)

So somehow once Sum is applied, Comet scan slows down...

I think these two screen shots are identical?

I think these two screen shots are identical?
Thanks @andygrove updated to the correct one

After #741 there are still some issues with if and case

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_decimals                                 11255          11385         184         25.6          39.1       1.0X
add_many_decimals: Comet (Scan)                   14372          14545         245         20.0          49.9       0.8X
add_many_decimals: Comet (Scan, Exec)              9846           9933         123         29.3          34.2       1.1X

Running benchmark: TPCDS Micro Benchmarks
  Running case: add_many_integers
  Stopped after 2 iterations, 7870 ms
  Running case: add_many_integers: Comet (Scan)
  Stopped after 2 iterations, 6910 ms
  Running case: add_many_integers: Comet (Scan, Exec)
  Stopped after 2 iterations, 7138 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_integers                                  3866           3935          98         74.5          13.4       1.0X
add_many_integers: Comet (Scan)                    3450           3455           7         83.5          12.0       1.1X
add_many_integers: Comet (Scan, Exec)              3548           3569          30         81.2          12.3       1.1X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_high_cardinality
  Stopped after 2 iterations, 3620 ms
  Running case: agg_high_cardinality: Comet (Scan)
  Stopped after 2 iterations, 5411 ms
  Running case: agg_high_cardinality: Comet (Scan, Exec)
  Stopped after 2 iterations, 2126 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_high_cardinality                               1769           1810          59         40.7          24.6       1.0X
agg_high_cardinality: Comet (Scan)                 2642           2706          90         27.3          36.7       0.7X
agg_high_cardinality: Comet (Scan, Exec)           1060           1063           4         67.9          14.7       1.7X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_low_cardinality
  Stopped after 5 iterations, 2032 ms
  Running case: agg_low_cardinality: Comet (Scan)
  Stopped after 3 iterations, 2179 ms
  Running case: agg_low_cardinality: Comet (Scan, Exec)
  Stopped after 7 iterations, 2089 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_low_cardinality                                 371            406          34        194.2           5.1       1.0X
agg_low_cardinality: Comet (Scan)                   717            727          13        100.4          10.0       0.5X
agg_low_cardinality: Comet (Scan, Exec)             278            298          19        258.7           3.9       1.3X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_decimals_no_grouping
  Stopped after 2 iterations, 14633 ms
  Running case: agg_sum_decimals_no_grouping: Comet (Scan)
  Stopped after 2 iterations, 78280 ms
  Running case: agg_sum_decimals_no_grouping: Comet (Scan, Exec)
  Stopped after 2 iterations, 16892 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
agg_sum_decimals_no_grouping                               7251           7317          93         39.7          25.2       1.0X
agg_sum_decimals_no_grouping: Comet (Scan)                38946          39140         274          7.4         135.2       0.2X
agg_sum_decimals_no_grouping: Comet (Scan, Exec)           8407           8446          55         34.3          29.2       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_integers_no_grouping
  Stopped after 2 iterations, 8131 ms
  Running case: agg_sum_integers_no_grouping: Comet (Scan)
  Stopped after 2 iterations, 8375 ms
  Running case: agg_sum_integers_no_grouping: Comet (Scan, Exec)
  Stopped after 2 iterations, 9179 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_no_grouping                               3914           4066         215         73.6          13.6       1.0X
agg_sum_integers_no_grouping: Comet (Scan)                 4079           4188         154         70.6          14.2       1.0X
agg_sum_integers_no_grouping: Comet (Scan, Exec)           4557           4590          47         63.2          15.8       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: case_when_column_or_null
  Stopped after 2 iterations, 3114 ms
  Running case: case_when_column_or_null: Comet (Scan)
  Stopped after 2 iterations, 5693 ms
  Running case: case_when_column_or_null: Comet (Scan, Exec)
  Stopped after 2 iterations, 3415 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------
case_when_column_or_null                               1440           1557         166        200.1           5.0       1.0X
case_when_column_or_null: Comet (Scan)                 2832           2847          20        101.7           9.8       0.5X
case_when_column_or_null: Comet (Scan, Exec)           1691           1708          23        170.3           5.9       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: case_when_scalar
  Stopped after 9 iterations, 2090 ms
  Running case: case_when_scalar: Comet (Scan)
  Stopped after 2 iterations, 2312 ms
  Running case: case_when_scalar: Comet (Scan, Exec)
  Stopped after 5 iterations, 2048 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
case_when_scalar                                    196            232          51        367.0           2.7       1.0X
case_when_scalar: Comet (Scan)                     1133           1156          33         63.5          15.7       0.2X
case_when_scalar: Comet (Scan, Exec)                376            410          38        191.5           5.2       0.5X

Running benchmark: TPCDS Micro Benchmarks
  Running case: filter_highly_selective
  Stopped after 11 iterations, 2132 ms
  Running case: filter_highly_selective: Comet (Scan)
  Stopped after 3 iterations, 2233 ms
  Running case: filter_highly_selective: Comet (Scan, Exec)
  Stopped after 10 iterations, 2054 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------
filter_highly_selective                                154            194          85        466.1           2.1       1.0X
filter_highly_selective: Comet (Scan)                  734            745           9         98.0          10.2       0.2X
filter_highly_selective: Comet (Scan, Exec)            156            205         108        462.3           2.2       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: filter_less_selective
  Stopped after 7 iterations, 2091 ms
  Running case: filter_less_selective: Comet (Scan)
  Stopped after 3 iterations, 2093 ms
  Running case: filter_less_selective: Comet (Scan, Exec)
  Stopped after 10 iterations, 2272 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------
filter_less_selective                                165            299         165        435.4           2.3       1.0X
filter_less_selective: Comet (Scan)                  691            698          11        104.3           9.6       0.2X
filter_less_selective: Comet (Scan, Exec)            193            227          39        373.7           2.7       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: if_column_or_null
  Stopped after 2 iterations, 2630 ms
  Running case: if_column_or_null: Comet (Scan)
  Stopped after 2 iterations, 3102 ms
  Running case: if_column_or_null: Comet (Scan, Exec)
  Stopped after 2 iterations, 5368 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
if_column_or_null                                  1315           1315           0        219.1           4.6       1.0X
if_column_or_null: Comet (Scan)                    1518           1551          46        189.8           5.3       0.9X
if_column_or_null: Comet (Scan, Exec)              2606           2684         111        110.5           9.0       0.5X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_anti
  Stopped after 2 iterations, 12404 ms
  Running case: join_anti: Comet (Scan)
  Stopped after 2 iterations, 12023 ms
  Running case: join_anti: Comet (Scan, Exec)
[528.105s][warning][gc,alloc] Executor task launch worker for task 2.0 in stage 802.0 (TID 14773): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 11999 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_anti                                          6128           6202         104         11.7          85.1       1.0X
join_anti: Comet (Scan)                            5774           6012         335         12.5          80.2       1.1X
join_anti: Comet (Scan, Exec)                      5971           6000          41         12.1          82.9       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_condition
  Stopped after 2 iterations, 4255 ms
  Running case: join_condition: Comet (Scan)
  Stopped after 2 iterations, 3245 ms
  Running case: join_condition: Comet (Scan, Exec)
  Stopped after 2 iterations, 3721 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_condition                                     2055           2128         103        264.4           3.8       1.0X
join_condition: Comet (Scan)                       1534           1623         125        354.2           2.8       1.3X
join_condition: Comet (Scan, Exec)                 1808           1861          75        300.6           3.3       1.1X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_exploding_output
  Stopped after 2 iterations, 3425 ms
  Running case: join_exploding_output: Comet (Scan)
  Stopped after 2 iterations, 3004 ms
  Running case: join_exploding_output: Comet (Scan, Exec)
  Stopped after 2 iterations, 3495 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------
join_exploding_output                               1603           1713         155        339.0           2.9       1.0X
join_exploding_output: Comet (Scan)                 1413           1502         126        384.5           2.6       1.1X
join_exploding_output: Comet (Scan, Exec)           1722           1748          37        315.6           3.2       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_inner
  Stopped after 4 iterations, 2194 ms
  Running case: join_inner: Comet (Scan)
  Stopped after 5 iterations, 2324 ms
  Running case: join_inner: Comet (Scan, Exec)
  Stopped after 3 iterations, 2110 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_inner                                          512            549          36        562.2           1.8       1.0X
join_inner: Comet (Scan)                            462            465           3        623.9           1.6       1.1X
join_inner: Comet (Scan, Exec)                      694            703          10        415.1           2.4       0.7X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_left_outer
  Stopped after 2 iterations, 192095 ms
  Running case: join_left_outer: Comet (Scan)
  Stopped after 2 iterations, 192225 ms
  Running case: join_left_outer: Comet (Scan, Exec)
  Stopped after 2 iterations, 191166 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_left_outer                                   95996          96048          73          3.3         303.0       1.0X
join_left_outer: Comet (Scan)                     95378          96113        1038          3.3         301.1       1.0X
join_left_outer: Comet (Scan, Exec)               95377          95583         292          3.3         301.1       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_semi
[1448.983s][warning][gc,alloc] Executor task launch worker for task 0.0 in stage 1153.0 (TID 20715): Retried waiting for GCLocker too often allocating 134217730 words
[1463.584s][warning][gc,alloc] Executor task launch worker for task 2.0 in stage 1159.0 (TID 20772): Retried waiting for GCLocker too often allocating 134217730 words
[1476.740s][warning][gc,alloc] Executor task launch worker for task 0.0 in stage 1165.0 (TID 20825): Retried waiting for GCLocker too often allocating 134217730 words
[1476.754s][warning][gc,alloc] Executor task launch worker for task 1.0 in stage 1165.0 (TID 20826): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 26371 ms
  Running case: join_semi: Comet (Scan)
[1490.216s][warning][gc,alloc] Executor task launch worker for task 2.0 in stage 1171.0 (TID 20882): Retried waiting for GCLocker too often allocating 134217730 words
[1515.531s][warning][gc,alloc] Executor task launch worker for task 1.0 in stage 1183.0 (TID 20991): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 25521 ms
  Running case: join_semi: Comet (Scan, Exec)
[1541.053s][warning][gc,alloc] Executor task launch worker for task 2.0 in stage 1195.0 (TID 21102): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 25372 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_semi                                         13164          13186          30          5.5         182.8       1.0X
join_semi: Comet (Scan)                           12522          12761         338          5.8         173.9       1.1X
join_semi: Comet (Scan, Exec)                     12276          12686         580          5.9         170.5       1.1X

After merging with latest main
OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_decimals                                 11828          12330         709         24.3          41.1       1.0X
add_many_decimals: Comet (Scan)                   15061          15181         171         19.1          52.3       0.8X
add_many_decimals: Comet (Scan, Exec)             12400          12885         686         23.2          43.1       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: add_many_integers
  Stopped after 2 iterations, 7415 ms
  Running case: add_many_integers: Comet (Scan)
  Stopped after 2 iterations, 6969 ms
  Running case: add_many_integers: Comet (Scan, Exec)
  Stopped after 2 iterations, 7371 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
add_many_integers                                  3590           3708         166         80.2          12.5       1.0X
add_many_integers: Comet (Scan)                    3467           3485          24         83.1          12.0       1.0X
add_many_integers: Comet (Scan, Exec)              3659           3686          38         78.7          12.7       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_high_cardinality
  Stopped after 2 iterations, 3604 ms
  Running case: agg_high_cardinality: Comet (Scan)
  Stopped after 2 iterations, 5400 ms
  Running case: agg_high_cardinality: Comet (Scan, Exec)
  Stopped after 2 iterations, 2131 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_high_cardinality                               1793           1802          13         40.2          24.9       1.0X
agg_high_cardinality: Comet (Scan)                 2683           2700          24         26.8          37.3       0.7X
agg_high_cardinality: Comet (Scan, Exec)           1056           1066          14         68.2          14.7       1.7X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_low_cardinality
  Stopped after 6 iterations, 2078 ms
  Running case: agg_low_cardinality: Comet (Scan)
  Stopped after 3 iterations, 2226 ms
  Running case: agg_low_cardinality: Comet (Scan, Exec)
  Stopped after 8 iterations, 2066 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_low_cardinality                                 333            346          10        216.5           4.6       1.0X
agg_low_cardinality: Comet (Scan)                   730            742          14         98.7          10.1       0.5X
agg_low_cardinality: Comet (Scan, Exec)             252            258           7        286.1           3.5       1.3X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_decimals_no_grouping
  Stopped after 2 iterations, 15998 ms
  Running case: agg_sum_decimals_no_grouping: Comet (Scan)
  Stopped after 2 iterations, 87493 ms
  Running case: agg_sum_decimals_no_grouping: Comet (Scan, Exec)
  Stopped after 2 iterations, 16900 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
agg_sum_decimals_no_grouping                               7828           7999         243         36.8          27.2       1.0X
agg_sum_decimals_no_grouping: Comet (Scan)                42539          43747        1708          6.8         147.7       0.2X
agg_sum_decimals_no_grouping: Comet (Scan, Exec)           8441           8450          12         34.1          29.3       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_integers_no_grouping
  Stopped after 2 iterations, 7564 ms
  Running case: agg_sum_integers_no_grouping: Comet (Scan)
  Stopped after 2 iterations, 8300 ms
  Running case: agg_sum_integers_no_grouping: Comet (Scan, Exec)
  Stopped after 2 iterations, 9181 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_no_grouping                               3761           3782          30         76.6          13.1       1.0X
agg_sum_integers_no_grouping: Comet (Scan)                 4125           4150          36         69.8          14.3       0.9X
agg_sum_integers_no_grouping: Comet (Scan, Exec)           4523           4591          96         63.7          15.7       0.8X

Running benchmark: TPCDS Micro Benchmarks
  Running case: case_when_column_or_null
  Stopped after 2 iterations, 2203 ms
  Running case: case_when_column_or_null: Comet (Scan)
  Stopped after 2 iterations, 5703 ms
  Running case: case_when_column_or_null: Comet (Scan, Exec)
  Stopped after 2 iterations, 3428 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------
case_when_column_or_null                               1099           1102           4        262.1           3.8       1.0X
case_when_column_or_null: Comet (Scan)                 2827           2852          35        101.9           9.8       0.4X
case_when_column_or_null: Comet (Scan, Exec)           1681           1714          47        171.3           5.8       0.7X

Running benchmark: TPCDS Micro Benchmarks
  Running case: case_when_scalar
  Stopped after 10 iterations, 2146 ms
  Running case: case_when_scalar: Comet (Scan)
  Stopped after 2 iterations, 2433 ms
  Running case: case_when_scalar: Comet (Scan, Exec)
  Stopped after 5 iterations, 2019 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
case_when_scalar                                    193            215          32        373.6           2.7       1.0X
case_when_scalar: Comet (Scan)                     1210           1217           9         59.5          16.8       0.2X
case_when_scalar: Comet (Scan, Exec)                378            404          44        190.4           5.3       0.5X

Running benchmark: TPCDS Micro Benchmarks
  Running case: filter_highly_selective
  Stopped after 13 iterations, 2221 ms
  Running case: filter_highly_selective: Comet (Scan)
  Stopped after 3 iterations, 2297 ms
  Running case: filter_highly_selective: Comet (Scan, Exec)
  Stopped after 11 iterations, 2084 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------
filter_highly_selective                                145            171          32        496.7           2.0       1.0X
filter_highly_selective: Comet (Scan)                  762            766           4         94.5          10.6       0.2X
filter_highly_selective: Comet (Scan, Exec)            171            190          18        420.4           2.4       0.8X

Running benchmark: TPCDS Micro Benchmarks
  Running case: filter_less_selective
  Stopped after 10 iterations, 2011 ms
  Running case: filter_less_selective: Comet (Scan)
  Stopped after 3 iterations, 2121 ms
  Running case: filter_less_selective: Comet (Scan, Exec)
  Stopped after 11 iterations, 2159 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------
filter_less_selective                                170            201          42        422.4           2.4       1.0X
filter_less_selective: Comet (Scan)                  694            707          12        103.8           9.6       0.2X
filter_less_selective: Comet (Scan, Exec)            176            196          20        409.9           2.4       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: if_column_or_null
  Stopped after 2 iterations, 2282 ms
  Running case: if_column_or_null: Comet (Scan)
  Stopped after 2 iterations, 3123 ms
  Running case: if_column_or_null: Comet (Scan, Exec)
  Stopped after 2 iterations, 3524 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
if_column_or_null                                  1136           1141           7        253.6           3.9       1.0X
if_column_or_null: Comet (Scan)                    1473           1562         126        195.6           5.1       0.8X
if_column_or_null: Comet (Scan, Exec)              1751           1762          17        164.6           6.1       0.6X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_anti
  Stopped after 2 iterations, 12717 ms
  Running case: join_anti: Comet (Scan)
  Stopped after 2 iterations, 11808 ms
  Running case: join_anti: Comet (Scan, Exec)
  Stopped after 2 iterations, 11755 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_anti                                          6230           6359         183         11.6          86.5       1.0X
join_anti: Comet (Scan)                            5789           5904         163         12.4          80.4       1.1X
join_anti: Comet (Scan, Exec)                      5720           5878         222         12.6          79.4       1.1X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_condition
  Stopped after 2 iterations, 3323 ms
  Running case: join_condition: Comet (Scan)
  Stopped after 2 iterations, 3201 ms
  Running case: join_condition: Comet (Scan, Exec)
  Stopped after 2 iterations, 3376 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_condition                                     1568           1662         133        346.6           2.9       1.0X
join_condition: Comet (Scan)                       1479           1601         172        367.4           2.7       1.1X
join_condition: Comet (Scan, Exec)                 1688           1688           1        321.9           3.1       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_exploding_output
  Stopped after 2 iterations, 2796 ms
  Running case: join_exploding_output: Comet (Scan)
  Stopped after 2 iterations, 2732 ms
  Running case: join_exploding_output: Comet (Scan, Exec)
  Stopped after 2 iterations, 3220 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------
join_exploding_output                               1395           1398           5        389.5           2.6       1.0X
join_exploding_output: Comet (Scan)                 1352           1366          20        401.8           2.5       1.0X
join_exploding_output: Comet (Scan, Exec)           1608           1610           4        338.0           3.0       0.9X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_inner
  Stopped after 4 iterations, 2045 ms
  Running case: join_inner: Comet (Scan)
  Stopped after 5 iterations, 2359 ms
  Running case: join_inner: Comet (Scan, Exec)
  Stopped after 3 iterations, 2083 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_inner                                          497            511          12        579.1           1.7       1.0X
join_inner: Comet (Scan)                            460            472           8        626.9           1.6       1.1X
join_inner: Comet (Scan, Exec)                      687            694           8        419.2           2.4       0.7X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_left_outer
  Stopped after 2 iterations, 193508 ms
  Running case: join_left_outer: Comet (Scan)
[1025.721s][warning][gc,alloc] Executor task launch worker for task 1.0 in stage 1178.0 (TID 21368): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 192356 ms
  Running case: join_left_outer: Comet (Scan, Exec)
  Stopped after 2 iterations, 191410 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_left_outer                                   96613          96754         200          3.3         305.0       1.0X
join_left_outer: Comet (Scan)                     95176          96178        1418          3.3         300.4       1.0X
join_left_outer: Comet (Scan, Exec)               94835          95705        1231          3.3         299.3       1.0X

Running benchmark: TPCDS Micro Benchmarks
  Running case: join_semi
[1485.285s][warning][gc,alloc] Executor task launch worker for task 3.0 in stage 1221.0 (TID 21917): Retried waiting for GCLocker too often allocating 134217730 words
[1510.652s][warning][gc,alloc] Executor task launch worker for task 3.0 in stage 1233.0 (TID 22027): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 25421 ms
  Running case: join_semi: Comet (Scan)
[1520.632s][warning][gc,alloc] Executor task launch worker for task 3.0 in stage 1239.0 (TID 22082): Retried waiting for GCLocker too often allocating 134217730 words
[1520.632s][warning][gc,alloc] Executor task launch worker for task 1.0 in stage 1239.0 (TID 22080): Retried waiting for GCLocker too often allocating 134217730 words
[1547.750s][warning][gc,alloc] Executor task launch worker for task 3.0 in stage 1251.0 (TID 22192): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 26686 ms
  Running case: join_semi: Comet (Scan, Exec)
[1573.897s][warning][gc,alloc] Executor task launch worker for task 2.0 in stage 1263.0 (TID 22301): Retried waiting for GCLocker too often allocating 134217730 words
  Stopped after 2 iterations, 25646 ms

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Mac OS X 14.5
Apple M1 Max
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
join_semi                                         11986          12711        1025          6.0         166.5       1.0X
join_semi: Comet (Scan)                           12957          13343         546          5.6         180.0       0.9X
join_semi: Comet (Scan, Exec)                     12694          12823         183          5.7         176.3       0.9X

We created many fixes. I think decimals are no longer issues. closing for now


[Medium] [Submodule+Server] Bug: Token refresh not working

It should be refreshing!
We already check if the token is expired using:
https://github.com/trypear/pearai-submodule/blob/971ad37637d257b6f7033805d4843f3df142619a/core/db/token.ts#L43
and it should be updated:
https://github.com/trypear/pearai-submodule/blob/971ad37637d257b6f7033805d4843f3df142619a/core/llm/llms/PearAIServer.ts#L18
Basically, there is an
Access Token and a Refresh Token
On the client side (pearai-submodule), it checks if it the token is expired. It either does not detect it is expired correctly, or does not update correctly.
It passes the new accessToken (or apiKey, same thing in this case) to the server.
On the server side (pearai-server), it checks if the token is expired, and it is saying it is expired
How to reproduce
Make the server side return 401 error locally, and this error will come up

Nvm i can fix this

We can implement auto-refresh like this:
https://supabase.com/docs/reference/javascript/auth-startautorefresh

interesting.. will look into this
so far i guess when we need to update it in local storage if it detects a change

Currently working on it

Monitor progress here: https://github.com/trypear/pearai-submodule/pull/112

Log of when it happens.

[Extension Host] Attempting to get credentials... (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
workbench.desktop.main.js:58161 [Extension Host] Access token is expired, attempting to refresh (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
workbench.desktop.main.js:58161 [Extension Host] Error refreshing token, redirecting to login: AuthApiError: Invalid Refresh Token: Already Used
    at handleError2 (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:84061:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at _handleRequest (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:84104:9)
    at _request (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:84087:21)
    at /Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:85877:20
    at /Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:83797:30 (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
workbench.desktop.main.js:123787   ERR [Extension Host] Error checking token expiration: ReferenceError: window is not defined
    at checkTokens (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:86903:7)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at PearAIServer._streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:87021:26)
    at PearAIServer.streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:36232:30)
    at llmStreamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374401:23)
    at UniqueContainer.value (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374667:29)
workbench.desktop.main.js:58161 [Extension Host] Error checking token expiration: ReferenceError: window is not defined
    at checkTokens (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:86903:7)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at PearAIServer._streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:87021:26)
    at PearAIServer.streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:36232:30)
    at llmStreamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374401:23)
    at UniqueContainer.value (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374667:29) (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
log @ workbench.desktop.main.js:58161
$logExtensionHostMessage @ workbench.desktop.main.js:325885
S @ workbench.desktop.main.js:326516
Q @ workbench.desktop.main.js:326501
M @ workbench.desktop.main.js:326442
L @ workbench.desktop.main.js:326369
(anonymous) @ workbench.desktop.main.js:326248
y @ workbench.desktop.main.js:16700
fire @ workbench.desktop.main.js:16729
fire @ workbench.desktop.main.js:89909
port.onmessage @ workbench.desktop.main.js:344157
workbench.desktop.main.js:58161 [Extension Host] Error: Error: HTTP 401 Unauthorized from https://stingray-app-gb2an.ondigitalocean.app/pearai-server-api2/server_chat

{"error":"Token has expired"}

    at customFetch (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:36104:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at withExponentialBackoff (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:35856:26)
    at PearAIServer._streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:87050:26)
    at PearAIServer.streamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:36232:30)
    at llmStreamChat (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374401:23)
    at UniqueContainer.value (/Applications/PearAI.app/Contents/Resources/app/extensions/pearai.pearai-0.0.1/out/extension.js:374667:29) (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
workbench.desktop.main.js:123787   ERR [Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "aba75193-9226-41b5-b9ba-1bcd027365c7",
    "messageType": "llm/streamChat",
    "data": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "```proxy_service.py (25-88)\n\ndef handle_server_chat(auth: Optional[Auth], span: Span, json_data):\n    # TODO: We should log to traces why the request failed not to the client (for security purposes)\n    if not auth:\n        return jsonify({\"error\": \"Authentication is required\"}), 403\n\n    token = auth.token\n    if not token:\n        return jsonify({\"error\": \"Token is missing\"}), 403\n\n    user_id = auth.get_user_id()\n    if not user_id:\n        return jsonify({\"error\": \"User ID is missing\"}), 403\n\n    span.set_attribute(\"user_id\", user_id)\n\n    # Check if user is subscribed\n    subscription = get_subscription_use_user(token, user_id)\n\n    if not subscription or subscription.status != \"active\":\n        code = 403\n        telem_utils.complete_span(span, code)\n        return (\n            jsonify(\n                {\n                    \"error\": \"User subscription is not active. Sign up at https://trypear.ai, or report an issue on the PearAI Discord.\"\n                }\n            ),\n            code,\n        )\n\n    # Check the quota based on users total requests this month\n    quota_group = \"total-requests\"\n\n    # initialize the users quota\n    if not quota.has_active_quota(user_id, quota_group):\n        profile = get_profile(token, user_id)\n        pricing = get_pricing_by_tier(subscription.pricing_tier)\n\n        quota.create_user_quota(\n            user_id=user_id,\n            quota_name=quota_group,\n            # use monthly_requests for quota because we reset the quota monthly\n            max_quota=pricing.monthly_requests if pricing else None,\n            initial_value=profile.requests_used if profile else 0,\n        )\n\n    quota_details = quota.get_user_quota_details(user_id, quota_group)\n\n    span.set_attribute(\"used_quota\", quota_details[\"used_quota\"])\n\n    if quota_details[\"quota_remaining\"] <= 0:\n        code = 403\n        telem_utils.complete_span(span, code)\n        return (\n            jsonify({\"error\": \"Quota exceeded\", \"details\": quota_details}),\n            code,\n        )\n    quota.increment_user_quota(user_id, quota_group)\n    # Todo: Abstract this to use classes so we can use different LLM's\n    # result = handle_stream_chat(span, json_data) OpenAI Service\n    result = handle_stream_chat(span, json_data)\n    return result\n\n```\nif get_config().DEV_ENABLED IS TRUE IT SHOULD SKIP ALL THE QUOTA STUFF"
            }
          ]
        }
      ],
      "title": "PearAI Server",
      "completionOptions": {}
    }
  }
}
workbench.desktop.main.js:58161 [Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "aba75193-9226-41b5-b9ba-1bcd027365c7",
    "messageType": "llm/streamChat",
    "data": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "```proxy_service.py (25-88)\n\ndef handle_server_chat(auth: Optional[Auth], span: Span, json_data):\n    # TODO: We should log to traces why the request failed not to the client (for security purposes)\n    if not auth:\n        return jsonify({\"error\": \"Authentication is required\"}), 403\n\n    token = auth.token\n    if not token:\n        return jsonify({\"error\": \"Token is missing\"}), 403\n\n    user_id = auth.get_user_id()\n    if not user_id:\n        return jsonify({\"error\": \"User ID is missing\"}), 403\n\n    span.set_attribute(\"user_id\", user_id)\n\n    # Check if user is subscribed\n    subscription = get_subscription_use_user(token, user_id)\n\n    if not subscription or subscription.status != \"active\":\n        code = 403\n        telem_utils.complete_span(span, code)\n        return (\n            jsonify(\n                {\n                    \"error\": \"User subscription is not active. Sign up at https://trypear.ai, or report an issue on the PearAI Discord.\"\n                }\n            ),\n            code,\n        )\n\n    # Check the quota based on users total requests this month\n    quota_group = \"total-requests\"\n\n    # initialize the users quota\n    if not quota.has_active_quota(user_id, quota_group):\n        profile = get_profile(token, user_id)\n        pricing = get_pricing_by_tier(subscription.pricing_tier)\n\n        quota.create_user_quota(\n            user_id=user_id,\n            quota_name=quota_group,\n            # use monthly_requests for quota because we reset the quota monthly\n            max_quota=pricing.monthly_requests if pricing else None,\n            initial_value=profile.requests_used if profile else 0,\n        )\n\n    quota_details = quota.get_user_quota_details(user_id, quota_group)\n\n    span.set_attribute(\"used_quota\", quota_details[\"used_quota\"])\n\n    if quota_details[\"quota_remaining\"] <= 0:\n        code = 403\n        telem_utils.complete_span(span, code)\n        return (\n            jsonify({\"error\": \"Quota exceeded\", \"details\": quota_details}),\n            code,\n        )\n    quota.increment_user_quota(user_id, quota_group)\n    # Todo: Abstract this to use classes so we can use different LLM's\n    # result = handle_stream_chat(span, json_data) OpenAI Service\n    result = handle_stream_chat(span, json_data)\n    return result\n\n```\nif get_config().DEV_ENABLED IS TRUE IT SHOULD SKIP ALL THE QUOTA STUFF"
            }
          ]
        }
      ],
      "title": "PearAI Server",
      "completionOptions": {}
    }
  }
} (at console.<anonymous> (/Applications/PearAI.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:61776:26))
log @ workbench.desktop.main.js:58161
$logExtensionHostMessage @ workbench.desktop.main.js:325885
S @ workbench.desktop.main.js:326516
Q @ workbench.desktop.main.js:326501
M @ workbench.desktop.main.js:326442
L @ workbench.desktop.main.js:326369
(anonymous) @ workbench.desktop.main.js:326248
y @ workbench.desktop.main.js:16700
fire @ workbench.desktop.main.js:16729
fire @ workbench.desktop.main.js:89909
port.onmessage @ workbench.desktop.main.js:344157
workbench.desktop.main.js:234705 Error: HTTP 401 Unauthorized from https://stingray-app-gb2an.ondigitalocean.app/pearai-server-api2/server_chat  {"error":"Token has expired"}

Working on Server-side: https://github.com/trypear/pearai-server/pull/111

Ohhh okay, this is interesting.
This is whats happening:
submodule has accessToken A and refreshToken B
refreshSession is totally fine, as many times as you want.
you get
accessToken C and refreshToken D
accessToken E and refreshToken F
and so on...
however, when the user logs in again, it gets refreshToken B again.
so then you only have refreshToken B, and its already used so you cant refreshSession

So we either need to:
On landing page: if theres an auth request, we need to refresh session when theres a login
On submodule: when we login, we try to refresh session. If it doesnt work ,dont take the params. Refresh session using our own refresh session

NVM.
https://cdn.discordapp.com/attachments/1267978653325856779/1268003472712667176/Screenshot_2024-07-30_at_5.33.01_PM.png?ex=66aad7c2&is=66a98642&hm=d6d969df3d39350c3c3cdfe7fa4042f4be86a50b77ad3eceb00eb0d35f062668&
My proposed solution is to have another sign in page on landing-page JUST for the app

Solved for now by allowing token refresh


NEXT.coin: no history

https://rinkeby.swaponline.io/#/NEXT/XHeAoTn4NECVuQjkpj6YWuXx8kAspEggkR

looks solved


Routes

When a directory level is selected from Breadcrumbs, it cannot locate a file. Enable users to set a Default page for each Section.

Same for unknown file types

File extensions are visible


"silence is not valid" on webui when using regex

What did you do?
I tried creating an alert with regex field like app_kubernetes_io_instance=~"application-.+"
What did you expect to see?
I expected alert to be created
What did you see instead? Under which circumstances?
I'm receiving Could not submit the form, Silence is not yet valid. error while the filed itself was green.

Environment

System information:

Linux 5.4.0-77-generic x86_64

Alertmanager version:

alertmanager, version 0.22.2 (branch: HEAD, revision: 44f8adc06af5101ad64bd8b9c8b18273f2922051)
  build user:       root@b595c7f32520
  build date:       20210602-07:50:37
  go version:       go1.16.4
  platform:         linux/amd64

Prometheus version:

prometheus, version 2.28.1 (branch: HEAD, revision: b0944590a1c9a6b35dc5a696869f75f422b107a1)
  build user:       root@2915dd495090
  build date:       20210701-15:20:10
  go version:       go1.16.5
  platform:         linux/amd64

Hello, you need to click the "+" button, but the field should indicate that you need to click it.

Aha. I was clicking on create button by habit.

But the field should indicate that you need to click it.

Or the UI can add the written filter automatically

This should be closed IMO, unless having a message post saying no filter criteria was added


disable Modernizer via @SuppressWarnings annotation

@SuppressWarnings allows finer-grained suppression of violations than the existing file-based mechanism.

If I understand modernizer correctly, this wouldn't work: @SuppressWarnings is only retained in the source, not in the generated class files.

@ankon I have limited background in annotation processing but I believe one can use the RUNTIME retention policy:
http://docs.oracle.com/javase/7/docs/api/java/lang/annotation/RetentionPolicy.html

@andrewgaul, that's true when one has control over the annotation. In this case, however, java.lang.SuppressWarnings is annotated @Retention(RetentionPolicy.SOURCE). Not much we can do about that :/

I see, so we would need a new SuppressModernizerWarnings class with the RUNTIME policy?

That would be a solution, yes. (Though not quite as nice as the original suggestion.) Have no better idea at this point. (Other than adding support for source code analysis to the plugin, of course!)

asm is operating on the bytecode, so I would think RententionPolicy.CLASS should be sufficient.

Ah, right. That's more appropriate indeed.

Thinking a bit outside of the box here: One "evil" way could be to add a annotation processing step, which finds the @SuppressWarnings that pertain to modernizer, and dumps those into an exclusion file which is then used by the actual modernizer goal.
This would avoid polluting sources with build-system specific things ... except that of course the value in the annotation is also modernizer-specific, and in fact eclipse warns about unrecognized values in it :/

way could be to add a annotation processing step, which finds the @SuppressWarnings that pertain to modernizer, and dumps those into an exclusion file which is then used by the actual modernizer goal.

This would be cool, and help with https://github.com/gaul/modernizer-maven-plugin/issues/28 as well...

Findbugs has its own annotation for suppressing warnings.  One interesting thing finbugs did is not caring which package the annotation is in, see : https://stackoverflow.com/a/44947252 .  This allows users to declare the annotation and avoid having a dependency on findbugs.

Just a heads up we're going to be working on this in our fork (https://github.com/HubSpot/modernizer-maven-plugin) and will upstream the changes once they're ready.

@jhaber Any updates on this?

We have it working where you can put @SuppressWarnings("modernizer") at the class level and that will get respected, and there is a PR to support this at the method level: https://github.com/HubSpot/modernizer-maven-plugin/pull/2
Once that is merged, we were thinking that it would be a good MVP of this feature and we would send a PR to upstream those changes.
Some notes about our approach:

it depends on an annotation processor, which means that users need to add a new dependency for these suppressions to work (using <scope>provided</scope> on this dependency is fine)
the annotation processor writes a CSV containing information about the excluded classes/methods which the plugin reads. We weren't sure if you had a preference for using something more structured like XML/JSON/YAML (the latter two would require additional dependencies in the processor/plugin)
@SuppressWarnings("modernizer") will disable all modernizer checks for the relevant code block. To support more granular suppressions, we would probably need to expand the violations file schema to include support for specifying an identifier for each violation. For example, the com/google/common/base/Function violation could have <id>GuavaFunction</id> and then you could do @SuppressWarnings("modernizer:GuavaFunction") to only suppress that specific check (this matches the syntax that other tools like checkstyle use to suppress specific checks, for example)

Sounds good.  I have concerns about the large number of spurious changes in that pull request so please minimize the noise before submitting a pull request.  I look forward to this feature!


Output of compareMS2 inconsistent

In the output of compareMS2, the values of min_basepeak_intensity and min_total_ion_current are separated by a = character instead of a tab (which is used for other values):
min_basepeak_intensity=10000.00
min_total_ion_current=0.00

Also, line with n_m/z_bins is duplicated

Fixed!


Settings UI gear is missing

Maybe related to @aeschli's icon updates. It's also now always visible, instead of only showing on hover.

Fixed yesterday


Make views and fields use defaults properly

don't need to add false to base level fields and views options
uses default fields and views paths as last resort.
does not throw if the default paths exist
move getPaths, getViews, and getFields into helper class as statics
update readme

Resolves #52 and #68

Whoop! @easternbloc, @bendiggle, @danielepolencic @sulthan-ahmed

@joefitter @sulthan-ahmed @easternbloc I am updating this; removing the detault fields and views values.
Should bootstrap throw an error if no fields are set or just a console warning?

it is possible that a route would have no fields, but this would only work if common fields are set on the base config object. If you want to be clever you could throw an error if no fields are found for the route and no common fields have been set, but easier option could be just allow no fields to be set

views shouldn't warn or throw an error IMO as if omitted it will always use hof-template-partials

thanks @joefitter. Views should never throw, and fields will throw if none set for either route or base.

LGTM!