support large binary objects

I just learned about cotire and was very impressed. Just what I always wanted! PCH in a box!
So I just tried and applied it to a project on VC11 that is rather heavy on templates boost and spirit and such and it almost worked, until:
error C3859: virtual memory range for PCH exceeded; please recompile with a command line option of '-Zm378' or greater
I usually have this error in my projects and generally set this option in my own CMake scripts, that are a collection of functions for ease of use and dependency management.
It would be nice to be able to pass such options into cotire or have them already in place there.
Cheers,
Moose

How many include directives do you have in the prefix header generated by cotire?

I have added support for setting the PCH scaling factor in cotire 1.7.0. To configure the scaling factor set the variable COTIRE_PCH_MEMORY_SCALING_FACTOR before including cotire:
set (COTIRE_PCH_MEMORY_SCALING_FACTOR "378")
include(cotire)

Wow, that totally slipped my mind. My apologies I didn't respond to your question. Just didn't notice it.
Anyway, I have just included that new option and as it looks like it does build now. I had to crank it up to 500 though but that's OK.
An issue that popped up with this configuration however, is that all of a sudden almost all the objects started throwing thousands of warnings like "decorated name length exceeded, name was truncated". Which are usually not that severe but clutter the build and hide real warnings. Also, I see that the sheer amount of such warnings slows down the compile. It's so many of them, I cannot even use the side bar to scroll. They all seem to come from within boost.
For example:
"warning C4503: 'boost::proto::exprns_::extends<Expr,Derived>' : decorated name length exceeded, name was truncated"
Also, to my surprise, the speed up seems to exclude some files. Some compile much faster as expected, but others take as long as before or only slightly less. Unfortunately those are the ones with template heavy code that go all the way through boost mpl, proto, phoenix, spirit and such. They generally have compile times of 6 or 7 minutes for one object. They are faster now but still take around 4-5 minutes.
I'm still trying to figure out if it all can work. Unfortunately all my attempts to create a build failed. In some modules, the compiler behaves differently than before.
I get errors with a few cases of templates that are specialized for specific types. The specialization appears to be missing or cannot be found when a typedef'ed symbol is used to deduce. Also, some headers that are aliasing other namespaces all of a sudden fail I guess that's in the nature of PCH though. I will continue to try create a build once I get a chance. Just wanted to give you a quick heads up on that one. Still very promising.
Cheers,
Moose

I always add the following flags to disable MSVC's noise warnings:
	# disable noisy "decorated name length exceeded, name was truncated" warning
	set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd4503")
	# disable noisy "unreferenced inline function has been removed" warning
	set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd4514")
	# disable noisy "function not inlined" warning
	set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd4710")
	# disable noisy "padding added after data member" warning
	set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd4820")
	# disable noisy "function selected for automatic inline expansion" warning
	set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd4711")

The target's first source file is used to precompile the generated header so this one might take longer to compile.

Hmmm so why would using cotire change the length of a template symbol?


Feature request: locate and paste-buffer commands

Description
This is a feature request.
There are two items might be good to add in fzf-lua. The first item is a locate command: it is to search by the locate command, then display results in a fzf list. The original fzf-vim has this one. The second is a paste-buffer command: it is to display reg contents, then the user can select which item to paste into the current buffer.
Thanks.

I’ll look into the locate command.
The second request is already possible with :FzfLua registers.

Thanks...

The locate command seems to be a vim only thing, :help locate returns nothing, are you sure this exists with fzf.vim running on neovim?

The locate command seems to be a vim only thing, :help locate returns nothing, are you sure this exists with fzf.vim running on neovim?

The command in fzf.vim is Locate. Please see: fzf-vim.txt#L131.

The locate command seems to be a vim only thing, :help locate returns nothing, are you sure this exists with fzf.vim running on neovim?

The command in fzf.vim is Locate. Please see: fzf-vim.txt#L131.

As I wrote this seems to be a vim only command, not a neovim command.

It works for neovim, and I use this one quite often.

It works for neovim. Just note that locate is not a vim command, but a command-line utility just like fzf. You need to install it first before use. On Debian, there is a package called locate -- other distributions might have slightly different names.

I see, sorry for the confusion.
This doesn’t quire a separate command with fzf-lua, you can use the files provider with any command you want, for example: lua require'fzf-lua'.files({ cmd = 'locate foo' }), it should be very easy to create a user command that replicates Locate by combining files cmd with the q-args from the user command.

I didn’t test this so you may want to inspect t.args (might need to use t.args[1], I’m not sure) but it should be as simple as:
vim.api.nvim_create_user_command("Locate",
  function(t)
    require'fzf-lua'.files({ cmd = 'locate '  .. t.args })
  end,
  { nargs = "*" }
)

Closing, lmk if this still doesn't meet your expectionations for a :Locate command and we can adjust it accordingly.


Error trying to use infracost comment with GHE

Hello!
I am getting the following error during the posting of an infracost diff report in the PR's comment session:
Error: The pull request comment was generated successfully but could not be posted:
non-200 OK status code: 404 Not Found body: "{\"message\":\"Not Found\",\"documentation_url\":\"[https://docs.github.com/enterprise/3.3/rest\"}"](https://docs.github.com/enterprise/3.3/rest/%22%7D%22)

See https://infracost.io/docs/troubleshooting/#5-posting-comments for help.

I am using the following syntax (Azure Pipelines + GHE 3.3.7)
      - bash: |
          infracost comment github --path=$(Agent.TempDirectory)/infracost-diff.json \
                                   --github-token=${{ parameters.ghToken }} \
                                   --github-api-url="https://API_SERVER_ADDR/api/v3/" \
                                   --commit=$(Build.SourceVersion) \
                                   --repo=$(Build.Repository.Name) \
                                   --behavior=update
        displayName: Post Infracost comment

I have a PR opened and I am using a token with all scopes selected.
Do I need to add any extra parameters when using this command against a GHE instance?

Hi @danielbmeireles, thanks for reporting. I don't have a GHE instance myself to test on just now, but looking at the code I wonder if it works if you pass --github-api-url="https://API_SERVER_ADDR/api" or --github-api-url="https://API_SERVER_ADDR"?

Presto!
--github-api-url="https://API_SERVER_ADDR/api"

Thank you so much for your help!

Thanks @danielbmeireles. For now I've updated the docs here https://www.infracost.io/docs/features/cli_commands/#github for Azure Repos.


Fix a typo in make_rst.py (Packedfloat64Array -> PackedFloat64Array)

Judging by how it's used, this may add the
The returned array is copied and any changes to it will not update the original property value

note to a few function docs.

Congratulations on your first merged contribution to the Godot repo!

Nice catch! Good thing we only have two PackedFloat64Array properties lol
@Mickeon And how dare you approve such a blunder! If you think about it that has to be worse!

Nice catch! Thank you!


Guard tput calls for errors from CI

Reopening with a better branch name, took out the sandbox config changes.

Pull Request Test Coverage Report for Build 05b8bafb026e80d69916d3df47609510e28b7aa5-PR-827

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.372%

Totals

Change from base Build 4aadabd5258ef1ac900f0387a2bfab54ba4e97f0:
0.0%

Covered Lines:
983

Relevant Lines:
1417

💛  - Coveralls


Vertical Bounce seems true after back to page

When I click my TouchableOpacity Component, I push a viewController. Then the content in the Carousel have some distance away from the Carousel's top when I pop back to.
And I can drag the content down.
It seems that Vertical Bounce changed to true.

@ZackZheng2014 Can you provide an example?

Closing for inactivity.


Add readthedocs folder

Created doc folder with initial structure
Moved installation and usage instructions from README.md to docs
added docs env to tox.ini

Please, be aware of https://github.com/rhos-infra/cibyl/pull/77

Please, be aware of #77

Changed the path


test: issue credits

issue 10 credits to @petar

 Follow up
On Wednesday, 17-Jan-24 13:09:55 UTC by Gov4Git dev
Issued 10 credits to member @petar.


index.py does not print 'saving' message

With the addition of end="\r" to this line, sushi does not print the "saving functions" message. This may be terminal specific, but this does not work for me in my IDE terminal (PyCharm) or terminal window (Tabby, Mac OS).
Output:
(venv) (base) easto@Eastons-MacBook-Air math-c % python app.py
sushi   saved indexed functions
-420735.392546
itt_add() took 0.3960579169999999 seconds
10%

When I remove , end="\r" from index.py:
(venv) (base) easto@Eastons-MacBook-Air math-c % python app.py
sushi   saving indexed functions (lib/main.h)
sushi   saving indexed functions (lib/main.h)
sushi   saved indexed functions
-420735.392546
itt_add() took 0.20237095900000002 seconds
10%                                                                                (venv) (base

It's probably because files are saving so fast that you can't even see it. I created it so old messages about saving are being overwritten by new ones (to reduce spam when sushi needs to save bunch of files). We could get rid of it or disable it only for verbose mode.

No, it does not print the 'saving' message with ,end="\r" added to the end of the rich_print().
I have begun working on a verbose print mode, controlled by the config file. I will submit a PR within the next day or two, it will at least get us to a working state.

Thanks for helping create verbose mode! Can you open the pull request as draft earlier so I can also contribute to it?

@ksawery29  Great idea, I'll do that


support tensorflow lite micro

is it possible to support tensorflow lite micro? in IOT field,esp32,stm32f7/f4 was widely used, but  no suitable binary neworks can run on these platforms with limited resourse。

Hey! That's a great question. We at Plumerai are developing Larq Compute Engine Micro. This leverages TFLite Micro and targets Cortex-M microcontrollers for efficient BNN inference. We are currently planning to keep this closed, but I'd love to chat with you to see if we can help you. You can reach me here<EMAIL_ADDRESS>Best,
Roeland

Thank you for your response,you guys are doing a great  job!  I would like to know your opinion about porting Larq Compute Engine to eps32 cam, it's a interesting and  challenging  job. I will contact you for your help if i want to start this job,if there are some other guys Interested in this question, we can discuss together.

We haven't really looked into porting Larq Compute Engine to ESP32 yet. Cortex-M was the obvious platform to start with, and there's relatively little documentation for ESP32. Anyway, it would be great to do this eventually!

Any updates on this (in particular the ARM Cortex M support)? I note that there was a talk at TinyML about Larq and also microcontrollers?

Any updates on this (in particular the ARM Cortex M support)? I note that there was a talk at TinyML about Larq and also microcontrollers?

It's good to hear that you watched our TinyML Talk. We're keeping our inference engine for Cortex-M (Larq Compute Engine Micro) closed source for now and it is currently only available for customers.

It's good to hear that you watched our TinyML Talk. We're keeping our inference engine for Cortex-M (Larq Compute Engine Micro) closed source for now and it is currently only available for customers.

Ok, thank you for the quick response!

Ok, thank you for the quick response!


Taglist

Hello!
First of all: I love your Theme, it's just great.
One thing I would really love is a Tag-Page like this:
http://sinai.aspirethemes.com/tags/
Since I have more and more Content on my page, this would be great to increase the usability.
Just an input. Thank you for your work.

That's a good idea, I guess in the near future I'll release a new version that includes this feature.

Hi @eddiesigner & friends please see if this could work:

Create a custom-tagsCloud.hbs
Add the following code:

{{!--
This template is used for the posts that are marked as pages.
--}}

{{!-- The tag below means: insert everything in this file
into the {body} of the default.hbs template --}}
{{!< default}}

{{!-- Everything inside the #post tags pulls data from the post --}}
{{#post}}

 {{!-- Special header.hbs partial to generate the  tag --}}
 {{> header background=feature_image}}
 {{!-- Inject styles of the hero image to make it responsive --}}
 {{> hero background=feature_image}}

{{title}}

   {{#if custom_excerpt}}

{{custom_excerpt}}

   {{/if}}

             {{content}}

             {{#get 'tags' limit='1' include='count.posts' order='count.posts asc'}}
             {{#foreach tags}}

             {{/foreach}}
             {{/get}}

             {{#get 'tags' limit='1' include='count.posts' order='count.posts desc'}}
             {{#foreach tags}}

             {{/foreach}}
             {{/get}}

             {{#get 'tags' limit='all' include='count.posts' order='name ASC'}}
             {{#foreach tags}}

               {{ name }}

             {{/foreach}}
             {{/get}}

{{/post}}

{{!-- The #contentFor helper here will send everything inside it up to the matching #block helper found in default.hbs --}}
{{#contentFor "scripts"}}

{{/contentFor}}

Create a Tags page, e.g. https://YOUR_BLOG_URL/tags
Choose tagsCloud template in the page settings
Done!

I got it working on my blog with Liebling theme, very glad! ^^
Credits:

https://forums.engineerworkshop.com/t/how-to-make-a-tags-page/93/2
https://github.com/Torqu3Wr3nch/nightCasper/blob/master/custom-tagsCloud.hbs

@hiensarahly Thanks for sharing! Would you mind to show us a live example just to see how it looks? 😄
Btw, I think you don't need to use {{#get}} twice in order to get the min and max values, I guess you can use it once and get those values by using @first and @last somehow, more info here: https://ghost.org/docs/api/v3/handlebars-themes/helpers/foreach/#first--last

@eddiesigner The example tag page is here if you would like a look: https://zenyrgarden.com/tags/ I truly appreciate it!!
You know I didn't write that code piece :) but obviously the author is very open for improvement. It's great to tweak it to make it cleaner, I like that idea a lot!

No problem ;) Thanks again for sharing!

Hey everyone 👋🏻
I just released a new version that finally includes a Taglist page.
You can see a live demo here: https://liebling.eduardogomez.io/tags/
And you can see how to enable it here: https://github.com/eddiesigner/liebling/wiki/How-to-enable-Taglist-page
Please download the latest version to get it ;)

I will close this issue since this feature is already implemented in the latest version ;)


AttributeError in DeepFace.stream()

module 'deepface.commons.functions' has no attribute 'preprocess_face'

https://github.com/serengil/deepface/issues/662


How tofit_constrained and handle missing values

Apologies if this is not the right place for this question.
I am evaluating dynamax for implementing simple, fast and efficient structural time series models for imputation/univariate forecasting and dynamic regression.  I am impressed with the API, however, there are a couple of blockers for me using the library.  I was wondering you would be able to provide some insight into:

How to constrain parameters before using fit_sgd.  I have looked into the source code and have seen the from_unconstrained function, however, I am unsure how to use this to constrain a single parameter before fitting.
When there are missing values the inference/fitting doesn't work.  Is handling missing data an improvement that will be considered?

The notebook I am using to investigate is here.
I would be happy to contribute if there is anything I can help with.

Hi @adoherty21, thanks for the kind words.
If you are interesting in structural time series models you might be interested in sts-jax a sister library built on top of dynamax for precisely this use case!
In response to your questions:

A bijector which maps to the constrained form of your parameters can be specified in ParameterProperties. For instance in your notebook this is generated by lgssm.initialize and then passed as an argument into fit_sgd.

You can add a tfp.Bijector to a particular parameter in this props object in order to constrain this parameter when using fit_sgd. Let me know if that makes sense.

I am not sure about the roadmap for supporting missing values - there is some discussion of this and related issue in #99.

Thanks a lot @gileshd.
That makes sense for constraining.
Regarding missing values I will start to think about the implementation and make time to contribute.

Closed.


CLI: Support --referenceBuildInfoDirs option

In some project setups, users of the Upgrades Core CLI may not want to change the names of upgradeable contracts for every new version that they deploy.  Instead, they just change the source code and want to perform storage layout comparisons with a previous version of their repository (e.g. from a different branch).
This PR provides a --referenceBuildInfoDirs option that takes a comma-separated list of additional build info directories that can be referenced for storage layout comparisons.  These build info directories can be built from previous versions of the project repository, containing the build artifacts (including storage layouts) of previously deployed versions of contracts.
When using this option, refer to one of these directories using prefix <dirName>: before the contract name or fully qualified name in the --reference option or @custom:oz-upgrades-from annotation, where <dirName> is the directory short name. Each directory short name must be unique.
Example usage:

From an older branch containing previously deployed versions of contracts, compile the contracts with storage layouts and save a copy of the resulting build-info directory (defaults are artifacts/build-info for Hardhat projects or out/build-info for Foundry projects), renaming it to a unique name.  For example, build-info-v1.
From the current branch containing the new versions of contracts that you want to upgrade to, compile the contracts with storage layouts for the new versions.
If you want to compare the storage layout of a contract called MyContract with its previous version, run the CLI command like:

npx @openzeppelin/upgrades-core validate --referenceBuildInfoDirs /old-builds/build-info-v1 --contract MyContract --reference build-info-v1:MyContract

Fixes https://github.com/OpenZeppelin/openzeppelin-upgrades/issues/1034

Here is a local development build that I packaged locally, in case anyone wants to try this out.
openzeppelin-upgrades-core-v1.36.0.tgz
For testing purposes, you can install this in your project dependencies using a file url, e.g.
"@openzeppelin/upgrades-core": "file:///my-dir/openzeppelin-upgrades-core-v1.36.0.tgz",

Then follow the example in the description.


XAML support

Hi @Clancey !
Thanks for this awesome lib!
I've tried to implement it (based on MAUI pr 🙊) to use on Xamarin Forms projects.
But I got conflicts with visual studio HotReload (XAML) and CodeBehind (xaml) classes cant be reloaded because the XAML part is missing.
I don't know if I managed to express myself right here ...
But you cant check this sample if you run the platform project (Android or iOS) and try to do any change on SecondPage.xaml.cs you will get a Exception for Activator.CreateInstance on HotReload.Views.cs Line 118
I'm afraid a little bit that we can get this error from MAUI too, what do you think?
Thanks again!

Yes, the Xaml code behind is not generated yet. We were talking about this during a team meeting the past week. It still needs to be solved :)


⚠ Bofrost.es has degraded performance

In 59b1b0d, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1462 ms

Resolved: Bofrost.es performance has improved in fc1bfc2 after 6 minutes.


Returns Affect Subsequent Rules

This one may be by design, but it makes removal of an item via rules a little clumsy.
Perhaps there's a better example that can be added to the docs for removing an object from an array by rules that check properties of the objects.
Succeeds
import { ruleFactory } from '@elite-libs/rules-machine';

const things = [
  {
    name: "ABC",
    value: "A Thing"
  },
  {
    name: "DEF",
    value: "A Stuff"
  },
  {
    name: "GHI",
    value: "A Thing"
  },
  {
    name: "JKL",
    value: "A Thing"
  }];

const thingRules = ([
  {
    if: {
      and: [
        'CONTAINS(thing.name, ["ABC", "GHI"])',
        'CONTAINS("thing", SPLIT(" ", LOWER(thing.value)))'
      ]
    }, then: "keep = false"
  },
  {
    if: 'LAST(SPLIT("", thing.name)) == "C"', then: 'keep = false'
  },
  {
    return: "keep"
  }
]);

const filterWithRules = ruleFactory(thingRules)

function isRateFiltered(thing: Record<string, string>): boolean {
  const results = filterWithRules({
    thing,
    keep: true
  });
  return results;
}

console.dir(things.filter(isRateFiltered));

Fails
PARSER FAIL: Error: Expected string, found: undefined undefined

import { ruleFactory } from '@elite-libs/rules-machine';

const things = [
  {
    name: "ABC",
    value: "A Thing"
  },
  {
    name: "DEF",
    value: "A Stuff"
  },
  {
    name: "GHI",
    value: "A Thing"
  },
  {
    name: "JKL",
    value: "A Thing"
  }];

const thingRules = ([
  {
    if: {
      and: [
        'CONTAINS(thing.name, ["ABC", "GHI"])',
        'CONTAINS("thing", SPLIT(" ", LOWER(thing.value)))'
      ]
    }, then: "thing = false"
  },
  {
    if: 'LAST(SPLIT("", thing.name)) == "C"', then: 'thing = false'
  },
  {
    return: "thing"
  }
]);

const filterWithRules = ruleFactory(thingRules)

function isRateFiltered(thing: Record<string, string>): boolean {
  const results = filterWithRules({
    thing
  });
  return results;
}

console.dir(things.filter(isRateFiltered));

Can you clarify what you mean by "returns affect subsequent rules"? It's not clear from your examples.
I'm not sure exactly where the error is coming from, but in your second example thing is being treated both as an object and as a boolean.
'CONTAINS(thing.name, ["ABC", "GHI"])'

then: "thing = false"

The following reproduces the error message in v1.4.9:
import { ruleFactory } from "@elite-libs/rules-machine";

const rule = 'SPLIT("", foo.bar)';

ruleFactory(rule)({ foo: true });

Closing because there is no bug here. The error message could certainly be more helpful, but that's a separate issue.

I think there may be a misunderstanding about the context the rules are run. The rules are run in sequence in the same mutable context. Your second example is doing something very much like this:
let thing = { foo: { bar: true } };

if (thing.foo.bar) thing = true;

// TypeError: Cannot read properties of undefined (reading 'bar')
if (thing.foo.bar) thing = "Whoops!";

Ah, @dara-rockwell - There's is a trace option that details what is going on at each step, I can add more examples of that & its output.

@justsml @chhatch Yeah, that's what I figured it was doing.
Is there a more elegant way to decide that we're removing an item besides adding a flag alongside the item?
We'd like to use the rules engine to filter a list of items based on rules, but something like keep = false is a proprietary hack, and we'd like to avoid each usage of rules engine to do filtering re-implement the mechanism for removing an item in some other way.
import { ruleFactory } from '@elite-libs/rules-machine';

const things = [
  {
    name: "ABC",
    value: "A Thing"
  },
  {
    name: "DEF",
    value: "A Stuff"
  },
  {
    name: "GHI",
    value: "A Thing"
  },
  {
    name: "JKL",
    value: "A Thing"
  }];

const thingRules = ([
  {
    if: {
      and: [
        'CONTAINS(thing.name, ["ABC", "GHI"])',
        'CONTAINS("thing", SPLIT(" ", LOWER(thing.value)))'
      ]
    }, then: "keep = false"
  },
  {
    if: 'LAST(SPLIT("", thing.name)) == "C"', then: 'keep = false'
  },
  {
    return: "thing"
  }
]);

const filterWithRules = ruleFactory(thingRules)

function isFiltered(thing: Record<string, string>): boolean {
  const { keep } = filterWithRules({
    thing,
    keep: true
  });
  return keep;
}

console.dir(things.filter(isFiltered));

Ongoing work in #47 - with filter support hopefully soon. @chhatch crushing it over here 💯.

@dara-rockwell With the work in https://github.com/elite-libs/rules-machine/pull/51, this adds support to filtering with multiple conditions.
This changes your example to be the following:
...

const things = [
  {
    name: "ABC",
    value: "A Thing to Remove"
  },
  {
    name: "DEF",
    value: "A Stuff to Keep"
  },
  {
    name: "GHI",
    value: "A Thing to Remove"
  },
  {
    name: "JKL",
    value: "A Thing to Keep"
  }];

const thingRules = ([
  {
    filter: 'list',
    run: { and: [
       'CONTAINS(thing.name, ["ABC", "GHI"])',
       'CONTAINS("thing", SPLIT(" ", LOWER(thing.value)))'
    ] },
    set: 'results',
  },
  { return: 'results' },
]);

...
const filterWithRules = ruleFactory(thingRules)

expect(filterWithRules(things)).toEqual([
  {
    name: "DEF",
    value: "A Stuff to Keep"
  },
 {
    name: "JKL",
    value: "A Thing to Keep"
  }
])


Allow users to clear their favorites

Description:

Users are complaining that they have to manually remove 1 by 1 of their favorite items.

Changes:

Add a --clear flag to +fav to remove everything at once.

Other checks:

[X] I have tested all my changes thoroughly.

Resolved.


[WIP] Add parser for textual ir

Currently, it can roundtrip all the information that was already saved by ir_writer. There's probably still some information that is lost in the process (register state?) but I'm not sure what.
Will fix #117 when done.

There seems to be a weird bug where occasionally an SSA value is just not emitted. It can be seen in when running minidec on ex-bins/key in the function sym._init:
define-fun sym._init(unknown) -> unknown {
    registers: $r15,$r14,$r13,$r12,$rbp,$rbx,$r11,$r10,$r9,$r8,$rax,$rcx,$rdx,$rsi,$rdi,$rip,$cs,$cf,$pf,$af,$zf,$sf,$tf,$if,$df,$of,$rsp,$ss,$fs_base,$gs_base,$ds,$es,$fs,$gs;
    bb_0x400438.0000():
<EMAIL_ADDRESS>%1: $Unknown64(*?) = {rsp} - #x8;
<EMAIL_ADDRESS>%2: $Unknown64(*?) = Load({mem}, #x600ff8);
<EMAIL_ADDRESS>%3: $Unknown64(*?) = %2 & %2;
<EMAIL_ADDRESS>%4: $Unknown64 = %3 - #x0;
<EMAIL_ADDRESS>%5: $Unknown64(*?) = %4 & #xffffffffffffffff;
<EMAIL_ADDRESS>%6: $Unknown64(*?) = #x1 ^ %5;
<EMAIL_ADDRESS>%7: $Unknown1(*?) = Narrow1(%6);
<EMAIL_ADDRESS>%8: $Unknown64(*?) = %4 & #xff;
<EMAIL_ADDRESS>%9: $Unknown64 = #x101010101010101 * %8;
<EMAIL_ADDRESS>%10: $Unknown64(*?) = %9 & #x8040201008040201;
<EMAIL_ADDRESS>%11: $Unknown64 = %10 % #x1ff;
<EMAIL_ADDRESS>%12: $Unknown64(*?) = %11 & #x1;
<EMAIL_ADDRESS>%13: $Unknown1(*?) = Narrow1(%12);
<EMAIL_ADDRESS>%14: $Unknown64(*?) = %4 >> #x3f;
<EMAIL_ADDRESS>%15: $Unknown1(*?) = Narrow1(%14);
        JMP IF %7 0x40044D.0000
    bb_0x400448.0000():
<EMAIL_ADDRESS>CALL #x400490($r15={r15}, $r14={r14}, $r13={r13}, $r12={r12}, $rbp={rbp}, $rbx={rbx}, $r11={r11}, $r10={r10}, $r9={r9}, $r8={r8}, $rax=%2, $rcx={rcx}, $rdx={rdx}, $rsi={rsi}, $rdi={rdi}, $rip={rip}, $cs={cs}, $cf=%16, $pf=%13, $af={af}, $zf=%7, $sf=%15, $tf={tf}, $if={if}, $df={df}, $of=%17, $rsp=%1, $ss={ss}, $fs_base={fs_base}, $gs_base={gs_base}, $ds={ds}, $es={es}, $fs={fs}, $gs={gs});
        ...

In the last call operation, $cf is set to the value %16, but %16 was never defined.
After some debugging, I found that ir_writer never emitted that value because SSAStorage.inorder_walk skipped that node because it was never added into a block.
If we want to round-trip IR through text, we have two options to fix this:

Rewrite ir_writer so that it doesn't depend on values being in blocks.
Say that it is invalid to have values that aren't in a block and check this in verifier::verify before it gets to ir_writer.

Option 1 is certainly more robust, but since values not being in blocks kinda breaks inorder_walk in a sense (and also bfs_walk), it might be better to go with option 2.

@HMPerson1 This looks great! I'll be trying this out today and merging it :+1:
For register state, I'd like an additional line at the beginning of every function to "define" registers, much like declaring variables in source languages. This can be parsed to generate the register state later on.
One thing I'd like to see are a few testcases, nothing too complicated -- just something that takes handwritten IR, generates the ssa structure and then assertions to check the lifting is correct.

I've added some test cases.
For register state, I already changed ir_writer to emit a line to "define" the registers and the register state at the very end of the function is roundtripped, but what I was confused about was the register state for all the other basicblocks. It seems like the register state for every basic block is recorded in SSAStorage (and can be retrieved with registers_in) but only that of the exit node is emited in ir_writer.
Also, could you comment on this problem?

@HMPerson1 The question is, why is %16 not a part of the block? Does it belong to some other block? Did you look into this? If not, that needs some investigation.

It looks like it happens in constant propagation, in emit_ssa. When creating a new node for a OpNarrowed constant, it only copies over the address of the old node and which block the old node was in:
let new_node = if w == 64 {
    const_node
} else {
    let address = self.g.address(*k).unwrap_or_else(..);
    let opcode = MOpcode::OpNarrow(w as u16);
    let new_node = self.g.insert_op(opcode, ndata.vt, None).unwrap_or_else(..);
    self.g.set_address(new_node, address);
    self.g.op_use(new_node, 0, const_node);
    new_node
};
self.g.replace_value(*k, new_node);

If I replace set_address with insert_into_block then it works again.

@HMPerson1 How close is this to be merged? Most of the PR lgtm, if this can be merged, I'll give it another pass and merge it in.

I think it's done now. Would you prefer a merge to fix the conflicts or a rebase?

Either one is fine, rebase could be preferable.

Great! SGTM, meaning this in :)

👍

On 17 Mar 2018, at 00:08, Sushant Dinesh<EMAIL_ADDRESS>wrote:
Great! SGTM, meaning this in :)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Runtime scales poorly with n_pvrows

Oftentimes it is desirable to neglect edge effects to model a row in the interior of a large array.  Unfortunately, it seems that the runtime scaling with n_pvrows is quite bad -- I'm having trouble pinning down the degree of the asymptotic polynomial complexity (if it even is polynomial, might be factorial?), but it's certainly not linear or even quadratic:

The good news is that going all the way to n_pvrows > 30 is overkill for making edge effects negligible -- n_pvrows=11 seems pretty good, at least for this array geometry:

The bad news is that n_pvrows=11 is still an order of magnitude slower than n_pvrows=3, the current default in the pvlib wrapper function.  The code for the above plots is available here: https://gist.github.com/kanderso-nrel/e88e3f7389b9d144a546dbe5651dfe1e
I've not looked into how we might go about improving this situation.  If I had to guess, it would require a pretty substantial refactor, if it's possible at all.  It would be a pleasant surprise if that guess is incorrect :)  But even if it can't be fixed, I think it's useful to have an issue documenting this effect.

Thanks a lot for this study @kanderso-nrel , that's really awesome data that I never collected before 🙏
Back in the days I focused my time trying to increase the calculation speed with respect to the # of time data points using vectorization. But that's only 1 dimension of the 3 dimensional view factor matrix that is inverted in the PV engine.
As the number of PV rows increases, I was expecting at least O(n^2) complexity just for calculating the vf matrix since the other 2 dimensions would increase like O(n); in the end it looks like the full calculation is a lot worse than that 😅
I would be curious to see where it spends the most time (if I have time maybe I'll try to run some quick profiling with pyinstrument or something like that); I would expect in the inversion process but I could be wrong. If that's the case, maybe doing the inversion using a GPU could help..? (I think it would be quite fast to transform the numpy array into a Pytorch tensor)
The only other way to speed things up that I can think of right now is to make some approximations: about the vf matrix, or maybe about irradiance, reflections, etc. Out of curiosity, did you try running the "fast" mode of the engine? I'm wondering how far off it is compared to the full mode (IIRC it only calculates back side irradiance for a single PV row in the whole array).

Thanks for the pointer to pyinstrument -- that's a very nice little tool that I will put to use outside of pvfactors as well!
I continue to find it difficult to get consistent timings, but indeed the inversion of a_mat is a large fraction (maybe 50-75%?) of total runtime for a simulation like the one in the original post.
I'm far from a computational linear algebra expert but it seems like it should be possible to somehow take advantage of a_mat's sparseness (often >99% zeros).  New array-like functionality was added to scipy.sparse in v1.8 (docs), but it looks like it's limited to 2D for now.  I played around a bit with the older matrix-based sparse interface (also 2D, so I had to loop and invert each MxM matrix individually) but didn't find anything faster than np.linalg.inv.  I did not try it out, but the pydata sparse package supports N-D sparse arrays, but doesn't seem to have an inverse function?  Although perhaps it can be used with scipy.sparse somehow: https://sparse.pydata.org/en/stable/roadmap.html#partial-scipy-integration
Much of the rest of the runtime in full mode is spent building the two vf matrices.
The fast mode is substantially faster: tenths of a second for fast vs tens of seconds for full.  I leave it to the reader to decide if the difference in output is significant :)

take advantage of a_mat's sparseness (often >99% zeros)

that's a good idea! I'll try to investigate on my end as well.

I leave it to the reader to decide if the difference in output is significant

Thanks for trying it : ) I was expecting worse results!! But yeah, I guess it really depends on the application (so the user) in the end.

Hmm, another idea is to never explicitly invert the matrix at all?  For large simulations I see 10-15% faster calls to engine.run_full_mode by replacing this:
https://github.com/SunPower/pvfactors/blob/e0ea9d5a03d130d25919b39d590645ea2997ad14/pvfactors/engine.py#L223-L226
with q0 = np.linalg.solve(a_mat, irradiance_mat.T).T.  I've never sat down to learn the einstein summation notation so I'm not positive this is a perfect replacement, but it seems to yield identical results... here are some timing plots if you're interested: https://gist.github.com/kanderso-nrel/6bf69c35314446b34ca2bfbd3bd41964

wow that's awesome @kanderso-nrel ! It's very true that inv_a_mat is not used anywhere else so if we can collapse both steps into 1 and at the same time obtain a consistent speed improvement let's go for it!! 🎉 👏  Honestly I don't really remember how the einstein sum works, it was just a trick to get the matrices multiplied in the way I needed 😅
IMO if the tests all pass after making the change that's good enough for me 👍

Some more progress, this time by combining the two previous ideas:  use a sparse linear solver instead of explicitly inverting the matrix.  I want to do some more testing to make sure it behaves nicely outside of my little benchmarks, but here's a runtime comparison in the meantime -- calls to the pvlib wrapper function are 30% to 60% faster in this case:

After continued efforts to speed up pvfactors (as always, collaborating with @spaneja), I think the timing interpretation I posted above is not wholly correct.  tl;dr:  I think just using a dense solver like np.linalg.solve is probably better than the irksome sparse solver from #140, but I want to do some more digging.
The runtime of np.linalg.inv and np.linalg.solve, at least on this computer (Win10, i7-8850H, numpy 1.22.3), varies significantly depending on whether numpy is installed from PyPI or from conda-forge.  I can only assume the difference is caused by which BLAS library is being linked against; for PyPI it seems to be OpenBLAS while I think conda-forge uses Intel's MKL, with the PyPI/OpenBLAS version being the slower of the two.  Debugging this is a little out of my depth, but I did discover that limiting the BLAS's own threading behavior makes a big difference:

Click to expand!
In [1]: from threadpoolctl import ThreadpoolController
   ...: import numpy as np  # v1.22.3, installed from PyPI
   ...: import time
   ...:
   ...: controller = ThreadpoolController()
   ...: a = np.random.random((500, 321, 321))
   ...: b = np.random.random((500, 321))

In [2]: st = time.perf_counter()
   ...: _ = np.linalg.solve(a, b)
   ...: ed = time.perf_counter()
   ...: print(ed - st)
5.766736300000002

In [3]: st = time.perf_counter()
   ...: with controller.limit(limits=1, user_api='blas'):
   ...:     _ = np.linalg.solve(a, b)
   ...: ed = time.perf_counter()
   ...: print(ed - st)
0.5034510000000019

In any case, I suspect the dramatic speedup I saw earlier in this thread and in #140 was more due to using the PyPI/OpenBlas numpy and the original np.linalg call being slow rather than the new scipy.sparse calls being fast.  Below is a runtime comparison across a few code variants as well as numpy from both PyPI and conda-forge.  The code variants are as follows:

1.5.2: pvfactors v1.5.2 as installed from PyPI (the current np.linalg.inv + np.einsum approach)
#140: the approach used in #140 (sparse linear solve using scipy.sparse.linalg)
np.linalg.solve: a new approach just using np.linalg.solve to replace the current approach
np.linalg.solve+limit: same as np.linalg.solve, but using the threadpoolctl trick to address the strange OpenBLAS slowdown.

Here are the timings of calls to pvlib.bifacial.pvfactors.pvfactors_timeseries using each of the above pvfactors variants, where all times have been normalized to the 1.5.2 variant using numpy from conda-forge (so values below 1.0 represent improvements):

Here are my takeaways:

#140 is an improvement if you're using an OpenBLAS numpy, but might be worse if you're using MKL.
Using a dense linear solver (np.linalg.solve) combined with the threadpoolctl trick gives the best and most consistent speedup across both BLAS packages.
The OpenBLAS issue is concerning and I plan to follow up with the numpy folks about it.  It's always possible I'm doing something wrong, but I still see a (smaller, but still substantial) slowdown w/ different processor and OS...

thanks a lot for this study @kanderso-nrel , I remember testing your PR locally back then before approving, and for me there was no slowdown but no crazy improvement either, so I assumed it was at least as good as the current implementation.
I think your study can be quite useful for people potentially using pvfactors in production etc, and I would be inclined to adding it in the package documentation.
But I've been MIA for a while so I'm not even sure there's anyone at SunPower that has ownership of pvfactors... most performance/software people I knew there seem to have left..? @campanelli-sunpower @pfranz-spwr


Update coffeescript.tmLanguage.json, fixing highlighting for embedded js

In coffee-script, `` means not normal strings but embedded js, but it's not correctly highlighted in vscode, like this:

After this fix, it should be like this:

Note that in embedded js the symbols like ()[]{} is green rather than white. This is intentional in order to indicate this is embedded js, not coffee-script.

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

Thanks for looking into this. However our coffescript grammar comes from this project: https://github.com/atom/language-coffee-script Can you try submitting a PR against that repo first? This allows your fix will be picked up by any editors that consume that shared coffescript grammar
Once the change is in the coffeescript grammar repo, back in VSCode, simply run npm run update-grammar in the coffescript extension to pick up latest version of the grammar.

sure

https://github.com/atom/language-coffee-script/pull/121

Closing this request. Waiting for atom/language-coffee-script#121 to be accepted.


RETRYING the same batch again

connector: lightweight
version: 2.1.0
I've noticed a lot of messages RETRYING the same batch again, but no errors around the message signalling something was wrong with the insert.
any info why could it happen and what to do with it?

@vpol the error message should be in the log file of course but you can probably check at the CH level. system.query_log where exception != ''
You should see it if it is a server side issue.

Unfortunately this is how my clickhouse-sink log looks like.

I'm using clickhouse-cloud, so had to open ticket in order to get something from the logs.
Please let me know if there is any debug I can enable to get better error from the sink side. Thank you.

Hi. I've tired to investigate it, but it looks like there are no errors in the log and no errors on the clickhouse side.
I'm not 100% sure if it's related, but I've noticed that almost always select * from replica_source_info returns nothing.

@subkanthi could it be related this breaking change in 2.10 ?
Breaking Changes.
The configuration clickhouse.server.database is now deprecated with the multiple database support.
By default the source MySQL/postgres database name will be used as the ClickHouse database name.

I've tried LOGGING_LEVEL=debug but it does not increase verbosity.
name: "postgres-sync-clickhouse"

# database.hostname: IP address or hostname of the PostgreSQL database server.
database.hostname: "<ip>"

# database.port: Integer port number of the PostgreSQL database server listening for client connections.
database.port: "<port>"

# database.user: Name of the PostgreSQL database user to be used when connecting to the database.
database.user: "<user>"

# database.password: Password of the PostgreSQL database user to be used when connecting to the database.
database.password: "<password>"

# database.server.name: The name of the PostgreSQL database from which events are to be captured when not using snapshot mode.
database.server.name: "api"

# schema.include.list: An optional list of regular expressions that match schema names to be monitored;
schema.include.list: "public"

# plugin.name:  The name of the PostgreSQL logical decoding plug-in installed on the PostgreSQL server. Supported values are decoderbufs, and pgoutput.
plugin.name: "pgoutput"

slot.name: "clickhouse_sync"

# table.include.list: An optional list of regular expressions that match fully-qualified table identifiers for tables to be monitored;
table.include.list: "<tables-list>"

# clickhouse.server.url: Specify only the hostname of the Clickhouse Server.
clickhouse.server.url: "https://<something>.us-central1.gcp.clickhouse.cloud"

# clickhouse.server.user: Clickhouse Server User
clickhouse.server.user: "<user>"

# clickhouse.server.password: Clickhouse Server Password
clickhouse.server.password: "<password>"

# clickhouse.server.port: Clickhouse Server Port
clickhouse.server.port: "<port>"

# clickhouse.server.database: Clickhouse Server Database
clickhouse.server.database: "<database>"

# database.allowPublicKeyRetrieval: "true" https://rmoff.net/2019/10/23/debezium-mysql-v8-public-key-retrieval-is-not-allowed/
database.allowPublicKeyRetrieval: "true"

# snapshot.mode: Debezium can use different modes when it runs a snapshot. The snapshot mode is determined by the snapshot.mode configuration property.
# snapshot.mode: "no_data"
snapshot.mode: "never"

# offset.flush.interval.ms: The number of milliseconds to wait before flushing recent offsets to Kafka. This ensures that offsets are committed within the specified time interval.
offset.flush.interval.ms: 60000
offset.flush.timeout.ms: 30000
buffer.max.records: 1000000
buffer.flush.timeout.ms: 120000
max.batch.size: 20000
max.queue.size: 100000

#snapshot.fetch.size: 2048
metrics.enable: "false"
cli.port: 7052
thread.pool.size: 10

# connector.class: The Java class for the connector. This must be set to io.debezium.connector.postgresql.PostgresConnector.
connector.class: "io.debezium.connector.postgresql.PostgresConnector"

# offset.storage: The Java class that implements the offset storage strategy. This must be set to io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore.
offset.storage: "io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore"

# offset.storage.jdbc.offset.table.name: The name of the database table where connector offsets are to be stored.
offset.storage.jdbc.offset.table.name: "api.replica_source_info"

# offset.storage.jdbc.url: The JDBC URL for the database where connector offsets are to be stored.
offset.storage.jdbc.url: "jdbc:clickhouse:https://<something>.us-central1.gcp.clickhouse.cloud:<port>/api"

# offset.storage.jdbc.user: The name of the database user to be used when connecting to the database where connector offsets are to be stored.
offset.storage.jdbc.user: "<user>"

# offset.storage.jdbc.password: The password of the database user to be used when connecting to the database where connector offsets are to be stored.
offset.storage.jdbc.password: "<password>"

# offset.storage.jdbc.offset.table.ddl: The DDL statement used to create the database table where connector offsets are to be stored.
offset.storage.jdbc.offset.table.ddl: "CREATE TABLE if not exists %s
(
    id                String,
    offset_key        String,
    offset_val        String,
    record_insert_ts  DateTime,
    record_insert_seq UInt64,
    _version          UInt64 materialized toUnixTimestamp64Nano(now64(9))
)
ENGINE = ReplacingMergeTree(_version)
ORDER BY id
SETTINGS index_granularity = 8198"
offset.storage.jdbc.offset.table.delete: "delete from %s where 1=1"
schema.history.internal: "io.debezium.storage.jdbc.history.JdbcSchemaHistory"
schema.history.internal.jdbc.url: "jdbc:clickhouse:https://<something>.us-central1.gcp.clickhouse.cloud:<port>/api"
schema.history.internal.jdbc.user: "<user>"
schema.history.internal.jdbc.password: "<password>"

schema.history.internal.schema.history.table.name: "api.replicate_schema_history"

# schema.history.internal.schema.history.table.name: The name of the database table where connector schema history is to be stored.
schema.history.internal.jdbc.schema.history.table.name: "api.replicate_schema_history"
schema.history.internal.jdbc.schema.history.table.ddl: "CREATE TABLE if not exists %s
(
    id String NOT NULL,
    history_data String,
    history_data_seq INTEGER,
    record_insert_ts TIMESTAMP NOT NULL,
    record_insert_seq INTEGER NOT NULL
)
ENGINE=ReplacingMergeTree(record_insert_seq)
order by id
SETTINGS index_granularity = 8198"

# enable.snapshot.ddl: If set to true, the connector wil parse the DDL statements as part of initial load.
enable.snapshot.ddl: "true"

# auto.create.tables: If set to true, the connector will create the database tables for the destination tables if they do not already exist.
auto.create.tables: "true"
auto.create.tables.replicated: "true"

# database.dbname: The name of the PostgreSQL database from which events are to be captured when not using snapshot mode.
database.dbname: "api"

# clickhouse.datetime.timezone: This timezone will override the default timezone of ClickHouse server. Timezone columns will be set to this timezone.
#clickhouse.datetime.timezone: "UTC"

# skip_replica_start: If set to true, the connector will skip replication on startup. sink-connector-client start_replica will start replication.
skip_replica_start: "false"

# binary.handling.mode: The mode for handling binary values. Possible values are bytes, base64, and decode. The default is bytes.
#binary.handling.mode: "base64"

# ignore_delete: If set to true, the connector will ignore delete events. The default is false.
ignore_delete: "false"

#disable.ddl: If set to true, the connector will ignore DDL events. The default is false.
disable.ddl: "false"

#disable.drop.truncate: If set to true, the connector will ignore drop and truncate events. The default is false.
#disable.drop.truncate: "false"

A little update: I don't see this issue it in my test environment where I use non-clustered clickhouse.

I've done a bit more debugging and identified that if returns error because
static boolean checkIfThereAreInflightRequests

returns true for the batch.

Hi @subkanthi, thanks for checking it out. Unfortunately I can't see any errors like the one you mentioned in the query log. To be honest I don't see ANY errors in the logs for the user/db I'm using for sync.

Hi
we have the same problem with normal (not lightweight) connector version 2.2.1, no errors in logs and clickhouse
[INFO ] 2024-08-09 12:43:40.783 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *************** EXECUTED BATCH Successfully Records: 239************** task(0) Thread ID: Sink Connector thread-pool-1 Result: [I@35563dc5 Database: db Table: table2
[INFO ] 2024-08-09 12:43:40.783 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *** INSERT QUERY for Database(sync2) ***: insert into [table1, redacted]
[INFO ] 2024-08-09 12:43:40.794 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *************** EXECUTED BATCH Successfully Records: 4************** task(0) Thread ID: Sink Connector thread-pool-1 Result: [I@7b4d921a Database: db Table: table1
[DEBUG] 2024-08-09 12:43:40.794 [Task: Sink Connector thread-pool-1askId] ClickHouseBatchRunnable - ***** RETRYING the same batch again
[INFO ] 2024-08-09 12:43:40.794 [Task: Sink Connector thread-pool-1askId] ClickHouseBatchRunnable - ****** Thread: Sink Connector thread-pool-1 Batch Size: 246 ******
[INFO ] 2024-08-09 12:43:40.794 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *** INSERT QUERY for Database(sync2) ***: insert into `debezium_signal`(`id`,`type`,`data`,`_version`,`is_deleted`) select `id`,`type`,`data`,`_version`,`is_deleted` from input('`id` String,`type` Nullable(String),`data` Nullable(String),`_version` UInt64,`is_deleted` UInt8')
[INFO ] 2024-08-09 12:43:40.798 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *************** EXECUTED BATCH Successfully Records: 3************** task(0) Thread ID: Sink Connector thread-pool-1 Result: [I@2ca472e9 Database: db Table: debezium_signal
[INFO ] 2024-08-09 12:43:40.816 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *** INSERT QUERY for Database(sync2) ***: insert into [table2, redacted]
[INFO ] 2024-08-09 12:43:40.834 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *************** EXECUTED BATCH Successfully Records: 133************** task(3) Thread ID: Sink Connector thread-pool-1 Result: [I@32aedbe4 Database: db Table: table2
[DEBUG] 2024-08-09 12:43:40.834 [Task: Sink Connector thread-pool-1askId] ClickHouseBatchRunnable - ***** RETRYING the same batch again
[INFO ] 2024-08-09 12:43:40.834 [Task: Sink Connector thread-pool-1askId] ClickHouseBatchRunnable - ****** Thread: Sink Connector thread-pool-1 Batch Size: 133 ******
[INFO ] 2024-08-09 12:43:40.839 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *************** EXECUTED BATCH Successfully Records: 117************** task(1) Thread ID: Sink Connector thread-pool-1 Result: [I@4d85b7ae Database: db Table: table2
[INFO ] 2024-08-09 12:43:40.839 [Task: Sink Connector thread-pool-1askId] PreparedStatementExecutor - *** INSERT QUERY for Database(sync2) ***: insert into [table1, redacted]

I'm testing it on small tables (1k rows) and sink just trying to insert rows in clickhouse indefinitely.

Hi!
Same issue with lightweight connector  2.1.0 and 2.2.1
I try sync from postgres15.6 to CH 24.8
I create empty ReplicatedReplacingMergeTree table, then add it to table.include.list
send execute-snapshot record to signal.data.collection table and  see [DEBUG] ***** RETRYING the same batch again
In clickhouse logs appears lot of  info-level lines about signal table and my replicated table :
<Information> dbname.signal_table  (Replicated OutputStream): Block with ID all_15085468674909155566_13758033302413788604 already exists locally as part all_56_56_0; ignoring it.
<Information> dbname.my_table (ba7899aa-66ba-45f0-ad60-a523fde759db) (Replicated OutputStream): Block with ID all_16026509731054093547_2011380493872661054 already exists locally as part all_6_6_0; ignoring it.


Set Global Reset Time For Scrapper

Per request, set the reset time for everyone to Midnight Pacific so that folks can more easily know when they can do another scrap without having to run the command just to be denied.

Double-checked and verified this is working properly now.


Hide method completions with incompatible this parameters

Suggestion
🔍 Search Terms
intellisense, completions, this parameter, this type, method receiver
✅ Viability Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion
When showing completion results for methods after a receiver with a known type, intellisense should hide methods that have an explicit this parameter if the receiver definitively isn't assignable to it. Trying to call one of these methods is always a type error, so displaying them just clutters things up and hampers discoverability by leading the user down dead ends.
📃 Motivating Example
Here's a simple wrapper class illustrating the problem. The map method is intended to only be applicable when the wrapped value is an array. This works fine in the type system (it's an error to try to use b.map when b is a Wrap<number>), but showing map as a possible completion after b. is somewhat misleading.
Playground link
class Wrap<T> {
  constructor(readonly value: T) {}

  map<A, B>(this: Wrap<A[]>, fn: (item: A, i: number) => B): Wrap<B[]> {
    return new Wrap(this.value.map(fn))
  }
}

// Wraps an array, so you can map it just fine.
const a = new Wrap(['hi'])
a.map((v) => `${v}, world`)

// Wraps a number.
// `b.map` is always a type error, but it shows up in completions anyway.
const b = new Wrap(0)
b.map((v) => 1)

💻 Use Cases
I ran into a situation very similar to the Wrap example above while writing a reactive programming library. Certain operations on reactive collections ("map over a reactive array", "get the value at key K from a reactive map", etc.) admit a more efficient implementation than just "rerun the operation when the collection changes", and it's convenient to expose these as methods on a Reactive<Map<K, V>> that wouldn't be present on a Reactive<number>, rather than having a separate  ReactiveMap<K, V> type. Since the type checking works fine, I'm likely to go with this approach anyway -- but it'd be nice if the completions were better.
I suspect this sort of thing is relevant to other frameworks with strongly-typed wrappers: reactive frameworks, ORMs, handles to typed objects provided by some external system, EDSLs that want to use the host type system to check the embedded language, etc.

We've looked into similar requests before and this unfortunately doesn't pencil out from a performance vs usability perspective. The cost to compute if the current this matches a given target this is effectively unbounded, and we'd have to do that work on every completion invocation up front since there's no mechanism to do a "deferred removal" of an item (to say nothing of how janky that would look in realtime). In some frameworks this was 30% or worse hit on every . keystroke. Even determining whether or not a method has a this parameter isn't free since base class type resolution might do extra work that wasn't going to happen already. Most methods with this parameters are this: this (or even this: Base where we'd have to do the work but the answer is effectively predestined), so even a first-pass filter to determine if there's any checking to do at all is still going to burn a lot of battery for not much UX enhancement.
If this-filtering were common it might maybe be worth considering, but in practice this is very often going to remove no methods at all.

Oh, yeah, that makes total sense -- I was thinking about it from the side of "the type system already knows all this" but of course this is on the hot path for a system that doesn't know it immediately and cares about the cost of having to figure it out.
For anyone who stumbles across this in the future with the same request, here's one alternative if you really care about the completions here and are willing to sacrifice other aspects of type ergonomics. You can make two separate interfaces, with a conditional type that chooses the right one based on its parameter. Then you continue writing Wrap<T> in declarations, return types, etc. and it will resolve to an interface that has only the applicable methods for the situation -- even if concretely it's always going to be the same class.
type Wrap<T> = T extends unknown[] ? WrappedArray<T> : WrappedValue<T>

interface WrappedValue<T> {
  value: T
}

interface WrappedArray<T extends unknown[]> extends WrappedValue<T> {
  map<A, B>(this: Wrap<A[]>, fn: (item: A, i: number) => B): Wrap<B[]>
}

// Can't write out the implements clause for it, but also implements WrappedArray<T>
// whenever T extends unknown[].
class Wrapper<T> implements WrappedValue<T> {
  constructor(readonly value: T) {}

  map<A, B>(this: Wrap<A[]>, fn: (item: A, i: number) => B): Wrap<B[]> {
    return new Wrapper(this.value.map(fn))
  }
}

function wrap<T>(value: T): Wrap<T> {
  // Not sure how to get this to be accepted without a cast; it's true for any individual type
  // that's either array-ish or not, but not for a generic T that might or might not be array-ish.
  return new Wrapper(value) as Wrap<T>
}

// Wraps an array, inferred to be a WrappedArray<string[]>, so you can map it just fine.
const a = wrap(['hi'])
a.map((v) => `${v}, world`)

// Wraps a number, inferred to be a WrappedValue<number>, has no map method on the interface.
const b = wrap(0)

As you can see, there are some infelicities here from breaking parametricity and using a conditional type that can't always be resolved to a specific branch. Not worth it for me, but it might be workable for someone else's use case.


Fix cleanup ruby logic

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[ ] Have you successfully run brew tests with your changes locally?

Fixes Homebrew/brew#7743
Align the logic in cleanup_portable_ruby with that in utils/ruby.sh:

check all available rubies*
Use the exact comparison below to check if a specific Ruby satisfies our requirements:

Gem::Version.new(system_ruby_version).to_s.split(".").first(2) ==
Gem::Version.new(required_ruby_version).to_s.split(".").first(2)

*: this part is OS-specific in utils/ruby.sh and I think it would be best to do the same here but I wasn't sure if we want to

do it right here in cleanup.rb,
extend cleanup.rb for Linux, or
implement a shell script that would be called by utils/ruby.sh and here.

Thanks, @MikeMcQuaid. Why did you add .3 to HOMEBREW_REQUIRED_RUBY_VERSION?

@maxim-belkin That's the required system version on macOS (that we're checking in brew doctor) and it's easier to discard the parts we don't need for supported versions.

Let's see how this goes, thanks for your work on this @maxim-belkin!


feat(confg): added chain config template and fgw example

force merged this since it doesnt affect the core


The text for youtube activities should say answer "the" questions

The text should say "Watch Video and answer the questions." rather than
Watch Video and answer for questions.

Text is already "Watch Video and answer the questions"


Determine the impact of blob size on Hadoop jobs

As part of our cold storage scenario, we are creating blobs that we expect to be processed by Hadoop at some point. Some implementations (such as our custom event processor) allow us some control over the size of the blobs. What are the considerations?
For background, here's a post on merging small files.
Also, we ran some informal tests a few months ago with a 20GB dataset.
Broken into 1MB files, our query took ~400s.
Broken into 2GB files, our query took ~200s.

In running the long running jobs - I had over 20GB in the blob storage and the hive job timed out at 1 hour.

Our findings on this topic are in agreement whit this post http://blogs.msdn.com/b/mostlytrue/archive/2014/04/10/merging-small-files-on-hdinsight.aspx
Note that if you are using ColdSotrageProcessor you can choose the size of the blobs from the configuration file.


[BUG] LiteDB 5.0.10 - Maximum number of transactions reached

.NET Standard 2.0
LiteDB version 5.0.10
//database is a singleton instance
protected ILiteCollection<T> GetCollectionWithIndex()
{
   var collection = database.GetCollection<T>(collectionName);
   collection.EnsureIndex(x => x.InternalId);
   collection.EnsureIndex(x => x.Name);
   return collection;
}

public async Task<T> FindOneAsync(BsonExpression expression)
{
   var collection = database.GetCollectionWithIndex();
   return await resilientPolicy.ExecuteAsync(async () =>
   {
        var result = collection.FindOne(expression);
         await Task.CompletedTask;
        return result;
   });
}

TransactionMonitor.GetTransaction (System.Boolean create, System.Boolean queryOnly, System.Boolean& isNew)
Maximum number of transactions reached

@byteAbit0101 LiteDB has a limit of 100 open transactions at a time. If youy program happens to create a lot of these FindOneAsync tasks at the same time, it may reach the limit.

Hello,
Me again. I analyzed the problem, it s happeingn after I did a rebuild of the database (reduced of 8Go).
Then each time I'm trying to insert in the filestorage I have this error (I'm not closing the database after the rebuild) :
-> LiteDB ENSURE: page type must be collection page
And then after 100 occurences of this error I get :
-> Maximum number of transactions reached
It's like the "LiteDB ENSURE: page type must be collection page" is not closing the collection
I think my problem is linked with #1958

Hello,
Please, did anyone find a solution to this problem ? I have this issue in production.

Hi,
I am also seeing this issue.
In LiteDB 4.x we had our own locking around the LiteDb database, and after upgrading to 5.0.16 we started seeing the application hanging trying to get the lock to the database.
After reading that LiteDb was now threadsafe, we removed the locking and are relying solely on the internal locking done by LiteDb.
This was working fine, occaisonally we would see a lock timeout, like this:
Caught exception: Collection 'redacted' lock timeout when entering in write mode after 00:01:00
at LiteDB.Engine.LockService.EnterLock(String collectionName)
at LiteDB.Engine.Snapshot..ctor(LockMode mode, String collectionName, HeaderPage header, UInt32 transactionID, TransactionPages transPages, LockService locker, WalIndexService walIndex, DiskReader reader, Boolean addIfNotExists)
at LiteDB.Engine.TransactionService.g__create|43_0(<>c__DisplayClass43_0& )
at LiteDB.Engine.TransactionService.CreateSnapshot(LockMode mode, String collection, Boolean addIfNotExists)
at LiteDB.Engine.LiteEngine.<>c__DisplayClass27_0.b__0(TransactionService transaction)
at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn)    at LiteDB.Engine.LiteEngine.Update(String collection, IEnumerable1 docs)
at LiteDB.LiteCollection`1.Update(T entity)
The application would recover after this and continue working just fine, however, it seems the lock timeouts became more prevalent, and then we started seeing the max transactions reached exceptions:
Caught exception: Maximum number of transactions reached
at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
at LiteDB.Engine.QueryExecutor.ExecuteQuery()
at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
at LiteDB.LiteQueryable1.<ToDocuments>d__26.MoveNext()    at System.Linq.Enumerable.Single[TSource](IEnumerable1 source)
at LiteDB.LiteQueryable1.Count()    at LiteDB.LiteCollection1.Count()
After this, the application does not recover and database access is not possible at all.
A restart of the application is the only way to recover.
I suspect each time we get the lock timeout, a LiteDb transaction is hung and never released, once we get this error 100 times, we get the max transactions reached error.
@lbnascimento, @mbdavid, any ideas on this problem?

I run into this exception also on LiteDB 5.0.18
LiteDB.LiteException: Maximum number of transactions reached
    at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
    at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
    at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
    at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
    at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)

+1
Same here.  As per #2418 , we have installed the latest version 5.0.18 and are getting this exception, after only a few accesses to the database:
Caught exception: Maximum number of transactions reached
tack trace following:
at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
at LiteDB.Engine.QueryExecutor.ExecuteQuery()
at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
at LiteDB.LiteQueryable1.<ToDocuments>d__26.MoveNext()    at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)

I have the same issue on v5.0.19, @PeterHagen did reverting to v5.0.17 solve the issue for now?

I have the same issue on v5.0.19, @PeterHagen did reverting to v5.0.17 solve the issue for now?

I had this issue on v5.0.18 and reverting to v5.0.17 fixed it.

Note that this issue is fixed by #2435 , but the fix is not included in any release yet.
We test this fix by rolling it into 5.0.19 and found that it resolves the "Maximum number of transactions reached" error, and also the DiskWriterQueue locking issue #2307  which has been preventing us from upgrading to v5.
So please push for a new release (5.0.20) which includes #2435 :)

I have the same issue. Please release the fix!

Downgrade to 5.0.17 helped in my case.

Had the same issue here. After deleting my log file, it no longer had the issue. My previous run was attached to a debugger which I killed the process which left a 4096kb log file. Deleting it resolved the issue. Bizzare

We had the same problem after upgrading from 5.0.16 to 5.0.19.

Same issue with version 5.0.19
Fixed by downgrade to version 5.0.17

Same here, windows 11 Pro,  .NET8.0. I was performing a lot of GetCollecton followed by FindAll(). Downgraded to 5.0.17 and the error did not occur. Did not try other versions.

With 5.0.17, I get "LiteDB ENSURE: page type must be collection page" from our test suite. With 5.0.19 Maximum number of transactions reached :( We have been stuck on v4 for awhile.

Same issue here: "LiteDB ENSURE..." with 5.0.17 and "Maximum number of transactions reached.." with 5.0.19.
We are definitely stuck since v4 is also marked as vulnerable.
Any expected date for new release?

Jellyfin (10.9 released this week) has the same problem too:
mai 13 13:41:38 awak jellyfin[2428612]: LiteDB.LiteException: Maximum number of transactions reached
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1] source, Boolean& found)
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.Data.DbRepo.<>c__DisplayClass10_0.<SetUserInfoSync>b__0(UserItemDataDto dto)
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Collections.Generic.List`1.ForEach(Action`1 action)
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.Data.DbRepo.SetUserInfoSync(List`1 dtos, List`1 itemRefs, String userId)
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.SaveUserChanges(List`1 dtos, List`1 itemRefs, String userName, String userId)
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.SendNotifications(IEnumerable`1 changes, List`1 itemRefs, CancellationToken cancellationToken)
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.UpdateTimerCallback(Object state)

This bug also exists in LiteDB 5.0.19.

Thank you for reporting this issue.
This issue has been resolved in the latest version of LiteDB.
Please update to the latest version of LiteDB to resolve this issue.
If the issue persists, please let us know.


rule or trigger cann't used on merge table & continue view?

HI,
I like to use trigger or rule to partition the merge table or continue view.
This is the thinking:
I should store the origin data and store statistic data at same time ,so i create a continue view for origin and some rela statistic continue view for purpose.
CREATE CONTINUOUS VIEW imps2_origin AS SELECT id::int,arrival_timestamp FROM imps_stream;
CREATE CONTINUOUS VIEW imps2_c1 AS SELECT count(*),date(arrival_timestamp) FROM imps_stream group by 2;
......
pipeline=# insert into imps_stream (id,x) values (1),(2);
INSERT 0 2
pipeline=# select * from imps2_origin;
id |       arrival_timestamp
----+-------------------------------
2 | 2015-11-04 14:10:25.085172+08
1 | 2015-11-04 14:10:25.085145+08
(2 rows)
pipeline=# select * from imps2_c1;
count |    date
-------+------------
2 | 2015-11-04
(1 row)
and then:
pipeline=# create table imps3(like imps2_origin);  -- this is can a partition table , there i ignore the so many code.
CREATE TABLE
pipeline=# create or replace function tg1 () returns trigger as $$
declare
begin
insert into imps3(id,arrival_timestamp) values (new.id,new.arrival_timestamp);
return null;
end;
$$ language plpgsql;
pipeline=# create trigger tg1 before insert on imps2_origin_mrel0 for each row execute procedure tg1();
CREATE TRIGGER
pipeline=# insert into imps_stream (id,x) values (1),(2);
INSERT 0 2
the trigger not trigged .
pipeline=# select * from imps2_origin_mrel0 ;
id |       arrival_timestamp
----+-------------------------------
2 | 2015-11-04 14:10:25.085172+08
1 | 2015-11-04 14:10:25.085145+08
1 | 2015-11-04 14:13:31.090223+08
2 | 2015-11-04 14:13:31.090347+08
(4 rows)
pipeline=# select * from imps3
pipeline-# ;
id | arrival_timestamp
----+-------------------
(0 rows)
So, what can i do ,if i should store origin data into my partition table?
thanks

@digoal currently it is not possible to use triggers on streams, continuous views or materialization tables. What it seems like you're trying to do here is store all events from a stream in a continuous view, and then replicate them out to a partitioned table, is that correct?
If so, you can probably write tuples directly to the partitioned table and use a trigger on it that then writes to the stream. Would that work?


Duplicated btn-primary active state rules in SCSS

Describe the issue
The _dark-theme.scss has a duplicated ruleset. We should remove the first set which is incorrect.
.btn-primary:not(:disabled):not(.disabled):active,
.btn-primary:not(:disabled):not(.disabled).display-active,
.show > .btn-primary.dropdown-toggle {
  background-color: #0189d2 !important;
  border-color: #0189d2 !important;
}
.btn-primary:not(:disabled):not(.disabled):active,
.btn-primary:not(:disabled):not(.disabled).display-active,
.show > .btn-primary.dropdown-toggle {
  background-color: #004f83 !important;
  border-color: #004f83 !important;
}

Thanks for spotting this @msankaran0712 !

Fixed in: https://bitbucket.trimble.tools/projects/TMDS/repos/modus-bootstrap/pull-requests/283


add me to repository

Erin Oleson
<EMAIL_ADDRESS>eoleson

Hi Erin - I've added you to the repository. The gear icon isn't showing up for others either, so you didn't do anything wrong. We will fix the instructions. Thanks!


Client crashes because of ArrayOutOfBoundsException

After playing for 20 minutes, the client will crash and will have to be closed.
To Reproduce

Play for a bit (I was at the clocktower when I crashed, coming from Fossil Island)
Client crashes
Close Runelite

Expected behavior
No crash
Environment (please complete the following information):

OS: Windows
RuneLite version: [e.g <IP_ADDRESS>]
Launcher version: [e.g 2.3.0]

Additional context
Client Log with Bug crash: client.log
Exact error:
2022-04-27 07:57:12 [Client] INFO  rs117.hd.HdPlugin - Using device: NVIDIA GeForce GTX 1070/PCIe/SSE2 2022-04-27 07:57:12 [Client] INFO  rs117.hd.HdPlugin - Using driver: 4.6.0 NVIDIA 511.23 2022-04-27 07:57:12 [Client] INFO  rs117.hd.HdPlugin - Client is 64-bit 2022-04-27 08:09:26 [Client] ERROR client-patch - Client error: null java.lang.ArrayIndexOutOfBoundsException: Index 19 out of bounds for length 16 	at rs117.hd.ModelData.getColorForFace(ModelPusher.java:24) 	at rs117.hd.ModelPusher.getVertexDataForFace(ModelPusher.java:126) 	at rs117.hd.ModelPusher.pushModel(ModelPusher.java:97) 	at rs117.hd.HdPlugin.draw(HdPlugin.java:2478) 	at client.ww(client.java:42125) 	at hv.au(hv.java:1162) 	at hv.uj(hv.java) 	at hv.ab(hv.java:18376) 	at gw.gq(gw.java:4670) 	at jr.jp(jr.java:10333) 	at jr.jp(jr.java:10364) 	at es.jl(es.java:10222) 	at client.gu(client.java:4321) 	at client.h(client.java:1025) 	at ao.g(ao.java:385) 	at ao.run(ao.java:339) 	at java.base/java.lang.Thread.run(Unknown Source)

#279

No it is the same thing
"java.lang.ArrayIndexOutOfBoundsException: Index 19 out of bounds for length 16 at rs117.hd.ModelData.getColorForFace(ModelPusher.java:24)"
vs
"java.lang.ArrayIndexOutOfBoundsException: Index 291 out of bounds for length 288
at rs117.hd.ModelData.getColorForFace(ModelPusher.java:24)"

I see. Does disabling model caching option prevent this specific crash too?

Yeah, that might be a good option though it's unfortunate. I think this is the only known crash and it definitely only happens with the live version of 117HD and model caching turned on, but these plugins will never be quite as bulletproof as the raw game.

Awesome! Thank you!

This should be fixed as of the latest update. You should consider re-enabling model caching if you have it disabled as it greatly improves performance in many scenes. :slightly_smiling_face:


[Bug]: Jaeger Operator Compatibility table is missing data for versions 1.44, 1.45, 1.46

What happened?
The compatibility table at https://github.com/jaegertracing/jaeger-operator/blob/main/COMPATIBILITY.md is outdated. Jaeger Operator v1.46 was released a few days ago but the table only goes till v1.43
Steps to reproduce
Visit the provided link and observe the table.
Expected behavior
The table to have rows and compatibility data for versions 1.44, 1.45 and 1.46.
Relevant log output
No response
Screenshot

Additional context
This isnt really a bug but the most applicable category was either this or a feature request. I ended up choosing bug, apologies if this confuses anyone.
I would also be more than happy to make a PR once the information regarding compatibility with K8s versions is provided 😄
Jaeger backend version
No response
SDK
No response
Pipeline
No response
Stogage backend
No response
Operating system
No response
Deployment model
No response
Deployment configs
No response

Duplicated #2219

Hmm... I didn't think I had the information to do that.. but looking again at issue https://github.com/jaegertracing/jaeger-operator/issues/2219 , maybe I do... let me see...


Physics Time Spikes in GodotSteam-3.4.4-stable-editor-win64

Hi! In the normal Godot 3.4.4 stable my project runs pretty well, without any real problems. In the GodotSteam-3.4.4-stable-editor-win64 version though, without me even using any Steam class methods the project gets weird Physics Time spikes according to the profiler, like every second it has a spike up to 70 ms. So the project stutters. Edit: some further testing shows bad performance around the globe. Editor is hanging, other maps/levels, if not spiking in physics time, spike in idle time etc. Any idea?
Windows 10 latest version

There was an old issue related to this but it was just how the editor was compiled.  Do you get this stutter in exported games?  Do you have a test project or something to reproduce?
I'll check out the Github Action used to create these new editors.  Also, for comparison, can you try the previous pre-compiled editor build to see if that is any different?

I haven't tried an exported game yet, will try now. Sadly i don't have anything ready to reproduce yet. I tried the version you posted, but it has the same issues.

Hmm, that's rather strange.  There shouldn't be anything in there that would cause it to spike.  Like I said, there was a weird performance bug a good while ago but it was just a compiling issue.  Thanks for testing the older pre-compile.  The only other thing you could test would be the plug-in version of GodotSteam with the vanilla Godot 3.4.4 editor.

Will do, thank you!

It doesn't solve the overall problem but it may at least rule some things out.
OK, that is different.  In the old issue, the editor version had a problem but the exported versions were fine.  It also was odd how it happened on some machines but not others.
I'll see what I can dig up on this and report back.

I tried the plugin version, without luck. As soon as i installed the plugin, the issues came up. Even when i turn it off. :D Thank you, i'll wait here! :)

Hmm, interesting and very weird.

And the exported games also have this issue...

OK, gonna make some special compiled version for you to test.  Will report back with links.

OK, here is the first test on DropBox.  This is using different compiling flags based on the ones vanilla Godot uses for editors.

Sadly same issue still.

That is rather bizarre and sounds quite different from the old issue.  Even in the old issue I was never able to reproduce it but could fix it for someone else with different compiling flags.
I can try another compiling method to see if that does anything.  That is rather strange you'd have high FPS but lag.  Nothing in the module should affect that at all; in which case it might be how it is compiled affecting the actual Godot code, module aside.

No problem.  Interestingly, when compiling with the production=yes there is a warning in Godot's core cowdata. file.  If you do it with their official SCONS flags -and- production this causes an error and it cannot compile.  I wonder...

At least i found out where the PT spikes came from. A fog of war scan on a really big map. Remade the function, to scan only parts of the map per frame, now they are gone.

Did this affect the lag at all?

I am not 100% sure, but i think you did it! It seems pretty damn fluid now, not perfect, but i think it is way better!

Hmm, haha, I have two more tests as well.  We'll have to figure out which ones are good and which ones aren't so I'm naming them specifically now.

Keep em comin! :D

OK, gonna issue about six test editors with very specific naming so I can keep better track.

Here are the first three to try out:

Normal with production flag
Normal without production flag
Godot's Github Actions, no production flag

Got all four, testing

OK, interesting.  Yeah, the GGA version with no production flag is what Godot uses when they compile with Github Actions.  That's about as close to the vanilla as you can get.  However, the module shouldn't do anything that interferes with the game or not as it doesn't interact with anything.
I'll look into mixing up our Github Actions SCONS flags for the next update.

In case i can help any further, im here.

Welcome!  I like to get issues resolved as fast as possible, so I don't forget about them.  Performance ones get tricky, obviously, but usually they can be solved.  I think we might have a patch before Godot 3.5 drops but we'll see.  Thank you for sticking around through these various builds!

Check out the new builds and see if those help!

I'll have to test 3.5 first :D I'm on it.

I see no difference between 3.5 and 3.5 steam editor build. :)

Glad to hear it!  Who knows where in the machine these things get wonky but glad it is working.  Keep me posted!

https://youtu.be/uIzUT-BlzUk
In case you would like to see what i am working on and what i used to test the builds :D

Holy crap!  Haha, that looks really cool.  I can see why performance matters greatly!

Thank you! :)


[Feature Request]: migrate badger logs to TRACE level

Implementation ideas
Currently, our badger logs are overwhelming other types of logs in DEBUG level.
In addition, the CPU and RAM consumption are sky-rocketing.
Hence, it will be beneficial to migrate the badger logs to TRACE level

I want to try this because it really bothers me.

This can be solved under log pkg. When we set Debug we should exclude it for the noisy modules

Thanks Wondertan for your suggestion.
I found in
https://github.com/celestiaorg/celestia-node/blob/cec6a2d6535870910478cac27f44c19dcd6316af/nodebuilder/store.go
Line: 129
We can remap the log level, but it is usually not a good idea.
Then, let me try your way.


An option to not add timestamp in the replay-file mode

Describe the solution you'd like
We would like to be able to not add timestamp into the generated file.
Input
<IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187

Output
2022-12-23T06:53:48.998+00:00 <IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187

Describe alternatives you've considered
We probably may use some logs parsers and ignore the first field.
Additional context
We would like to test our logging system and it implies a kind of predefined log format. We can use replay-file mode but in the current behavior a timestamp will be added to every generated line. We would like to be able to change this behavior.
Timestamp is still useful for our case, but in a different format and location.

Looks there is some cosmetic typos. Now we get the following in the output
loadgen replay-file \
  --input-file-path in.log \
  --duration 2 \
  --lines-per-second 1 \
  --disable-timestamp \
  --output stdout

t <IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187`

An extra symbol at the start
t

ACK. fixing it right away

fixed. Kindly retest when you get a chance

@air3ijai also note that we have added an additional flag for you to be able to customize the timestamp
      --custom-timestamp-format string        Optional custom timestamp format.The default format is 2006-01-02T15:04:05.000-07:00

I just did and update
go install github.com/intuit/go-loadgen@latest
go: downloading github.com/intuit/go-loadgen v0.0.0-20221231063551-674e189a9504

Sorry, for you probably it is a very basic stuff like a go developer, but for newbies like me it is not so transparent, this is why I opened another request #8.
And now I'm trying to check if I have the latest version installed
loadgen --version
Error: unknown flag: --version

And tested again
2022-12-31T07:13:39.735+00:00 t <IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187

So, t  is still here...
rm -rf /root/go/bin/go-loadgen
go install github.com/intuit/go-loadgen@latest

2022-12-31T07:16:50.667+00:00 <IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187

And now it works as expected. Thank you for the quick fix!

Now, I'm trying to play wit the --custom-timestamp-format
Google showed me the following topic - Go: time.Format: how to understand meaning of '2006-01-02' layout?
loadgen replay-file \
  --duration 1 \
  --input-file-path in.log \
  --lines-per-second 1 \
  --output stdout \
  --custom-timestamp-format "[02/Jan/2006:15:04:05 -0700]"

[31/Dec/2022:07:32:13 +0000] <IP_ADDRESS> - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187

So, I was able to reproduce timestamp in Nginx log format - thank you for implementing that!


docs: fix typo about comment tasks in getting-started

Description
This disclaimer is trying to state that "Obsidian doesn't read text in comments" => "Tasks Plugin doesn't read tasks in comments", but it's missing a negation.
Motivation and Context
n/a
How has this been tested?
It has not. I'm a first-time contributor with no local setup, just offering to fix a small typo.
Screenshots (if appropriate)
Old Callout

New Callout
(using chrome inspect tool)

Types of changes
Changes visible to users:

[ ] Bug fix (prefix: fix - non-breaking change which fixes an issue)
[ ] New feature (prefix: feat - non-breaking change which adds functionality)
[ ] Breaking change (prefix: feat!! or fix!! - fix or feature that would cause existing functionality to not work as expected)
[x] Documentation (prefix: docs - improvements to any documentation content for users)
[ ] Sample vault (prefix: vault - improvements to the Tasks-Demo sample vault)
[ ] Contributing Guidelines (prefix: contrib - any improvements to documentation content for contributors - see Contributing to Tasks)

Internal changes:

[ ] Refactor (prefix: refactor - non-breaking change which only improves the design or structure of existing code, and making no changes to its external behaviour)
[ ] Tests (prefix: test - additions and improvements to unit tests and the smoke tests)
[ ] Infrastructure (prefix: chore - examples include GitHub Actions, issue templates)

Checklist

[x] My code follows the code style of this project and passes yarn run lint.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] My change has adequate Unit Test coverage.

Terms

[x] My contribution follow this project's contributing guide
[x] I agree to follow this project's Code of Conduct

Hi @PikaBlue107, good spot - thank you.
It's now published:
https://publish.obsidian.md/tasks/Getting+Started/Getting+Started#Tasks in Comments

wonderful! thank you for facilitating my first open source PR ^-^


Add login/signup

Is your feature request related to a problem? Please describe.
I want to add login/signup for the website for robust user management.
Describe the solution you'd like
I would create login and signup routes

Duplicted issue #52, I have already added auth but admins are not merging my issue.


bug fix: merge results of group by limit push down

Description

Fixed the bug:
when merge limit push down results with long dimension not in limitSpec, merge result is incorrect.

This PR has:

[x] been self-reviewed.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[x] been tested in a test Druid cluster.

this PR may also resolve issue #11758

could u do a code review for this pr? @gianm

@clintropolis Could u plz do a code review for this pr?

@clintropolis Could u plz do a code review for this pr?

sorry yes, will try to have a look soon

邮件收到，谢谢！

@mchades - can you merge the latest master branch in your PR source branch? This is because there is any recent change incompatible with your PR, we will be able to catch it. There have been a lot many changes to druid after the last Travis run for your PR.

@abhishekagarwal87 -yeah,It's done now, all checks have passed as well.

Thanks for the fix @mchades! It also looks to me like it should address #11758.


DBZ-7912 Add content to the docs - 'Metrics'

address : https://issues.redhat.com/browse/DBZ-7912

@roldanbob Could you please take a look? This will be more about form/formatting of the clarification

@chungeun-choi @roldanbob Applied, thanks


Hide overlays if present when changing options in runtime

Changes

Added buttons on the Demo app to toggle options:

Make sure to hide all active overlays when toggling options

@jtrivedi did you had time to check the PR?

Hey there, sorry for the delay. This patch includes some UI work, which I don't think is relevant to this issue. If you'd like to update the patch to just hide the overlays if needed, I'm happy to review it.

@jtrivedi reverted the changes
I wanted to add an UI to easily test the change + make it easier to understand how TouchInspector works for new users

Thank you!


More flexible test naming in XML reports

Currently there's an "umbrella" report.xml renerated by run_tests_matrix.py  and
several partial reports (e.g. report_c_linux_dbg.xml) for different run_tests.py runs.
This PR changes the "suite name" in the JUnit reports so that all tests results are nicely organized when viewed with Jenkins JUnit test result browser.
An example (just for C#)  is here:
https://grpc-testing.appspot.com/job/report_experiment/7/testReport/(root)/
under aggregate_tests one can see separate run_tests.py runs (and how long they took - useful for determining what to optimize).
under csharp_linux_dbg you can browse individual tests that ran for given configuration.

Should be sufficient to address #8322

CC @matt-kwong  @nicolasnoble

Another example of how it looks on Jenkins when some tests fail is here: https://grpc-testing.appspot.com/job/gRPC_pr_win/137/

Test failure unrelated, merging.


[Backport 1.3] Skip SymbolicLinkPreservingTarIT when running on Windows

Signed-off-by: Poojita Raj<EMAIL_ADDRESS>
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Failing with known flaky test - #1828


waypoint: 0.2.0 -> 0.2.1

Motivation for this change

General cleanup
Restructured so make bin succeeds
No need to destribute waypoint-entrypoint

Last time we had issues with the resulting binary not being able to successfully inject the waypoint-entrypoint without adding extra static flags.
(#103754)

Using a more modern and maintained version of go-bindata resolved this issue.
@winpat if you could double check this works that'd be great but waypoint build and waypoint up with a the example project and no actual deploy target worked fine.
This version of go-bindata is also used by influxdb2.
https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/servers/nosql/influxdb2/default.nix#L77-L91
Potentially it should replace the current version we're using for go-bindata. It's used by homebrew.
If that's desireable I suggest that be done in a separate PR as there are plenty of packages that use it and it'll probably need to go into staging.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS (x86_64)
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 112148 run on x86_64-linux 1

1 package built:

waypoint

I've made a PR for the maintained version of go-bindata #112164
If it can be merged easily into master that one should go in before this one (and I can clean this copy up)
Or if it has to go into staging then I think this can merge into master until it makes its way through

Not sure why, but my local build fails with:
...
patching sources
configuring
building
vendor folder exists, please set 'vendorSha256 = null;' in your expression
builder for '/nix/store/yw81b341hz0hdxf9j8n9bhnjvmx1x9rl-waypoint-0.2.1-go-modules.drv' failed with exit code 10
...

Any idea why this did not happen in your case @06kellyjac ?

So there is a vendor folder with just a git submodule.
I think deleteVendor = true; allowed me to get the sha for the new vendor
Then when I removed it, because the sha was in my nix-store it just used that before checking if there was the pre-existing vendor

Result of nixpkgs-review pr 112148 run on x86_64-linux 1

1 package built:

waypoint


cranelift/llvm: these backends no longer work right due to missing signal support

Originally wasmer used signals to implement signalling (heh) of traps and various other exceptional cases such as stack overflow.
We found that to be flaky in nearcore and have moved away from signal based traps in singlepass, the compiler that we use, however the llvm and cranelift backends remain written in such a way that they expect traps to be used. This means that neither of these backends are capable of passing the test suite and will only work “right” in complete absence of any exceptional situation.
On one hand keeping these backends going is somewhat desirable, since they are a good comparative target, on another, though, we already have access to a production-quality clift backend via wasmtime, so the value there is not as high as one might think.
Should we just remove these two backends entirely?

+1 to removing!


Why you use Google Chrome to Run Application? Whereas you have your own best browser i.e. Edge ?

Hi Microsoft
Microsoft is always been Excellent above all Technology companies I know.
But I am not getting on your documentation page, why there is a screenshot of Visual Studio showing Debug session to begin with Google Chrome?
Is Microsoft so small that, Google is taking over it ?
Is Microsoft Browser ... Edge... Not capable of running Power BI reports with all features?
Does Microsoft intend to Market other Companies Products over to their own products?
Do you have any doubt on your own capabilities in Browser Compatibilities?
Hope you can understand what I realy mean with all these.
I would suggest to show your own Products in all omni channels, so that we feel confident with Microsoft always, as me and my team is always confident with you.
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0fdac5f7-cd43-34c5-5905-dceeb9813952
Version Independent ID: 6fd53f84-dbb2-2a9c-2745-ea1999779113
Content: Embedded analytics to embed Power BI content in your application for your customers - Power BI
Content Source: powerbi-docs/developer/embed-sample-for-customers.md
Service: powerbi
GitHub Login: @rkarlin
Microsoft Alias: rkarlin

Thanks for the feedback @csplVinodKumar. We typically show Chrome to emphasize that a given process can be completed successfully regardless of the browser you choose.


⚠ Bofrost.at has degraded performance

In e558b8b, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1440 ms

Resolved: Bofrost.at performance has improved in 48d86cf after 8 minutes.