[Feature Request] Table structure extraction at the API

There is already some table detection mechanism in tesseract but unfortunately, there is seems to be no possibility to access the table structure at the API.
This could be done only minimal changes to the API, just by expanding the PageIteratorLevel enum by two additional members RIL_TABLEROW and RIL_TABLECELL or similar.
Those would only be relevant inside PT_TABLE blocks, just like PT_PARAGRAPH is only meaningful for text blocks.

Are you able to send PR for this including simple test case (similar to #1614)

@zdenop I didn't mean to imply that it was easy to implement, just that the interface changes are small. I have honestly no idea what it takes but if I find the time, I'll give it a try.

I assume tesseract handle tables in one of these two ways:

Tables columns are held in tesseract blocks and cells are held as lines within blocks.
Tables rows are held in tesseract blocks and cells are held as lines within blocks.

I bet on option (1).

@amitdo I'm pretty sure that this is not the case. I fear that the information is lost completely.
IMO it's also not a very good representation. Cells can be multi-line, so they should be comparable to paragraphs, not lines.
The table I'm testing with seems to be recognized as a single block (which makes sense IMO).
But then the table is split into two paragraphs (one for the first row and one for the rest), which does not make much sense.
The lines span the whole table. For multiline cells, the lines of each cell are combined into nonsensical long lines.
If this reflects the internal table structure, that would mean that the table detection is really bad and I can just disregard it.
If not, the results could be presented much better.
The fact that the table separators are actually recognized as horizontal and vertical lines makes me think that the information might be there.
I'm going to investigate a bit more, once I've successfully set up the debug viewer.

The table detection code is here:
https://gitlab.com/ocropro/bzikOCR/commit/3789787fda7
There are a few variables you can set related to table detection. See the above link.

The table detection code is here:
https://github.com/tesseract-ocr/tesseract/blob/master/src/textord/tablefind.cpp

Play with the variables:
https://github.com/tesseract-ocr/tesseract/blob/509a6f0ce0e636a9ed92553439f1ed6a56b346c5/src/textord/tablefind.cpp#L143

They published a paper about the table detection module.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>8.7400&rep=rep1&type=pdf

https://github.com/tabulapdf/tabula/issues/409#issuecomment-327050906
from someone who has scripted the process of splitting a PDF into cells and
OCR'ing them separately:
On Sat, Jun 30, 2018 at 3:24 AM Amit D<EMAIL_ADDRESS>wrote:

They published a paper about the table detection module.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>8.7400&rep=rep1&type=pdf
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/1714#issuecomment-401483687,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0F4ejGsDx0rzlfR0jLv5hpy-L9Yks5uBqIngaJpZM4U9IXO
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

Related issue - How to detect table region after the update in Tablefind.cpp? #825

Also see https://github.com/DanBloomberg/leptonica/blob/5dca24f9674c7fd057ab55bbfc71efa87a83a520/version-notes.html#L180

 Improved table detection on scanned page images (tests: pageseg_reg.c)

Thanks for all the pointers. I don't want to change the table detection though, or even implement it myself. I just want the results accessible at the public API, if available.
Is there a high-level description of the internal processing pipeline of tesseract somewhere?

Ok, I've got the debug viewer running.
It seems, that the table detection works perfectly:

But then, the contents of the table are just processed as any other text, which doesn't make sense to me:

So, this means that the data is actually there, but it's not actually used.
Is this, because the whole table is a simple block? Would it be better to treat every cell as single block and represent the table structure on a higher level?

Hi, I'm thinking about writing the API / structure part necessary to hold and access the otherwise lost information (as described by @troplin ). Is the approach described in @troplin s first comment feasible? Would a commit from a tesseract-team outsider be acceptable? Is there a Guideline for your c++ code-styling? And: Is this the right place to ask these questions?

Hi @Sintun!

I'm thinking about writing the API / structure part necessary to hold and access the otherwise lost information (as described by @troplin ). Is the approach described in @troplin s first comment feasible?

I don't know.

Would a commit from a tesseract-team outsider be acceptable?

Yes, of course.
https://github.com/tesseract-ocr/tesseract/graphs/contributors
AFAIK, only 2 people in this list are from Google.
Whether a specific PR will be accepted or rejected will depend on it's code quality.
Don't break an existing API.

Is there a Guideline for your c++ code-styling?

Not officially, but since most of the code comes from Google, it's a good idea to use Google C++ Style Guide.

Is this the right place to ask these questions?

I think so.
.

Nice, I will start this and a traceback and possible fix of https://github.com/tesseract-ocr/tesseract/issues/1712 as a weekend-project.

I suggest that you provide an example that demonstrates the use of the new API.
Good luck!

@Sintun Actually, it might actually be better to invert my API suggestion and do something like:

RIL_TABLE

RIL_TABLEROW

RIL_BLOCK (== cell)

RIL_PARA

instead of my earlier suggestion, which was:

RIL_BLOCK (== table)

RIL_TABLEROW

RIL_TABLECELL

RIL_PARA

I don't know which one is better, it depends on what a block means to the engine. I already tried to figure out how the recognition process works on a high level, but I'm a bit lost.
Maybe someone with a deeper understanding of the internals could give a hint?

Maybe someone with a deeper understanding of the internals could give a hint?

I think only @theraysmith can help here.

@troplin At a first sight your new suggestion looks more logical. I hope that the existing structures will give me a clear path to hold the table information, otherwise i will stick with the most logical (at least for me) non-API breaking approach. Thanks for the hint, I will take it into account when looking at the code.
@amitdo I think i have to invest some time and dive through the surrounding code before i can understand helping hints of people who are familiar with the code base.

@zdenop Please label as Feature Request.

@Sintun : any progress on this issue? Is API needs to be changes I would like to get it for 4.0 release...

@Sintun : any progress on this issue? Is API needs to be changes I would like to get it for 4.0 release...

Unfortunately not yet, i'm still working on #1192 / #1712 , because it seems to be a more pressing matter.

what happened to table extractor feature api

Hi, I wanted to make the information accessible through the api.
Unfortunately i wasn't able to find enough free time to do it.
Fortunately my employer needs this feature, and I'm going to propose (next week) using this tesseract code and updating the tesseract api. I'm pretty confident that this will go through, so i could focus on it, which should enable me to finish it in the near future.

Just side note: there is python project for extracting table data from pdf: Camelot and there is also web interface for it: excalibur

Hi, today i started looking into this. I'm working on 5.0.0-alpha.
I realised, that there are many TODOs in the 3 table recognition classes. Some of them are pretty serious. So i tested the table recognition on my examples. And unfortunately the results were unusable for me. So there is more work necessary than i hoped. On sufficiently dense documents (books, journals) it should be better than on my documents. The TODOs mainly aim at things that need to be done in order to deal with frequent "edge" cases.
The main class is TableFinder and it uses the TableRecognizer, which in turn uses StructuredTable.
The latter two only use table regions found by the TableFinder class. Finding the Tables seems to need 2k lines of code, whereas finding the structure in them needs 1k lines plus supporting classes and structures in other files, which seems to be a little excessive. Nonetheless, maybe some people will improve it in the future.
The found table rectangles are already exposed by the api, and I am not entirely shure yet, that the table structure isn't. Maybe it's just not documented. Unfortunately i can't invest more work time into this, because of the table extraction failure on my example documents. Nonetheless I will have another look at the weekend in order to find that out.
I would like to add my minimal example, just in case someone else wants also to take a look. I used it to obtain the debug output & to test it on different documents and languages. But it is ~120 lines and i can't attach it, github says "We don't support that file type". So i crammed it into a zip file, together with shell scripts, that contain the compiling commands for tesseract and this program, you just need to adjust the paths. Since some things can be platform dependent: i used this under Kubuntu 18.04.
compile.sh compiles the example program tess and compile_tess.sh compiles tesseract, in case you checked out a fresh version.
OpenSourceDev.zip
Finally, the magic lines in order to get Table - boundaries through the api:
api-> SetVariable("textord_tabfind_find_tables", "true");
api-> SetVariable("textord_tablefind_recognize_tables", "true");

which needs to be set before the recognition.
Then a loop over the blocks gives the boundaries:
tesseract::ResultIterator* ri = api->GetIterator();
tesseract::PageIteratorLevel level = tesseract::RIL_BLOCK;

if (ri != 0) {
  do {
    if( ri->BlockType() == PT_TABLE )
    {
      printf("found a table\n");
      int x1, y1, x2, y2;
      ri->BoundingBox(level, &x1, &y1, &x2, &y2);
      printf("table BoundingBox: %d,%d,%d,%d;\n", x1, y1, x2, y2);
    }
  } while (ri->Next(level));
}

The found table rectangles are already exposed by the api, and I am not entirely shure yet, that the table structure isn't.

It isn't. That's the whole point in this feature request.

Jep, I just wanted to say, that I'm still trying to understand the code, and hadn't reached the point where i understand all the side effects.
Introducing a structure like troplin proposed seems to be difficult, because a paragraph or a textline (even words) can go over table cell boundaries. At least at a word level, it is easy to fix them to a single table cell.
In Order to use iterators many things would need to be introduced in order to properly iterate over table & table parts and make funktions like GetUTF8Text available. Without using lists in the table cells, that hold pointers to the contained words, this could also become inefficient.
So a question:
Should tesseract offer iterators over tables (like RIL_WORD, RIL_TEXTLINE ...), table columns and table rows ?
Or would it be enough, to just add a function like
tesseract::ResultIterator* ri;
ri->TablePosition( &table_num, &table_row, &table_col );

And then the api user can decide what to do with that.
It would be simple to do the second one.
Then i Could just add these three infos to the WERD_RES class as
int8_t table;
int8_t table_row;
int8_t table_col;

and fill them after the table recognition.

Introducing a structure like troplin proposed seems to be difficult, because a paragraph or a textline (even words) can go over table cell boundaries. At least at a word level, it is easy to fix them to a single table cell.

IMO lines and paragraphs should only be detected within a table cell. It does not make sense for a line or paragraph to cross cell boundaries.
Just look at the pictures that I posted above, the first one (table structure) looks fine, but in the second one (final table partitions) the content of different cells is combined to a single structure, which just doesn't make any sense to me.
I don't know exactly how the engine actually works, but I imagine that there's a layout analysis that comes first, dividing the image into blocks of different types (text, image, table, etc). After that, each block is processed separately depending on the type.
The problem is IMO that a table is a single block and processed as a whole.
I think it would be better to create a block for each cell in the first place. That way all the problems with structures that cross cell boundaries can be avoided.

I think that implementing what troplin asked for will be very hard task.

Then i Could just add these three infos to the WERD_RES class as

int8_t table;
int8_t table_row;
int8_t table_col;

Can you save the tables info in a vector of structs/classes of tables inside  WERD_RES?
struct MyTable {
  TBOX box;
  std:vector<TBOX> table_row;
  std:vector<TBOX> table_col;
};

WERD_RES seems to be responsible for word - level information. So adding table information on this level would result into every word holding the information about every table.
Nonetheless at the next weekend i will take a look for a hierarchy level, where this information can be stored and accessed.
Creating a vector of table structs really warms my c++ heart.

You're right about WERD_RES.
It seems that with my suggestion you need to place the vector of tables in
PAGE_RES.

Any update on this API? Is anyone working on this?

Since there were a few questions about the progress: I made experimental changes that are sufficient to get the table information through the api. My understanding of the internal iterators and lists is currently not good enough, so i used a singleton approach to "warp" the information from the page segmentation to the api and hopefully i catched the reskewing and coordinate inversion steps. This approach probably fails on LTR languages, because I haven't taken them into account yet.
I forked the repo and my changes can be found in https://github.com/Sintun/tesseract.
A minimal example to get it to work:
#include <tuple>
...
tesseract::TessBaseAPI *api = new tesseract::TessBaseAPI();
for(unsigned i = 0; i < api->GetNumberOfTables(); ++i)
{
  int x1, y1, x2, y2;
  std::tie(x1, y1, x2, y2) = api->GetTableBoundingBox(i);
  printf("table BoundingBox: %d, %d, %d, %d;\n", x1, y1, x2, y2);
  std::vector<std::tuple<int,int,int,int>> rows = api->GetTableRows(i);
  std::vector<std::tuple<int,int,int,int>> cols = api->GetTableCols(i);

  for(const std::tuple<int, int, int, int>& t: rows)
  {
    std::tie(x1, y1, x2, y2) = t;
    printf("row: %d, %d, %d, %d;\n", x1, y1, x2, y2);
  }
  printf("\n");
  for(const std::tuple<int, int, int, int>& t: cols)
  {
    std::tie(x1, y1, x2, y2) = t;
    printf("col: %d, %d, %d, %d;\n", x1, y1, x2, y2);
  }
  printf("\n");
}

My small (but not minimal) program for testing:

#include <vector>
#include <iostream>
#include <string>
#include <tesseract/baseapi.h>
#include <leptonica/allheaders.h>
#include <tuple>

#include <argp.h>

class CmdArguments
{
  private:
    /// single argument parser invoked by argp
    static int parseOneArg(int key, char* arg, argp_state* state);

  public:
    std::string file_path;
    std::string data_path;
    std::string lang;
    std::string exec_path;

    /// parse (argc, argv) into CmdArguments
    CmdArguments(int argc, char** argv);
};

CmdArguments::CmdArguments(int argc, char** argv)
{
  exec_path = argv[0];

  // {long option, short option, argument, arg optional?, option description}<
  const argp_option option_list[] =
  {
    { "file-path", 'f', "/path/img", 0, "path to an image" },
    { "data-path", 'd', "/path/img", 0,
      "path to the tessdata folder containing the LSTM files" },
    { "language", 'l', "/path/img", 0, "language you want to use" },
    { 0 }
  };

  const argp setting {option_list, parseOneArg};
  argp_parse(&setting, argc, argv, 0, 0, this);
}

int CmdArguments::parseOneArg(int key, char* arg, argp_state* state)
{
  CmdArguments* data = (CmdArguments*) state->input;
  switch(key)
  {
    case 'f': data->file_path = arg; break;
    case 'd': data->data_path = arg; break;
    case 'l': data->lang = arg; break;
  }
  return 0;
}

int main(int argc, char** argv)
{
  CmdArguments cla(argc, argv);
  if( cla.data_path.empty() )
    cla.data_path = "/usr/share/tesseract-ocr/4.00/tessdata/"; //ubuntu 18.04
  if( cla.lang.empty() )
    cla.lang = "eng";

  char *outText;

  tesseract::TessBaseAPI *api = new tesseract::TessBaseAPI();

  if(api->Init(cla.data_path.c_str(), cla.lang.c_str(),
    tesseract::OcrEngineMode::OEM_LSTM_ONLY))
  {
      fprintf(stderr, "Could not initialize tesseract.\n");
      exit(1);
  }

  /// variables for table detection
  api-> SetVariable("textord_tabfind_find_tables", "true");
  api-> SetVariable("textord_tablefind_recognize_tables", "true");

  /// variables for table recognition debug output
//   api-> SetVariable("textord_show_tables", "true");

  /// Open input image with leptonica library
  Pix *image = pixRead(cla.file_path.c_str());
  api->SetPageSegMode(tesseract::PSM_AUTO);
  api->SetImage(image);
  /// Get OCR result
  outText = api->GetUTF8Text();
  /// hold the program in order to inspect debug output
  std::cout<<"Tesseract recognition finished"<<std::endl<<std::endl;
  std::cout<<"Press any key to continue"<<std::endl; getchar();
  printf("OCR output:\n%s", outText);

  /// iterate over results
  tesseract::ResultIterator* ri = api->GetIterator();
  tesseract::PageIteratorLevel level = tesseract::RIL_BLOCK;

  printf("old API:\n");
  if (ri != 0) {
    do {
      if( ri->BlockType() == PT_TABLE )
      {
        int x1, y1, x2, y2;
        ri->BoundingBox(level, &x1, &y1, &x2, &y2);
        printf("\ntable BoundingBox: %d,%d,%d,%d;\n", x1, y1, x2, y2);
      }
    } while (ri->Next(level));
  }

  printf("\nexperimental API:\n");
  for(unsigned i = 0; i < api->GetNumberOfTables(); ++i)
  {
    int x1, y1, x2, y2;
    std::tie(x1, y1, x2, y2) = api->GetTableBoundingBox(i);
    printf("table BoundingBox: %d, %d, %d, %d;\n", x1, y1, x2, y2);
    std::vector<std::tuple<int,int,int,int>> rows = api->GetTableRows(i);
    std::vector<std::tuple<int,int,int,int>> cols = api->GetTableCols(i);

    for(const std::tuple<int, int, int, int>& t: rows)
    {
      std::tie(x1, y1, x2, y2) = t;
      printf("row: %d, %d, %d, %d;\n", x1, y1, x2, y2);
    }
    printf("\n");
    for(const std::tuple<int, int, int, int>& t: cols)
    {
      std::tie(x1, y1, x2, y2) = t;
      printf("col: %d, %d, %d, %d;\n", x1, y1, x2, y2);
    }
    printf("\n");
  }

  api->End();
  delete [] outText;
  pixDestroy(&image);
  delete api;
    return 0;
}

+1

Ok, I've got the debug viewer running.
It seems, that the table detection works perfectly:

But then, the contents of the table are just processed as any other text, which doesn't make sense to me:

So, this means that the data is actually there, but it's not actually used.
Is this, because the whole table is a simple block? Would it be better to treat every cell as single block and represent the table structure on a higher level?

Hi troplin,
Can you help me understand how I can get the debug viewer running? Also, do we have any sample codes on table detection using tesseract?

https://github.com/tesseract-ocr/tessdoc/blob/master/ViewerDebugging.md

@saiprasadjnv TBH I don't know exactly what I've done anymore. I guess I just followed the instructions linked by @zdenop .

Thank you so much. I got it running.

Is this feature request still open? I am interested in working on it. If someone is already working on it, we can collaborate and speed up the process. Please feel free to contact me at<EMAIL_ADDRESS>

@saiprasadjnv you can work on this. @Sintun  started to work on this, but never send PR here, so IMO this is unfinished task.

I suggest to start by testing @Sintun's patches in his fork.

do we have any sample codes on table detection using tesseract?

https://github.com/tesseract-ocr/tesseract/issues/1714#issuecomment-588183356

I tried with some sample tables, @Sintun's solution works well. Any idea when this would be merged into master?

I could update it and create a pull request around the next weekend, if someone gives me an absolution for the usage of a singleton approach (an object that shares properties with global variables :(  ).

@Sintun any update on the pull request?

Hi, sorry for the delay, and thanks for the reminder. Since no one objected my approach i will create a pull request and a more in-depth example tomorrow.

Thank you @Sintun. Appreciate your response.

I just created a PR #3330 , my approach circumvents the tesseract iterator infrastructure. This enables the feature, but it is uncertain if it will be merged into master.

@Sintun Thanks.
I also suggest to make the demo example as part of tesseract repo.
@zdenop @stweil @egorpugin @amitdo
What is the recommended location for keeping the examples? Should we add a directory in this repo to keep this and other samples of usage of api?

/example dir is fine for any examples.

I adjusted the demo file License to Apache 2. It can be added to any repo. I was not able to find an /example directory.

Yes, /example directory does not exist. Currently api examples are there as part of documentation, tessdoc repo (having been transferred from Wiki).
I had added one apiexample to 'test' repo for use in ci.
It would be good to move them all to one place and have option to build them.

I wonder whether it would be even better to add a test case in unittest (which could also serve as an example).

I wonder whether it would be even better to add a test case in unittest (which could also serve as an example).

That's a good idea!
For starters, though, I suggest that we add /example directory to tesseract repo itself (similar to unittest) and the images needed for running can be from the test repo.
@stweil Does the directory need to be created before @Sintun can add the demonstration example to it?

No, I think we should put examples in separate repo (and make it a submodule if you want example dir in main repo).

Then I suggest a new repository tesscontrib (not to be confused with tessdata_contrib) and provide API examples there in a directory examples.
That new repository could include tesseract as a submodule and provide continous integration tests which check whether all examples work with the latest tagged Tesseract release.
It could also add tesserocr and maybe other third party Tesseract APIs as submodules and test those as well.

Does the directory need to be created before @Sintun can add the demonstration example to it?

No. If new files are added to new directories, git will handle that automatically.

Then I suggest a new repository tesscontrib (not to be confused with tessdata_contrib) and provide API examples there in a directory examples.

Sorry, I missed this message earlier.
@stweil Can you set this up?
I had extracted the api examples that were in the wiki at https://github.com/tesseract-ocr/tessdoc/tree/master/examples
Those can also be added.
Thanks!

Pull request #3330 has to be reverted because of two severe regressions, so we still need another implementation.

See PR #3505.

What code triggered the issues? Does it come from tablerecog_test.cc? Why it was not detected before?

The Python wrapper tesserocr (which is used by OCR-D) has a global TessBaseAPI object which is destroyed at program exit. That interferred with the singleton introduced by PR #3330. The resulting segmentation fault was detected before by people working with OCR-D but not analyzed. Our CI tests don't include a global TessBaseAPI (that should be added, maybe to baseapi_test).
The other regression was an assertion which occurred with some images with tables, see https://github.com/tesseract-ocr/tesseract/pull/3330#issuecomment-879995123. tablerecog_test did not detect that issue.

See pull request #3509 which detects one of the two regressions.

Did you solved the issue with tables?


GKE Cluster provision error

Rancher versions:
rancher/rancher:2.0.0-rc5
Component | Version
Rancher | master
User Interface | v2.0.41
Helm | dev
Machine | dev

Docker version: (docker version,docker info preferred)
Client:
Version:	18.04.0-ce
API version:	1.37
Go version:	go1.9.4
Git commit:	3d479c0
Built:	Tue Apr 10 18:24:33 2018
OS/Arch:	linux/amd64
Experimental:	false
Orchestrator:	swarm

Server:
Engine:
Version:	18.04.0-ce
API version:	1.37 (minimum version 1.12)
Go version:	go1.9.4
Git commit:	3d479c0
Built:	Tue Apr 10 18:22:45 2018
OS/Arch:	linux/amd64
Experimental:	false

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
GKE
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Kubernetes
Steps to Reproduce:

Create GKE Service account
Create GKE Cluster with any configuration
Watch your console (:

Results:
2018/05/01 08:27:28 [INFO] Provisioning cluster [c-lh2bj]
2018/05/01 08:27:28 [INFO] Creating cluster [c-lh2bj]
E0501 08:27:30.883393       1 generic_controller.go:204] ClusterController c-lh2bj recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil>) Call stack:
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:153
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:367
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/kontainer-engine/types/rpc_client.go:39
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/kontainer-engine/types/rpc_client.go:38
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/kontainer-engine/cluster/cluster.go:151
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/kontainer-engine/cluster/cluster.go:79
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/kontainer-engine/service/service.go:201
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/driver.go:22
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/provisioner.go:225
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/provisioner.go:105
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/provisioner.go:147
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/condition/condition.go:149
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/condition/condition.go:132
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/condition/condition.go:123
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/provisioner.go:146
/go/src/github.com/rancher/rancher/pkg/controllers/management/clusterprovisioner/provisioner.go:142
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/types/apis/management.cattle.io/v3/zz_generated_cluster_lifecycle_adapter.go:19
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/lifecycle/object.go:158
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/lifecycle/object.go:57
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/lifecycle/object.go:40
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/types/apis/management.cattle.io/v3/zz_generated_cluster_lifecycle_adapter.go:49
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/types/apis/management.cattle.io/v3/zz_generated_cluster_controller.go:122
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:250
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:190
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:178
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:170
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2337

I've had the same issue with the same error outputted to me. For me, it was the missing permissions for the service account. Maybe double check them and try again?

I too get a nil pointer error.

does that look correct permissions wise?

Please try it with v2.1.0
Document is here https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/#create-a-service-account
Please let us know in a comment if you can still reproduce the issue, and we'll reopen it


styles in deployment list should be with the same color

network:Devnet
Description
styles of action buttons in deployment list and contract should be with the same color

in taiga same goes for admin panel:

Steps to reproduce

go to playground
go to peertube or any solution with domain feature
make a deployment with domain
check the colors of the action buttons they should be the same

@ehab-hassan

@zaelgohary, let's make all icons white.

verified on devnet all icons are now white


Generate epub document

Similar to the old .../perl6.html page, generate a single document epub version of the docs.

I'll have a go at this. In part just to demonstrate how Collection can be used to generate different outputs from the same documents.
Note that the generation of the epub will not interfere with the generation of the website.

Another output that would be cool: a normalized sqlite database that we could test queries on. Not to power the website, mind you, but to power other offline tools.

@dontlaugh what should be in the db? Fields etc.what sort of queries should it respond to?

A good example might be all of the built-in classes and roles they implement.
Something like
select c.name, r.name from classes c
join class_does_role cdr on c.class_id = cdr.role_id
join roles r on cdr.role_id = r.id

Basically another view into all the type information parsed from the pod files.

@dontlaugh Perhaps we can do something quite quickly without a full new Mode.

I think, from the above, that the information required is in the table displayed in the 'Routines' tab of the website.
Even if more information is needed, we can create a proof of concept from that table.
The Routines page is created at the 'compilation' stage of the web site build by a plugin called ListFiles
We can therefore create a new plugin to be triggered at the  compilation stage, say db-creation. This would take the data structure used by ListFiles (the data structure is created in the plugin secondaries), and output it to a data file that sqlite uses to create a database.
move the database file to a place in the directory tree where it can be accessed / copied elsewhere

This is something I can do quite easily, once I know the format requirements for the sqlite input. I have done this before (used a Perl program to generate an sqlite input).
The next step would be to set up the infrastructure that

takes the database file from a
instantiates an sql database
receives a query (presumably through via the Internet), serves the response.
This task is beyond me at the present because I do not know how you might want to host this.
Another docker container?
Using Caddy in some way?

I have begun work on an EBook mode.
@coke @dontlaugh some questions and comments:

would it be possible to grant me commit rights without review for a week or so?

Only for a new directory EBook. If I make changes to the Website (as in a button to download), I would create a PR for review.
Once the new mode is meeting tests, we can go back to the current scheme.

The aim would be for the ebook, in a file called RakuDocumentation.epub (or something) to be generated into a directory allowing for a download.

is this a good name?

We would need to change the docs.raku.org index page to have a button that starts a download.

the directory that the epub file ends in should be inside the docker container
what to call the directory e-book/ ?

A significant number of resources needed to generate the Website would be also used to generate the ebook, so it would make sense to have them generated at the same time.

would this cause a problem?

I have begun work on an EBook mode.
@coke @dontlaugh some questions and comments:
would it be possible to grant me commit rights without review for a week or so?
Only for a new directory EBook. If I make changes to the Website (as in a button to download), I would create a PR for review.
Once the new mode is meeting tests, we can go back to the current scheme.

Would it make sense to do the work in a branch instead?

The aim would be for the ebook, in a file called RakuDocumentation.epub (or something) to be generated into a directory allowing for a download.
is this a good name?

This is fine.

We would need to change the docs.raku.org index page to have a button that starts a download.

+1

the directory that the epub file ends in should be inside the docker container
what to call the directory e-book/ ?

epub?

A significant number of resources needed to generate the Website would be also used to generate the ebook, so it would make sense to have them generated at the same time.
would this cause a problem?

Makes sense to me to have them updated at the same time.

Also: Thanks for tackling this!

would it be possible to grant me commit rights without review for a week or so?

No, let's not do this. But I have a compromise: I will change the automatic deployment of docs-dev.raku.org in the following ways:

It will build from a branch called, say, docs-dev. You will be able to push to this at will.
I will make the frequency of the automatic deployment more aggressive. Perhaps every 30 minutes. This will give you faster feedback.

Give me 24-48 hours for this. I will comment on this issue with more details when it's done.
Otherwise, your plan sounds good 👍.
If you focus on generating the epub first, I can look into the Caddy server config that lets us do this "correctly" from a web standards point of view: responding with the correct MIME type headers, for instance. We also definitely want downloading to work with pure curl, even if the website requires JavaScript.

@dontlaugh the compromise is better! I need something to test in situ, but I don't want to affect the main branch until we have things working.

The server automation is almost done. Here is what happens next.

PR #355 gets merged to main; go take a look at that
@finanalyst create a new branch called "docs-dev" off of main
Push your docs-dev branch to github

https://docs-dev.raku.org is already configured to pull a "docs-dev"-tagged container image from quay. I cloned latest for the first one. https://quay.io/repository/colemanx/raku-doc-website?tab=tags
Every push to docs-dev will build/push/overwrite the docs-dev tag on quay
Every 30 minutes that container gets deployed to docs-dev

@finanalyst we should be good to go. Pull down the docs-dev branch and then make any commits you want, push it back up, and every 30 minutes the the (current) docs-dev container tag will be deployed.

@dontlaugh I pulled the git repo and found docs-dev. I changed to the branch and added directories and files. When I went to commit I got a message that I have a detached head.
I'm not sure how to proceed. Do I go ahead with the commit? Do I have to fix the detached head in my local repo?

Sounds like you checked out the commit of the branch instead of the branch itself?
I would, if it's not too much trouble, start over with:
$ git clone git@github.com:Raku/doc-website.git
$ cd doc-website
$ git checkout -b docs-dev

Then in this state git add (note you cannot add directories, only files), and finally git commit and git push.
You might want to try this just with one file to make sure you haven't missed anything.
I'm sure it's possible to fixup the clone you already have but without a screen share it will be tedious.

@finanalyst It is difficult to give precise advice here without knowing about the state of your local repository, and what your local changes are. I don't want to make you lose your work.
Paste the output of git status in a comment and it will give me a better idea of what's up.

@dontlaugh

Another output that would be cool: a normalized sqlite database that we could test queries on. Not to power the website, mind you, but to power other offline tools.

Would you create a separate issue for this. I have started thinking about this but it is different to the epub issue, which I have also started on.
By the way, thanks for pointing to the reddit thread as it showed I needed to work on the epub.

Closing as this is being tracked in #356


Detail panel doesn't open when the rows are grouped

#394 breaks the details panel once the rows are grouped by a column. I managed to track the problem down to the following line.
https://github.com/material-table-core/core/blob/47b7080a51cce5597393de4dc1bd78c400d92952/src/utils/data-manager.js#L571 This bug only exists on the latest version (4.3.10), once I downgraded to 4.3.9, the bug was resolved.
Describe the bug
The bug occurs because the findDataByPath function returns undefined.
If you try to open the details panel while no grouping is active, the path array contains only one integer value, and the details panel opens normally.
If you group the rows by some column, the path array now contains two values, the first one being the group index, and the second one being the unique row id (uuid). The uuid is a string, and result.data["uuid"] returns undefined.

To Reproduce
CodeSandbox

Group the rows by the column "Words"
Expand the group "Paper"
Try to open the details panel for one of the rows inside the "Paper" group
The following error will get thrown in the console

Seems like there is a mismatch between the grouping and tree data arrays. Would you mind looking into where the wrong thing comes from ?

Did i miss something? or the code in the sandbox works very well!

@kouraf Yes, you sorted the values by the column "Word", which should work well.
The bug occurs when you group the rows by the column "Word". To group the rows drag the header of the column "Word" into the group bar, and then try to expand the detail panel.
See the example bellow:

@Domino987 From what I can see the grouping and data object logic works correctly, I think this bug is connected to the path array. This bug was introduced when this line was changed in #394:

Reverting that change fixes the bug, but it also reverts back to using the index instead of the row uuid for the path.
I think the solution is to look for a uuid match when trying to get the specific object from the data array:

I tested this fix, and it does fix the bug, while also not breaking any other features from what I can see.
Do you think this is a viable fix for this bug? If it is I'd be happy to open a PR for it.

Yes this looks good. Seems like I missed this Point when I converted it to the id.
A PR would be appreciated.

Are we good to close this one out?


Icon to remove a recipient in compose screen is almost invisible in dark theme

Dark theme with "X" button almost invisible:

Light theme for comparison:

Please reopen. Issue still exists in newest update.

The fix will be included in v5.204


Switch to Quilt Loader, use QMJ

This update should prevent many cases of dependency snarls by using Quilt Loader's provides. The QMJ format is similar to the automatically generated ones by QSL (and therefore, has tabs), although I took some liberties.
While I did my best to match the old FMJs (and equivalent FAPI versions), I fear that I probably have slipped somewhere and did a mistake, so, yeah, please take a deeper look into this when reviewing

Getting this merged would be really nice, we kinda need this to build


added support for react 18

I have udpated the packages file so that it now support the React18
we are using your great package in one of our project and we needed the support for react 18 so I thought to help you with this.
Let me know if there is any problem I would love to contribute

Thank you for your contribution!

Great job @aryanwebmob 🎉  !
@rileystephens28, are you planning to create a release for this PR?

Hey, I was waiting for your response until then I published my own module on npm ... we needed that urgent https://github.com/rogdevil/crypto-icons ... here I published it under my personal github ... will remove it later ... thanks for merging : ) @rileystephens28 🎉


Update index.md types of backend sizes

update of types of backends used for jobs, don't merge before we release the Dynamic backend size Python,R!

@odinuv jeste mergni tuhle upravu kreditu tedy :-)


[IMPORTANT] Invoice fails but transaction still proceeds

I broke the connection with the mail server and tried posting an invoice, but sure enough, the email failed. However, in the back end, the transaction was recorded. This should not happen, if the email fails to post then the transaction should also fail and not proceed any further till the connection is restored. No Entry should be in the sales data unless the email is also sent out

Fixed by using database transaction to push to dB only after email sends


Added dblink_fdw doc

What changed, and why it matters
Added the dblink_fdw documentation for PostgreSQL, this gives details on how to connect to a remote database and originally comes from the help document
https://help.aiven.io/en/articles/977358-using-the-postgresql-dblink-extension
final doc https://deploy-preview-473--developer-aiven-io-preview.netlify.app/docs/products/postgresql/howto/use-dblink-extension.html
Fixes #472

Fix #294


mixed types in columns?

Should we be setting a default type in DataFrame columns?

@matt-long, is this still an issue?

@matt-long,
While working on #66, I realized that this issue causes a lot of other issues when it comes to the .search() method (when the default value for a column is not compatible with the column data type). For example, if the column is supposed to contain integers, and pandas encounters None, the column data type is cast to float.
As a result, the search() method becomes unreliable.

Should we be setting a default type in DataFrame columns?

This sounds like a step in the right direction though It may not be an easy task as the number of collections supported in intake-esm increases.

Can we replace .csv with a different file format such as parquet or hdf5 that maintains the original column data types instead of infering them at runtime?
One disadvantage of these formats is that they are not human-readable. Would this be a problem to the users?

I don't see a problem with non-human-readable formats.


No Telegram notification in Alarmo - Bug or missing feature

Checklist

[X] I am using the latest version of Alarmo (latest version can be found here)
[X] I checked for similar existing requests (both open and closed) before posting.

Alarmo Version
v1.10.4
HA Version
2024.7.7
Bug description
Hi.
Many thanks for this great ALARMO integration :)
It is a real game changer in handling alarm systems in Home Assistant.
During configuring ALARMO I encounter a problem for my notifications....
Since I send all Home Assistant relevant notifications via Telegram (Channel), I would also realy like to send the ALARMO notifications via Telegram, but it seems it is not supportet by the ALARMO integration !?
When I configure a "notification action" in ALARMO, the Telegram notification bot is not available for selection!?
Ist that a bug or perhaps a missing feature...? I d'ont know and will realy appreciate any information and help.
Mans thanks
--> screenshot with availabilities under ALARMO and under HomeAssistant automations:

Steps to reproduce
nz
Relevant log output
nz

Hi,
Does anyone have an idea?
I'm still looking for a solution and I'm wondering what other users are doing for Telegram notification...
Is there anyone here who also uses Telegram for their HA notifications and could tell me what it looks there like in ALARMO or what solution is used?
I would be very happy if I could somehow get further, or at least know if it's a missing function in Alarmo or a bug or maybe it just doesn't work for me.
many thanks
Best regards

@kitty99hub I am using n8n with http endpoint and all my Home Assistant triggers point to that:
configuration.yaml
rest_command:   home_assistant:     url: "http://<IP_ADDRESS>:5678/webhook/home_assistant"     method: POST     headers:       Content-Type: application/json       User-Agent: "Home Assistant Webhook"     payload: >       {         "title": "{{ title }}",         "priority": "{{ priority }}",         "message": "{{ message }}"       }
and here you have my alarmo action configuration:

@kitty99hub You can use Alarmo actions to trigger any script you want. I think it should be possible to trigger the Telegram notification this way.


fix(config): cleaned up config keys, made naming and defaults more intuitive

Description

version/bundle []string -> startingVersion/startingBundle string
remove unused index handling keys, since these modes will not be supported

Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] unit tests
[x] e2e tests

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

That could be cleaner, yeah, although I think the term "latest" here is not clear. Wdyt about "diffFromLastRun" or something self-explanatory. @afflom thoughts?

That could be cleaner, yeah, although I think the term "latest" here is not clear. Wdyt about "diffFromLastRun" or something self-explanatory. @afflom thoughts?

latest is problematic because of channels. diff is problematic because that assumes that users understand that concept. Though I have questioned if most users understand the concept of channel/git HEAD, I think that it might be the most precise. I think everyone understands the concept of an upgrade. what if we make a boolean called upgrade that is true by default and correlate it with packages? This is on my mind: https://issues.redhat.com/browse/WRKLDS-281#:~:text=Messer
 added a-,comment,-- 11 hours ago

/lgtm


[CT-369] [Feature] Use variables declared in dbt_project.yml file in the file

Is there an existing feature request for this?

[X] I have searched the existing issues

Describe the Feature
When I declare a variable in a dbt_project file, I sometimes need to use it in the same file. For example:
vars:
  special_schema: "my_special_schema"

models:
  my_staging_models:
    staging:
      +materialized: view
      common:
        +enabled: true
        +schema: "{{ var('my_special_schema') }}"

special_schema is declared in the project-level vars because I need it in several places. I also need to specify this schema as the schema for some models. Currently when I do the above, I get an error:

Compilation Error
Could not render {{ var('my_special_schema') }}: Required var 'my_special_schema' not found in config:
Vars supplied to  = {}

Describe alternatives you've considered
Currently, I am adding a config block to every model to specify the schema, and that works, but with many models, it gets very repetitive. It would be nice to specify the schema once (using a variable), in the project file, and not have to do it again.
I suppose I could also include --vars "my_special_schema: some_schema" when running DBT, but for ease of use by others, I'd prefer to use the vars: block in the project file. We have many variables, and there are a few more I'd like to use in this way. Passing them all from the CLI makes for messier scripts.
Who will this benefit?
Everyone who wants to make use of variables in the project file, for variables that are declared in the project file.
Are you interested in contributing this feature?
Yes, but I'm not sure I have the necessary skills or time.
Anything else?
No response

It would make a lot of sense to be able to separate out the variables. I could see doing this in two different ways: moving the vars into their own file or pulling the vars out of the dbt_project file first and using them to process the rest of the file.

Having the vars in a separate file as one option could make life easier for some devops people as they could move a project around between environments and instead of modifying the vars in the project file, they could just swap different vars files in different environments and leave everything else untouched.

Looking forward to have something like this as well. Thanks for the initiative @solomonshorser

So will this feature be available in the future?

Upvoting this as I am also running into this issue

Not sure if it's been mentioned and albeit redundant, you could look into the default parameter to the var() function. Docs.
vars:
  special_schema: "my_special_schema"

models:
  my_staging_models:
    staging:
      +materialized: view
      common:
        +enabled: true
        +schema: "{{ var('special_schema', 'my_special_schema') }}"

I'm trying to set a config for metadata and reference it like below
alerting_config:
    team_alerting:
      owner: ['@test_user1, '@test_user2']
      slack_channel: '#data-team-dbt-alerting'
    ...etc

models:
  core:
   +schema: 'core'
   +meta:
      owner: {{ var('alerting_config')['team_alerting']['owner'] }}"
      channel: "{{ var('alerting_config')['team_alerting']['slack_channel'] }}"
   ...etc

But am also running into the issue outlined above

Same here. It'd be awesome to have vars defined in dbt_project.yml also be reference-able within that file too. It would help keep things DRY

Reopening for now. No update on priority or timeline; I just know this is a small pain felt by many.

Are there any updates on this ticket? Is there any chance that this is going to be considered anytime soon?

+1 on this feature request. I'm hitting the same constraint when trying to leverage vars in yaml config within dbt_project.yml -- specifically, trying to define env specific grants...
models:
  my_project:
    staging:
      +materialized: view
      +schema: staging
      +grants:
        select:
          - "developer"
          - >
            {%- if target.name == 'prod' -%}reporter
            {%- else -%}{{ var('default_group') }}
            {%- endif -%}

vars:
  default_group: "transformer"

Would love this feature!

If variables are usable in dbt_project.yml then I would want a means to be able to externally access values after interpolation. ie, dbt info --format json (maybe also needing the --quiet flag) to output to stdout a json blob of the processed dbt_project.yml file. (json is just an example, I would like it to output yaml by default)
I can go into technical details if wanted but simply, I have some library code that wants to know the location of the dbt artifacts. For production it is easy enough to parse/compile the dbt project and put the artifacts in a known/static location during the (CI) build process for the library code to find. During development, the library code will parse/compile the project at startup to ensure the artifacts are fresh.
The complication is that the dbt target path is configurable, so now the library code needs to know where the target path is. We can enforce a given target path when running dbt, but what about other tooling? Maybe the developer has changed the target-path value for a reason. Who knows 🤷, and its not up to the code to tell the developer they can't.
A simple solution is to read and parse the dbt_project.yml file. target-path is a simple scalar value, just a quick yaml-load + value-fetch away. Now we know where artifacts will be produced, done. If variables are allowed everywhere in dbt_project.yml, ie, if target-path could be a variable, then the library code now needs to perform variable-lookup and jinja interpolation 😨.
So now I'm no longer sure that I want variables to be used in the dbt_project.yml file 🤔, (but I still would like to be able to use variables in sub-section configs like flags, models, sources, etc)
Possible solutions exist to allow the use of variables in the dbt_project.yml file, and I'm sure that one day this will happen. When it does I will definitely want a means for external tooling to be able to know information about the dbt project without having to handle parsing.

This should be easily doable/supported if we can get #2955 to be implemented first, because we could define the variables outside of dbt_project.yml.

@GKTheOne: You could just require the devs to implement a macro which returns the data you need in JSON format, and run it with run-operation:
dbt run-operation some_macro
And dbt already has a nice tojson() macro to return the data in a workable format. :)
Unfortunately target-path is not a value which is available to be surfaced from within dbt/jinja/macros, but there are workarounds (such as using a variable).


Is it still being developed?

Any chances for an update? or is it abondoned?

I think its abandoned. Too bad. I like the charts

It would be a shame if this has been abandoned, it has some great features.
If people are looking for an alternative, d3fc (which I develop) has a very similar feature set. I've been expanding our gallery (https://d3fc.io/examples/) to show how it can be used to recreate a number of the techan examples:


PV nodeAffinity conflict with Pods - AzureDisk persistent volume claim

What did you do?
I deployed prometheus statefulset in self managed kubernetes cluster(Not AKS) in Azure. The storage was configured as persistent volume claim. The storage class for PVC is given below.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azuredisk-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_ZRS
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Statefulset got deployed successfully. But the pods are in pending state and never got scheduled to the nodes.
What did you expect to see?
The pods should get scheduled to nodes and start running.
What did you see instead? Under which circumstances?
The pods are continuing in pending state. On analysis understood that, whenever PVC is created in Azure as per the storage class the PVs are also created with nodeAffinity with certain labels given below.
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - westus2-1
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - westus2-2
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - westus2-3
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - ""

As per my understanding, the pods are going to pending state because, the nodes dont have the above labels and hence when the pods scheduled to the nodes they never get the PV to mount.
When describing the pending pods getting the below messages.
Warning  FailedScheduling  <unknown>        0/9 nodes are available: 1 node(s) had taint {dedicated: storage}, that the pod didn't tolerate, 2 node(s) had taint {dedicated: telemetry}, that the pod didn't tolerate, 3 node(s) didn't match Pod's node affinity, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

Please note the taint - toleration is working without any issues.

Prometheus version: 2.25.2

Thanks for your report. It looks as if this is actually a question about usage and not development.
It makes more sense to ask questions like this on the prometheus-users mailing list or our forums rather than in a GitHub issue. In our community channels, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.


Fix Nuget restore issue

The reason why we run into scenario where we have XYZ.2.0.0 and XYZ/2.0.0 in the same folder is because both the global package folder and the repositoryPath variable point to the same location.
XYZ.2.0.0 is from the packages.config restore.
XYZ/2.0.0 is from the global package folder item.
Normally we should reference the bootstrapped packages in our proj, so use XYZ.2.0.0 paths instead of the XYZ/2.0.0.

https://github.com/Microsoft/testfx/issues/311


Unproper commas in the contributor list

Current:moon-rabbit ,openfoodfacts-contributors
Should be:moon-rabbit, openfoodfacts-contributors

Will take this issue


How to use atoti on Google Colab

Description
docs.atoti.io is missing a documentation on how to use atoti on google colab

It is not possible to use atoti-jupyterlab in colab, but atoti can work without its jupyterlab extension as follow:
!pip install atoti
!pip install pyngrok

import atoti as tt
from pyngrok import ngrok
import pandas as pd

port = 9090
ngrok_connection = ngrok.connect(port)

config = tt.config.create_config(url_pattern=ngrok_connection.public_url, port=port)

session = tt.create_session(config=config)
print(session.url)

base = session.read_pandas(
    pd.DataFrame({"ID": [1,2,3,4], "Product": ["a", "b","c","d"], "Amount": [10.0, 20.0, 30.0, 40.0]}), store_name = "sales"
)
cube = session.create_cube(base)

pyngrok make possible the access to the atoti dashboard to create interactive graphs

After more work on this :

pyngrok free usage limitations (about 30queries/minute) is not enough to have a good experience with atoti
We tried to use colab advanced outputs but colab does not support websockets (see this issue) therefore it is not possible to use atoti with colab features.
Third parties cookies will have to be authorized for atoti to work properly on colab with advanced outputs.

Can you please comment if you use Colab ?

Hello Team,
I am trying to use atoti on Google Colab and able to visualize graph. I found this issue in atoti about how to use atoti with Google Colab:
#273
But when I try to use cube.schema command it is showing below error:
Missing Graphviz library which is required to display the graph. It can be installed with Conda: conda install graphviz or by following the download instructions at https://www.graphviz.org/download/.

I have installed graphviz. pydot, pygraphviz in colab still its failing.

Hi @sumitrathi95, how did you install graphviz in your environment? Did you try restarting the kernel and the atoti session after installing it? Maybe this would help: https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/.

What about using https://github.com/antoniomika/sish instead of pyngrok @bloussou ?


counters updating but the date is stalled

Hello.
The command line I use to run goaccess:
/usr/bin/goaccess - --port=7890 -o /var/www/html/report.html --keep-db-files --load-from-disk --db-path=/var/lib/goaccess/ --compression bz2 --real-time-html < <(/usr/bin/socat -t3 -u UNIX-RECV:/tmp/haproxy.sock -) >/dev/null

For some reason the date period is stalled with "21/JUN/2017-30/JUN/2017", however the counters are still updating.
Addon: the only counters that get updated are total, valid and failed requests on top of the page
Any ideas what to check?

Does refreshing/reloading the page on your browser work? Also, do you see the expected dates within the visitors panel? Thanks.

No, reloading the page as well as restarting goaccess does not help
No, the last date everywhere is 30/Jun/2017

Looks like the data source could the issue. Can you please verify that the input log given to goaccess has the latest dates? For instance, use tail -F  to see if < <(/usr/bin/socat -t3 -u UNIX-RECV:/tmp/haproxy.sock -) has those dates. Thanks.

Please also make sure you have a valid log/date/time format that matches the source log.

I've just run socat without goaccess and here is its output:
# /usr/bin/socat -t3 -u UNIX-RECV:/tmp/haproxy.sock -  | head -1
Jul  6 15:11:03 80.70.x.x:18415 [06/Jul/2017:15:11:03.202] ...

# grep format /etc/goaccess.conf | grep -v "^#"
time-format %H:%M:%S
date-format %d/%b/%Y
log_format %^ %^ %^ %^ [%d:%t.%^] %^ %^ %^/%^/%^/%^/%L %^ %^ %^ "%r" %s {%^|%^|%^|%u} %^ %^=%b %^ %h:%^ %^ %^ %^

Got it. Can I ask you to run a test by getting rid of --load-from-disk and see if that's where the issue is. If that still doesn't work, and if you don't mind loosing the current stored data-set, please try without --keep-db-files. Thanks.

Without "--load-from-disk":

Should I try without "--keep-db-files" only or without "--keep-db-files" AND without "--load-from-disk"?

Before doing that and since you got a bunch of failed requests, can you please post a few lines from your log so I can test the log-format you posted above. Thanks.

Jul  6 16:14:05 <IP_ADDRESS>:64163 [06/Jul/2017:16:14:00.748] lb-useast~ frontend_app1_backend/sp-useast-001 4669/0/1/11/4681 --VN 83/74/0/1/0 0/0 "POST /app1/navmessages/api/alerts HTTP/1.1" 200 {||142|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59} ireq_size=2829 resp_size=168 <IP_ADDRESS>:443 <IP_ADDRESS>:443 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 B6:A3_39:BB_C8_D5:B2
Jul  6 16:14:05 <IP_ADDRESS>:54888 [06/Jul/2017:16:14:05.283] lb-useast~ frontend_app1_backend/sp-useast-001 141/0/2/10/153 --VN 93/86/1/2/0 0/0 "POST /app1/navmessages/api/alerts HTTP/1.1" 200 {||162|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59} ireq_size=2106 resp_size=168 <IP_ADDRESS>:443 <IP_ADDRESS>:443 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 51:68_39:BB_CD_A8:8B
Jul  6 16:14:05 <IP_ADDRESS>:16078 [06/Jul/2017:16:14:00.306] lb-useast~ frontend_ls_backend/sp-useast-006 5139/0/1/22/5162 --NI 93/86/0/1/0 0/0 "POST /ls/apiLocation/location HTTP/1.1" 200 {||386|Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-N910P Build/MMB29M)} ireq_size=639 resp_size=157 <IP_ADDRESS>:443 <IP_ADDRESS>:443 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 A8:39:C8_5A:8B

Please try this:
log-format %^[%d:%t.%^] %^ %^ %^/%^/%^/%^/%L %^"%r" %s {%^|%^|%^|%u} %^ %^=%b %^ %h:%^
date-format %d/%b/%Y
time-format %T

it works. adding "--load-from-disk" does not load the past data

Good. Now try to see if everything else is working fine and let me know. Note that if you didn't use --keep-db-files on a previous command, then all previous data is deleted upon exiting goaccess.

OK. I didn't remove --keep-db-files from the command line

It works so far, thank you! Any chances to get data back from the database for the previous period?

You will need to parse the log again. I think the issue was after removing --load-from-disk and passing a new data source. So when you exited the program, it kept the new data but since the old data was not loaded, it got discarded. Sorry about the confusion on that.

Closing this. Feel free to reopen it if needed.


gamecube-tools: 1.0.2 -> 1.0.4

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/devkitPro/gamecube-tools/releases.
meta.description for gamecube-tools is: "Tools for gamecube/wii projects"
meta.homepage for gamecube-tools is: "https://github.com/devkitPro/gamecube-tools/"
meta.changelog for gamecube-tools is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

/nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4/bin/gcdsptool passed the binary check.

/nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4/bin/elf2dol passed the binary check.

/nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4/bin/gxtexconv passed the binary check.

2 of 3 passed binary check by having a zero exit code.

2 of 3 passed binary check by having the new version present in output.

found 1.0.4 with grep in /nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4

found 1.0.4 in filename of file in /nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4

directory tree listing: https://gist.github.com/a1a8dbc2e2d9674b644d5278334b37c3

du listing: https://gist.github.com/c848fbb6e9749c362ea6a4a7c8d05109

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
0 i686-linux rebuild(s)
0 x86_64-darwin rebuild(s)
0 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
gamecube-tools

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A gamecube-tools https://github.com/r-ryantm/nixpkgs/archive/d37cd94a4ad3a9a892ec0f828de22e1d41f86165.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4
ls -la /nix/store/sfmas3bgkaq99yk9jakw85312qv1wv0h-gamecube-tools-1.0.4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

gamecube-tools

Maintainer pings
cc @tomsmeets for testing.

Result of nixpkgs-review pr 127298 at d37cd94a run on x86_64-linux 1

1 package built successfully:

gamecube-tools

1 suggestion:

warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.
Near pkgs/development/tools/gamecube-tools/default.nix:21:5:
   |
21 |     license = licenses.gpl2;
   |     ^


Recommended Python Version?

I couldn't find this in different help forums, etc., but is there a recommended version of python to use with denoiser? Pardon for the simple question!

Yep, maybe python 3.13 is suitabale to you.
Some of the new major new features and changes in Python 3.13 are:
A new and improved interactive interpreter based on PyPy(https://pypy.org/)'s, featuring multi-line editing and color support, as well as colorized exception tracebacks.


Please re-tune DeviceSegmentedSort policies

Looking into code it looks polices for DeviceSegmentedSort are not consistent (or possible have some typos):
for Policy700 ITEMS_PER_MEDIUM_THREAD is defined as Nominal4BItemsToItems<DominantT>(KEYS_ONLY ? 11 : 7);
for Policy800 ITEMS_PER_MEDIUM_THREAD is defined as Nominal4BItemsToItems<DominantT>(KEYS_ONLY ? 7 : 11);
for Policy860 ITEMS_PER_MEDIUM_THREAD is defined as Nominal4BItemsToItems<DominantT>(LARGE_ITEMS ? 9 : 7);

For my use case (suffix array and BWT construction) segmented sorting is accounting for 30-40% of overall performance. And with current polices compiling for SM 7.5 / 8.0 produces faster versions: I measure 5% to 10% overall performance degradation if I target SM 8.6+.
So it would be great if DeviceSegmentedSort policies gets re-tuned.

Hello @IlyaGrebnov!
Thank you for reporting this. The current tuning is not a typo. We've benchmarked various workloads and selected tuning based on some aggregate metrics. The issue with ITEMS_PER_MEDIUM_THREAD is that increasing it deteriorates performance for small segments while providing a speedup for a bit larger ones. Observing speedup in a particular case is possible, but the change will slow down others. We are currently working on a public tuning infrastructure that would illustrate (and probably alter) our choices. After it's released, you should be able to check a particular tuning against a set of use cases that we have in mind.
Meanwhile, you should be able to tune CUB for your use case by providing a custom tuning policy. Below is a code sample illustrating custom tuning for segmented sort:
#include <cub/cub.cuh>
#include <thrust/device_vector.h>
#include <thrust/sort.h>
#include <thrust/execution_policy.h>

template <bool KEYS_ONLY, class KeyT>
struct policy {
  struct policy350 : cub::ChainedPolicy<350, policy350, policy350> {
    // Threads in block
    constexpr static int BLOCK_THREADS = 256;

    // Bits per digit
    constexpr static int RADIX_BITS = 6;

    // Number of segments to trigger fallback kernel (CTA/segment)
    constexpr static int PARTITIONING_THRESHOLD = 500;

    using LargeSegmentPolicy = cub::AgentRadixSortDownsweepPolicy<
        BLOCK_THREADS, 19 /* items per thread in large segments */, KeyT,
        cub::BLOCK_LOAD_DIRECT, cub::LOAD_DEFAULT, cub::RADIX_RANK_MEMOIZE,
        cub::BLOCK_SCAN_WARP_SCANS, RADIX_BITS>;

    constexpr static int ITEMS_PER_SMALL_THREAD =
        cub::Nominal4BItemsToItems<KeyT>(7);

    constexpr static int ITEMS_PER_MEDIUM_THREAD = 7;

    using SmallAndMediumSegmentedSortPolicyT =
        cub::AgentSmallAndMediumSegmentedSortPolicy<
            BLOCK_THREADS,
            // Small policy
            cub::AgentSubWarpMergeSortPolicy<
                (KEYS_ONLY ? 4 : 8), // Threads per segment
                ITEMS_PER_SMALL_THREAD,
                cub::WarpLoadAlgorithm::WARP_LOAD_DIRECT,
                cub::CacheLoadModifier::LOAD_DEFAULT>,
            // Medium policy
            cub::AgentSubWarpMergeSortPolicy<
                32, // Threads per segment
                ITEMS_PER_MEDIUM_THREAD,
                cub::WarpLoadAlgorithm::WARP_LOAD_DIRECT,
                cub::CacheLoadModifier::LOAD_DEFAULT>>;
  };
  using MaxPolicy = policy350;
};

int main() {
  constexpr bool is_descending = false;
  constexpr bool keys_only = true;

  using key_t = int;
  using value_t = cub::NullType; // <- means SortKeys
  using offset_t = std::size_t;
  using offset_iterator_t = offset_t *;

  const bool is_overwrite_ok = true; // <- means DoubleBuffer

  // input problem
  const int num_items = 1024;
  const int num_segments = 2;
  thrust::device_vector<key_t> keys_input(num_items);
  thrust::device_vector<key_t> keys_buffer(num_items);
  thrust::device_vector<offset_t> offsets(num_segments + 1);

  // init problem
  thrust::sequence(keys_input.rbegin(), keys_input.rend());
  cub::DoubleBuffer<key_t> d_keys(thrust::raw_pointer_cast(keys_input.data()),
                                  thrust::raw_pointer_cast(keys_buffer.data()));
  cub::DoubleBuffer<value_t> d_values(nullptr, nullptr);
  offset_iterator_t segment_start = thrust::raw_pointer_cast(offsets.data());
  offset_iterator_t segment_end = segment_start + 1;

  // initialize segment offsets
  offsets[0] = 0;
  offsets[1] = 512;
  offsets[2] = num_items;

  // dispatch layer
  using policy_t = policy<keys_only, key_t>;
  using dispatach_t =
      cub::DispatchSegmentedSort<is_descending, key_t, value_t, offset_t,
                                 offset_iterator_t, offset_iterator_t, policy_t>;
  void *d_temp_storage{};
  std::size_t temp_storage_size{};
  cudaStream_t stream{};

  // get temporary storage size
  dispatach_t::Dispatch(d_temp_storage, temp_storage_size, d_keys, d_values,
                        num_items, num_segments, segment_start, segment_end,
                        is_overwrite_ok, stream);

  thrust::device_vector<std::uint8_t> temp_storage(temp_storage_size);
  d_temp_storage = thrust::raw_pointer_cast(temp_storage.data());

  // sort
  dispatach_t::Dispatch(d_temp_storage, temp_storage_size, d_keys, d_values,
                        num_items, num_segments, segment_start, segment_end,
                        is_overwrite_ok, stream);

  // validate result
  if (thrust::is_sorted(thrust::device, d_keys.Current(), d_keys.Current() + 512))
  {
    std::cout << " first segment is sorted " << std::endl;
  }

  if (thrust::is_sorted(thrust::device, d_keys.Current() + 512, d_keys.Current() + 1024))
  {
    std::cout << " second segment is sorted " << std::endl;
  }
}

It would also be helpful if you could attach a workload sample that is important for you so we can consider incorporating it into the current set of workloads.

Hi @senior-zero,
Thank you for response! My issue is that I do not have access to wide variety of GPU to complete tuning, but I would be certainly interested in contributing my workload for benchmark. Is any reference I can use?
And just to confirm, ITEMS_PER_MEDIUM_THREAD for Policy700 and Policy800 are defined in reverse (11 / 7 vs 7 / 11) is not the typo?

Is any reference I can use?

@IlyaGrebnov you can use something along the following lines as a reference for writing benchmark, but you could just attach a single file with segments and their contents along with a code illustrating how to read that data. It would be preferable to generate data by code instead, though.

And just to confirm, ITEMS_PER_MEDIUM_THREAD for Policy700 and Policy800 are defined in reverse (11 / 7 vs 7 / 11) is not the typo?

It's not a typo.

Folks, sorry for going back to this topic.
But I still see some odd things like Policy860 uses CacheLoadModifier::LOAD_LDG while Policy800 uses CacheLoadModifier::LOAD_DEFAULT. Non of this should matter for new architectures with unified L1 / Textures cache (based on my testing). Also, "Threads per segment" for Medium policy was always 32, yet for Policy860 it is switched to 16. Should it be full warp?

But I still see some odd things like Policy860 uses CacheLoadModifier::LOAD_LDG while Policy800 uses CacheLoadModifier::LOAD_DEFAULT. And non of this should matter for new architectures with unified L1 / Textures cache (based on my testing).

The default load is not equivalent to LDG one. The default load is equivalent to dereferencing a pointer. LDG load explicitly marks the load as a load through a non-coherent cache. You must've meant that there should be no difference between the CA and LDG loads in some cases, but that's not the case with the default load. Also, note that, unlike other loads, the memory consistency model is not applied to ld.global.nc, which might have consequences in the presence of threads synchronization.

And non of this should matter for new architectures with unified L1 / Textures cache (based on my testing). Also, "Threads per segment" for Medium policy was always 32, yet for Policy860 it is switched to 16. Should it be full warp?

Medium segments don't have to be handled by full warps. In general, larger tile sizes do not necessarily mean better performance. An increase in tile size leads to a slowdown of smaller segments and a speedup of larger ones. This is a compromise. Having smaller medium segments here means that the slowdown for small segments overweighted potential speedup for the segment size distributions we looked at.
We are going to retune this algorithm as part of the following issue.


specify revocation version add v0.1.0 spec

address issue 67 https://github.com/decentralized-identity/jwt-vc-presentation-profile/issues/67

@dtmcg i think we should delete spec/ and just keep everything in one folder for now until we officially fork. it will get too confusing to try to maintain two versions.

@dtmcg  can we please break this PR into two - revocation spec version (which has issues per #19 ) and v0.0.1 of a whole spec?


[Bug]: Tally not works on server (SOURCE)

What happened?
Hello I installed TA 3.0.4 on Ubuntu 20.04.5 LTS - this is a server. For vdeo switching I use OBS studio 27.2.4 and at this computer is webcsocket server plugin and Tally arbiter 3.0.4. In TA on switching computer everything seems good and everything works correctly. I connected switching computer to server with key and on the website appeared every sources and devices but on the server tally lights not works and OBS source is red.
For OBS on video switching computer I use settings: IP <IP_ADDRESS>, port 4444 and password. In the desktop app tallylights works and on the local web <IP_ADDRESS>:4455 from this computer everything looks allright.
On server in logs tab there I see every changes when I change source video - example:
[3/11/23 6:16 PM] Source: obs Address: f71df342 Busses: Preview,Program[3/11/23 6:16 PM] Source: obs Address: 6a749efc Bus: Program[3/11/23 6:16 PM] Source: obs Address: f71df342 Bus: Program[3/11/23 6:16 PM] Source: obs Address: 6a749efc Bus: Preview[3/11/23 6:16 PM] Source: obs Address: f71df342 Bus: Program[3/11/23 6:16 PM] Source: obs Address: 6a749efc Busses: Preview,Program[3/11/23 6:16 PM] Source: obs Address: f71df342 Bus: Preview[3/11/23 6:16 PM] Source: obs Address: 6a749efc Bus: Program
Can you any help me?
Version
3.0.4
Distribution
Source
OS
Linux
What browsers are you seeing the problem on?
Firefox, Chrome
If applicable, What Listener Clients are You Using?
Web (Firefox)
TallyArbiter configuration
{
  "security": {
    "jwt_private_key": "PmZxqxLzU7am2RYdhx8UHjIsUjUNbcC4aSIi87lI539JRETwcfAVOvsTq4GXlNZg+bjyYJqNKPpRD3J04TvfjGQXTyzYb7e/G8L2fI9gEJEaPzdv5rmuORAUqwghJIK7V4D0vpf9Tz/tX1by4rURj4Ss12UVjr9muMobBJ4nLpgdQRf6uunTAZsaJ4RVdR5dRz5/Vm8HkGMMdjpn9gZVp6vPIlp561KIvakfQkhwJ9YecSXhGrjVyz6ISUirEsxk4JcFKjEffAjv3GXE588JKulTQJkvdb37B9LnyiRcifHvN0FWoN4KzWnS4LKWNKzUF9uq38RCnuQ+xF2Jj3Hu0g=="
  },
  "users": [
    {
      "username": "producer",
      "password": "$2b$10$s5nv3cTTkInEGd6wsYcVN.D7zfbrhfdppBcpZSz278AhJEAHM2n9u",
      "roles": "producer"
    },
    {
      "username": "admin",
      "password": "$2b$10$/RVsrjTeLJ72YilEgtWgJ.u37PmOH5Adyhv8YnlfV.SzbXe11.M1m",
      "roles": "admin"
    }
  ],
  "cloud_destinations": [
    {
      "host": "<IP_ADDRESS>",
      "port": 4455,
      "key": "j",
      "id": "f02aac62",
      "status": "connected",
      "connected": false
    }
  ],
  "cloud_keys": [],
  "device_actions": [],
  "device_sources": [
    {
      "deviceId": "1cb2415b",
      "sourceIdx": "0",
      "address": "Text (GDI+)",
      "sourceId": "600c2cb7",
      "id": "6a749efc"
    },
    {
      "deviceId": "ae19bb28",
      "sourceIdx": "0",
      "address": "xxx",
      "sourceId": "600c2cb7",
      "id": "bf536d2f"
    },
    {
      "deviceId": "60c21982",
      "sourceIdx": "0",
      "address": "zpráva",
      "sourceId": "600c2cb7",
      "id": "f71df342"
    }
  ],
  "devices": [
    {
      "name": "ahoj",
      "tslAddress": 1,
      "enabled": true,
      "id": "1cb2415b"
    },
    {
      "name": "xx",
      "tslAddress": 2,
      "enabled": true,
      "id": "ae19bb28"
    },
    {
      "name": "upozorneni",
      "description": "upozorneni",
      "tslAddress": 3,
      "enabled": true,
      "id": "60c21982"
    }
  ],
  "sources": [
    {
      "data": {
        "ip": "<IP_ADDRESS>",
        "port": 4444,
        "password": "test"
      },
      "name": "obs",
      "sourceTypeId": "4eb73542",
      "reconnect": true,
      "enabled": true,
      "id": "600c2cb7",
      "connected": true,
      "max_reconnects": 5
    }
  ],
  "tsl_clients": [
    {
      "id": "f5087910",
      "ip": "<IP_ADDRESS>",
      "port": 4455,
      "transport": "tcp"
    },
    {
      "id": "f5087910",
      "ip": "<IP_ADDRESS>",
      "port": 4455,
      "transport": "udp"
    }
  ],
  "tsl_clients_1secupdate": true,
  "bus_options": [
    {
      "id": "e393251c",
      "label": "Preview",
      "type": "preview",
      "color": "#3fe481",
      "priority": 50
    },
    {
      "id": "334e4eda",
      "label": "Program",
      "type": "program",
      "color": "#e43f5a",
      "priority": 200
    },
    {
      "id": "12c8d699",
      "label": "Aux 1",
      "type": "aux",
      "color": "#0000FF",
      "priority": 100
    },
    {
      "id": "12c8d689",
      "label": "Aux 2",
      "type": "aux",
      "color": "#0000FF",
      "priority": 100
    }
  ],
  "externalAddress": "http://<IP_ADDRESS>:4455/#/tally",
  "remoteErrorReporting": false,
  "uuid": "54f9cb06"
}

Relevant log output
[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for ahoj to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for xx to <IP_ADDRESS>:4455[3/11/23 6:19 PM] Sending TSL data for upozorneni to <IP_ADDRESS>:4455

Error stacktrace (if applicable)
No response

This looks like a network problem if you are trying to reference <IP_ADDRESS> from your server as OBS when it is at another IP.


token: fix ts for @solana/spl-token

Fix typescript definitions for https://github.com/solana-labs/solana-program-library/pull/1802

Hi there, thanks for noticing this and providing a contribution!  Unfortunately, since this is just a declaration, TS doesn't allow setting a default, so it'll have to be:
allowOwnerOffCurve: boolean,

Could you make that fix, and also apply it to the module.flow.js while you're at it?

Hey! Thank you!!!
BTW, this PR also can solve
https://github.com/solana-labs/solana-program-library/issues/1997
https://github.com/solana-labs/solana-program-library/pull/2145

@joncinque thanks for the fix of fix :smile:
Updated!

I think it should be allowOwnerOffCurve?: boolean, not required.

@TimonPeng thanks!

Great point, thanks @TimonPeng ! Once this passes CI I'll merge it in


Do not promote to Any

Before:
julia> push!!([Init(+)], 0)
2-element Array{Any,1}:
  Init(+)
 0

After:
julia> push!!([Init(+)], 0)
2-element Array{Union{InitialValues.InitialValueOf{typeof(+)}, Int64},1}:
  Init(+)
 0

Codecov Report

Merging #27 into master will decrease coverage by 1.83%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master      #27      +/-   ##
==========================================
- Coverage   91.83%   90.00%   -1.84%
==========================================
  Files           2        2
  Lines          49       50       +1
==========================================
  Hits           45       45
- Misses          4        5       +1

Impacted Files
Coverage Δ

src/InitialValues.jl
85.71% <0.00%> (-2.53%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 065cc1c...6ff8816. Read the comment docs.


Add apcu as hard dependency in composer.json

Do you think apcu extension "ext-apcu": "*"?

I've added it to require-dev mistakenly, should be moved to require.


Add pointer-events declarations to modal for easier custom close handling

make outer .modal-dialog pass through events for custom click handling (making it possible to just listen on the .modal-backdrop for any clicks), then counteract it for the actual .modal-content
Closes https://github.com/twbs/bootstrap/issues/22493

Cool! Saving this for Beta 2.


DS-3710 Fix ehcache config conflict (port to master)

This is a master port of the https://github.com/DSpace/DSpace/pull/1860 PR, already tested, reviewed and merged to 6.x

merging without review as this is a simple one-liner port of a PR that's already been reviewed and merged to 6.x


[INLONG-2230] add jackson-core dependency

Title Name: [INLONG-XYZ][component] Title of the pull request
where XYZ should be replaced by the actual issue number.
Fixes #2201
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Codecov Report

Merging #2231 (e5d597d) into master (4a5da2b) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2231   +/-   ##
=========================================
  Coverage     12.58%   12.58%
  Complexity     1206     1206
=========================================
  Files           417      417
  Lines         35735    35735
  Branches       5590     5590
=========================================
  Hits           4497     4497
  Misses        30435    30435
  Partials        803      803

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a5da2b...e5d597d. Read the comment docs.


Rename to WikiSpeedruns

Update headers, any references, etc.

I think we should still keep references to Wikipedia like on the home page "The goal of this game is simple. Try to get from one wikipedia page to another as fast as possible by clicking on the links in the page." (also maybe capitalize that) but for all places where we are describing our game, it should be WikiSpeedruns.
Having Wikipedia on the first line helps with SEO a lot I think

and people mentioned that they found our site by just googling "wikipedia speedruns"


fix(compile): update external module indicator

Closes #341
Typescript determines whether or not to mutate the module part of a file based on a few factors, including if the module is an external module.
This computation is only done on file parse, but sometimes functionless may change a non-external module into an external module when adding the functionless import at the top of the file (see #341).
Added a hack to re-compute the module-ness of the file after we make our changes.
      functionlessContext.requireFunctionless &&
        // @ts-ignore
        !updatedSourceFile.externalModuleIndicator &&
        // @ts-ignore
        updatedSourceFile.setExternalModuleIndicator &&
        // @ts-ignore
        updatedSourceFile.setExternalModuleIndicator(updatedSourceFile);

Rolling this into https://github.com/functionless/functionless/pull/337


Fix - callx encoding

Currently callx encodes its target register in the imm field, but it should use the src register field instead.
The instruction info in llvm would need to be patched here:
https://github.com/solana-labs/llvm-project/blob/7b8db05b564faffb522434b73b7082662171f94a/llvm/lib/Target/BPF/BPFInstrInfo.td#L508

Codecov Report

Merging #491 (bbbe89c) into main (5debf64) will increase coverage by 0.01%.
Report is 1 commits behind head on main.
The diff coverage is 96.00%.

@@            Coverage Diff             @@
##             main     #491      +/-   ##
==========================================
+ Coverage   89.69%   89.70%   +0.01%
==========================================
  Files          23       23
  Lines       10109    10123      +14
==========================================
+ Hits         9067     9081      +14
  Misses       1042     1042

Files Changed
Coverage Δ

src/disassembler.rs
92.92% <0.00%> (ø)

src/assembler.rs
99.60% <100.00%> (+<0.01%)
:arrow_up:

src/elf.rs
87.34% <100.00%> (+0.02%)
:arrow_up:

src/interpreter.rs
98.23% <100.00%> (+0.01%)
:arrow_up:

src/jit.rs
92.90% <100.00%> (+0.01%)
:arrow_up:

src/verifier.rs
97.50% <100.00%> (+0.04%)
:arrow_up:

... and 1 file with indirect coverage changes


Help with error

Hi there,
I tried to use the tool again today after a loong time, but I got met with an error. Any idea if it's something on my end? I'm using python 3.8 because I need that version specifically for another project. I'm on Windows 11.
this is my config.json

{
"downloadDirectory": "C:\Users\danie\OneDrive\Documents\TS4\#TS4 Mods\CC",
"maxActiveDownloads": 4,
"saveDownloadQueue": true,
"debug": false
}

and this is the error i got.

C:\Users\danie\OneDrive\Documents\TS4\The-Sims-Resource-Downloader-0.4.0>python src/main.py
--- Logging error ---
Traceback (most recent call last):
File "C:\Users\danie\AppData\Local\Programs\Python\Python38\lib\logging_init_.py", line 1081, in emit
msg = self.format(record)
File "C:\Users\danie\AppData\Local\Programs\Python\Python38\lib\logging_init_.py", line 925, in format
return fmt.format(record)
File "C:\Users\danie\AppData\Local\Programs\Python\Python38\lib\logging_init_.py", line 664, in format
record.message = record.getMessage()
File "C:\Users\danie\AppData\Local\Programs\Python\Python38\lib\logging_init_.py", line 369, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "C:\Users\danie\OneDrive\Documents\TS4\The-Sims-Resource-Downloader-0.4.0\src\logger.py", line 23, in exceptionHandler
logging.exception(type, value, tb)
Message: <class 'TypeError'>
Arguments: (TypeError("unsupported operand type(s) for |: 'type' and 'NoneType'"), <traceback object at 0x0000015CB5DBACC0>)
Traceback (most recent call last):
File "src/main.py", line 1, in
from TSRUrl import TSRUrl
File "C:\Users\danie\OneDrive\Documents\TS4\The-Sims-Resource-Downloader-0.4.0\src\TSRUrl.py", line 6, in
class TSRUrl:
File "C:\Users\danie\OneDrive\Documents\TS4\The-Sims-Resource-Downloader-0.4.0\src\TSRUrl.py", line 16, in TSRUrl
def __getItemId(self, url: str) -> int | None:
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

It does seem like an issue with type annotations on Python 3.8, thankfully there is a module called __future__ which seems to fix the issue.
Branch: https://github.com/Xientraa/The-Sims-Resource-Downloader/tree/Python-Future


Verification error when verifying the fully-specified example program

Hi there,
I guess there is a bug in silicon-gv. I am reporting it because it fails the example program of the paper.
Problem Statement
I tried to install this tool and run the fully-specified example listInsertBug.c0 of insertLast() in Fig. 2 of GC0-24, but gvc0 reports an error in a fold statement:
sbt:gvc0> run src/test/resources/quant-study/listInsertBug.c0
[info] running (fork) gvc.Main src/test/resources/quant-study/listInsertBug.c0
[info] [*] — Mon Jun 17 14:41:39 GMT 2024
[error] Exception in thread "main" gvc.VerificationException: Folding acyclicSeg(curr.Node$next, null) might fail. There might be insufficient permission to access acyclicSeg(from.Node$next, to). (<no position>)
[error] 	at gvc.Main$.verifySiliconProvided(main.scala:405)
[error] 	at gvc.Main$.verify(main.scala:363)
[error] 	at gvc.Main$.$anonfun$run$5(main.scala:175)
[error] 	at gvc.benchmarking.Output$.printTiming(Output.scala:47)
[error] 	at gvc.Main$.run(main.scala:174)
[error] 	at gvc.Main$.delayedEndpoint$gvc$Main$1(main.scala:78)
[error] 	at gvc.Main$delayedInit$body.apply(main.scala:45)
[error] 	at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] 	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] 	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] 	at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at scala.App.main(App.scala:80)
[error] 	at scala.App.main$(App.scala:78)
[error] 	at gvc.Main$.main(main.scala:45)
[error] 	at gvc.Main.main(main.scala)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 4 s, completed Jun 17, 2024, 2:41:41 PM

Similarly, if I get the viper file listInsertBug.vpr by option -d silver and run it in silicon-gv, it still fails for the same reason (shown below). However, this viper file can pass viper-online.
sbt:Silicon> run listInsertBug.vpr
[info] running (fork) viper.silicon.SiliconRunner listInsertBug.vpr
[info] Silicon found 1 error in 2.26s:
[info]   [0] Folding acyclicSeg(curr.Node$next, null) might fail. There might be insufficient permission to access acyclicSeg(from.Node$next, to<EMAIL_ADDRESS>[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 3 s, completed Jun 17, 2024, 3:08:32 PM

My Analysis
In the c0 source file of the example program, function insertLast() contains 3 fold statements after allocating a new node at the end of the list:
//@ fold acyclicSeg(curr->next->next, NULL);
//@ fold acyclicSeg(curr->next, NULL);
//@ fold acyclicSeg(curr, NULL);

I feel it strange because we have curr->next->next == NULL and the predicate acyclicSeg(curr->next->next, null)  should hold. However, the verification complains about the "insufficient permission to acyclicSeg(curr->next->next, NULL)", which fails the fold statement of fold acyclicSeg(curr->next, NULL).
I think the bug comes from silicon-gv, since this example program should be correct; and viper-online can verify it successfully while silicon-gv cannot.
Temporary Fix
I can remove the fold statement: //@ fold acyclicSeg(curr->next->next, NULL);, then both c0 and viper file can pass gvc0 and silicon-vc, respectively.

Plus: My environment is built by Dockerfile, where the tools are in directory /tool in the container.

The root cause of this issue appears to be the same as #62 (refer to my comment there) and #33.
When consuming acyclicSeg(curr->next, null), acyclicSeg(curr->next->next, null) appears in the heap with permission 2.


Missing method GridViewDataSet.LoadFromQuery()

I have been trying run the application few days but a can't fix an error in AppCrudFacadeBase.cs, where in a method FillDataSet is used dataSet.LoadFromQuery(...) which doesn't exist. Maybe I did some mistake, but after 2 days I spent, I'm not able to fix it...

It seems like some packages didn't restore correctly. I have updated the DotVVM package to the latest version, so hopefully it will work for you.

Iťs a little bit confusing... I'm not able to restore Riganti.Utils.Infrastructure.* packages. Nuget packages versions 2.0.1 were not found. But I've tried write my own PoC application, but I can't use a method GridViewDataSet.LoadFromQuery(IFilteredQuery)...

Sorry, we have forgot to publish the 2.0.1 version on a public NuGet feed.
Now it should work.


Write tests for CrystallizedState

Write tests for the CrystallizedState class based on these tests.

Closing since this is no longer relevant.


BUG: Address divide by zero error in SignedMaurerDistance

PR Checklist

[x] No API changes were made (or the changes have been approved)
[x] No major design changes were made (or the changes have been approved)
[x] Added test (or behavior not changed)
[x] Updated API documentation (or API not changed)
[x] Added license to new files (if any)
[x] Added Python wrapping to new files (if any) as described in ITK Software Guide Section 9.5
[x] Added ITK examples for all new major features (if any)

Refer to the ITK Software Guide for
further development details if necessary.

This commit has already been merged into the master branch.

The original PR #2025, had 2 commits. This one and b706cf64c7e10f8985e6fdb18c77b607f1991f75, the latter adding const and fixing the style issues of the linter. Having the same commit in both the release and master makes the history cleaner and merging easier.
I see 2 options to move forward:

Merge as is, causing release to not be completely compliant with style ( but master OK ).
Merge both commits from #2025, including the style/const changes.
Diverge this commit from master, and make the style compliant, causing minor conflict merging release into master.

Please advise.

My preference is to merge both commits.