Training fails with batch size > 1

When I use batch size 1 with multiple TitanX GPUs (4 12GB GPUs), it trains perfectly fine, but when I increase the batch size to 8 or 12 or even 2, training fails with the following trace after loading the weights:
Called with args:
Namespace(batch_size=12, checkepoch=10, checkpoint=0, checkpoint_interval=10000, checksession=20, class_agnostic=False, cuda=True, dataset='kaist', disp_interval=100, large_scale=False, lr=0.001, lr_decay_gamma=0.1, lr_decay_step=15, mGPUs=True, max_epochs=45, net='res50', num_workers=0, optimizer='sgd', resume=False, save_dir='/mnt/nfs/scratch1/dchakraborty/infrared/models', session=10, start_epoch=1, use_tfboard=False)
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [0.05, 0.1, 0.25, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4],
'CROP_RESIZE_WITH_MAX_POOL': True,
'CUDA': False,
'DATA_DIR': '/mnt/nfs/scratch1/dchakraborty/Kaist',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'res50',
'FEAT_STRIDE': [16],
'GPU_ID': 0,
'MATLAB': 'matlab',
'MAX_NUM_GT_BOXES': 30,
'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
'FIXED_LAYERS': 5,
'REGU_DEPTH': False,
'WEIGHT_DECAY': 4e-05},
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'POOLING_MODE': 'crop',
'POOLING_SIZE': 7,
'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '/mnt/nfs/scratch1/dchakraborty',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'MODE': 'nms',
'NMS': 0.3,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': False,
'BATCH_SIZE': 256,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'BN_TRAIN': False,
'DISPLAY': 20,
'DOUBLE_BIAS': False,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res50_faster_rcnn',
'STEPSIZE': [30000],
'SUMMARY_INTERVAL': 180,
'TRIM_HEIGHT': 600,
'TRIM_WIDTH': 600,
'TRUNCATED': False,
'USE_ALL_GT': True,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0001},
'USE_GPU_NMS': True}
Loaded dataset combined_train for training
Appending horizontally-flipped training examples...
wrote gt roidb to /mnt/nfs/scratch1/dchakraborty/Kaist/cache/combined_train_gt_roidb.pkl
before filtering, there are 100368 images...
after filtering, there are 44316 images...
44316 roidb entries
Loading pretrained weights from /home/dchakraborty/.torch/models/resnet50-19c8e357.pth
Traceback (most recent call last):
File "trainval_net.py", line 338, in
rois_label = fasterRCNN(im_data, im_info, gt_boxes, num_boxes)
File "/home/dchakraborty/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/dchakraborty/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/dchakraborty/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/dchakraborty/.local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
raise output
RuntimeError: bool value of Tensor with more than one value is ambiguous
Please help identify the issue if possible. Let me know if you need any more info.

does it work if you try to train with single GPU?
multi-GPU sometimes does not work.

@kentaroy47 Yes, it works on a single GPU, and it also works on multiple GPUs when batch size is set equal to 1 or equal to number of GPUs used. It doesn't work however if I use any batch size which is not in the set {1, nGPU}.

I think the batch size should be a multiplier of nGPU used, or it cannot be
distributed.
2019年3月28日(木) 20:21 Deep Chakraborty<EMAIL_ADDRESS>
@kentaroy47 https://github.com/kentaroy47 Yes, it works on a single
GPU, and it also works on multiple GPUs when batch size is set equal to 1
or equal to number of GPUs used. It doesn't work however if I use any batch
size which is not in the set {1, nGPU}.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jwyang/faster-rcnn.pytorch/issues/446#issuecomment-477554312,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlxuJaWFGhrzb47Tby7gQ1y1bAFj4-wrks5vbKW1gaJpZM4a88bC
.

I'm aware of that, but it doesn't work with any multiplier of nGPU other
than 1. I.e, if I'm using 4 GPUs, I am either being able to use batch size
of 1 or 4, but not anything else - 8,16, etc.
On Thu, 28 Mar 2019, 11:23 a.m. kenken<EMAIL_ADDRESS>wrote:

I think the batch size should be a multiplier of nGPU used, or it cannot be
distributed.
2019年3月28日(木) 20:21 Deep Chakraborty<EMAIL_ADDRESS>
@kentaroy47 https://github.com/kentaroy47 Yes, it works on a single
GPU, and it also works on multiple GPUs when batch size is set equal to 1
or equal to number of GPUs used. It doesn't work however if I use any
batch
size which is not in the set {1, nGPU}.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/jwyang/faster-rcnn.pytorch/issues/446#issuecomment-477554312
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AlxuJaWFGhrzb47Tby7gQ1y1bAFj4-wrks5vbKW1gaJpZM4a88bC
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jwyang/faster-rcnn.pytorch/issues/446#issuecomment-477643101,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF7tIBsB5fpmCdcklST23X6iiHotqMjNks5vbN6FgaJpZM4a88bC
.

I am also using single GPU and when i set batch size greater than 1 it gives error. While training with batch size 1 there is no problem. But it takes lot of time to train.

@AtriSaxena
You should be able to set to a larger batch size (if your GPU has enough memory).
What kind of error do you get?

I solved the problem. I was using Cuda10 with pytorch 1.1. I was getting some dimension error in minibatch.
When i switched to pytorch 1.0.0 problem is solved.

when i kept the training with batch-size 2 for my own dataset then my system was restarted ?
any idea , please help .


Get rid of less files etc in /static

/static is pretty heavy data wize -- you should consider to get the CSS only dist version of Font Awesome etc.

I stripped the assets down the the minified Font Awesome stylesheet.


How can i add it in my laravel project

I need to add the vuex-search in my Laravel project.

I don't think it's relevant to vuex-search specifically. Though I'm not familiar with Laravel, I think you should look for how to use npm packages inside a Laravel project instead.
Please reopen if I miss something.


How to create rounded edges for my images

Hello, I can't find a solution to create rounded edges for my images. I work with node js, jspdf , sharp and other libraries. I don't use html page. I have already looked at the previous answers on the subject but it does not work anymore.

const rect = '';
      sharp('./uploads/1669196401950.jpg')
        // .resize(200, 200)
        .composite([{ input: new Buffer.from(rect), gravity: 'northwest'}])
        .toFile('565.png', function(err, info) {
          console.log(info);
        });

I found the solution. Here is my method:
1: install safe buffer
install "safe buffer
2 code
const rect = '<svg><rect x="0" y="0" width="1000" height="1000" rx="50" ry="50"/></svg>';

          sharp('./uploads/1669196401950.jpg')
            // .resize(200, 200)
            .composite([{ input: new Buffer.from(rect), gravity: 'northwest', blend: 'dest-in'}])
            .toFile('565.png', function(err, info) {
              console.log(info);
            });

Const rect represents the vector shape that will be used to make an exclusion with the image. rx and ry are the value of the rounded edges.
.composite will inform the vector shape, the position, and the most important element the type of fusion. Here an exclusion.
It is probably possible to make a cleaner code if you have any question don't hesitate.


StackSetsResource: Pass Role Error on update_stack_set

When using custom admin/execution roles there are errors updating stack sets with a PassRole permissions error. The solution to the issue is updating the update_stack_set method signature to include the admin role and execution role and follow the same strategy as create_stack_set. Attached is the code I updated to make it work in my account.
lambda_function.py.zip

Cleaning up stale issues in preparation for refreshing this repository. Please re-open if this still needs to be addressed.


Various refactorings and tweaks to the steps

The UPDATEREPOS change should be the only significant change in behaviour.

what was the _ prefix meant to be? to make clear it's an alias not for external use?
and why do we drop it now?
and why do _upgrade and _testupdate still have it?

what was the _ prefix meant to be? to make clear it's an alias not for external use?

Yes, see #319.

and why do we drop it now?

Because it could be useful to call from the CLI.

and why do _upgrade and _testupdate still have it?

Because noone yet suggested it would be useful to call those from the CLI.

+-1  :)
In general i like the idea. What I don't like is that we introduce even more names which do not clearly state if they are expanding from the beginning up to a certain point or if they are expanding just some intermediate steps.
Currently its inconsistent as well, but we only have: instonly, plain and all.
Just a dumb idea: What about making the names of the steps upper, that start at the beginning:
all -> All
instonly -> Instonly
plain -> Plain
blank_admin -> Blank_admin
crowbar_admin -> Crowbar_admin
updated_crowbar_admin -> Updated_crowbar_admin

Other ideas welcome (even other names).

@jdsn I agree it's a mess, and this PR doesn't totally clean it up, but I think it is several steps (pun intended) in the right direction.  I'm not sure about the uppercase idea - it would help a bit, but it would also break all the existing Jenkins scripts, and it's not a full solution.  The problem is clearly expressing the tree of step expansion, and this could probably be done better by defining the aliases in a separate config file which we could also parse and feed to graphviz or similar to draw a tree showing how all the expansions happen.  For example
digraph g{
  node [shape=ellipse,fillcolor="burlywood",style="filled"];
  plain -> { instonly proposal }
  instonly -> { updated_crowbar_admin nodes }
  updated_crowbar_admin -> { blank_admin updates crowbar }
  crowbar -> { prepareinstcrowbar instcrowbar }
  blank_admin -> { cleanup prepare setupadmin }
  updates -> { addupdaterepo runupdate }
  nodes -> { setupnodes instnodes}
  cleanup [shape=box,fillcolor="palegreen",style="filled,rounded"];
  prepare [shape=box,fillcolor="palegreen",style="filled,rounded"];
  setupadmin [shape=box,fillcolor="palegreen",style="filled,rounded"];
  addupdaterepo [shape=box,fillcolor="palegreen",style="filled,rounded"];
  runupdate [shape=box,fillcolor="palegreen",style="filled,rounded"];
  prepareinstcrowbar [shape=box,fillcolor="palegreen",style="filled,rounded"];
  instcrowbar [shape=box,fillcolor="palegreen",style="filled,rounded"];
  setupnodes [shape=box,fillcolor="palegreen",style="filled,rounded"];
  instnodes [shape=box,fillcolor="palegreen",style="filled,rounded"];
  proposal [shape=box,fillcolor="palegreen",style="filled,rounded"];
}

gives:

and I think that's a far more valuable way of documenting the steps.

Introducing the uppercae names does not mean to break the lowercases as well.
The lowercases can have a prominent warning message that they will be dropped, and we can change the jenkins jobs after the merge without to hurry.
If there are other solutions than uppercase names I'm fine with it. But having to search for the latest graph whenever I'm unsure is more work than grep or "/string" in vi.

Introducing the uppercae names does not mean to break the lowercases as well.  The lowercases can have a prominent warning message that they will be dropped, and we can change the jenkins jobs after the merge without to hurry.

But having to search for the latest graph whenever I'm unsure is more work than grep or "/string" in vi.

I'm not proposing having a fancy graph instead of useful CLI help text.  I'm proposing having it in addition.  Like I said, if we defined the aliases in a separate config file, then mkcloud -h or mkcloud steps or mkcloud help $alias or whatever could easily print on STDOUT what the step expansions are.
And while we're at it, we should replace the big case in expand_steps with an associative array.

BTW what did you mean by "search for the latest graph"?  Surely the docs would always be in docs/mkcloud.md?

No. But this PR is currently adding to the confusion around steps and aliases (and full aliases = expanding to start with cleanup)

I'm proposing having it [the graph] in addition.

OK, thats fine.

And while we're at it, we should replace the big case in expand_steps with an associative array.

Go ahead and make a PR for it :)
@aspiers ok, so what is your answer concerning the "do not add more aliases (and full aliases) that could be mixed up with steps"?
I really hesitate to merge this as it makes things worse. We should work on making in simpler, more intuitive. Bernhard has a nice idea and will add it in a minute.

my proposal for naming: use the step-names we already have, like allocate and have an alias like "all_up_to_allocate" that includes the steps before (and maybe the step itself).

@jdsn Why does it make it worse?  If because it introduces more aliases without improving the documentation system then maybe I agree.  But that's not a big difference, because the aliases are already hard to understand, and if we fix that properly then it will be fixed for all of them (see my proposal below).
Also I'm not sure why you are worried about confusion between steps and aliases?  The user does not care whether it's a step or alias, they only care what work gets done, and that can be solved by properly documenting steps and aliases.  An auto-generated graph (or ASCII representation) goes a long way towards that, but we can also document things like "blank_admin creates a fresh admin node from scratch, ready to have Crowbar installed".
@bmwiedemann wrote:

my proposal for naming: use the step-names we already have, like allocate and have an alias like "all_up_to_allocate" that includes the steps before (and maybe the step itself).

It sounds like you are assuming that there is only a need for a) atomic steps and b) aliases which start from the beginning (i.e. cleanup), in other words that the expansion tree is only allowed to be two levels deep (aliases at the top level, and steps below it).  If you look at the tree above, I think it's pretty clear that's insufficient, and in fact even without this PR, the code already uses an expansion graph which is 3 levels deep, e.g. all expands to _new_admin ... and _new_admin expands to 7 steps which have 5 steps in common with the expansion of instonly.  This is one of several DRY violations in the current code.
What we really need aliases for arbitrary sets of steps/aliases via recursive expansion:

it makes the definition of the higher level aliases more concise
it avoids DRY violations in alias definitions
it caters for many use cases where the user wants to do multiple steps without starting right at the beginning

Think of a normal, well-written program which a bunch of functions/methods, with a main() entrypoint.  Does main() call every single other function/method directly, so that the call graph is only a single level deep?  No, of course not.  Instead the sophistication is built up compositionally, starting with small building block functions which then get called by other functions, which then get called by other functions, and so on.  The same applies here, except we are dealing with aliases and steps instead of function/method calls.
I am proposing something like an aliases.yaml file, e.g.
- alias: plain
  steps: instonly proposal
  description: Installs a simple cloud from scratch.
- alias: instonly
  steps: updated_crowbar_admin nodes
  description: Installs all Crowbar nodes from scratch, ready for proposals to be applied.

and so on.  Then it would be easy to auto-generate the above graph via dot(1) and also some ASCII representation for quick reference via mkcloud steps or similar if a graphical environment is not immediately available.  And of course it would be easy for the code to parse it in order to perform the expansion.

@aspiers needs rebase

@toabctl I'm still waiting for replies from @jdsn and @bmwiedemann on the design discussion before spending more time on this.  But perhaps I should take a leaf out of my own book and schedule a mumble meeting about this, since we've already seen how design discussions in PRs are doomed to failure.

@aspiers yes lets mumble, when?

@jdsn Next week please.

@jdsn Please can you tell / remind me which parts of this proposal you still disagree with, bearing in mind my responses to your earlier comments? Thanks!

please reopen if this is still relevant (and it requires a rebase)


Fix description breaking when it's None

👌

I just realized this doesn't work. I'll create a fix for this soon.


Update README.md

In this project I followed a react tutorial

Status: REQUIRED CHANGES ♻
Hi @obed300 👋 👋
Good job so far! 🥇
There are some issues that you still need to work on to go to the next project but you are almost there!
Highlights
✔ Professional PR. 👍
✔ No linter errors👍
✔ Awesome design👍
✔  Repo looks professional :briefcase:
Required Changes ♻

[ ] kindly As well-documented and licensed Readme file but you set up and some links are for math-magician links . ✨✨

Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.
As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Thank you. I will do the necessary

Hi @obed300
I'm Yemisirach your reviewer for this PR. :hand:
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉

Highlights
✔ Professional PR. 👍
✔ No linter errors👍
✔ Awesome design👍
✔  Repo looks professional :briefcase:
Optional suggestions
Please notice these are mandatory requirements and are only optional because it has not been mentioned to you as mandatory requirement previously.

[Optional] Please consider updating the Link to your MIT file, 🛑 This is one of the main requirements for a professional Readme file.
[Optional] Please remove the live demo link from your readme file. It's mandatory to delete the live demo section if you decide not to deploy the application.

Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Experiment with using libgraphqlparser for query validation

Currently a bottleneck in performance is query validation using graphql-core, which can consume up to 75% of execution time for the synthetic query given below. Without requiring a larger internal refactor, a potential exists to increase runtime performance by using another library to perform only the validation step of the query, and still rely on graphql-core for performing document parsing / response creation. Preliminary benchmarks reveal that anywhere between a ~2-5x speed-up in validation performance can be achieved using this approach. Using different bindings this performance is expected to be even higher, see disclaimer below.
Validation with graphql-core

Validation with libgraphqlparser

Benchmark Code
from __future__ import annotations

from timeit import timeit

from pyinstrument import Profiler

import strawberry
from strawberry.types.info import Info

@strawberry.type
class Query:
  @strawberry.field
  def foo(self, info: Info) -> None:
      return None

for label, use_libgraphqlparser in {
  "libgraphqlparser": True,
  "graphql-core": False,
}.items():
  schema = strawberry.Schema(Query, use_libgraphqlparser=use_libgraphqlparser)
  profiler = Profiler(interval=0.0001)
  profiler.start()
  N = 10000
  duration = timeit(
      lambda: schema.execute_sync("query { foo }", root_value=Query()),
      number=N,
  )
  profiler.stop()
  with open(f"{label}.html", "w") as f:
      f.write(profiler.output_html())

Description
This commit adds a new use_libgraphqlparser argument to strawberry.Schema that parses queries using the CFFI bindings to libgraphqlparser created by the developers of tartiflette
Disclaimer: This is a quick and dirty proof of concept and not intended to be representative of the desired implementation which would not dynamically link to libgraphqlparser using CFFI. Further experimentation is required into generating bindings to relevant portions of libgraphqlparser using pybind11, and compare that to using pyo3 with apollo-rs. As pointed out by @patrick91 during the discussion on Discord, apollo-rs would probably recieve quicker updates to new GraphQL specifications than libgraphqlparser.
Types of Changes

[ ] Core
[ ] Bugfix
[ ] New feature
[x] Enhancement/optimization
[ ] Documentation

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Closing in favor of using apollo-rs. See rustberry for more details


Incremental pca

closes #617
Most part of the script is stolen from scikit-learn but updated to be compatible with dask-arrays.
Tests are also copied.

I apologize for the delay @fujiisoup .  Many of the Dask maintainers are at a workshop this week.
At first glance I am surprised that the Scikit-Learn code works well with small modifications.  I haven't looked at it deeply yet, but I have a few questions:

Could we use the Scikit-Learn estimator and the general approach for partial_fit?
How does this perform?  In particular, if it is easy for you i would love to see a performance report.  I think that this would help us to talk about bottlenecks that might exist in this algorithm.  https://docs.dask.org/en/latest/diagnostics-distributed.html#capture-diagnostics
I'm curious about when the da.linalg.svd_compressed algorithm  in dask.array might be a better fit, particularly with the compute=True keyword.

Again, I apologize for the delay, and for not yet giving this a thorough review

Thank you for the comments ;) and sorry for my late reply.
We tested this PR with our real data and found some problems.
Now we are fixing them.
We are also preparing some benchmarking of our PR.
I'll push and post the update (probably later this week or the next week.)
@yasahi-hpc

Thanks. I marked the updated part from scikit-learn by comment.
We will make sure its performance, and report it back!

Hello, I have made a performance analysis with the performance_report and a naive timer. The testbed is a Intel Skylake CPU. Using the incremental PCA to our large data ~20GB, it gave the dask-report for the fit method . For some reason, if we call the method under performance_report, it gets 50 times slower.
A naive timer gave 184s, 148s and 63s for incremental_mean_and_var, svd, and compute(), respectively. For the naive measurement, we deliberately compute() all the quantities for synchronization. Other functions are immediately finished.

For some reason, if we call the method under performance_report, it gets 50 times slower.

Hrm, that is concerning.  Thank you for reporting this.  I don't immediately see why.  I raised https://github.com/dask/distributed/pull/3654 to make sure that we are not counting the time taken to generate the plots.
However after I looked more closely at your performance report I see that this is not the problem.  I see that the computations are actually taking many hours, which is surprising. There are tasks like mean_agg-aggregate and getitem which take 60 seconds, which is very surprising.  Computing means and getting slices should be very fast.  I do not know why these would be slow, or why running your code with the performance_report context manager would make things slow.
The performance report code doesn't run during execution, it only collects data afterwards.
When I look at the administrative profiles, which observe Dask itself (not the numpy/xarray code) I don't see anything that would cause this.  I do however see that it is taking some time to read excess data from disk (you are maybe running out of memory?)  But this adds only a few minutes of total execution time across all workers.
In summary, I am confused :/

@fujiisoup apologies for the CI failures. If you merge master they should be resolved.

@mrocklin
Thanks so much for your detailed explanation and help.
I am also sorry for my late reply. Our cluster has been unavailable for a week just until today.
I guess some of the issues stem from the Lustre file system of our cluster.
(Still not very clear, why the problem occurs only with the performance_report.)
The memory shortage may be a thing since there is a warning message implying a memory leak (the message appears when creating a Client object).
In order to focus on the performance of Dask itself, I have simplified the script with the smaller input data size (~7.5GB).
I no longer use xarray/numpy to read our actual input data, rather I use the mimic data generated by dask.random.random. In addition, it turned out that inserting the line dask.config.set({'temporary_directory': './tmp'})  to avoid creating the scratch directory gives a significant performance improvement.
In the end, the script runs three times slower than without performance_report.
The new performance report seems to show more reasonable performance. Unfortunately, I am not sure why adding the performance_report changes the performance.


ARROW-5863: [Python] Use atexit module for extension type finalization to avoid segfault

As reported on JIRA, the following script provokes a segfault
#! /usr/bin/env python

import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

+1
Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

I opened https://github.com/cython/cython/issues/3039

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?
https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67
The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

Updated the comment here.


ESLint 9 upgrade

:clipboard: Description
This project currently uses ESLint 8.
ESLint 9 makes a big breaking change requiring the move to flat config.
For a simple project, this migration is not a huge deal. For example, I migrated most of my projects to ESLint 9 a while back. Here's a couple of examples of what the commits migrating the config looked like:

https://github.com/chris48s/jq-playground/pull/4/commits/d4c38a0f225905cdedca08242d1911c0031575d0
https://github.com/chris48s/unfurl-preview/commit/68a87f030762c2c5632dd04f010170ecdc58ffba

Shields is a bit more of a complicated beast though, for two reasons:

The biggest issue is that Shields uses a lot of ESLint extensions. In order to migrate to ESLint 9, all the plugins and configs we use need to be compatible with ESLint 9/Flat config. Even if a lot of the extensions we use have versions which are compatible, if we are using some plugins that aren't yet compatible with ESLint 9 they may prevent us from upgrading, or we might need to look at replacing them.
Shields has quite a big ESLint config https://github.com/badges/shields/blob/master/.eslintrc.yml - there is certainly more to it than the relatively simple examplesI posted above. Migrating it to the new format will be a bit more of a job, but we'll get over it.

We are now starting to see some ESLint plugins we use which now require ESLint 9. For example eslint-plugin-cypress requires ESLint 9. We're also starting to see unresolvable peerDependency conflicts with some NPM plugins.
I think the first step is to go through this list of all the eslint- packages we rely on, work out which ones do/don't have compatibility with flat config and see what the gaps are:

eslint-config-prettier
eslint-config-standard
eslint-config-standard-jsx
eslint-config-standard-react
eslint-plugin-chai-friendly
eslint-plugin-cypress
eslint-plugin-icedfrisby
eslint-plugin-import
eslint-plugin-jsdoc
eslint-plugin-mocha
eslint-plugin-no-extension-in-require
eslint-plugin-node
eslint-plugin-promise
eslint-plugin-react
eslint-plugin-react-hooks
eslint-plugin-sort-class-members

Then we can work out how many blockers we have. I don't really have a vibe for how big the gap is yet.
Annoyingly, I already know about one of the problem children here. eslint-plugin-icedfrisby does not yet have a release which is compatible with flat config, and it is maintained by.. me :smile: I've been ignoring it due to https://github.com/bmish/eslint-doc-generator/issues/526 but it is probably time for me to at least revisit the topic on that repo.

Having spent a bit of time on this, the situation is not too bad.
Most of the plugins we use do have a version that supports flat config and either we're already using it or we can upgrade to it.
The exceptions are:

eslint-config-standard
eslint-config-standard-jsx
eslint-config-standard-react

issues:

https://github.com/standard/eslint-config-standard/issues/410
https://github.com/standard/eslint-config-standard/issues/411

The recommendation seems to be to migrate to https://github.com/neostandard/neostandard for those 3
eslint-plugin-no-extension-in-require is not compatible but also now that we've almost completely switched to ESM we can just get rid of this with basically no impact. We have so few require() statements left its just not an issue really.
eslint-plugin-react-hooks does have a compatible release but also there is a really long issue about it https://github.com/facebook/react/issues/28313 which makes me think this one might be a case where its a bit more complicated than that we'll have to see.
..and then we've got eslint-plugin-node which I think is not compatible. But also having looked at out eslintrc, it doesn't look like we are actually using any of the rules or configs from it. I think this was something we were using in the past but are not any more so I think we could just ditch this one with no impact.
So although there are some bits of the jigsaw that need putting together, I don't think there is anything which represents a hard blocker there :crossed_fingers:

we could just ditch this one with no impact

Are there any other ones we could ditch? Our eslint config has always felt very complex and obscure to me, I'd be all for simplification... :)

I'm currently knee-deep in an eslint.config.js with a crapload of TODO comments in it :sweat_smile:
I agree that our eslint setup is pretty fiendish.
I think there are several things going on here:

Our codebase has several distinct entities in it. The badge server, the frontend, the NPM package. Each of these has slightly different characteristics. The badge server is a node app and uses ESM. The frontend is a docusaurus/react app that runs in the browser. The NPM package is node and still uses CommonJS.
Also, we are using several distinct test frameworks in different places for different reasons. We use mocha for the core tests. The service tests use IcedFrisby. We have end-to-end tests written in cypress, etc.
All of that means that

We need (or at least choose to use) a lot of plugins
We need to selectively use certain bits of config in certain places

With some simpler projects you can make a statement about the codebase as a whole like "we use react" or "the tests are written in mocha". The situation with all that stuff in this codebase is "its complicated" so there's lots of things that don't make sense to do globally. As such, we've got an ESLint setup that accounts for all that. e.g: Allow browser globals in frontend/**/*.js but node globals elsewhere. Apply mocha lint rules in **/*.spec.js but not in services/**/*.tester.js and so on ad infinitum.
So I think in that respect our ESLint config is quite complicated, but a lot of it is there for sensible reasons.

Our ESLint setup has evolved over a period of many years. There are certain bits of cruft we have accumulated over time that may now be obsolete that we could possibly remove or simplify.

There are certain plugins or configs we adopted at a point where we already had a lot of code in place that maybe violates one or two of the recommended rules. In order to adopt a config without having to make massive changes everywhere, there are certain places where we had (or chose) to adopt a subset of rules in a more customised configuration rather than just go with the default.

So I think there are probably some opportunities to do a bit of cleanup and I will try and do this as part of the process, but I think on the whole we are probably still going to end up with quite a complicated config and a lot of that is just for valid reasons.


Add ability to correlate Resource pool to vSphere cluster in node templates for VMware

What kind of request is this (question/bug/enhancement/feature request):
Enhancement
Steps to reproduce (least amount of steps as possible):
Rancher 2.3.3
It would be useful when specifying a Resource pool in VMware node template, to be able to identify to which vSphere cluster it maps to. An idea to improve this would be to provide full path of the cluster or show next to the dropdown menu a label with the name of the cluster.
Currently if there are N numbers of vSphere clusters in VMware, there will be N default values of Resources values in the dropdown menu for Resource Pools in the node template in Rancher, making it hard to identify the right vSphere cluster.
This can be even harder if the vSphere clusters don't have any resource pools defined(flat view). In that case the Resource Pool dropdown menu will only show Resources values.
One would have to either keep trying each Resources value and confirm in the API view, or just edit the node template in API directly, which is not a preferable method from user experience perspective.

A screenshot from v2.3.6:

A screenshot from version 2.3.6:

In a response to Rancher UI XHR request to vCenter, vCenter sends a list of values like "//host//Resources", and the total number of values equals the number of hosts in vCenter.

This is a serious issue for people with multiple vsphere cluster, but no RessourcePool.
This bug prevent from choosing the appropriate cluster to create nodes :-(


Mic is unmuted even though "Mute on startup" is enabled

I updated MicMute yesterday (not sure which version I came from - but it was most likely months old) to 1.2.2 and now the mic is enabled after a reboot even though I have enabled "Mute on startup". At first I thought this might be because in the new version the app now start on startup using Task Scheduler, but I have tested by going back to having a shortcut in shell:startup without seeing any difference. I have also disabled and re-enabled "Mute on startup" to see if there was some kind of syncronization issue with the settings. This did not help either.

I just now removed config.json but the issue remain.

Can you post the contents of your config.json file?
Also could you post a screenshot of the logs window?
To open it, just right click the MicMute icon in the taskbar and then while holding Shift on your keyboard, click on Help.

Commit 0fdc2a3 should fix this.
If you want, you can test the development build, or as a workaround, you can temporarily select your microphone in the config window instead of the default option


GPS SV05 update info

B1062.2 static fired yesterday at 2021-06-13 19:30Z
https://twitter.com/cbs_spacenews/status/1403797886853390341?s=20
It will land on JRTI.

Window is from 2021-06-17 16:09Z to 2021-06-17 16:24Z, or 900 seconds.

HOS Briarwood departed (it's a temprary contracted ships for fairing recovery). Also JRTI is towed by Finn Falgout. As of now, no info about if fairings are brand new or reused.

Booster status has to be updated: now it's unkown, but it's for sure active

Fairings will be brand new. SpaceX published the patch.

Both fairings were recovered.


add banner of crosswoc to the readme file

I will add banner of crosswoc to the readme file so that, People will know it is associated to crosswoc.
Moreover, I will associate link of the crosswoc website to its banner such that, if someone clicks the banner on readme file, it will redirect them to the website of crosswoc.

You have to banner and make a redme good see in link

I just followed you

ok

if you don't need any further modifications, this issue can be closed after merging the above pull request

Remove that crosswoc banner other is good

hahaha,
matlab jiss kaam ke iye issue banaya wahi hata doon?
okay no problem

I have updated,
Please have a look once again

Yes I don't think its important but you have done good

I have done the required changes,
Please have a look

Why aren't you merging the PR?

Can I close this issue now? or you need it?


[WIP] Add API executeStatement

Add function executeStatement Sesssion and SessionPool, referring to java.sql.Statement#execute(java.lang.String).
Need help to point out how to test.
Ticket link
https://issues.apache.org/jira/browse/IOTDB-1088

Coverage increased (+0.01%) to 67.242% when pulling b66002d6b658547680c6064e5854558f359ff728 on LeicongLi:executeStatement_api into 9931149fb2e8e7ddec8187539d9940869edd1a7a on apache:master.

Need help to point out how to test.

Thanks for your contribution!
To add test cases, you can see the following IT classes.
https://github.com/apache/iotdb/blob/master/session/src/test/java/org/apache/iotdb/session/IoTDBSessionSimpleIT.java
https://github.com/apache/iotdb/blob/master/session/src/test/java/org/apache/iotdb/session/pool/SessionPoolTest.java

Hi, I checked the execute(String sql) method on IoTDB JDBC. It seems that it keeps a static ResultSet in this class.
I think we should do the same thing.
https://github.com/apache/iotdb/blob/3333f3b2a6a5035b7f42a33dd23b3e841b0768b1/jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java#L251-L287
I also checked the clickhouse and shardingsphere JDBC code and they all keep the ResultSet.

Hi, I checked the execute(String sql) method on IoTDB JDBC. It seems that it keeps a static ResultSet in this class.
I think we should do the same thing.
https://github.com/apache/iotdb/blob/3333f3b2a6a5035b7f42a33dd23b3e841b0768b1/jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java#L251-L287
I also checked the clickhouse and shardingsphere JDBC code and they all keep the ResultSet.
IOTDB-1088 also states
You must then use the methods * getSessionDataSet to retrieve the result.

Therefore, I think we also need a method getSessionDataSet.

Thanks.
Will reopen the PR when the changes are made.


Why not use built in two way binding?

Out of curiosity, why didn't you just leverage the native two-way binding on an input field?

Hello there,
sorry for the late response.
I need to watch for changes inside the jQuery Plugin by using its 'change' method because I didn't got it to work in any other way.
Also there were issues when I wanted to set an initial value, but I can't really remember why exactly I did it the way I did it.


AP-WF1E HVAC not loaded anymore in integration when logging to FGLair.

System Health details
I have two Fujitsu HVACs, one that prefix AC-UTY and another that prefix AP-WF1E. Both were working perfectly under FGLair HACS Custom Integration since I have integrated them with Home Assistant. However, I have just noticed that the AP-WF1E device was not taking any changes made to the controls. I tried to delete the integration load it back but its now only loading my AC-UTY unit; not the AP-WF1E unit.
This is what i get in the log detail:
Logger: homeassistant.components.climate
Source: helpers/entity_platform.py:728
integration: Climate (documentation, issues)
First occurred: 10:20:55 AM (4 occurrences)
Last logged: 10:47:41 AM
fglair_heatpump_controller: Error on device update!
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 728, in _async_add_entity
await entity.async_device_update(warning=False)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1302, in async_device_update
await self.async_update()
File "/config/custom_components/fglair_heatpump_controller/climate.py", line 426, in async_update
self._fan_mode = self.fan_mode
^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/climate/init.py", line 324, in getattribute
return super().getattribute(__name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/fglair_heatpump_controller/climate.py", line 466, in fan_mode
DICT_FAN_MODE[self._fujitsu_device.get_fan_speed_desc()],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pyfujitsugeneral/splitAC.py", line 213, in get_fan_speed_desc
return FAN_SPEED_DICT[fan_speed]
KeyError: 6

Does anybody have this issue?

### Checklist

- [X] I have enabled debug logging for my installation.
- [X] I have filled out the issue template to the best of my ability.
- [X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
- [X] This issue is not a duplicate issue of currently [previous issues](https://github.com/bigmoby/fglair_for_homeassistant/issues?q=is%3Aissue+label%3A%22Bug%22+)..

### Describe the issue

I have two Fujitsu HVACs, one that prefix AC-UTY and another that prefix AP-WF1E. Both were working perfectly under FGLair HACS Custom Integration since I have integrated them with Home Assistant. However, I have just noticed that the AP-WF1E device was not taking any changes made to the controls. I tried to delete the integration load it back but its now only loading my AC-UTY unit; not the AP-WF1E unit.

### Reproduction steps

1.In Device/Integration, in FGLair heat pump controller integration, delete entries
2.Enable integration debugging
3.Add Entry; Email, Passwd, Region (US) of your FGLair account and keep default for other parameter then submit
4.Success; Created configuration; Finish
5.It has only created one entity for my AC-UTY HVAC; No Entry was crated for my AP-WF1E HVAC
6.Pull the log from Home assistant Core for FGLair.

### Debug logs

```text
Logger: homeassistant.components.fujitsu_fglair.coordinator
Source: helpers/update_coordinator.py:382
integration: FGLair (documentation, issues)
First occurred: November 27, 2024 at 10:41:33 AM (10 occurrences)
Last logged: November 27, 2024 at 3:53:18 PM

Unexpected error fetching Fujitsu HVAC data data
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 382, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/fujitsu_fglair/coordinator.py", line 46, in _async_update_data
    devices = await self.api.async_get_devices()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ayla_iot_unofficial/ayla_iot_unofficial.py", line 289, in async_get_devices
    devices.append(FujitsuHVAC(self, d, europe=self.europe))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ayla_iot_unofficial/fujitsu_hvac.py", line 98, in __init__
    raise DeviceNotSupportedError("This device is not supported by FujitsuHVAC.")
ayla_iot_unofficial.fujitsu_hvac.DeviceNotSupportedError: This device is not supported by FujitsuHVAC.

Logger: homeassistant.components.climate
Source: helpers/entity_platform.py:728
integration: Climate (documentation, issues)
First occurred: November 27, 2024 at 10:20:55 AM (5 occurrences)
Last logged: November 27, 2024 at 4:01:59 PM

fglair_heatpump_controller: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 728, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1302, in async_device_update
    await self.async_update()
  File "/config/custom_components/fglair_heatpump_controller/climate.py", line 426, in async_update
    self._fan_mode = self.fan_mode
                     ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/climate/__init__.py", line 324, in __getattribute__
    return super().__getattribute__(__name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/fglair_heatpump_controller/climate.py", line 466, in fan_mode
    DICT_FAN_MODE[self._fujitsu_device.get_fan_speed_desc()],
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyfujitsugeneral/splitAC.py", line 213, in get_fan_speed_desc
    return FAN_SPEED_DICT[fan_speed]
           ~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 6
```

### Diagnostics dump

_No response_

Hi @OlivierBlain, unfortunately, I can’t quite understand whether it’s truly an issue related to the integration or not. Could you please let me know if the problem persists when downgrading the library?

Mine too has stopped working since the upgrade. Downgraded and now it still won't work :(
Also doesn't work with the official integration something about doesn't accept HVAC.
Hase been working flawlessly for years since last update though!

Hi guy, If it had been an issue caused by an integration bug, a downgrade would have fixed the problem. Don’t you think? Have you tried installing two or three versions back? And with each new installation, did you restart Home Assistant?!

I have the same, it started after I had to updates on Home Assistant itself, the core and the os?

updates on Home Assistant itself

oook, so not the custom integration?!

I actually just updated home assistant to the december update 12.0 and operating system to 14. I did not see the error come up after updating to the december update. I saw it come after updating the operating system to 14

I downgraded home assistant to version 2024.11.3 and everything works fine again

Same here. not working anymore

Guys, I apologize but in this issue that @OlivierBlain opened, things have become completely unclear! The logs posted are apparently two completely unrelated ones: the first one refers to the integration that has been in the Home Assistant core for some months now:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 382, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/fujitsu_fglair/coordinator.py", line 46, in _async_update_data
    devices = await self.api.async_get_devices()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ayla_iot_unofficial/ayla_iot_unofficial.py", line 289, in async_get_devices
    devices.append(FujitsuHVAC(self, d, europe=self.europe))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ayla_iot_unofficial/fujitsu_hvac.py", line 98, in __init__
    raise DeviceNotSupportedError("This device is not supported by FujitsuHVAC.")
ayla_iot_unofficial.fujitsu_hvac.DeviceNotSupportedError: This device is not supported by FujitsuHVAC.

I consider this unrelated to the support for my custom integration. The second log:
fglair_heatpump_controller: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 728, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1302, in async_device_update
    await self.async_update()
  File "/config/custom_components/fglair_heatpump_controller/climate.py", line 426, in async_update
    self._fan_mode = self.fan_mode
                     ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/climate/__init__.py", line 324, in __getattribute__
    return super().__getattribute__(__name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/fglair_heatpump_controller/climate.py", line 466, in fan_mode
    DICT_FAN_MODE[self._fujitsu_device.get_fan_speed_desc()],
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyfujitsugeneral/splitAC.py", line 213, in get_fan_speed_desc
    return FAN_SPEED_DICT[fan_speed]
           ~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 6

is of a completely different nature and has nothing to do with the issue title AP-WF1E HVAC not loaded anymore in integration when logging to FGLair because this second log shows that the air conditioner has been correctly integrated, but is encountering an exception because it's unable to map the correct value returned by the API (specifically 6).
Now, there is a suggestion that it might be related to the Home Assistant 2024.12.0 release, which could indeed be the case... I see, from quickly reading the changelog, that this PR was merged: https://github.com/home-assistant/core/pull/125578... it could be... I'm not sure at the moment.
I kindly ask you to open issues on GitHub in a clear and detailed manner, following the proposed template.
Thank you for your collaboration,
Fabio


mdict supports duplicate keys

mdict supports duplicate keys, but the python standard dict doesn't. a list with tuple elements in the demo script.

Dear @seaspeak, could you reopen this issue, please, or explain how did you solve the problem? I have a dictionary with dozens or duplicate keys.

Is it enough to change one line?
--- writemdict.py
+++ writemdict.py
@@ -278,7 +278,7 @@
 		#  e.record_null: encoded version of the record, null-terminated
 		#
 		# Also sets self._total_record_len to the total length of all record fields.
-		items = list(d.items())
+		items = d.copy()
 		items.sort(key=operator.itemgetter(0))

 		self._offset_table = []


Can this php debugger work with apache24 and php7 on a windows 10 PC (i.e. can it debug laravel)

Can this debugger can be used on a pc with a laravel app running via apache?
If so, how does one setup apache and/or PHP?

This is an issue tracker, not a support forum. Please ask general questions on StackOverflow or Gitter.


Support squoosh

squoosh is a wasm alternative to sharp. While sharp is much faster and supporting modern formats, it has its own downsides:

NodeJS dependency. Cannot be used in environments like Deno or Cloudflare workers
Native dependency. This reduces the portability of the library. Both bundle distribution and target-machine (if there is no pre-built version for a specific node version, installation requires building with node-gyp) (example https://github.com/nuxt/image/issues/204)

We can keep sharp as an opt-in dependency and install/use squoosh by default. This needs a refactor for handlers to create a shared interface. (as a result, we lose some existing features)
Note: squoosh api seems is not released on npm nor documented! We might use/redistribute http://npmjs.com/package/@squoosh/cli, inspire and inline source similar to https://github.com/vercel/next.js/pull/22253, or publish as squoosh.
Related: https://github.com/GoogleChromeLabs/squoosh/issues/934 / https://github.com/GoogleChromeLabs/squoosh/pull/1002 (PR depends on NodeJS API like worker_threads therefore removing first point)

This looks like a cool idea! I expect that @squoosh/api will be released on npm in a few days, but it will only be an experimental release. It has a list of known issues, it has spotty documentation, and the API isn't finalised yet.
I would recommend keeping sharp as the default backend, but adding squoosh as an option and letting the end user choose which one to use. Then when the squoosh API hopefully one day becomes more stable, you can consider making it the default.

Awesome to see this. Just want to add that this will also enable full compatibility inside of WebContainers as well (more info in the blog post / repo

const fs = require('fs/promises');
const { ImagePool } = require('@squoosh/lib');
const { cpus } = require('os');

const imagePool = new ImagePool(cpus().length);

export async function convert2WebpAndAvif(originalImage: Buffer) {
  const image = imagePool.ingestImage(originalImage);

  const encodeOptions = {
    webp: {},
    avif: {},
  };

  await image.encode(encodeOptions);

  const [webp, avif] = await Promise.all([
    image.encodedWith.webp,
    image.encodedWith.avif,
  ]);
  return {
    webp,
    avif,
  };
}

@pi0
I have used  @squoosh/lib": "^0.3.1 to optimize my backend images
Usage
const fs = require('fs/promises');
const { ImagePool } = require('@squoosh/lib');
const { cpus } = require('os');

const imagePool = new ImagePool(cpus().length);

async function convert2WebpAndAvif(originalImage: Buffer) {
  const image = imagePool.ingestImage(originalImage);

  const encodeOptions = {
    webp: {}, //empty object means using the default configuration
    avif: {},
  };

  await image.encode(encodeOptions);

  const [webp, avif] = await Promise.all([
    image.encodedWith.webp,
    image.encodedWith.avif,
  ]);
  return {
    webp,
    avif,
  };
}

// ==== Usage ==== //
const file = await fs.readFile('image.png');
const { webp, avif } = await convert2WebpAndAvif(imageBuffer);
await Promise.all([
  fs.writeFile('image.webp', webp.binary),
  fs.writeFile('image.avif', avif.binary),
]);

Caveats and Concerns

When I used the latest version 0.4.0, avif format does not work, so I revert back to version 0.3.1.
The latest version (0.4.0) is 9 months old with no recent updates or bug fixes .
It needs nodejs v.16.* and doesn't work on the latest lts nodejs version (v18.*)
It takes a lot of time to convert images, which makes it unsuitable for converting images on demand (per API request), however, it can be used to process images in a work queue in the background and serve the optimized images when ready.

The Squoosh library has unfortunately been discontinued indefinetely. You can still use the WASM enc/-decoders directly from the Squoosh source code, but other than that you are out of luck.
Thankfully, you can instead use wasm-vips, which is a WASM implementation of the library that Sharp uses under the hood. In the future, it is possible that we can even see a cross-platform version of sharp that uses wasm-vips.

Closing this and waiting for wasm support for sharp.

The Squoosh library has unfortunately been discontinued indefinetely

Do you have a link for that @atjn?

waiting for wasm support for sharp

Is this an ongoing project? Do you have a link for this @pi0 ?

Sharp support for WASM seems ongoing here: https://github.com/lovell/sharp/pull/3522

@karlhorky I am not entirely sure what you are asking for, but yes, the notice on @squoosh/lib is probably the most official communication available. I worked on @squoosh/lib, so I knew from talking to Jake that they were stopping work on it, and since it wasn't stable yet, that essentially killed the project for good.

Seems like landed!
https://github.com/lovell/sharp/pull/3522#issuecomment-1805398977
Any plans to support it?

@brendonmatos See https://github.com/unjs/ipx/pull/190 ;)


Fix the calculation of reserved nodes for ElastiCache

The ElastiCache report was comparing the number of reserved nodes to the
number of clusters, rather than to the number of nodes. This meant that
any cluster with more than one reserved node appeared appeared to lack
reservations, leading to much confusion.
Rather than counting each cluster toward the reservation, each cluster's
node is now counted instead.

Coverage remained the same at ?% when pulling 84a93101bb822a125e26d5ac4f7c5630ef301426 on beanbaginc:fix-elasticache-counts into ed4126733aa6b733ad2a6459392cf3476973656e on TerbiumLabs:master.


Visual artifacts / missing characters on windows vegawallet

When you run the vegawallet on Windows with Powershell or Command Prompt, you see occasional weird characters on the left of the window, small boxes with question marks. See attached photo. This is in a build which includes the fix for https://github.com/vegaprotocol/vegawallet/issues/333
Repro:
1: go install<EMAIL_ADDRESS>2: On Powershell or Command Prompt, run:
vegawallet create --wallet TEST
3: Fill in passphrase when prompted.
Expected behaviour:
All output is readable, with no visual artifacts or spurious characters.
Observed behaviour:
Some rectangles with question marks inside

OK so this is just because the default font for cmd.exe (which is consolas) doesn't support the ✗ ('ballot X') glyph. If you change the font to, e.g. Lucida Console, it works fine.
I'm not sure if there is anything to do here. I tried it in the new 'Windows Terminal' app and it worked with the default font there. Given that the terminal app is slated to replace cmd.exe, I think this problem will resolve itself with time.
I don't think there's any way to detect what font is being used so we can't dynamically turn them on or off based on if they are going to display nicely. The only thing we could do is remove all the fancy unicode ticks and crosses for all windows users, which seems like a bit of a shame.
If we think that's the right way to go, I can do that. Otherwise I think we should just close this issue.. thoughts?

I don’t like the fact we should condition the display of glyphs based on the platform because it's not worth the overhead in the source code, it's just niceties. But I also don’t like that some users are just left out because of some outdated font.
I would opt for ASCII characters, such as X  for errors and #  for section title, instead of the ✗ ('ballot X') and the ➜ ('arrow right').


Insufficient logging in /oauth2/callback handler when using SSO Authentication

Summary
Desire for more detailed error logging in the /oauth2/callback handler when using SSO
Use Cases
When enabling SSO Authentication for the first time, we may encounter subtle errors and misconifucations that can delay our setup process.
In such cases, having more detailed logs in the server would greatly assist us in identifying and resolving these misconfigurations and errors.
For instance, I deployed Keycloak and used it as an Identity Provider. I attempted to configure Argo Workflows SSO using Keycloak. However, misconfigured the OIDC issuer setting by mistake. Instead of setting it to http://keycloak-http.keycloak/auth/realms/myrealm, I incorrectly set it to http://keycloak-http.keycloak:80/auth/realms/myrealm. (i.e., I should have omitted the port).
Thie misconfiguration resulted in a simple 401 error without any logs in the server and lacked a detailed error message in the HTTP response to clients.
To gather more information, I added logging on every line in the HandleCallback function (ref: https://github.com/argoproj/argo-workflows/blob/97b6fa84441c423c68ecc8a8f1af5e26402d118e/server/auth/sso/sso.go#L227).
Through this, I managed to identify that the error occurred during the verfication of ID tokens after their issuance.
Notably, I discovered that the HandleCallback function, unlike other functions, simply responds with w.WriteHeader(401), which led to insufficient logging or information about the errors specific to this function.
I believe there are others who may face similar challenges in setting up SSO, and having more detailed logging woulde be immensely helpful.

Message from the maintainers:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

This is excellent feedback. Thank you! Since you already went through the debugging process, would you like to put together a PR that improves the logging?

@terrytangyuan Sure! I'd love to. Thank you for suggesting it first. I'll work on this ASAP.


Add bang methods

Add bang functions for HTTP requests.

Return ErrorResponse struct by default when there is an issue.

Add catch all for Response#success?

I bumped version to 2.3.0 not sure how you would want to handle it.

now that I think about it..maybe the version should be bumped to 3.0.0 since technically they are breaking changes.

About the version – I'll bump it to 3.0.0 later, please don't touch the version in pull requests.
Seems reasonable overall, except you're duplicating the raise code everywhere. Make a request! method that does the exception throwing and make other bang methods call it!

i will make changes and return the version back to what it was...sorry about that.

@myfreeweb better?  BTW, thanks for the feedback.

Uh… almost. Except not really better. Please don't add any code duplication — Don't Repeat Yourself! (or others)
I meant something like this:
      @doc """
      Like `request`, but raises  `HTTPotion.HTTPError` if failed.
      """
      @spec request!(atom, String.t, [{atom(), any()}]) :: %HTTPotion.Response{} | %HTTPotion.AsyncResponse{}
      def request!(method, url, options \\ []) do
        case request(method, url, options) do
          %HTTPotion.ErrorResponse{message: message} ->
            raise HTTPotion.HTTPError, message: message
          response -> response
        end
      end

Also, the @spec for the non-bang request should include %HTTPotion.ErrorResponse{} in the list of return types.

@myfreeweb 3rd time is a charm =]

Looks good, thanks!

Thank you!

@myfreeweb could you bump the version?


incorrect 'upload in progess' view

a validation error has occurred, but it still says 'uploading':

Resolved: https://github.com/fle-internal/content-curation/pull/211/commits/1873176f31ec229435df19137e7ffeb44aef3158


FEATURE REQUEST: Custom timeout elasticsearch and disable install of plugins

FEATURE REQUEST
If I disable plugins in the plugin.toml file that the plugins do not get downloaded or updated with malice plugin update -all. This would save time and diskspace.
Thanks !!!

This should work?
https://github.com/maliceio/malice/blob/master/config/config.toml#L33

please see #80


Remove some manifests from this bucket!!!

Remove some manifests from this bucket!!!
https://github.com/anderlli0053/DEV-tools/commit/ed248a71a71ff5f6cd1890f70ba60c451294f8fc
#44 why???

@Ryanjiena
https://github.com/anderlli0053/DEV-tools/issues/44#issuecomment-1592661747

@Ryanjiena
All good now? I think i have cleared all of them


when i train a classifier on own images,the accuracy = 0

the facenet code is error when calculating accuracy by using calssifier.py.


All Extensions not working

VS Code Version: 1.53.1
OS Version: CENTOS7   7.9.2009

Steps to Reproduce:

Update to VSCODe 1.53.1 (tried uninstalling, reinstalling, tried removing ~/.vscode folder)
Open git repo and look at any file. No extensions work. No error messages.

Does this issue occur when all extensions are disabled?: Yes
Manually installing VSCode 1.52.1 fixed the problem.
The C++ extension does not download dependencies when started with 1.53.1 but does with 1.52.1
Specifically I'm running these extensions:
ms-vscode.cpptools
eamodio.gitlens
Manually updated git to: 2.24.3 from Rackspace ius repo.
No error messages on startup of either VSCode version 1.53.1 or 1.52.1.

Starting with 1.53 CentOS 7 is no longer a supported platform. See https://github.com/microsoft/vscode/issues/115784#issuecomment-775364356

/duplicate https://github.com/microsoft/vscode/issues/115784


Invalid binary path gives no error message

Actual Behavior
Running the command without providing the full path to the executable binary creates an executable that doesn't do anything.
suid-wrapper -o ls_la ls -- -la

Expected behavior
One of these options:

The runner creation should fail if the argument is not a path to an executable file.
The runner should check if the execution failed, and display an error message.

Solved in v0.1.4


Update to 0.1.20

Closing and re-opening the PR due to a bad merge in the aggregate .gitmodules that's causing the builds to fail.


HSEARCH-997: Adding numeric bridges for short/byte

I've added a bridge for byte, too, for the sake of completeness.

Is this really a good idea? If I understand this change correctly, then byte and short will automatically be now indexed numerically. As @Sanne was saying, this involved quite some overhead and indexing as string might be better.
As of Search 5, @NumericValue is optional. The engine decides based on type, which type to index numerically. The default of using string encoding for short and byte seems correct.
I guess the question is, what to do, if a byte/short is explicitly annotated with @NumericValue?

Ignore the annotation
Throw a configuration error
Only in this case index numerically

If I understand this change correctly, then byte and short will automatically be now indexed numerically.

Yes, that's true. Indeed we may consider to use String-based indexing by default for byte and short if you guys think that makes more sense. OTOH it'd be kind of inconsistent to the other numeric types and thus may come at user's suprise.
Btw. how does one disable numeric indexing for int if it enabled by default now?

I guess the question is, what to do, if a byte/short is explicitly annotated with @NumericValue?

IMO it should be indexed numerically. Why would one mark it as numeric otherwise?

Yes, that's true. Indeed we may consider to use String-based indexing by default for byte and short if you guys think that makes more sense.

I think so

OTOH it'd be kind of inconsistent to the other numeric types and thus may come at user's suprise.

Well, the docs in this case describes the types which are numerically indexed. There is a reason why Lucene does not do it. In this context it would be interesting to have some performance measurements. If string indexing indeed performs better than numerical indexing, then it seems wrong to index numerically just for consistency.
Btw, with the automatic encoding discovery of Search 5, the surprise comes really at query time where you have to choose the right query type for an indexed field. So for byte/short you need to use query types which target string encoded fields. There might indeed be some 'surprise'. However we have some error reporting around this (eg targeting a string field with a numeric query). And then there is the docs of course.

Btw. how does one disable numeric indexing for int if it enabled by default now?

By explicitly setting a non numeric field bridge using @FieldBridge. It's in the docs. Maybe read it and let us know it it makes sense. This is quite new.

IMO it should be indexed numerically. Why would one mark it as numeric otherwise?

true, true - the user add @NumericField, because he wants it to be indexed numerically. But why does he do it? Because he things if something is a number, it is "better" to index them as such. However, in this case this might just be wrong. But I guess, if you want to do things like sorting etc, it might make sense. AFAIR, an explicit @NumericValue is honored anyways at the moment. So indexing byte/short numerically, if explicitly requested is probably ok. I still think, however, we should stick to string encoding for the default behaviour.

If string indexing indeed performs better than numerical indexing, then it seems wrong to index numerically just for consistency.

I'm not sure. Just when looking at a class which has several numeric properties, isn't reasonable to expect that all these numeric props are indexed as numbers (if it is doable of course), allowing number-based queries to run against them?
So in that light, if string-based indexing is faster, it seems like an optimization which one could opt-in if really needed.

And then there is the docs of course.

Sure, but let's try to make the API work as one would reasonably expect it, so to avoid the need for checking with the docs in the first place.
Anyways, that's just my view as a newcomer to the field. If you folks say that using string-based fields for byte/short by default is more sensible, I'm all for it. Seems we agree on numeric indexing if @NumericField is given explicitly.

Sure, but let's try to make the API work as one would reasonably expect it, so to avoid the need for checking with the docs in the first place.

I see your point. As said, to make a sounds decision, we would need to know/measure the actual overhead

Seems we agree on numeric indexing if @NumericField is given explicitly.

+1

As we agreed on IRC, using numeric fields for byte/short is generally a good idea for the sake of consistency and to avoid surprises to the user. Only we shouldn't do that change by default in a micro release in order to ensure index compatibility.
Added therefore some more commits which make byte/short mapped to String fields by default; Only if @NumericField is given, the new bridges will be used. There is HSEARCH-1779 to change the default behavior at the next suitable time.
@hferentschik, @Sanne: Can you give it another look? Thx!

Btw. also added a bridge for encoding bytes in Strings, as it apparently was missing.

@hferentschik Is there anything else you think should be done? Or is it good to go? Thx!

Is there anything else you think should be done? Or is it good to go? Thx!

Looking (did not see your addititional commit)

rebased and merged

Sweet, thx!


About singler.rmd , any human reference specifized tissue, like liver

Dear Sanbomics,
Thanks a lot for your efforts, that did inspire me a lot,
while I have an issue that is there any solution for human reference specifized for various tissues, exactly like what "TabulaMurisData" did in mus musculus.
Thanks again.
Best regards,
Na

Dear Snabomics,
I found [Tabula Muris Consortium], who developed TabulaMurisData, recently have reported similar data for human, https://pubmed.ncbi.nlm.nih.gov/35549404/ , but still no open data available in R. Hopeful soon for automatically usage.
Maybe it will take some time for usage.
Best regards,
Na

Hi, sorry for the late response. I see you found tabula sapiens, which is what I would have recommended. You can convert the adata to an R object then do label transfer.


Training on a gpu <=8Gb

Hello,
Thank you for your project.
I have a 8Gb RTX 4070.
By any chance do you know how to limit the memory usage during training at checkpoint save?
RuntimeError: CUDA out of memory. Tried to allocate 508.00 MiB (GPU 0; 7.53 GiB total capacity; 5.31 GiB already allocated; 355.88 MiB free; 5.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[1]    7367 IOT instruction (core dumped)  ./build/bin/train --config configs/train_normalnet.ini  tt_train    1  1  256

Hi,
just to make sure, you started training with
./build/bin/train --config configs/train_normalnet.ini --TrainParams.scene_names tt_train --TrainParams.batch_size 1 --TrainParams.inner_batch_size 1 --TrainParams.train_crop_size 256 ?
(just asking as the error output is missing the argument names)
and you are using the tt_train scene from our supplemental?

Hi Linus,
Yes, exactly like that. Additionally, I have tried not to save images with every checkpoint save and/or to further reduce the crop size, but without success.
I am going to dive into the code to understand why there is an attempt to allocate an additional ~508.00 MiB on the checkpoint save.

Hello,
I have found that it is due to lt_vgg = loss_vgg->forward(x, target); and so I have forced want_eval = false; to ignore eval and test epochs

Hi, good work-around :)
This might also apply for you: https://github.com/lfranke/TRIPS/issues/26#issuecomment-1956377052
Not using VGG reduces VRAM, however this will also impact quality


Update LocationIQ Endpoint

I have no affiliation with LocationIQ besides as a user.
However, per an email received recently, the endpoint used by this gem is being deprecated.
Details below...
Hello,

We’re pushing a few changes to ensure continued service, higher availability, and lower latencies for our APIs.

What’s changing?

On 1st June 2022 at 00:00 UTC, we’re going to deprecate some endpoints used to reach our Geocoding, Routing & Timezones APIs. These include older & root domain endpoints such as [us1.locationiq.org/v1](http://us1.locationiq.org/v1), [eu1.locationiq.org/v1](http://eu1.locationiq.org/v1), [locationiq.org/v1](http://locationiq.org/v1), and [locationiq.com/v1](http://locationiq.com/v1).

Starting 1st June, our Geocoding, Routing & Timezone APIs will only respond to requests sent to these two endpoints - https://us1.locationiq.com/v1, and https://eu1.locationiq.com/v1.

There are no changes to other APIs:
Autocomplete API requests - [api.locationiq.com/v1](http://api.locationiq.com/v1)
Static Maps - [maps.locationiq.com/v3](http://maps.locationiq.com/v3)
Map Tiles - [tiles.locationiq.com/v3](http://tiles.locationiq.com/v3)

What you need to do

If you’re using one of our new endpoints - you’re good! If you’re using any of the older endpoints, please update your request URLs to the above-mentioned endpoints.

Please check out our [documentation page](https://locationiq.com/docs) for more information about our APIs. Just hit reply if you have any questions, we’d be happy to help.

Warm Regards,
LocationIQ Team

Excellent, thanks!


i18n(es): add reference/integrations-reference.md

What kind of changes does this PR include?

Translated content

Description

add reference/integrations reference.md
add page to nav.ts

Thanks so much for this @kevinzunigacuellar and @alfredogonzalezmartinez! :heart:
Hope you don't mind I updated the translation to include some recent code and link related changes from https://github.com/withastro/docs/commit/f449538343d16b15fb5d07469224504665437625 & https://github.com/withastro/docs/commit/23ea009f28a75df021b9fbf37ce18e741a75c07c.
I'll wait for a LGTM just to be sure I didn't break anything and then merge it! :raised_hands:

And it's time to merge :rocket:! Amazing work everyone :raised_hands:


Reading TFlite model metadata

Originally posted by @phlash in https://github.com/floe/backscrub/issues/77#issuecomment-835513517

Sounds like a nice solution, but is the metadata normally included in models? Sounds like models with metadata should essentially be a zip file?

Not in the models we're currently using, but only because we didn't need it. Models with metadata in are available from the Google model zoo (https://tfhub.dev/). [edit] I lied, that only contains Deeplabv3 with metadata. Looks like the Media Pipe team haven't added any yet, although they do have model cards. Oh well. end
I'm currently looking at sane ways to read the metadata without adding a new dependency and build pain (it needs Bazel) for the tf_lite_support library that's officially required. So far, getting a blob of metadata out works ok (see snippet below), since TfLite already supports random blobs in a file, however that blob then needs parsing (from flatbuffers format, schema here: https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/metadata_schema.fbs) to pull out the input normalization constants.. Currently poking through this: https://google.github.io/flatbuffers/md__internals.html and a hex dump of the raw buffer.
int init_tensorflow(...) {
	//...
	auto model = flatmodel->GetModel();
	auto *md = model->metadata();
	if (md) {
		for (uint32_t mid=0; mid < md->size(); ++mid) {
			const auto meta = md->Get(mid);
			printf("found: %s\n", meta->name()->c_str());
			if (meta->name()->str() != "TFLITE_METADATA")
				continue;
			// grab raw buffer and dump it..
			const flatbuffers::Vector<uint8_t> *pvec = model->buffers()->Get(meta->buffer())->data();
			printf("metadata dump (size=0x%X)\n", pvec->size());
			parse_metadata(pvec->data(), pvec->size());  // currently just a hex dump :)
		}
	}

This might turn out to be horribly fragile though!
Question: What do we think about using Bazel for builds (standard Tensorflow tooling)?

Maybe also a bit off topic in regards to meta data reading; Tried to use that hipster build system for a standalone lib that uses TensorFlow. Turned out to be a huge time sink..
Goal: just make some quick experiments with post processing of model output in Python

Problem: Not possible to set custom ops in current (released) tflite Python [1]
Solution: Try to create a small wrapper lib to do inference and use it from Python with ctypes
Reality: Spend way too much time fighting with Tensorflow building and linking errors

Got it to build for my use case in the end. Tried to make a quick adaption of Bazel for backscrub but never managed to find a good way to integrate opencv dep.
Some take home messages:

Building TensorFlow as Bazel external dependency does not seem officially supported, but is doable [2], [3]
C API seems to be recommended over C++ by tf devs [4]
TF Devs indicate CMake is no longer supported? (old comment) [5]
Bazel has nothing like pkg-config or .cmake files. Externals deps that are not built by Bazel can be added by listing paths and globs on what headers and libs to look for

1: https://github.com/tensorflow/tensorflow/issues/44043
2: https://github.com/tensorflow/tensorflow/issues/12761
3: https://stackoverflow.com/questions/48497006/how-to-add-tensorflow-to-existing-bazel-project-as-external-dependencies
4 : https://github.com/tensorflow/tensorflow/issues/35689#issuecomment-642903301
5: https://github.com/tensorflow/tensorflow/issues/30183#issuecomment-506279570

[still OT slightly] Thanks for trying the 'hipster way' :smile:. I'm not sure CMake is going away now (it may have been then), as it's properly documented and marked as 'experimental since 2.4' here: https://www.tensorflow.org/lite/guide/build_cmake. I for one would rather use CMake for it's good documentation, popularity and capability (even if I don't like the mess it spews out!). It took me just a few minutes to get a basic 'backscrub + tensorflow' combined build working, so I could enable XNNPACK and double the CPU-based performance: https://github.com/phlash/backscrub/tree/xnnpack-test
[back on topic] Thoughts on my rough metadata extraction hack? It could/should probably use [de]serialisation code generated by flatc rather than hacked up by hand (but then we have to build & run flatc - which has CMake and Bazel support). It should read the 'associated files' metadata rather than assume 'labelmap.txt' exists. Is there other metadata we would want? It might be better to rebase on experimental rather than my tflite-extract branch...

Does the flatc stuff always need to full spec to be built? Could we try to build a reduced version of it and freeze it for our purposes? Haven't looked at the generated flatc source code (and I'm too afraid I'd rather not do that to reduce the amount of nightmares), but how sane is it? Would interfacing just that reduced part work? Or is the data structure needed sane enough to whip up our own reduced parser for it?

Google do exactly that for parts of the build already, pre-built headers from flatc (resulting in schema_generated.h), to avoid the pain of building and running it, but only for the parts that they support in Make and CMake builds, not the metadata library (nor it turns out the GPU delegate). I'll have a go at building/running flatc...

There seems to be a package [flatbuffers-compiler[(https://packages.debian.org/bullseye/flatbuffers-compiler) (at least on Ubuntu 20.04 LTS and Debian Bullseye) containing flatc, thus this might come with an acceptable level of PITA … It could even be remarked upon as an optional build dependency then …

Ah-ha! Also in buster-backports so this might be an easy way out. That said flatc builds easily enough with CMake, unfortunately it's a struggle to do that via the main tensorflow-lite build (the dependency mechanism using FetchContent is fiddly and doesn't pass options through).

OK, it looks neater with the compiled metadata serializer: https://github.com/phlash/backscrub/tree/tflite-metadata
This branch is now:

based on main
uses flatbuffers-compiler package (stock Ubuntu or backport for Debian buster)
reads the output labels file name from metadata

Awaiting somewhere to use the metadata values... coming in: https://github.com/floe/backscrub/pull/77


feat: update build function to create lua mappings

Just added a step to build function inside build.js to create lua mappings alongside shell ones for those that are using lua

I didn't know Lua was a thing with sketchybar, nice!

Would there be a way to preserve the wildcard matching btw? Could maybe use regex instead?

I thought about that. But than we would have basically O(n) ifs/regex instead of just O(1) with table.
I had two ideas in mind to solve patterns:

Create two separate mappings. One that would include all regexes and second that would include all strings and return some function that will do both mappings under the hood
Just do regex before mapping to remove numbers for example

First solution sounds super complicated to me, so I just went with remove numbers by pattern before actual mapping in my config.
There probably is better solution, but I happy at least with that for now)
It already better then manual mapping to font😅


Unable to run umami behind Traefik after update 1.39.5

Hi,
I update umami to the 1.39.5 the update of db went fine but I'm unable to see the dashboard in the Chrome Developer console I only see:
 Failed to load resource: the server responded with a status of 404 () /_next/static/chunks/api/collect
and blank page.
The strange thing is that I'm able to retrieve the tracker script.

Anything under the _next folder is from the app itself. Have you tried rebuilding the app?

I am having the same issue. I currently have a version 1.33.1 running in a docker container with a MySql database.
To make sure an update will work fine, I made a copy of the database and then downloaded the latest docker container. I fired it up, pointing to the DB copy and it started up just fine.
yarn run v1.22.19
$ npm-run-all check-db update-tracker start-server
$ node scripts/check-db.js
✓ DATABASE_URL is defined.
✓ Database connection successful.
✓ Database tables found.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": MySQL database "umami_test" at "vs02.nhz.ch:3306"
4 migrations found in prisma/migrations
Following migrations have not yet been applied:
02_add_event_data
03_remove_cascade_delete
04_add_uuid
To apply migrations in development run yarn prisma migrate dev.
To apply migrations in production run yarn prisma migrate deploy.
Running update...
Prisma schema loaded from prisma/schema.prisma
Datasource "db": MySQL database "umami_test" at "vs02.nhz.ch:3306"
4 migrations found in prisma/migrations
Applying migration `02_add_event_data`
Applying migration `03_remove_cascade_delete`
Applying migration `04_add_uuid`
The following migrations have been applied:
migrations/
  └─ 02_add_event_data/
    └─ migration.sql
  └─ 03_remove_cascade_delete/
    └─ migration.sql
  └─ 04_add_uuid/
    └─ migration.sql

All migrations have been successfully applied.
✓ Database is up to date.
$ node scripts/update-tracker.js
$ node server.js
Listening on port 3000

I then tried to connect to the dashboard, but just receive an empty page. The Console says
_ @ main-af7420747e3d718a.js:1
q @ main-af7420747e3d718a.js:1
V @ main-af7420747e3d718a.js:1

After checking, I found this thread, so I tried to build the app again by connecting to the docker container and run the yarn build. But then I got these errors.
/app $ yarn build
yarn run v1.22.19
$ npm-run-all build-db check-db build-tracker build-geo build-app
$ npm-run-all copy-db-files build-db-client
$ node scripts/copy-db-files.js
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'fs-extra'
Require stack:
- /app/scripts/copy-db-files.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Function.Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1067:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/app/scripts/copy-db-files.js:2:13)
    at Module._compile (node:internal/modules/cjs/loader:1165:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1219:10)
    at Module.load (node:internal/modules/cjs/loader:1043:32)
    at Function.Module._load (node:internal/modules/cjs/loader:878:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/scripts/copy-db-files.js' ]
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "copy-db-files" exited with 1.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "build-db" exited with 1.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

How to fix this upgrade problem?

I tried a bit more, but receive the same issue:

I stopped the new test container.
I deleted the database in MySql and created an empty database again
I fired up the test container again.
The container created the tables
When I try to browse to the dashboard, I get the 404 error that _next/static/chunks/api/collect is not found

Is there an issue with the docker container?

Can confirm, that this is due the bug discovered in #1769
After I changed the script name, I do get the login page for the test install.

I will leave #1769 open, which is currently an open bug. Thanks for reporting.


Support array in isLoading getter

Hi,
I think it might be a good idea having isLoading handle strings and arrays:
$loading.isLoading(['getData', 'saveData'])
https://github.com/f/vuex-loading/blob/d45d7fd2df6d9d751040947c14f9f83d2021d403/src/vuex-loading.js#L51

I created a pull request for this, #12.

Merged. #12


curl error when attempting to fetch issues from github for custom tap

Please always follow these steps:

[x] Confirmed this is a problem with running a brew command and just brew installing one or more formulae? If it's a formulae-specific problem please file this issue at https://github.com/Homebrew/homebrew-core/issues/new

[x] Ran brew update and retried your prior step?

[x] Ran brew doctor, fixed all issues and retried your prior step?

[x] Ran brew config and brew doctor and included their output with your issue?
brew-config output
$ brew doctor
Your system is ready to brew.

Please note we may immediately close your issue without comment if you delete or do not fill out the issue checklist and provide ALL the requested information.
To help us debug your issue please explain:

What you were trying to do (and why)
Attempting to install a formula from a custom tap. Important detail: the custom tap is hosted on our internal  GitHub Enterprise instance.

What happened (include command output)
Received 2 bits of error output. The first was some error output from the failed formula itself (which was expected). The second was error output from a failed curl command while brew was attempting to fetch related issues from the tap. Full command output
relevant snippet:
If reporting this issue please do so to (not Homebrew/brew or Homebrew/core):
rockstack/rock

/usr/local/Homebrew/Library/Homebrew/utils/github.rb:226:in `raise_api_error': curl failed!
(GitHub::Error)
curl: (22) The requested URL returned error: 422 Unprocessable Entity
curl: (3) <url> malformed
	from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:184:in `open'
	from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:233:in `issues_matching'
	from /usr/local/Homebrew/Library/Homebrew/utils/github.rb:272:in `issues_for_formula'
	from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:368:in `fetch_issues'
	from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:364:in `issues'
	from /usr/local/Homebrew/Library/Homebrew/exceptions.rb:418:in `dump'
	from /usr/local/Homebrew/Library/Homebrew/brew.rb:138:in `rescue in <main>'
	from /usr/local/Homebrew/Library/Homebrew/brew.rb:31:in `<main>'

What you expected to happen
Expected just the error output from the formula. (ie, the error output from make) Brew should not be attempting to fetch issues from api.github.com for taps which are not on public github.

Step-by-step reproduction instructions (by running brew commands)

brew-tap a repo that is not on github.com
attempt to install a formula that is is known to fail/error

I would imagine this is where the additional guard should take place: https://github.com/Homebrew/brew/blob/b9cfd2cb58a2f360dadf691dc3eeaf22b694e960/Library/Homebrew/exceptions.rb#L418
It should be checking if the tap has a custom_remote? and/or if the tap is a private_repo? (which is currently not exposed on a Tap but probably should be?)

This would be where the change should be made: https://github.com/Homebrew/brew/blob/b9cfd2cb58a2f360dadf691dc3eeaf22b694e960/Library/Homebrew/utils/github.rb#L270-L273
Closing as we don't support GitHub Enterprise but we'll accept a fix for this case.

Don't need to support issues for GHE. (I can see that being a huge pain in not knowing which version of the API any particular GHE instance was using.) But brew already supports taps for any given remote, so brew shouldn't be throwing its own errors when one such tap is in use, no?

as evidenced by the custom_remote? method

I don't think this method necessarily implies "not on GitHub".

so brew shouldn't be throwing its own errors when one such tap is in use, no?

Yes. If this problem is reported more widely I'll look at fixing it but otherwise I'll leave it to those affected to work on the fairly straightforward fix.

I don't think this method necessarily implies "not on GitHub".

Fair. (A github-hosted tap that doesn't follow the homebrew-{repo} convention would be considered a custom_remote? while still being on GitHub.)
However, this documentation clearly indicates that non-github taps are supported.

If it’s hosted outside of GitHub, they have to use brew tap user/repo , where user and repo will be used to refer to your tap and  is your Git clone URL.

But again, I think it's totally fair to define "support" of a non-github tap as: "will not perform the niceties like fetching open issues on formula failure". But I strongly think that brew shouldn't be throwing exceptions when it encounters non-github taps. Especially when documentation indicates they are a supported (though second-class) use case.
I'll work on a PR to fix!

However, this documentation clearly indicates that non-github taps are supported.

 Things maintainers say are supported are supported.

But I strongly think that brew shouldn't be throwing exceptions when it encounters non-github taps.
I'll work on a PR to fix!

That's great, thanks.

I'm not trying to be contentious here, but the claim that non-github taps aren't supported is in direct contradiction to all the documentation in the repo and in brew's own manpages. Obviously, if you say they're unsupported, then they're unsupported. But all the documentation below indicates (unequivocally) that they are supported. Was support dropped recently and the docs need updated to reflect the change?
docs/How-to-Create-and-Maintain-a-Tap.md

If it’s hosted outside of GitHub, they have to use brew tap user/repo , where user and repo will be used to refer to your tap and  is your Git clone URL.

docs/brew-tap.md

Unlike the one-argument version, URL is not assumed to be GitHub, and it doesn't have to be HTTP. Any location and any protocol that Git can handle is fine.

brew-tap manpage (literally says "can be cloned from places other than GitHub")

With URL specified, taps a formula repository from anywhere, using
any transport protocol that git handles. The one-argument form of tap
simplifies but also limits. This two-argument command makes no
assumptions, so taps can be cloned from places other than GitHub and
using protocols other than HTTPS, e.g., SSH, GIT, HTTP, FTP(S), RSYNC.

Was support dropped recently and the docs need updated to reflect the change?

I think we're just disagreeing over what the word "supported" means. To me, it means "things that we will prioritise fixing if it's broken" (and "we" mostly means "@MikeMcQuaid" on Homebrew/brew). To others it means, "all documented Homebrew functionality". I don't agree with this; we document things that people want and try to do even but they are not the "happy path" that we recommend. For example, installing Homebrew into a non-/usr/local location is functional but not supported; we may fix issues you have with it or we may not.

Okay, I can understand that. And the spectrum of "support" makes sense. Even with this bug, I would never expect non-github taps to fetch open issues, as github.com taps do.
Thanks!

Thought I'd add that brew bump-formula-pr is also broken for taps on GitHub Enterprise in what appears to be the same way. Once the next version of brew gets released I'll see if it gets resolved.


BSP v5.9 with no patched branches

Includes commits from #69 because we modify yaml modified in that PR

bsp 5.9: get rid of '-patched' branch for meta-renesas
bsp 5.9: get rid of meta-rcar

Tested on:

h3ulcb-4x2g-kf DomD + DomU
h3ulcb-4x2g-kf DomD + AGL
h3ulcb-4x2g DomD+ AGL

In all mentioned cases we see proper work of

network (ping google.com)
weston-simple-egl
glmark2-es2-wayland
pulseaudio (pactl list)

Substituted by #74


API Raises 400 if user license is nonexistent/wrong

In functions like user_call_center() if the input UserID doesn't have an agent license the function will return with Http Error 400 instead of returning an empty Json value.
Current:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url<EMAIL_ADDRESS>
Expected:
[]

Looks like the same thing happens for group_call_center_forced_forwarding() if the call center has OOH times setup

Noticing the same issue is happening for call centres:
out = magic.get.group_call_center_forced_forwarding(service_user_id=service_user_id)
print(out, type(out))

Return the same error code.
Possibly related to the fact its Call Forwarding Selective, instead of just generic forwarding.