Introduce rubocop:set/rubocop:reset directives

A lot of work. A LOT of work. (@alexdowad at #2716)

So, here we go! This PR should be considered rather of a draft-y quality, to discuss the whole idea and its implementation. But I consider it mostly complete.
This PR introduces rubocop:set Cop{Setting: value} and rubocop:reset Cop|all directives.
Design/implementation decisions:
0. Directive syntax is rubocop:set Metrics/AbcSize{Max: 20}. The {Max: 20} part is expected to be a valid YAML.
1. Directives are not nested:
# rubocop:set Metrics/AbcSize{Max: 20}
# ...here it is 20...
# rubocop:set Metrics/AbcSize{Max: 30}
# ...here it is 30
# rubocop:reset Metrics/AbcSize
# ...here it is default (fully reset), not back to 20

I believe it is less confusing, but open to discussion. One exception is single-line directives:
# rubocop:set Metrics/AbcSize{Max: 20}
# ...here it is 20...
def very_really_complicated # rubocop:set Metrics/AbcSize{Max: 50}
  # 50 for this method only
end
# ...but here it is 20 again, not reset to default

2. Directives work only for #on_<nodetype> style cops, not #investigate-style cops. That's because in #investigate, there is no place where Commissioner or base Cop is hooked to change something line-by-line (rubocop:disable works this around by allowing cop to analyze, but preventing it to report on disabled lines). This is a hard limitation, I believe the feature would be useful nevertheless, but it should be communicated clearly (may be with additional feature with auto-documentation, which checks for def investigate and specifies whether cop supports inline settings, depending on it).
3. I took the liberty to rewrite CommentConfig completely. I've introduced a notion of Directive class, which should open a way for, probably, implementing new ones.
On PR quality:

Currently, the code of CommentConfig is written in the style that allows to see algorithms clearly (large-ish methods, and usage of rubocop:set to enlarge thresholds, just to make a point). When/if we agree on the final implementation, I am ready to change the style (though, see PS about it)
The tests are a bit shallow now (again, awaiting discussion of architectural details)

PS: On methods complexity: TBH, it is my current belief that Rubocop's default thresholds for complexity metrics suit nice for business logic (where if you have three branches of if, they probably represent Nameable Cases, and should be extracted to methods or even classes), but much less for algorithms. To show one case, here is method CommentConfig#extra_enabled_comments that provides list of redundant rubocop:enable directives

def extra_enabled_comments # rubocop:set Metrics/AbcSize{Max: 26}
  disabled = all_cop_names.to_h { |name| [name, false] }

  directives
    .select(&:disable_enable?).reject(&:single_line?)
    .chunk { |d| d.enable? && d.all? }
    .each_with_object([]) do |(enable_all, dirs), redundant|
      # if this group of directives is exploded from
      # `# rubocop:enable all` comment, it should be handled
      # as one statement
      if enable_all
        redundant << [dirs.first.comment, 'all'] if disabled.values.none?
        disabled.transform_values! { false }
      else
        dirs.each do |dir|
          redundant << [dir.comment, dir.cop] if dir.enable? && !disabled[dir.cop]
          disabled[dir.cop] = dir.disable?
        end
      end
    end
end

I'd argue that this is the best representation of the algorithm. The method itself has one local variable, and one large iteration (with several smaller "preparatory" Enumerable methods), with one `if` inside it; it depends on 2 other, very generic methods, and only for setting initial data. It fits on half a screen even on small laptop, and makes it easy to follow "what'll be in what variable" at every step. But it violates both method length (14 max, method is 20), and AbcSize (15 max, method is 26) cops.
Alternatively, the current implementation is split into multiple small methods throughout the class, works with two mutable collections (array and hash) passed from method to method and changed in each of them; and as a person who tried to understand what and how it does, I can witness it was not easy to follow through all the micro-methods.
I am not sure what is my point here, and it is probably not a discussion for this PR, but still :)
PPS: I'll comment on some non-obvious changes inline.

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[ ] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[ ] Run bundle exec rake default. It executes all tests and RuboCop for itself, and generates the documentation.

About CI fails:

I left some of Rubocop self-investigation fails till final refactoring.
Two of failing tests: I am not sure they test the right thing! For example, one tests that if the cop has wrong namespace # rubocop:disable Style/LineLength, then (besides other errors), we'll have a "Redundant enable" warning about subsequent # rubocop: enable Layout/LineLength, which fails for my implementation, and I believe it is better this way (even if the disable was in the wrong namespace, we "got what you meant", so subsequent enable shouldn't be warned about).

I marked PR as draft, but it means "I want to discuss it", not "I'll be working on it for some more time till it would be ready for discussion".

It's really great to see someone tackling this! I'll review the PR in the next couple of days, as lately it has been pretty busy on my end.
Have you measured the performance impact of the changes? (especially in the absence of annotations). I'm assuming it's not insignificant if you're hooking into the node callbacks.

@bbatsov

Have you measured the performance impact of the changes?

Not yet. I wanted to hear an opinion on the whole idea first, but I definitely plan to profile/benchmark/optimize before calling it a day.
As for viability of the approach, I checked it empirically (running update rubocop against its own codebase), and it hasn't become unbearably slow (or even noticeably slower) on my machine (which is pretty rough metric, I agree!). That's why I believe it is possible to proceed this way.

@bbatsov So, I profiled the situation a bit, and it appears that new method indeed becomes a bottleneck, in a quite bad way. The thing is, "whether we have config for this cop" is of course called once-per on_ method, and it leads to numbers like ~2mln calls on 254 not-very-large files being analyzed. At this rate, performance penalty strucks on the most innocent methods. For example, ProcessedSource#comment_config spends ~2 seconds for the simple @comment_config ||= CommentConfig.new(self) -- it calls real (and trivial) CommentConfig.new just once per file (e.g. 224 times), but turns out that at least on my machine's Ruby 2.6 that simple method call + ||= is heavy enough to provide a noticeable slowdown. The same with Cop#cop_name (again, 2mln of calls of simple memoization ||= manifests itself into 1 second of slowdown).
I don't believe it is acceptable, so I am postponing this PRs further development and will try to think of another approach (not sure what it might be... maybe pre-dissecting ProcessedSource into chunks with different configs once on reading). Sorry for taking your time.

@zverok Do you plan to revisit this any time soon or can we close the PR for now?

@bbatsov This one definitely might be closed. I plan the second attack to the problem for ~December, but will try a different approach.
I'll hopefully also (sooner) will extract small PR with found bugs in tests.


Mime type load errors

We've been seeing some errors that seem to be related to mime-type. Mostly that wasm is being served with the wrong mime-type (supposed to be served application/wasm), but I also saw a JS file be served with text/html. The fact that they work after refresh most of the time makes it seem caching related. Kind of sounds like this (unresolved) issue:
https://community.cloudflare.com/t/wrong-content-type-added-to-cloudflare-pages-site/373967/9

this guy claims to have a solution:
https://community.cloudflare.com/t/wrong-content-type/446632/12

another thread on the topic has a js file for rewriting headers... not my ideal solution! https://community.cloudflare.com/t/pages-messes-with-mime-types/254794/16

I believe this is related:

this guy claims to have a solution: https://community.cloudflare.com/t/wrong-content-type/446632/12

I set this up on signet-app for js, let me know if it fixes the problem over the next week or so.


Fix _lobby_message signal

Hello! According to API _lobby_message should return lobby_id.
BTW Thanks for your awesome work! :)

Hey there!  Thank you!  Yeah, the result it is currently sending back is actually the size (in bytes) of the message and totally useless.  Good catch!


Display column names in error message

Hi Seddryck,
I would like to ask for a feature: Could you provide the display of column names within error messages? Right now, we get the information if it is a key or value-column. The Name would be helpful :)
For example:
Expected:
ResultSet with 1 row

KEY (Text) | VALUE (Numeric) (+/- 15.00%)
---------- | -----------------------------
RowCount | 273649

If I would know, which values are in the values columns (especially if there are more than one), I could be able to communicate easier about the issues.
Thanks :)
Tilo

Should be relatively easy to add this.

Preview available at https://ci.appveyor.com/project/CdricLCharlier/nbi/build/1.15.0-ci.18
This preview also includes #241

I have downloaded the code from  and build it in my local machine.
But still I am not able to see the column names in the result. Am I doing the right way?
	<column name="productKey"  role="key" type="text" ></column>         <column name ="ProductAlternateKey" role="value" type="text" /> 		<column name="EnglishProductName" role="value" type="text" />         <column name="StandardCost" role="value" type="numeric" />

This branch is not merged with the branch where the feature you're looking for is implemented: https://github.com/Seddryck/NBi/tree/feature/Display_column_name_(%23105)
Personnaly, I'd try this build to get this feature: https://ci.appveyor.com/project/Seddryck/nbi/build/1.15.0-ci.27/artifacts
It should work, but (there is a "but") ... I've many issues with the rest-api and this feature was implemented within this work-package ... I'll need to rebranch and play a lot of git to get this feature in the release 1.15 (most of the rest-api features will be discarded).

It works for me with build/version number <IP_ADDRESS>.
Expected:
ResultSet with 1 row
 #0 (ImportantMeasure)
 KEY (Text)
 ---------------------
 2

Good news :-)

Yeah Its Working. Thanks.

Available in beta: https://github.com/Seddryck/NBi/releases/tag/v1.15-beta


Instructions on how to load the downloaded weights or where can the ckpt file be found for pretrained model

Hi, I have prepared the Python environment, and downloaded data from the PDEBench, as well as the model weights from Google Drive. I can train the model from scratch on my current data available and I wanted to know your instructions on how to use the load the pre-trained model. Thank you for your time and contribution.
P.S.
I also have questions about the paper MPP 2023 paper, what is your preferred method of communication about the project?

I have added the path of the MPP_AViT_S tar file to YAML setting pretrained_ckpt_path. From here I arrive to an attribute error when accessing self.model.module
python train_basic.py --run_name first_test_run --config finetune --yaml_config config/mpp_avit_s_config.yaml
Loading configuration file: multiple_physics_pretraining/config/mpp_avit_s_config.yaml
Configuration name: finetune
Initializing data on rank 0
Initializing model on rank 0
Model parameter count: 28979436
Starting from pretrained model at weights/MPP_AViT_S

Traceback (most recent call last):
  File "multiple_physics_pretraining/train_basic.py", line 547, in <module>
    trainer = Trainer(params, global_rank, local_rank, device, sweep_id=args.sweep_id)
  File "multiple_physics_pretraining/train_basic.py", line 80, in __init__
    self.restore_checkpoint(params.pretrained_ckpt_path)
  File "multiple_physics_pretraining/train_basic.py", line 201, in restore_checkpoint
    self.model.module.unfreeze()
  File "multiple_physics_pretraining/multiple_physics_pretrained_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'AViT' object has no attribute 'module'. Did you mean: 'modules'?

I will look into it so just updating the thread here

I have added the path of the MPP_AViT_S tar file to YAML setting pretrained_ckpt_path. From here I arrive to an attribute error when accessing self.model.module
python train_basic.py --run_name first_test_run --config finetune --yaml_config config/mpp_avit_s_config.yaml
Loading configuration file: multiple_physics_pretraining/config/mpp_avit_s_config.yaml
Configuration name: finetune
Initializing data on rank 0
Initializing model on rank 0
Model parameter count: 28979436
Starting from pretrained model at weights/MPP_AViT_S

Traceback (most recent call last):
  File "multiple_physics_pretraining/train_basic.py", line 547, in <module>
    trainer = Trainer(params, global_rank, local_rank, device, sweep_id=args.sweep_id)
  File "multiple_physics_pretraining/train_basic.py", line 80, in __init__
    self.restore_checkpoint(params.pretrained_ckpt_path)
  File "multiple_physics_pretraining/train_basic.py", line 201, in restore_checkpoint
    self.model.module.unfreeze()
  File "multiple_physics_pretraining/multiple_physics_pretrained_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'AViT' object has no attribute 'module'. Did you mean: 'modules'?

I will look into it so just updating the thread here

The issue I was encountering seems to be consistent with trying to access self.model.module when self.model is not wrapped in a DistributedDataParallel (DDP) object; the 'module' attribute is added by the DDP wrapper.
It seems the original method assumed the presence of the module attribute even when not using DDP. The updated restore_checkpoint method checks if self.model is an instance of DistributedDataParallel (DDP) and only then access the module attribute; both when loading the state dict and under the self.params.pretrained if statement. Just minor changes but its working on my 1 local GPU workstation.
    def restore_checkpoint(self, checkpoint_path):
        """ Load model/opt from path """
        checkpoint = torch.load(checkpoint_path, map_location='cuda:{}'.format(self.local_rank))
        if 'model_state' in checkpoint:
            model_state = checkpoint['model_state']
        else:
            model_state = checkpoint
        try:  # Try to load with DDP Wrapper
            self.model.load_state_dict(model_state)
        except:  # If that fails, either try to load into module or strip DDP prefix
            if isinstance(self.model, DDP):
                self.model.module.load_state_dict(model_state)
            else:
                new_state_dict = OrderedDict()
                for key, val in model_state.items():
                    # Failing means this came from DDP - strip the DDP prefix
                    name = key[7:]
                    new_state_dict[name] = val
                self.model.load_state_dict(new_state_dict)

        if self.params.resuming:  # restore checkpoint is used for finetuning as well as resuming. If finetuning (i.e., not resuming), restore checkpoint does not load optimizer state, instead uses config specified lr.
            self.iters = checkpoint['iters']
            self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
            self.startEpoch = checkpoint['epoch']
            self.epoch = self.startEpoch
        else:
            self.iters = 0

        if self.params.pretrained:
            if isinstance(self.model, DDP):
                model_to_modify = self.model.module
            else:
                model_to_modify = self.model

            if self.params.freeze_middle:
                model_to_modify.freeze_middle()
            elif self.params.freeze_processor:
                model_to_modify.freeze_processor()
            else:
                model_to_modify.unfreeze()

            # See how much we need to expand the projections
            exp_proj = 0
            # Iterate through the appended datasets and add on enough embeddings for all of them.
            for add_on in self.params.append_datasets:
                exp_proj += len(DSET_NAME_TO_OBJECT[add_on]._specifics()[2])
            model_to_modify.expand_projections(exp_proj)

        checkpoint = None
        self.model = self.model.to(self.device)


BUGFIX: Replace deprecated ereg* with preg_*

@chillu this looks good to me - I believe that the affected functionality should be covered by the existing test suite?

Hmh, I tried the test before on browser where it failed because he could not found the AssetTableField ...
On shell it "works".
I will fix this issue:
Convert Test::test Html2raw
preg_replace(): Unknown modifier '['
<ul>preg_replace(/</[Tt][Dd]([^A-Za-z0-9>][^>]*)?> */,    ,This has a *strong tag*.)

This is much slower than the old, outdated, unsupported, and insecure ereg_replacei().


Searching by 'state' doesn't work

When searching (didn't test with advance search), the checkboxes to indicate which state should be included in the results are not respected, and instead all states are returned (rather, no states are filtered out).

done


Should we correct mispronunciations (phonetic spelling correction)?  This is especially trooublesome for foreign-sounding names

homoGraphPhone.pptx


Aeotec ZWA012 has incorrect parameters 22-24 and missing parameter 255

Reported in https://github.com/zwave-js/node-zwave-js/commit/ba6dbf63ce0bda1dbddb6bcb675ed0a98da9a587#commitcomment-55857202:

The AEOTEC Dorr/Window 7 Pro Sensor does not support parameters 22,23,24 and they should be removed from this. Please see the bottom of this page: aeotec.freshdesk.com/support/solutions/articles/6000230465 where you will see parameter 21 and then parameter 255. The sensor will not report status for 22,23, or 24 and will cause timeout issues during the interview process.

We seem to be missing Parameter 255 too.

Odd, those are in the engineering sheet. They must have removed them.


Updated worm DE tutorial to use new API

Hello, I finished updating the worm DE tutorial to use new API. I also added an example of using scvi.data.poisson_gene_selection to select genes.
I'm not 100% sure if my descriptions of raw_mean and raw_normalized_mean (copied below) are correct. In the docstring of model.differential_expression description of the returned object it only says "Returns: Differential expression DataFrame." It would be good if you could update the docstring to include a description of what each column in the returned DataFrame is.
Thanks!

Print the differential expression results

bayes_factor: The bayes factor for cell type 1 having a higher expression than cell type 2

proba_mi: p-value for cell type i having higher expression than the other

raw_meani: the average number of UMI counts sampled cell type i during the DE. Because scvi is a generative model, it is generating new cells and counting how many times each gene is sampled to compute the DE.

raw_normalized_meani: the average UMI counts sampled cell type i weighted by the number of draws made.

non_zeros_proportioni: proportion of non-zero expression in cell type i

norm_meani: average UMI counts in cell type i normalized by cell size

scalei: average scVI inferred gene expression scale in cell type i. This is the same value that can be obtained using model.get_normalized_expression() using library_size=1, equivalent to the "expression frequency" of the gene: what fraction of the UMIs sampled from that cell would belong to that gene.

I will review this soon. Thanks for contributing!

I can't review this notebook as is I think because it was directly imported from Colab. Could you download locally and open and save with your local jupyter? Also it would be helpful if you overwrite the notebook of yours in the contributed directory.

Print the differential expression results

bayes_factor: The bayes factor for cell type 1 having a higher expression than cell type 2

proba_mi: p-value for cell type i having higher expression than the other

raw_meani: the average UMI counts for gene in group i

raw_normalized_meani: the average library-size normalized expression for gene in group i

non_zeros_proportioni: proportion of non-zero expression in cell type i

scalei: average scVI inferred gene expression scale in cell type i. This is the same value that can be obtained using model.get_normalized_expression() using library_size=1, equivalent to the "expression frequency" of the gene: what fraction of the UMIs sampled from that cell would belong to that gene.

Where do you see norm_mean?

Ah sorry the norm_mean is old, I missed removing it. My doubts are about what raw_mean and raw_normalized_mean are.
I'm gonna submit a new PR with the notebook overwrite

Sorry I had put the definition in the post above
raw_mean is just the average UMI count for all the cells in that group.
raw_normalized_mean is the average library size normalized expression for that group (divide by library size, multiply by 10,000).


Scaleable Broadcast Issue

Disregard.  Mistake.  Number of viewers seems to be working ok.  I am playing with the Scaleable Broadcast and noticed that if I originate a broadcast from my Android phone, it will also display the "Number of Broadcast Viewers: "  but if I originate from my PC, it does not display the number of viewers?  Any idea why?
I was going to post this in the Scaleable Broadcast github,  but it is archived and read only?
Thanks,
Ray

Another issue with Scaleable Broadcasting.  After broadcaster starts, next viewer is ok,  2nd viewer joins ok, but then has wrong roomid stored. He has userid stored instead of roomid, so 3rd viewer cannot join? I am using Chrome and new tab for each viewer.  So roomid should be the same.
Anyone else see this?

Another issue with Scaleable Broadcasting. After broadcaster starts, next viewer is ok, 2nd viewer joins ok, but then has wrong roomid stored. He has userid stored instead of roomid, so 3rd viewer cannot join? I am using Chrome and new tab for each viewer. So roomid should be the same.
Anyone else see this?

To test, use Chrome browser:  Start with new default broadcast.  Random roomid assigned.
Open another tab and browse to page. Same roomid used. Join broadcast.
Open another tab and browse to page. Same roomid is used.  Join broadcast.
Open another tab.  Different roomid is used.  Cannot join broadcast,  but can start new broadcast.
This viewer should have received same roomid as other viewers!!
How to fix???

I tried replacing PC Chrome with 'replaceTrack' as well and it worked. E.g.
       connection.getSocket(function(socket) {
            socket.emit('can-relay-broadcast');

			connection.getAllParticipants().forEach(function(p) {
				if (p + '' != event.userid + '') {
					connection.replaceTrack(event.stream, p);
				}
			});
        });


if run:detached is used with -a app the instructions on viewing logs should have -a app

current
heroku run:detached bin/something -a an-app
Running `bin/something` detached... up, run.6553
Use `heroku logs -p run.6553` to view the output.
heroku logs -p run.6553
##either no output if you are in a dir with a git heroku repo, or no app error

desired
heroku run:detached bin/something -a an-app
Running `bin/something` detached... up, run.6553
Use `heroku logs -p run.6553 -a an-app` to view the output.
heroku logs -p run.6553 -a an-app
##yay logs

is this the same issue? https://github.com/heroku/heroku/issues/1264

not afaict

oh I see the problem, it's that the output command heroku logs -p run.6553 should show the app since it's not in the same context you're in now

yep!
On Thu, Jan 29, 2015 at 10:48 AM, Jeff Dickey<EMAIL_ADDRESS>wrote:

oh I see the problem, it's that the output command heroku logs -p run.6553
should show the app since it's not in the same context you're in now
—
Reply to this email directly or view it on GitHub
https://github.com/heroku/heroku/issues/1390#issuecomment-72080720.

check out #1391


Allow the use of flake8 (http://pypi.python.org/pypi/flake8) instead of pyflakes to check python code

Define a user-configurable variable g:syntastic_python_checker and
otherwise default to an available checker.
Make the highlighting catch all pyflakes errors
Add a header to the file
Make the errorformat handle columns provided by pep8 through flake8

I didn't see there was already something in Pull request https://github.com/scrooloose/syntastic/pull/102
I actually think one thing it has and that this pull request didn't include is the addition of
%f:%l:%c:\ E%n\ %m to efm
I didn't include it since then it might then make sense to also add %f:%l:%c:\ W%n\ %m but one ends with ambiguous numbers...
Since the choice was made in thus checker to only consider really bad errors as errors, I propose to keep the EXXX and WXXX as parts of the message...

I have merged this code in and will notify @claytron (author of pull request 102), in case he has any input  - since TBH i am not familiar with python.
Cheers :)

Ok, great and thanks. If @claytron wants the %f:%l:%c:\ E%n\ %m efm back, I won't have any complain (though I might prefer %f:%l:%c:\ %t%n\ %m then).

I had debated that particular error format. I think it is fine to leave it as you have it now.

Ok then.
To be honest, I think I'd prefer %f:%l:%c:\ %t%n\ %m but that would require to fact that currently, any "Error" will be replaced by the message "Syntax error" (I have no idea where this came from in the first place...)


Different color for current pixel

A different colored "current" pixel would be helpful to quickly tell what the latest status is. For example, a green pixel, or black.

Future is now gray, implemented by stolk in commit 16caf735e233b0c806e150d12bde5e3596c3596c


WIP. (nae nae.)


Executing a query with Skip() and Take() alters the query

Executing a query with Skip()/Take() alters the query unexpectedly, and breaks subsequent executions of the same query.
Eg:
When I run:
var result = await query.Skip((pageNum - 1) * pageSize).Take(pageSize).ListAsync();
I am not expecting the query to be modified, but it is, the Skip()/Take() statements are permanently added to the query.
Then when I run the same instance of the query but increment the pageNum, the SQL call fails because Offset/Fetch has been added twice.
Expected behavior:
Skip()/Take should not alter the query, unless I explicitly add them as part of the query.  Like so:
query = query.Skip((pageNum - 1) * pageSize).Take(pageSize)
Workaround:
Create a new instance of the query each time it is executed.
When I get a chance I'll dig into it, and send a PR.

Ok. Then all actions on a query should return new instances.

Ah, I didn't realize it was happening universally, or that it was by design.
This would be a big breaking change then.  Are you cool with that?

It's it mutates globally then we need to keep this behavior. And fix the fact that the inner sql is altered.
We could instead add a new method to create a snapshot of the query.


Conflicting constraints in the sample

This is due to the default constraints set by the system to match the autoresizing mask.
Log:
BaseView[16125:2837815] Unable to simultaneously satisfy constraints.
	Probably at least one of the constraints in the following list is one you don't want.
	Try this:
		(1) look at each constraint and try to figure out which you don't expect;
		(2) find the code that added the unwanted constraint or constraints and fix it.
	(Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints)
(
    "<NSAutoresizingMaskLayoutConstraint:0x7fecea4037f0 h=--& v=--& H:[UILabel:0x7fecea70b2e0'Some text'(0)]>",
    "<NSLayoutConstraint:0x7fecea417bd0 H:|-(0)-[UILabel:0x7fecea70b2e0'Some text'](LTR)   (Names: '|':BaseView.View:0x7fecea70c780 )>",
    "<NSLayoutConstraint:0x7fecea417c70 UILabel:0x7fecea70b2e0'Some text'.right == BaseView.View:0x7fecea70c780.right>",
    "<NSLayoutConstraint:0x7fecea404790 'UIView-Encapsulated-Layout-Width' H:[BaseView.View:0x7fecea70c780(375)]>"
)

Will attempt to recover by breaking constraint
<NSLayoutConstraint:0x7fecea417c70 UILabel:0x7fecea70b2e0'Some text'.right == BaseView.View:0x7fecea70c780.right>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKit/UIView.h> may also be helpful.

#11 closed by #12


Feature request: Add support for Multi-Agent Collaboration with Amazon Bedrock Agents

Use case
It would be great if the library supports creation of supervisor / collaborator agents as announced during Re:Invent 2024: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-multi-agent-collaboration.html
Solution/User Experience
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/create_agent.html has the "agentCollaboration" parameter and there are other methods like https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/associate_agent_collaborator.html to associate a collaborator to a supervisor
Alternative solutions
No response

Hi @maxritter ,
Thanks for using the Multi agent orchestrator.
I just want to clarify that we are not linked to the Bedrock Agent Collaboration.
Our framework doesn't create any resources. You create resources and you can bring them into the orchestrator.
As per Multi agent collaboration, the agent is an Bedrock Agent and bedrock agent is already supported as part of the framework: https://awslabs.github.io/multi-agent-orchestrator/agents/built-in/amazon-bedrock-agent/
So if you have created a Bedrock Multi agent collaboration agent, and you want to orchestrate it along with other agents that you have, you can already do so in the orchestrator.
Let me know if this clarify.
Anthony

Hey @brnaba-aws,
thanks a lot for the clarification, now I understand it!
Best,
Max

Hello everyone, I appreciate you setting up this forum for important conversations!
My name is Kundan Vyas, and I love technology and creativity. I think that by sharing my knowledge of full-stack programming, cloud optimisation (particularly on AWS), and AI-powered products, I can benefit this community.
Start fascinating conversations by posing challenging questions about topics such as effective CI/CD pipeline integration or methods for growing web apps.
Discuss my thoughts and experiences working on projects like creating a sign language translator with AI and automating processes to save cloud expenses.
Participate in the community to share your distinct viewpoints and work together on projects.
I'm eager to offer my experience and absorb knowledge from everyone.


[Suggestion] - secrets.json for sensitive data.

In my exploration with this project, I created a secrets.json file and added it to .gitignore (example below).
I feel it's good practice to get cloners off to a good start, discouraging them from commiting Gruntfiles with sensitive information.
@leemunroe - I'm happy to fork and send a PR if this is something you're into.
Gruntfile.js
Using <%= secrets.mailgun.api_key %> in the mailgun task.
grunt.initConfig({
        secrets: grunt.file.readJSON('secrets.json'),
});

secrets.json
{
    "mailgun": {
        "api_key": "YOUR MG PRIVATE API KEY"
    },
    "litmus": {
        "username" : "LITMUS USER NAME",
        "password" : "LITMUS PASS"
    }
}

I think it's a great idea. Been trying to figure out a solution to this that I can use (push/pull) seamlessly and that new users can understand.


Major cleanup to get the acceptance tests working

I can split this up into multiple PRs for a better changelog but for now I want to know how much of the tests would pass.

https://github.com/voxpupuli/puppet-mongodb/pull/494 is the next to be merged.

Includes https://github.com/voxpupuli/puppet-mongodb/pull/507 & https://github.com/voxpupuli/puppet-mongodb/pull/509

Updated to remove Debian-based distros since they include a 3.x version of mongodb. The tests simply do not work on them because of an auth issue.

I did consider leaving the shared_examples style in if we want to add testing with external repos again but perhaps it's better to revisit that when we get there.

Awesome!


Permissions

I have created several permissions in AdministrationPermissionKeys.cs and registered with the assembly as follows:
Administration:Security:Users
Administration:Security:Roles
I have added the permission attributes to the corresponding row classes and ServiceAuthorize and PageAuthorize attributes to the corresponding end points and pages, respectively. However, when I login with a user (not admin) that has no permissions they can still see and do everything. Breakpoints in the PermissionService and GetUserData method never get hit. It's like there is no security at all. Obviously I'm doing something completely wrong, but I have spent 2 days trying to figure it out and it just won't work. Please help! Thank you!

please see your permissions are listed under edit permission screen. if so please see if you can revoke them

It still does not work. I verified there are no records in the database assigning permissions to the user or role.

Please check at the level user screen edit permission tree .. is it listed there?

If a user has ReadPermission to a page they will see the page. Make sure that the user is profile on the client side has not been associated with a role or permission that allows them to see the restricted page.


listUser in Java client bug

Checklist

[x] I have looked into the README and have not found a suitable solution or answer.
[x] I have looked into the documentation and have not found a suitable solution or answer.
[x] I have searched the issues and have not found a suitable solution or answer.
[x] I have upgraded to the latest version of OpenFGA and the issue still persists.
[x] I have searched the OpenFGA Community and have not found a suitable solution or answer.
[x] I agree to the terms within the OpenFGA Code of Conduct.

Description
Hi, when i use the listUser method for OpenFgaClient in Java it arise an exception that say method listUser not found. Description and stack trace are reported in the log section.
OpenFGA version
openfga-sdk:0.7.1
How are you running OpenFGA?
As a binary
What datastore are you using?
Postgres
OpenFGA Flags

Logs
listUsers
dev.openfga.sdk.errors.FgaApiNotFoundError: listUsers
at dev.openfga.sdk.errors.FgaError.getError(FgaError.java:51)
at dev.openfga.sdk.api.client.HttpRequestAttempt.lambda$attemptHttpRequest$2(HttpRequestAttempt.java:97)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

My bad. The problem was with openfga server version I used (1.4.3). Now I have update to new version (1.7.0) and all works


[Event Request] table 167 Job - OnBeforeShouldSearchForCustomerByName

Please add new event in the Job table analog to the Sales Header:
[IntegrationEvent(false, false)]
local procedure OnBeforeShouldSearchForCustomerByName(CustomerNo: Code[20]; var Result: Boolean; var IsHandled: Boolean; var CallingFieldNo: Integer; var Job: Record Job)
begin
end;

procedure ShouldSearchForCustomerByName(CustomerNo: Code[20]) Result: Boolean
var
    Customer: Record Customer;
    IsHandled: Boolean;
begin
    IsHandled := false;
    OnBeforeShouldSearchForCustomerByName(CustomerNo, Result, IsHandled, CurrFieldNo, Rec);
    if IsHandled then
        exit(Result);

    if CustomerNo = '' then
        exit(true);

    if not Customer.Get(CustomerNo) then
        exit(true);

    exit(not Customer."Disable Search by Name");
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 11637.


Setting/Option to turn off RPi system LEDs

Not really an issue, more than a quality of life.
How tough would it be to add a setting to turn off the system LEDs to the 1.7 update?
No rush, was just something I had to do some searching for to turn it off and will probably have to re-do it for the 1.7 update.  Maybe.
Thanks though!

If you can find a link that shows how to-do that or have code/edits to share that would be a nice option for the future.

There is a thread here referring to how to turn leds off.
https://github.com/rootzoll/raspiblitz/issues/1564

@Vanmar1 I wrote this some time ago on telegram but I dont have lcd to test
From here https://raspberrypi.stackexchange.com/questions/84401/turn-pi-screen-on-and-off-programmatically
Command line:
OFF:
sudo -E sh -c 'echo 1 > /sys/class/backlight/rpi_backlight/bl_power'
or
echo 1 > /sys/class/backlight/rpi_backlight/bl_power
ON:
sudo -E sh -c 'echo 0 > /sys/class/backlight/rpi_backlight/bl_power'
or
echo 0 > /sys/class/backlight/rpi_backlight/bl_power
TIP: I define the above as backlight_off and backlight_on in ~/.bash_aliases. Be sure to save that file elsewhere, since you'll lose it if you re-image.
Never saw someone here mention this, so try with caution. Check if it is the same file, also notice this is for RPI3B, could be different on the 4.


Changing Llama Guard safety check to HF classes

What does this PR do?
Changes the Llama Guard safety check to use HF classes. Removes the previously used code to load Llama Guard.
Feature/Issue validation/testing

Llama guard enabled:

Command: python examples/inference.py --model_name ../llama/models_hf/7B --prompt_file examples/test_prompt_2.txt --enable_llamaguard_content_safety --enable_salesforce_content_safety False

Result

Llama Guard disabled:

Command: python examples/inference.py --model_name ../llama/models_hf/7B --prompt_file examples/test_prompt_2.txt

Result

Salesforce, Llama Guard and sensitive topics enabled:

Command: python examples/inference.py --model_name ../llama/models_hf/7B --prompt_file examples/test_prompt_2.txt --enable_llamaguard_content_safety --enable_sensitive_topics

Result

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Thanks for contributing 🎉!

@HamidShojanazeri The reference to the Llama Guard scripts in the relevant Readmes!
@QingHu1227 Referencing the prompt formatting methods in the stand alone script only. For the safety checker, I'm using the method provided by the tokenizer. We can evaluate adding custom categories to that as well in the future.


test: add EISDIR test

I couldn't find tests that check for the presence/absence of EISDIR. If the CI run turns up green, I'll see about adding some.

As https://github.com/libuv/libuv/pull/2025 already landed just a reminder.

I have a test in #2023, would that be good enough?

@thefourtheye You want to open a PR for that?

@bnoordhuis Sure, I have created #2029


[RKE2] Unable to add custom encryption configuration to yaml

Rancher Server Setup

Rancher version: v2.6-head 13482ca
Installation option (Docker install/Helm Chart):

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2

Proxy/Cert Details:

Information about the Cluster

Kubernetes version:
Cluster Type (Local/Downstream):

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

EC2 node driver cluster
User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

If custom, define the set of permissions: Admin

Describe the bug

When provisioning a cluster, user is not able to add custom encryption configuration to the yaml. The yaml file is editable during cluster creation, but once the user clicks Create, the custom configuration is removed and changes are not preserved.
To Reproduce

Create RKE2 cluster
Edit YAML and add the following configuration under rkeConfig and Click Create

services:
  kube-api:
    secrets_encryption_config:
      enabled: true
      custom_config:
        apiVersion: apiserver.config.k8s.io/v1
        kind: EncryptionConfiguration
        resources:
        - resources:
          - secrets
          providers:
          - aescbc:
              keys:
              - name: k-fw5hn
                secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I=
          - identity: {}

Result
The custom configuration is removed and secret is not created
Expected Result

The custom configuration should be preserved and secret should be created

Closing this issue as this is only applicable to rke1 and not rke2


fix warnings for braces on old clang

@tdegeus these were the two places where you pointed me in your issue about warnings for older versions on clang on OS X.
Hope that catches all!?

So far I can compile without any warnings!

cool, just gotta fix the beloved MSVC then...


lossless compression applied to sheets using ImageOptim with >12% savings

see https://slack-files.com/files-pub/T024BE7LD-F029K5Z1L-5b3453/pasted_image_at_2014_05_03_05_16_pm.png

this is too far behind master now. i should probably hook up optipng/whatever to the build process


orchestration

Hi,
Great tool ! Thank you very much for sharing it !
I use pepp on a daily basis and I run the same template of queries for numerous recordings. Currently, I need to store 1 config file / recording and then launch script one by one. It would be great to use a config file including several recording id that is called to execute the same script.
I've a similar issue with filters. For instance, i need to generate regularly 3 reports focused on France / UK and US. Countries may change according to the client needs. I use constants to restrict my queries, but I still need to store 3 different config files. Do you have any possibility to make more efficient tasks orchestration ?
Erwan / The Metrics Factory.

Hi Erwan,
Glad your finding the tool useful. If I understand correctly, you could just setup multiple index values. See https://github.com/haganbt/pepp#index-credentials. This enables you to set multiple recordings and reference the appropriate one from each task. There is an example of where the index keys sits in /config/demo.json.
That key has been in for a while, but if you do pull the latest changes, there has been a major refactoring (v2.0) to support the new upcoming /task api and as a result there are two changes you need to make to your old recipes to make them work. See here and instructions on pulling the latest changes are here.
Let me know if that meets you needs.
Ben.

@erwanlenagard - thinking about this further, do you need to generate 3 Tableau reports along with the output data?

Thanks. I already know this features but it doesn't fit my needs :
1 : you duplicate queries in the same config file, which is not very clean especially for maintenance
2 : number of recordings I need to extract in bulk change often
3 : recordings id may also change often
From my marketer perspective (I don't master all the stakes of developping with node.js), it would  be more efficient to have attached to my config file, a csv or txt including all the recording ids in need to extract in bulk. When I execute the script, it reads line by line the .txt passing recording id to query pylon api.
I don't generate tableau templates directly from pepp. I parse all csv with Talend and load in mysql to add complementary informations such as the date of extraction, name of the country, Facebook population, etc..
If you want we can schedule a call next week to show you how we use pepp and what would be our key features for a next version of pepp.

OK I see, that makes sense. Yes a call would be great. I will sync with Rakan and set something up for next week.

great ! :)


use orc 1.7

Checklist before submitting

[ ] Did you read the contributor guide?
[ ] Did you update any affected documentation?
[ ] Did you write any tests to validate this change?

Description
Update orc pin to 1.7.  Requires - https://github.com/open-ce/orc-feedstock/pull/1
Review process to land

All tests and other checks must succeed.
At least one maintainer must review and approve.
If any  maintainer requests changes, they must be addressed.

Closing this as it is not from my fork.


Update aws-java-sdk-dynamodb to 1.12.641

About this PR
📦 Updates com.amazonaws:aws-java-sdk-dynamodb from 1.11.1034 to 1.12.641
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.amazonaws", artifactId = "aws-java-sdk-dynamodb" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.amazonaws", artifactId = "aws-java-sdk-dynamodb" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #534.


pprof: add support for profiler labels

I propose adding a mechanism to the runtime and profiling code to allow for annotating profile samples with key-value labels. These labels are an already-existing feature of pprof that we do not support yet.
The user interface would look something like this:
    package context

    type ProfileLabel struct {
        Key string
        Value string
    }

    // DoWithLabels calls f with a copy of the parent context with the
    // given labels appended. The set of labels on that context will be set
    // for the duration of the call to f and restored once f returns.
    func DoWithLabels(parent Context, labels []ProfileLabel, f func(ctx Context))

More details in the proposal document

The proposal document has been submitted to the proposals repo. Please take a look.
https://github.com/golang/proposal/blob/master/design/17280-profile-labels.md

At first glance, I find the high-level API confusing. You're proposing taking the Context type, which already encapsulates a key-value store, and adding a second separate key-value store to it. I especially find the name "DoWithLabels" confusing - I naively expect it to behave similarly to WithValue, but it doesn't.
To my mind, this would be resolved by moving this outside the context package. I think this can just be a regular consumer of the Context type. i.e. we could move this to pprof.DoWithContext, and it can store the profiling labels in a regular Context key using context.WithValue.
It feels unfortunate to me that this has to be DoWithLabels and not just WithLabels returning a new Context. I understand how that is necessary, though, given the runtime API.
The fact that the argument is func(context.Context) means it is cumbersome to pass back any values such as an error from the function. Perhaps the func type should be func(context.Context) error? "Generics would help here".
Finally, don't you need a way to read out the current labels to pass them along with an RPC? It is my understanding that this is one of the underlying goals of the proposal.

I don't see how it makes sense to store all the Context keys as profiling labels.
Personally I think it's fine to use func(context.Context) as the closure can be used for any other values.  I don't feel strongly about it, though.
For an RPC I think the expectation is that the specific appropriate labels would be applied by the RPC server.  There is no necessary or appropriate correspondence between the complete set of labels used by the client and those used by the server.

I don't have a strong opinion about the package DoWithLabels appears in, but I think context is a sensible place to place it. runtime/pprof is okay also.
I have a slight preference for func(context.Context) just because it's a shorter signature. I think it's a wash between the clunkiness of assigning an error in the closure vs. returning an error and immediately assigning it when DoWithContext returns. But the context is necessary, so what we have now is the bare minimum.

Here's a change with a prototype of the interface (but not actually applying the labels to profiles).

@ianlancetaylor Not sure if that was directed at me, but I did not propose storing all the Context keys as profiling labels. I proposed, roughly:
package pprof
type ctxKeyType struct{}
var ctxKey ctxKeyType

func DoWithLabels(ctx context.Context, new Labels, f func(context.Context)) {
  labels := ctx.Value(ctxKey)
  newLabels = mergeLabels(labels, new)
  setLabels(newLabels)
  defer setLabels(labels)
  f(context.WithValue(ctx, ctxKey, newLabels))
}

This can be done in any package, and does not require any change to the Context interface or implementation.

@quentinmit I think we're all in agreement then... That's exactly how we want DoWithLabels to work. See the prototype here: https://go-review.googlesource.com/c/31145/6/src/context/context.go
I think the only question here is where DoWithLabels belongs. It coulb be in package context or package pprof. I don't have a strong preference.

Yep, I think we're all in agreement about how DoWithLabels should work. I still think that it needs a better name and/or a different package, because otherwise I find the current state confusing.
As for reading labels, your proposal currently says:
// ProfileLabels is an immutable map of profiler labels. A nil
// *ProfileLabels is an empty map of labels.
// There is intentionally no way to access the profile labels contained
// inside the ProfLabels because doing so could create a goroutine-local
// storage mechanism.
type ProfileLabels struct { /* runtime-internal unexported fields */ }

From talking to @ianlancetaylor, the thing we are trying to prevent is goroutine-local storage. I missed the fact that runtime.SetProfileLabels returns a pointer to the previous ProfileLabels struct. I don't see why we need that instead of just restoring the labels from the context as my code above does. Then ProfileLabels would not need to be opaque.
Either way, we need to add a way to get at the labels in a context. This could be a func pprof.LabelsFromContext(context.Context) [][2]string or similar (in the same package that DoWithLabels ends up in).

@quentinmit That comment is obsolete. I've sent a cl to update it. The "compromise" solution is to allow reading labels, but not on ProfileLabels returned by the runtime. So that allows us to read labels we keep in the context (because they're never set from a ProfileLabels returned by the runtime), while disallowing GLS.
It would be nice to have LabelsFromContext, but I don't think it's strictly necessary (at least for now). You can get all the nice pprof benefits of adding labels without needing to read the labels on a context. It would be really easy to add LabelsFromContext later. -- I'm not saying that we shouldn't add it, but we can (if we want) wait until later to do so.
Another option is to have LabelsFromContext return a runtime.ProfileLabels. Any ProfileLabels set on a context using DoWithLabels should be readable.

@quentinmit Here's an example of why we'd want to restore the previous ProfileLabels set on the goroutine rather than the parent context's ProfileLabels: Let's say a user wants to do some work that they don't want to have accounted to the current context. They might have code that looks like this:
 DoWithLabels(context.Background(), nil, func(ctx context.Context) { ... } )

When the DoWithLabels returns, they probably want to revert to the context already set on the goroutine rather than the background context.

Yeah, makes sense. (But really, don't call it "the context already set on the goroutine"! The runtime is not in fact tracking a Context object. :) This is a great example of the kind of confusion I think putting it in package context will cause.)

Eek! That's pretty bad...
I wouldn't mind moving DoWithLabels to package runtime/pprof if @ianlancetaylor and @aclements don't have any objections.

So the only point of discussion now is which package DoWithLabels belongs in, and maybe whether it should have a different name?

I've been stressing all along that I believe that since we already have a request-specific type, namely context.Context, that profiling labels should be associated with that type.  The user should be able to manage profiling labels entirely in conjunction with a context.Context value, because that is the value that is already being passed to every request-specific functions.  I agree that this needs to be carefully documented, but the goal should be for it to work for normal cases without requiring a deeper understanding.

@ianlancetaylor Is that an argument for adding LabelsFromContext? I think that's something we can do later [because it's not necessary to support profile labels], but I have no objections to adding it to the proposal.

@matloob I have no opinion about LabelsFromContext and I'm not sure I even understand what you are asking.  I am only saying what I said above: the user should be able to manage profiling labels entirely in conjunction with a context.Context value.  In particular I am rejecting the suggestion above that the user should be expected to call runtime/pprof/DoWithLabels to work with profiling labels.

@ianlancetaylor Okay. I think I misunderstood your previous comment. Please ignore the stuff about LabelsFromContext.
But I don't see why runtime/pprof.DoWithLabels wouldn't allow the user to manage profiling labels entirely in conjunction with a context.Context value. The only difference between context.DoWithLabels(from the proposal) and the proposalruntime/pprof.DoWithLabels` is the package the function is located in. They'd work with the context type the same way.

I think we should keep context.DoWithLabels as is. Is that okay?

Discussed offline with @ianlancetaylor and @quentinmit. It seems like there's general agreement for naming the function context.DoWithProfileLabels

Can we wait until we have more experience with it inside Google before adding it to the standard library? Or maybe we do. I haven't been paying attention.
Or could we add it to golang.org/x/net/ctxlabel or something for now, labeling its API as experimental?
I'd prefer not to add runtime API now. In particular, [2]string all over the place is gross. I'd make a type for that. Because this needs runtime support for the profiling bits, perhaps we could do a short-term hack (or GOEXPERIMENT?) to add support for the runtime without add new API for Go 1.8.

I think we have a fair amount of experience with this, but I would also prefer an experimental API if it's possible. What would a short term hack look like? Would we have a flag that switched on part of the API?
I don't have a strong preference between [2]string and struct {key string; value string}.

As one strawman, you could add some undocumented behavior to runtime/pprof and sneak in some undocumented private interface to any of the interface methods.
Like on,
https://golang.org/pkg/runtime/pprof/#StartCPUProfile
or
https://golang.org/pkg/runtime/pprof/#Profile.Add
Or add a global variable in the package (new API in Go 1.8), like:
   // Experiment is an internal hook for pprof experiments.
   // This is not a documented or stable API.
   var Experiment = experiment{}

And the lowercase experiment type have some public methods you could assert from the golang.org/x/net/ctxlabel package.

Put it behind a build tag?

In particular, [2]string all over the place is gross. I'd make a type for that.

This was not done without reason. If we use a type, context must use the exact same named type to avoid having to copy the entire structure just to change it from some type in context to some type in runtime (a flaw in the Go conversion rules, IMO, but it is what it is). Alternatively, we could declare a type in the runtime and use that in the context interface, but then you have to import runtime to call the context method (maybe that's okay).

I think a named type in the stdlib is okay.

I spoke to @matloob, @aclements, and @ianlancetaylor about this last week. I believe we agreed that:

The new API can go into runtime/internal/profile, so that it can be used by runtime/pprof but not other clients. This lets us have some flexibility in evolving the API.
The per-goroutine profiling tag should be an opaque value passed into the runtime and then passed back by the runtime when obtaining profiling data. The value needs to be a reference type that participates in garbage collection, or else it is too hard for the client code to GC its own associated state. That reference type could be an interface{} or maybe even an unsafe.Pointer (remember, this API can only be used by runtime/pprof).
The new call for doing something with extra labels can be pprof.Do. Ian had been arguing for a new method on context.Context, but since that's not possible (context.Context is an interface) he agreed that pprof.Do makes sense.

I suggest:
package pprof

func Do(ctx context.Context, labels []Label, func(context.Context))
func Labels(...string) []Label

and then code would look like
pprof.Do(ctx, pprof.Labels("foo", "bar"), func(ctx context.Context) {
    ...
})

I think in addition to Do and Labels, we'll want some way of extracting the labels currently set on the context- something similar to Len/AppendLabels in https://go-review.googlesource.com/c/31145/11/src/runtime/proflabel.go.

On hold for a few months since I hear this isn't happening for Go 1.8.

After talking to @rsc and @aclements I think we want something like this. This is essentially rsc's proposal above but with (WithLabels, SetLabels) to explicitly set labels, and (HasLabel, GetLabels) to look at the labels set on the context. Another important thing is that Do does not restore the labels that were set on the goroutine before it was called, but instead restores the labels on the parent context.
package pprof

type Label struct {
    // No exported fields
}

// WithLabels returns a context with the given labels appended to its
// label set.
func WithLabels(ctx context.Context, labels []Label) context.Context

// SetLabels sets the context's label set onto the current goroutine.
// This is a lower-level API than Do, which should be used instead when possible.
func SetLabels(ctx context.Context)

// Do calls f with a context with the given labels added. It calls SetLabels with that
// context before calling f, and SetLabels with the parent context after.
func Do(ctx context.Context, labels []Label, f func(context.Context)) {
    cctx := WithLabels(ctx, labels)
    SetLabels(cctx)
    f(cctx)
    SetLabels(ctx)
}

// Labels takes an even number of strings representing key-value pairs
// and adds them to
func Labels(...string) []Label

// HasLabel reports whether the context has a label set with the given key.
func HasLabel(ctx context.Context, key string) bool

// GetLabels returns an iterator over the labels set on the context.
func GetLabels(ctx context.Context) *LabelIterator

// LabelIterator is an iterator over the set of labels on the context.
type LabelIterator struct {}

// Next returns the next label in the iteration.
func (*LabelIterator) Next() (label Label, ok bool)

@alandonovan FYI

Some thoughts:

The Label type and Labels function seem merely decorative.  You could just use an even-length []string slice instead of []Label.  This would eliminate one of the three representations of key/value pairs (the other two being ...string and the opaque mapping accessed by GetLabels.

LabelIterator could be replaced by an anonymous function, that is, func GetLabels(context.Context) func(k, v *string) bool.  Client could would then look like:

iter := GetLabels(ctx)
var key, value string
for iter(&key, &value) { ... }

There is no single-key lookup function in the API only because we don't have a need for it today. Perhaps it should be added to the API since it is easy to do so, doesn't constrain the implementation in any way, and may be useful for debugging.

@alandonovan:

I could go either way with this. I tried to follow Russ's definition of Do.
I think that's a good idea.
I'd like to add as little as we can, but I don't have any other objections to a single key lookup.

Alternatively, LabelIterator could be replaced with a for-each callback. Something like
// ForLabels invokes cb with each label set on the context.
// cb should return true to continue iteration or false to stop iteration early.
func ForLabels(ctx context.Context, cb func(key, value string) bool)

instead of GetLabels.

@aclements Are there any pros/cons or performance issues either way? It looks like the usages of the two APIs would look very similar to each other.

I find the returned closure slightly obtuse, but that's a personal preference. It does have the advantage that you get to use a real "for" loop, so things like break and continue work like you'd expect.
Performance-wise, the returned closure would require a heap allocation for the closure object, while ForLabels could probably be done without any allocation. Otherwise I would expect them to be similar, since it's the same number of function calls, assignments, etc.

Ok, then I think we should go with ForLabels, unless anyone has a strong objection. Then we end up with this:
package pprof

type Label struct {
    // No exported fields
}

// WithLabels returns a context with the given labels appended to its
// label set.
func WithLabels(ctx context.Context, labels []Label) context.Context

// SetLabels sets the context's label set onto the current goroutine.
// This is a lower-level API than Do, which should be used instead when possible.
func SetLabels(ctx context.Context)

// Do calls f with a context with the given labels added. It calls SetLabels with that
// context before calling f, and SetLabels with the parent context after.
func Do(ctx context.Context, labels []Label, f func(context.Context)) {
    cctx := WithLabels(ctx, labels)
    SetLabels(cctx)
    f(cctx)
    SetLabels(ctx)
}

// Labels takes an even number of strings representing key-value pairs
// and adds them to
func Labels(...string) []Label

// HasLabel reports whether the context has a label set with the given key.
func HasLabel(ctx context.Context, key string) bool

// ForLabels invokes cb with each label set on the context.
// cb should return true to continue iteration or false to stop iteration early.
func ForLabels(ctx context.Context, cb func(key, value string) bool)

s/SetLabels/SetGoroutineLabels/ (to discourage accidental use)
s/ForEachLabel/ForLabels/
s/cb/f/
s/cb should/The function f should/ (sentences begin with capital letters)
body of Do should be
defer SetGoroutineLabels(ctx)
ctx = WithLabels(ctx, labels)
SetGoroutineLabels(ctx)
f(ctx)

otherwise this proposal looks good to me.

Okay. With @rsc's changes we end up with:
package pprof

type Label struct {
    // No exported fields
}

// WithLabels returns a context with the given labels appended to its
// label set.
func WithLabels(ctx context.Context, labels []Label) context.Context

// SetGoroutineLabels sets the context's label set onto the current goroutine.
// This is a lower-level API than Do, which should be used instead when possible.
func SetGoroutineLabels(ctx context.Context)

// Do calls f with a context with the given labels added. It calls SetLabels with that
// context before calling f, and SetLabels with the parent context after.
func Do(ctx context.Context, labels []Label, f func(context.Context)) {
    defer SetGoroutineLabels(ctx)
    ctx = WithLabels(ctx, labels)
    SetGoroutineLabels(ctx)
    f(ctx)
}

// Labels takes an even number of strings representing key-value pairs
// and adds them to
func Labels(...string) []Label

// HasLabel reports whether the context has a label set with the given key.
func HasLabel(ctx context.Context, key string) bool

// ForLabels invokes f with each label set on the context.
// The function f should return true to continue iteration or false to stop iteration early.
func ForLabels(ctx context.Context, f func(key, value string) bool)

LGTM. Possible we should change type Label and []Label to type LabelList and LabelList (not slice). As written there's nothing particularly useful you can do with the slice, so maybe the sliceness should be opaque.

It seems like the right next steps are:
CL #1: Implementation of and tests for type labels, the functional label set that we'll store in contexts. (Labels, WithLabels, HasLabel, ForLabels)
CL #2: Implementation of and tests for SetLabels and Do, including what happens with new goroutines and goroutines that die.
CL #3: Adding labels to runtime-generated profiles. This is the part that I said I'd help with, if I remember correctly.
It's OK to send those CLs during the Go 1.8 freeze. I will make some time to review them so that they can be ready at the start of Go 1.9.

Can you add a comment to WithLabels clarifying the behavior when a label with the same key already exists in the context. I think we agreed on "replace semantics".

When you say that Do calls SetLabels, I assume you  mean SetGoroutineLabels.
It's not obvious to me why HasLabel is useful, but if it is useful, it's really not obvious why it doesn't return the value of the label.

@ianlancetaylor. I think HasLabel could be useful because in some instances we might need the following behavior: "If set we do nothing. If not set then set with some value."
I guess the same can achieved by iterating thru the labels but I it will be very expensive.

@acetechnologist Well, I guess, but do you have an answer for the second part of the question--why not return the value?

Discussed with @acetechnologist offline. I don't think there's any reason not to return the value. I'm also clarifying below that setting a (k, v) where the key already exists will overwrite the previous value for the key (map semantics).
package pprof

type Label struct {
    // No exported fields
}

// WithLabels returns a context with the given labels appended to its
// label set. If labels with a given key already already set on the context, or
// appear more than once in labels, the last label appearing in labels with
// that key will replace the others.
func WithLabels(ctx context.Context, labels []Label) context.Context

// SetGoroutineLabels sets the context's label set onto the current goroutine.
// This is a lower-level API than Do, which should be used instead when possible.
func SetGoroutineLabels(ctx context.Context)

// Do calls f with a context with the given labels added. It calls SetLabels with that
// context before calling f, and SetLabels with the parent context after.
func Do(ctx context.Context, labels []Label, f func(context.Context)) {
    defer SetGoroutineLabels(ctx)
    ctx = WithLabels(ctx, labels)
    SetGoroutineLabels(ctx)
    f(ctx)
}

// Labels takes an even number of strings representing key-value pairs
// and adds them to
func Labels(...string) []Label

// HasLabel reports whether the context has a label set with the given key.
func HasLabel(ctx context.Context, key string) bool

// ForLabels invokes f with each label set on the context.
// The function f should return true to continue iteration or false to stop iteration early.
func ForLabels(ctx context.Context, f func(key, value string) bool)

In terms of documentation, I think we all put quite a bit of effort into wording the documentation on the previous version of this at https://go-review.googlesource.com/c/31145/. Obviously most of that can't be copied verbatim, but rather than starting from scratch, we should at least start from there. For example, the current doc for Do is a inward description of what it does rather than outward description of what it's for. In contrast, the documentation from DoWithLabels from that CL is much better and could mostly be reused here:
// DoWithLabels calls f with a copy of the parent context with the
// given labels added to the parent's label map.
// Each key/value pair in labels is inserted into the label map in the
// order provided, overriding any previous value for the same key.
// The augmented label map will be set for the duration of the call to f
// and restored once f returns.

Let's move forward with the CLs I outlined above and then we can fine-tune the doc comments in the actual code review.

@matloob, could you update https://github.com/golang/proposal/blob/master/design/17280-profile-labels.md with the new API? It is impossible to link to the final proposal at this point.

Hi, I've sent out a CL to update the proposal document with the actual changes: golang.org/cl/43530


(Error) Tactic got stuck!

val eval_tactic: tactic unit -> unit
let eval_tactic tac = assert_by_tactic True tac

This is expected, assert_by_tactic will run the tactic immediately at the point where it is written, much in the same way assert proves things statically before your function is called. What's happening here is that tac is not a concrete tactic and will unavoidably get stuck. We probably need a better error message here, though.
To make functions "tactic-parametric", you need the by_tactic marker, which will just wrap a formula with a tactic, but will not run anything until the formula needs to be proven. When that happens, the tactic is run on the formula and the remaining goals are fed to SMT.
For example, you can do this:
module E

open FStar.Tactics

val eval_tactic : tau:(tactic unit) -> Pure unit (requires (by_tactic tau (squash True)))
                                                 (ensures (fun _ -> True))
let eval_tactic tau = ()

let _ = eval_tactic (print "1")
let _ = eval_tactic (print "2")
let _ = eval_tactic (print "3")

That will not run anything for eval_tactic, but it will on each call site, as the precondition by_tactic tau (squash True) needs to be proven. You then get lines of output as expected.
As for the error, maybe we can quit early if the tactic expression has free variables. That needs some work regarding quoting, to not count those variables as free. I'm working on improving quoting too, so this should be easy in a couple of days.

I revisited this, and it doesn't seem as clear as "fail if there are free variables". For most examples, that's correct, but what about something like this (from #1272):
let broken (a: Type0) =
  assert_by_tactic a (fun () ->
                        apply (quote (unsquash #a));
                        let xx : a = admit () in
                        exact (quote xx))

While there might be different ways to accomplish this, we have a free a here in the annotation for xx, so that criterion would reject this tactic. We could distinguish between terms and annotations in this check, but that's bound to be brittle because of uniform syntax.
I'm tempted to leave this as-is, or maybe, if it gets really annoying, we could provide a different error when

the tactic got stuck
its head is an unbound variable
In that case, we give a less dramatic error message and say "Hey, this is probably wrong"


ENH: Implement Jacobian weighting during unwarp [backport gh-391]

Slightly different shaping than in the latest master, but I believe this is the equivalent. It would be good to get a test. @themeo would you be able to test? If you're using fmriprep-docker, it's pretty easy:
$ git clone https://github.com/effigies/sdcflows.git
$ git -C sdcflows checkout bp/2.5.x/391
$ pipx run "fmriprep-docker==23.1.4" [usual arguments] --patch sdcflows=$PWD/sdcflows/sdcflows

This will mount the sdcflows directory into the correct location to be picked up by fMRIPrep.

I'd be happy to give it a spin, but our cluster doesn't have docker (we use singularity). Is it possible to put the patched version together using singularity?

Yes, it's just a little bit less automated. You'll need to add the following flag to your binds, before the image:
-B $PWD/sdcflows/sdcflows:/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sdcflows

You can replace $PWD/sdcflows with the full path to wherever you clone the repository. Note that on the left side of the colon sdcflows should appear twice, like sdcflows/sdcflows:...

This fix led to a dramatic improvement in how fmriprep handles SDC.
original raw bold:

topup/applytopup on the raw image:

fmriprep with the fix:

This is not an entirely clean comparison, as the fmriprep-processed image has also undergone motion correction etc.
Congrats on cracking this!

Thanks for the fix! The boldref is SDCed now, but the T1w and MNI space outputs are still extra-distorted. Is it because the pipelines are not totally merged? PE-dir is RL, SDC with PEPOLAR.
Raw bold:

fmriprep 23.1.4. "vannila" boldref:

fmriprep 23.1.4. SDC-fixed boldref:

fmriprep 23.1.4 vannila and SDC-fixed T1w space boldref:

This is a good point, this will currently only affect files resampled into boldref space, since the resampling to other spaces is handled differently. I'm currently working on a tool that will resample directly to any volumetric space, but that will only be available with fmriprep 23.2.

This will need to wait.


Fix incorrect variable expansion when parsing PalServer.sh start options

The way the script is currently implemented, it passes all of the start options as one quoted string to PalServer.sh. This results in Palworld itself setting all of the options (including spaces) as a single argument, leading to the options not being properly applied.
This PR changes START_OPTIONS to a Bash array, where the required options are added incrementally as required to the array, and when ready, the whole array is expanded as individual arguments to PalServer.sh.
Fixes #186.

The cloud instance I use to run Palworld doesn't have fantastic single thread performance, but before I containerised it was working quite well regardless, with good distribution of load across all cores.
With this container, however, I was seeing this:

I looked at the output from ps to see what was actually running:
root@palworld1:~# ps aux | grep Pal | grep -v grep
ubuntu     26176  0.0  0.0   2480   520 ?        S    19:34   0:00 /bin/sh ./PalServer.sh  -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS
ubuntu     26183 97.3 10.4 4362972 1704516 ?     Rl   19:34   0:38 /palworld/Pal/Binaries/Linux/PalServer-Linux-Test Pal  -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS

You can see that there are two spaces between ./PalServer.sh and -useperfthreads. If variable expansion is working properly, that space wouldn't be there, and proves that  -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS is being passed as a single argument.
With this PR, however, I'm getting this:
ubuntu@palworld1:~/docker-palworld-dedicated-server$ ps aux | grep Pal | grep -v grep
ubuntu     27676  0.0  0.0   2480   520 ?        S    19:55   0:00 /bin/sh ./PalServer.sh -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS
ubuntu     27683  102 13.1 5116716 2152212 ?     Sl   19:55   9:09 /palworld/Pal/Binaries/Linux/PalServer-Linux-Test Pal -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS

And much better load distribution:

Hey @Callum027 - Thanks for the contribution. I merged your fix to develop, but somehow Github doesnt realize this, it even gave me the commands i used:
Step 1: From your project repository, check out a new branch and test the changes.

git checkout -b Callum027-186-fix-singlethread-running develop
git pull git@github.com:Callum027/docker-palworld-dedicated-server.git 186-fix-singlethread-running
Step 2: Merge the changes and update on GitHub.

git checkout develop
git merge --no-ff Callum027-186-fix-singlethread-running
git push origin develop

Did that and moved your code from the old delete servermanager.sh in the root to the one in scripts now.
Not sure why this PR is still open.
Sorry 😭

II think it might be because I force-pushed to rebase it with develop, and you probably were working on merging it at the same time. It's all good. Thanks for looking at it so quickly.

@Callum027
Im very sorry IF you did NOT get the contributor tag or so.
If that is the case, please send another PR with like a typo fix or so and i will merge that one!
Also please be aware this is only on develop, normally i do a image with develop tag, but i canceled this for extensive testing.
Im also planning to add a user-jail mechanic for the main process which enables me to add dynamic UID/GID switching.
As you might be able to imagine, im testing many many stuff, after the big refactor right now, there is no current develop image, thats to come tomorrow i hope.
Feel free to join Discord if you like.

@Callum027 Im very sorry IF you did NOT get the contributor tag or so. If that is the case, please send another PR with like a typo fix or so and i will merge that one!
Also please be aware this is only on develop, normally i do a image with develop tag, but i canceled this for extensive testing. Im also planning to add a user-jail mechanic for the main process which enables me to add dynamic UID/GID switching. As you might be able to imagine, im testing many many stuff, after the big refactor right now, there is no current develop image, thats to come tomorrow i hope.
Feel free to join Discord if you like.

No worries. If there's anything else I notice I'll make another PR. I'm already part of Discord (as Callum), so I'm already able to contribute and be contacted from there.

I'm already able to contribute and be contacted from there.

We have 2 "Callum" which one are you? Dont see a Callum027 😮

I'm already able to contribute and be contacted from there.

We have 2 "Callum" which one are you? Dont see a Callum027 😮

Both. The first one, I accidentally joined without signing in with my Discord account. I deleted that one and joined with my actual account, which is the second one listed in #welcome, and the one that has been posting in #chat.


Has input refactor (redux)

Do not merge until https://github.com/oborel/obo-relations/pull/291 is merged
Fixes #11066

Replaced by #16910


Doesn't work in 1.18 - needs update asap

Crashes out on 1.18, needs update immediately!!!

Entitled much?
Duplicate of #60


Update per-group-guest-access.md

Changing the cmdlet as Get-AzureADUser -Filter "userType eq 'Guest'" does not report the guest who are members.
Instead user we can use get-AzureADUser -all $true | ?{$_.CreationType -eq "Invitation"}

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/8434

@JohanFreelancer9 Please copy review

@MikePlumleyMSFT Please merge this PR. Thanks


ingest issue

ingest err.log
Traceback (most recent call last):
File "./ingest/mitie/mitie_entity_ingest_file.py", line 65, in
r = extract(email_id, body)
File "./ingest/mitie/mitie_entity_ingest_file.py", line 29, in extract_entities
for rng, tag in entities ]
ValueError: too many values to unpack

ingest tee.log
ingesting -<EMAIL_ADDRESS>working dir /srv/software/newman
ingest data
tx: 1
ingested count - 0
entity extraction
loading NER model...
Tags output by this NER model: ['PERSON', 'LOCATION', 'ORGANIZATION', 'MISC']
class 'mysql.connector.errors.InternalError

am i doing something wrong then ?

The ingest process has changed significantly.  Issue suggested closed by S. Mahoney.


Improve Custom D3 Formatting for Source Columns on table viz type

Is your feature request related to a problem? Please describe.
Currently not all viz types allows users to user d3 formatting for their data.  For example viz type big number with trendline allows this, but table viz type doesn't. There is a hacky workaround, where users can specify this in the edit Datasource --> Calculated Column section, but it would be much better user experience, is they can do it in the explore itself.
Current hacky solution:

Describe the solution you'd like
Enable users to easily define their column type on table viz, specifically custom d3 formatting.
Describe alternatives you've considered
Current solution works but is really hacky and non intuitive for users.
Option 1:  Let users edit when clicking on column name in Explore table viz.

Option 2: Add a dropdown with formatting options in Edit Datasource --> Calcualted Column section

.. Similar is available on big trendline viz type and other viz types

Option 3: Enable both option (1) and (2), where option (1) saves the settings, but can also be visible and edited via option (2). This would provide users most flexibility.
Additional context
It would be great to also go through all the viz types and if there are other which don't allow this functionality, enable this for all viz types with as much consistency in the settings as possible across.

If my understanding is correct, this is still missing.
We can't specify any format in data table viz except if we setup calculated columns. This is resulting in such tables:

Or else I am missing somthing...
I am using 0.38.0


Failed to read iceberg TPCH generated by snowflake

The table generated in this way:
CREATE OR REPLACE ICEBERG TABLE lineitem (
    l_orderkey       BIGINT,
    l_partkey        BIGINT,
    l_suppkey        BIGINT,
    l_linenumber     INT,
    l_quantity       DECIMAL(15,2),
    l_extendedprice  DECIMAL(15,2),
    l_discount       DECIMAL(15,2),
    l_tax            DECIMAL(15,2),
    l_returnflag     STRING,
    l_linestatus     STRING,
    l_shipdate       DATE,
    l_commitdate     DATE,
    l_receiptdate    DATE,
    l_shipinstruct   STRING,
    l_shipmode       STRING,
    l_comment        STRING
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'iceberg_external_volume'
BASE_LOCATION = 'lineitem'
CATALOG_SYNC = 'icebergPolaris';

INSERT INTO lineitem
SELECT * FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF100.LINEITEM;

I'm using iceberg-rust to read data from given table via polaris catalog:
select * from lineitem limit 1;

Got the following error:
1001=>iceberg data stream read: DataInvalid => Parquet schema table {
1: L_ORDERKEY: optional long
2: L_PARTKEY: optional long
3: L_SUPPKEY: optional long
4: L_LINENUMBER: optional int
5: L_QUANTITY: optional decimal(4,2)
6: L_EXTENDEDPRICE: optional decimal(8,2)
7: L_DISCOUNT: optional decimal(3,2)
8: L_TAX: optional decimal(3,2)
9: L_RETURNFLAG: optional string
10: L_LINESTATUS: optional string
11: L_SHIPDATE: optional date
12: L_COMMITDATE: optional date
13: L_RECEIPTDATE: optional date
14: L_SHIPINSTRUCT: optional string
15: L_SHIPMODE: optional string
16: L_COMMENT: optional string
}
and Iceberg schema table {
1: L_ORDERKEY: optional long
2: L_PARTKEY: optional long
3: L_SUPPKEY: optional long
4: L_LINENUMBER: optional int
5: L_QUANTITY: optional decimal(15,2)
6: L_EXTENDEDPRICE: optional decimal(15,2)
7: L_DISCOUNT: optional decimal(15,2)
8: L_TAX: optional decimal(15,2)
9: L_RETURNFLAG: optional string
10: L_LINESTATUS: optional string
11: L_SHIPDATE: optional date
12: L_COMMITDATE: optional date
13: L_RECEIPTDATE: optional date
14: L_SHIPINSTRUCT: optional string
15: L_SHIPMODE: optional string
16: L_COMMENT: optional string
}
do not match.

It appears snowflake writes parquet files with different schema when using iceberg tables. Should we support this scenario?

Yes, we should support this. I think https://github.com/apache/iceberg-rust/pull/786 fixes this.
Looking at the schema, only the decimal ones are different. And we support increased precision, as long as the scale stays the same:

Keep in mind that when people migrate from an existing Hive table to Iceberg (which is a pretty common scenario), we'll bump into this kind of thing more often, and I think that's okay to accommodate that, as long that it is perfectly safe.

@Xuanwo are you running into anything else? Curious to learn if it works :)

@Xuanwo are you running into anything else? Curious to learn if it works :)

Hi, it works great! Thank you for the fix (just in time I needed it!). Let's close.


Deprecate SETUPTOOLS_USE_DISTUTILS=stdlib

Soon, as Python 3.12 starts to roll out, users will be unable to use SETUPTOOLS_USE_DISTUTILS=stdlib. Moreover, Setuptools supporting this mode also adds complexity to the project and reduces the ability of distutils/setuptools to evolve to consolidate behaviors. Let's deprecate this mode.

If i am correct, this will break the workaround needed on #3301 on old pythons on Debian based systems.

Yes, perhaps. The deprecation wouldn't, but the subsequent removal might. There are different ways we could possibly ameliorate the situation:

those environments would pin to older Setuptools (maybe by way of pip constraints?).
extend distutils to support those environments.
wait for those environments to drop support
??

I've been proactively building Fedora packages without distutils to see what breaks. Figured out some setuptools tests fail. Hence: https://github.com/pypa/setuptools/pull/3636

I don't think that Python 3.11.2 is considered old.  However, I am unable to import setuptools without setting the environment variable that you want to remove.  Wouldn't it be better to fix setuptools first?
This is what I am seeing:
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-13<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import setuptools
Traceback (most recent call last):
File "", line 1, in
File "/Users/culler/Library/Python/3.11/lib/python/site-packages/setuptools/init.py", line 8, in
import _distutils_hack.override  # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/culler/Library/Python/3.11/lib/python/site-packages/_distutils_hack/override.py", line 1, in
import('_distutils_hack').do_override()
File "/Users/culler/Library/Python/3.11/lib/python/site-packages/_distutils_hack/init.py", line 77, in do_override
ensure_local_distutils()
File "/Users/culler/Library/Python/3.11/lib/python/site-packages/_distutils_hack/init.py", line 64, in ensure_local_distutils
assert '_distutils' in core.file, core.file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/distutils/core.py
import os
os.environ['SETUPTOOLS_USE_DISTUTILS'] = 'stdlib'
import setuptools
setuptools.version
'67.6.0'

Hello, Can I jump your project?


Allow debug flag to be a function.

I was thinking about making some Express middleware that will allow showing the queries that are being executed and how long they take.
At this point, the best that I can come up with is making the debug config a callable. I don't really like this, though, and I think it'd be better to listen for some events. The following events should suffice to build this:

query
queryComplete

The current query event doesn't seem to make it's way back to the knex instance all the time, though. I've created another pull request, #336, to resolve issues with the events making their way back.
Would you guys be open to adding another event for when the query completes?

Did this get abandoned? In the current release .8.5 there is no event that fires per query AFTER the query runs. Therefore no way to modify @wbyoung logger to tell the number of rows returned. I .7.3 I was able to do so.

@pursual I started working on a new tool for projects that I'm doing, so I haven't looked at this recently.


Error when using Chewy.transport_logger= method with elasticsearch 7.14

Chewy.transport_logger= raises error when used with elasticsearch 7.14.
But It woks with elasticsearch 7.13
Steps to reproduce the problem
require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  git_source(:github) { |repo| "https://github.com/#{repo}.git" }
  gem 'chewy', '~> 7.2'
  gem 'elasticsearch', '~> 7.14'
end

Chewy.transport_logger = nil

#  => /.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/chewy-7.2.2/lib/chewy/config.rb:67:in `transport_logger=': undefined method `logger=' for #<Elasticsearch::Transport::Client:0x00007f4a301c70b0> (NoMethodError)

Version Information

Chewy version 7.2.2
Elasticsearch version 7.14.1
Ruby version 2.7
Rails version 6.0

Please don't use elasticsearch >= 7.14.0: https://github.com/toptal/chewy/pull/811


Add 'How to fund a mainnet account'

What problem does your feature solve?
Adding an instruction on how to initial fund a stellar account on mainnet. On Stellar, a keypair isn't valid unless it's funded. Some wallets and exchanges like Lobstr and Coinbase fund an account for you if it isn't funded, but not all wallets do that for you so it can cause a confusion among users.
What would you like to see?
Add a description on which wallets and exchanges to use for initial funding on mainnet
What alternatives are there?

Hi @jeesunikim! Thanks for the recommendation here. Where do you foresee this going? We do have this guide: https://developers.stellar.org/docs/build/guides/basics/create-account and it directs folks to https://stellar.org/learn/lumens on acquiring lumens from exchanges. I think we could add more specific instructions on that page! I'll work on it.

Can I work on this issue @briwylde08

@Ileolami of course! We can collaborate.

That's great. How should we do that


[Enhancement] Turn on default block cache configuration in be.conf.

As we also have a system variable enable_scan_block_cache to control whether use block cache, so we turn on the default enable_block_cache in be.conf, to avoid restarting BE process if users want to use block cache.
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.1
[x] 3.0
[x] 2.5
[ ] 2.4

@Mergifyio backport branch-3.1

@Mergifyio backport branch-3.0


Support for async response handling and multicast/parallel. Included …

…local implementation of SpanManager to add support for transferring managed span stack between threads.

@davsclaus This PR adds support for handling responses in different threads, and multicast/parallel patterns.

Thanks for the PR it has been merged. Do you mind closing this?

@davsclaus Thanks!