Feat/add pymechanical

added pymechanical cheat sheet

Thank you @rs-bh-n for opening the PR, please provide the screenshot/pdf file of the PDF generated  in the description of the PR, so that it will be easy for docs team for review. Adding @EstherProsperi for the proof reading.

Hi @Revathyvenugopal162 and @EstherProsperi  , here is the snapshot of the pdf generated.

Hi @Revathyvenugopal162 and @EstherProsperi , here is the snapshot of the pdf generated.

Any chance I could have the actual PDF?

pymechanical_cheat_sheet.pdf
Here it is , @EstherProsperi .

pymechanical_cheat_sheet.pdf
Here it is , @EstherProsperi .

pymechanical_cheat_sheet.pdf

@rs-bh-n can you please adapt the review from Ester? if you encounter any issue/ roadblock, let me know i will try to adapt Thanks.

Thanks for your guidance , @Revathyvenugopal162  , in git push.
@EstherProsperi , Thanks for your inputs on wednesday :  Here is the pdf with the below changes . Let me know if this is okay.
pymechanical_cheat_sheet.pdf

Titles changed(same as in docs now. with underscores intact) :
A.Connect to a Remote Session of Mechanical from Python
B.Load an Embedded instance of Mechanical in Python

Retained the title "Launch and Connect to a Session " ( a and b are methods of doing this )

Local and Remote : Reworded , comment modified

Subtext changes :
"Example: Import file and print the count of bodies"
"Example: Import file and print the count of bodies"
"To turn on logging for warnings:"

5)python corrections : added missing import command .added black format .and  set the reference links as two rows.

Thanks @EstherProsperi , for your inputs.
Attached are the pdfs.
@Revathyvenugopal162 , the documentation approval is done.

merging!


use of @import when C++ modules are disabled

Expected Behavior
Unity project builds iOS target without issues
Actual Behavior
use of @import when C++ modules are disabled error during build
setting of Build Settings->Enable the modules (C and Objective-C)->True does not resolve the issue as the header file is being imported by .mm files which are import as Objective-C files and not Objective-C++ files by XCode.
Steps to Reproduce the Problem

Import 1.6.5 Unity SDK
Build iOS target in Unity
Start build in XCode
Error occurs

Specifications

Version: 1.6.5
Platform: Unity +iOS
Subsystem: N/A
XCode Version: 10.1

Is there a time frame on this fix? Running into this issue on Unity Cloud Builds at the moment and it would be nice to know when this is expected to be working.
Thanks.

I'll double check again and i'll release new version if everything looks good.

@milos1290 How's that update looking?

Fix was merged on iOS side, waiting for Android release before publishing new unity version, sorry for delay.

@milos1290 Was this fix included in the latest unity package? I see there was an update a few days ago.

not yet sorry, still waiting for iOS release before publishing new version of unity SDK.

Closing issue, fixed in 1.7.0


geteltorito: init at 0.5

Built on platform(s)

[x] NixOS

[x] Tested execution of all binary files (usually in ./result/bin/)

Update: Doesn’t depend on fetchdir anymore, uses a github mirror repo instead.

it's your own git... Just commit the .roff \ man page instead of depending on ronn :/

it's your own git... Just commit the .roff \ man page instead of depending on ronn :/

It’s nice, but I’m not particularly fond of adding my own stuff to the repo. This means I’d have to manually adapt it each time upstream updates.

Your call.
For reference:
{ stdenv, perl, fetchFromGitHub }:

stdenv.mkDerivation rec {
  name = "geteltorito-${version}";
  version = "0.6";

  src = fetchFromGitHub {
    owner = "Profpatsch";
    repo = "geteltorito";
    rev = version;
    sha256 = "05bcn3pam29xmsz1ykyqsdbkz8y23kcrvvhm987f65wd1g741f75";
  };

  nativeBuildInputs = [ perl ];
  dontBuild = true;
  geteltorito_mdoc = ''
    .Dd 2015-02-25
    .Dt GETELTORITO 1
    .Os Unix
    .Sh NAME
    .Nm geteltorito
    .Nd an El Torito boot image extractor
    .Sh SYNOPSIS
    .Nm
    .Op Fl hv
    .Op Fl o Ar outputfilename
    .Ar cd-image
    .Nm
    .Ar cd-image
    >
    .Ar outputfilename
    .Sh DESCRIPTION
    .Nm
    will try to extract the initial (default) El Torito image from a bootable CD (or cd-image) given by
    .Ar cd-image
    and write the data extracted to STDOUT or to
    .Ar outputfilename \.
    All other information is written to STDERR (e.g. type and size of image).
    .Pp
    .Nm
    will not extract any of other possibly existing bootimages that are allowed by the El Torito standard.
    .Pp
    .Bl -tag -width "outputfilename" -compact
    .It cd-image
    bootable CD or cd-image
    .It outputfilename
    optional output file name
    .El
    .Bl -tag -width -indent
    .It Fl h
    Build-in help.
    .It Fl v
    Print version of script and exit.
    .It Fl o
    Write extracted data to
    .Pa outputfilename
    instead of
    .Pa STDOUT \.
    .El
    .Sh ABOUT
    .Bl -tag -width "Version:"
    .It Author:
    Rainer Krienke
    .It Email:
<EMAIL_ADDRESS>    .It License:
    GPL v2
    .It Version:
    0.6
    .El
    .Sh FILES
    .Bl -tag -width "/usr/share/man/man1/geteltorito.1" -compact
    .It Pa /usr/share/bin/geteltorito
    The
    .Nm
    perl script.
    .It Pa /usr/share/man/man1/geteltorito.1
    This help file.
    .El
    .Sh SEE ALSO
    .Xr perl 1

  '';

  installPhase = ''
    mkdir -p $out/man/man1
    echo "${geteltorito_mdoc}" > $out/man/man1/geteltorito.1
    install -vD geteltorito $out/bin/geteltorito
  '';

  meta = with stdenv.lib; {
    description = "Extract the initial/default boot image from a CD image if existent";
    homepage = "https://userpages.uni-koblenz.de/~krienke/ftp/noarch/geteltorito/";
    maintainer = with maintainers; [ profpatsch ];
    license = licenses.gpl2;
    platforms = with platforms; all;
  };

}


Change Function .toString() output to be Emscripten compatible

Change Function .toString() output in two minor ways to make it Emscripten compatible without the need for fixups:

If the function is anonymous, don't use a dummy "anon" name for it. For example, instead of function anon() {...} output function () {}.
Use a string literal (similar to a directive) instead of a comment to indicate function type. For example, instead of function foo() {/*ecmascript*/} output function foo() {"ecmascript"}

While both the old and new formats are equally valid from a compliance perspective, it'd be nice if no Emscripten fixups were needed. The anonymous function behavior is, in my opinion, an improvement; the function type indication is more a matter of style.
Tasks:

[ ] Finalize change
[ ] Fix breaking testcases
[ ] Add a dev testcase for new format with explanation
[ ] Internal documentation, esp. Emscripten status
[ ] Update fix_emscripten.py to indicate it's no longer required (at least with current Duktape and Emscripten versions)
[ ] Update Makefile to avoid the hopefully unnecessary fixups
[ ] Releases entry

I agree with the anonymous function change.  The old way was misleading: an anonymous function has no name by definition, while Duktape actually returned a named function. (the name happened to be "anon" but it was named nonetheless).

What interests me about the Emscripten behavior is that it can potentially break under any engine which provides the real function source code in .toString().  SpiderMonkey does this, for example.  It amazes me that this hasn't yet caused real-world issues outside of Duktape.


Doesn't work after bundling application

After I bundle my application with the badass runtime plugin, it doesn't work. I get a long error: https://hastebin.de/avaxetebuf.sql

This is not enough information. Use the bug template and describe your project, build and the error in detail.
We cannot guess what you are doing.


Mac: file events are not reported when using workspace path with different casing

Interestingly on Mac when opening VS Code on a workspace path that does not match 100% the casing of the workspace as it is on disk, all file events are not working anymore, even though the rest of the program is functional.
Filed: https://github.com/paulmillr/chokidar/issues/418

Verify: (Mac only) you can open a folder from the command line into VS Code that uses not the letter casing as it is on disk and you still get file events properly.

Reopening because it looks like node.js 6.4.0 will revert the nice fs.realpath to go back to the previous JS implementation and behaviour.

Merging into https://github.com/Microsoft/vscode/issues/12448

[ ] To verify: on macOS run Code on a folder path that is not case identical to the version on disk and verify that you get file events to work (e.g. changes outside are reflected in explorer)


Memory can not been freed

Delete the key of a map,memory can not been freed.  How to Solve the Problem

Not sure what you mean. If the thing that was in the map is no longer reachable anywhere else, it will be garbage-collected.


add shm support for NetBSD and OpenBSD

initial work from @landryb for OpenBSD, various fixes and NetBSD support from me.
Fixes #1585
shm support is need for firefox building (for slice-deque)
cc @jakllsch @ryoon for NetBSD

r? @gnzlbg
(rust_highfive has picked a reviewer for you, use r? to override)

@bors: r+

:pushpin: Commit 487b454d9f8c73b6a611d85d1e7d6820fece9938 has been approved by gnzlbg

@bors p=100

:hourglass: Testing commit 487b454d9f8c73b6a611d85d1e7d6820fece9938 with merge 5e5c1ee62a52558dee08984bdc3811c6e0072219...

:sunny: Test successful - checks-cirrus-freebsd-10, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, status-azure
Approved by: gnzlbg
Pushing 5e5c1ee62a52558dee08984bdc3811c6e0072219 to master...


PostLoad event on fetch joined entities - Lifecycle Callback not firing

I have a problem with a Lifecycle Callback of Doctrine ORM: the callback is not executed on fetch-joined entities, while it's regularly executed on entities that are lazy-loaded.
Is there a bug in the ObjectHydrator, or it is a lack of documentation about that?
See this question on stackoverflow

Any load operation should trigger postLoad events, so this is a bug. Can
you submit a test case to be added to our test suite?
On 21 Jul 2017 09:28, "stoccc"<EMAIL_ADDRESS>wrote:

I have a problem with a Lifecycle Callback of Doctrine ORM
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/events.html#lifecycle-callbacks:
the callback is not executed on fetch-joined entities, while it's regularly
executed on entities that are lazy-loaded.
Is there a bug in the ObjectHydrator, or it is a lack of documentation
about that?
See this question on stackoverflow
https://stackoverflow.com/questions/45211316/doctrine-postload-lifecycle-callback-not-executed-in-fetchjoin
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/doctrine/doctrine2/issues/6568, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJakNngALqVjHN9rC7paj5DntVTSGtVks5sQFMigaJpZM4OfDYL
.

I'm not so expert, I hope I did the right thing!

@stoccc test looks good, thanks!

@Ocramius
I fixed the patch, and now all tests pass correctly.
I think you should remove Bug label, but you gained a test case :)
The problem was in my application, I was using in PostLoad of the root entity some informations setted in PostLoad of the fetchjoined entities, but certainly no one can guarantee me that one PostLoad is executed before or after some other PostLoad.
I close, thanks for your time.


Organize example folder structure for archive upload by interface slug?

If an archive item consists of multiple files, the example cases are currently organized into folders by case. The upload script then relies on the user listing the files in EXPECTED_CASES in the correct order (file for ci-1 first, file for ci-2 second) per case. It might be more fool-proof to have subfolders per case for each interface with each folder containing only one file. The file paths in EXPECTED_CASES would then be self explanatory. This is also how the input is presented to the algorithms etc, so would be nicely consistent.
Just a suggestion though ;-)

Good suggestion! I even had that as the initial structure! However, I am pretty sure the current format is far more common to see for structured cases: just a directory per case.
Of course, that is just an assumption and likely no two users will have the same data structure. I opted for simplicity here, it should be pretty straight-forward for people to generate a list of files per case. Instead of including pointers to slugs I have explicitly made a map_case_content_to_interfaces() as to highlight that there is an ordinal step in there.

Ok, fair enough. If it turns out that organizers have issues with this, we can always change it in the future. For now we just need to make sure that we explain to them that the order matters.

Also, fools are ingenious.


Test UI should surface continuous run on gutter test icon

CC @connor4312
The Turn On Continuous Run option should also be surfaced in the right click menu for the gutter icons

I'll try to fix this

@connor4312 Hi, is this issue not accepting PR?

We would, I just don't think the solution of what to display is very obvious so I removed the tag

We would, I just don't think the solution of what to display is very obvious so I removed the tag. Ideas/contributions welcome though.

Thanks!, By the way, isn't it only issues with the help want tag that accept external PRs? That's what I've always thought XD

Not necessarily. Personally I use the tag when the issue has a clear+scoped solution and is not something I'm going to get to in the immediate future.

Why wouldn't it just be exposed as Turn on Continuous Run at the gutter level and then hook to the same function/event? What am I missing that's non-obvious?

I think that if we have the context menu option, we should also reflect the state of continuous run being on on that icon as well, which is the trickier part. I would take a PR adding the action alone, but would want the state to be reflected before I would consider this "complete"

The status would be reflected in the test explorer, and conversely when you flag a test for continuous run in the test explorer it doesn't update the gutter, either way though it probably would be good to implement if things are changing anyways.
My thoughts:

Make the gutter icon the same eye codicon upon activation, with color still matching the last test result (light green/green/red)
All other "processing" occurs as normal

Issue:

What if a test is in suspended state or otherwise where the color conflicts with an existing status color? How do we represent a test is both watched and suspended or do we care?

Alternatives:

Some alternate indicator that doesn't affect the gutter icon directly but would be next to it. Highlighted line maybe?


But after saving then i go to file manager i cant find the captured png image

What else can I do

Please check in the file manager.
You will see it under your main storage.

Please check in the file manager.
You will see it under your main storage.


add icon to collabora app provider in deployment example

Description
Add an icon to the Collabora app provider driver in the deployment example

Code LGTM, question is how it ends up in my frontend code? 🤔

Code LGTM, question is how it ends up in my frontend code?

you will have it in the /app/list endpoint ;-)
"application/vnd.ms-excel": {
      "app_providers": [
        {
          "name": "Collabora",
          "icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
        }
      ]
    },


Generation should allow user to specify max length of generated portion, rather than total

The max_length arg requires us to specify a max length of the input and output combined. Also, it represents the number of tokens rather than chars. This isn't useful because we can't precisely calculate the number of tokens in the input; we can only get an estimate. So there is no way to specify to the model whether we want it to generate 1 extra token or 200 extra tokens. I don't understand why there isn't already a way to specify the number of tokens to generate rather than total tokens, or why it would ever be more useful to specify total number rather than generated number.
Is there any workaround to this? I looked into GPT2Tokenizer but still am not sure how to get the # of tokens in the input.

Bump, in case anyone knows the answer (I suspect there is an easy way to get the number of tokens in the prompt and then calculate the desired tokens to generate as a workaround, and I'm just not understanding it)

I am a little confused because the documentation seems to imply the length of input_ids will be the number of tokens https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt2/tokenization_gpt2.py#L115
But when running the code input_ids is always a length of 1

For the workaround: It's a 1-element list whose only element is the real list, so Just do input_ids[0] to get token length, and add this to the specified max tokens.

I only just discovered the max_new_tokens arg in the documentation which seems to be expressly for this purpose. Weird that no one's mentioned it yet

I'm sorry that you're frustrated by the fact we weren't able to get to this issue, but I'm glad you found what you were looking for by reading the documentation.


feat: build and release for arm platform

fixes #739

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Line break in heredoc is not honoured

Prerequisites
Put an x into the box that applies:

[x] This issue describes a bug.
[ ] This issue describes a feature request.

For more information, see the Contributing Guidelines.
Description
String in terraform heredoc requires extra line break to render properly in markdown. This was fixed in 0.8.0. But broken again in 0.8.2. See https://github.com/segmentio/terraform-docs/issues/167
Actual Behavior
Given follow variable, I expect the description to be formatted as shown in markdown. However, no line break is rendered between image: and uid:
pretty version works as expected
variable "envoy_config" {
  description = <<-EOT
    Configuration for Envoy proxy container

    image:            The image tag in ECR for appmesh envoy. Default to aws-appmesh-envoy:v<IP_ADDRESS>-prod
    uid:              The user identifier used to run container under a specific user. Default to 1337
    ingress_port:     The port used by envoy to accept inbound traffic. Any requests to the contianer port will be redirected to this
    egress_port:      The port used by envoy for outbound traffic. Any outboud traffic will be redirected to this
    log_level:        The log level of envy. Default to info
    egress_ignore_ip: The egress traffic going to the specified IP addresses is ignored and not redirected to envoy_egress_port
  EOT
  type        = object({ image = string, uid = string, ingress_port = number, egress_port = number, log_level = string, egress_ignore_ip = list(string) })
  default = {
    image            = "aws-appmesh-envoy:v<IP_ADDRESS>-prod",
    uid              = "1337",
    ingress_port     = 15000,
    egress_port      = 15001,
    log_level        = "error",
    egress_ignore_ip = ["<IP_ADDRESS>", "<IP_ADDRESS>"]
  }
}

Do you have long logs to share? Please go to https://ghostbin.com and insert the link here.
Expected Behavior
Render new lines as it is in heredoc for markdown

Replication Case
Tell us which steps to take to replicate your problem.

...
...
...

Version
terraform-docs version v0.0.0- v0.8.2 darwin/amd64 BuildDate: 2020-02-03T18:39:25+0000

Hmm, strange. How did you test this, with downloading the released binary or building the HEAD?
This is supposed to be fixed in #169, specifically I added these lines to example and tests keep passing. https://github.com/segmentio/terraform-docs/blob/5a05b953496081556fd4c723161994c3f83b06d7/examples/main.tf#L9-L10
The way it's supposed to work is if consecutive lines are:

of a paragraph: double <space> will be added to the end of each lines to make markdown engine treat them as a "carriage return"
of a markdown list: no extra <space> will be added, because markdown will know what to do with them.

I'm in the middle of adding extensive test cases in #208 and I'll make sure to add specific test case for this in it, and if need be fix it, otherwise I suspect you might have a mechanism (e.g. .editorconfig) to strip put extra spaces from .md files which will be problematic in this case and I'd suggest to make sure that's not the case.

Hi @khos2ow,
Thanks for looking into this. I use the binary directly downloaded from github Release.
I also think I find out which works, which not.
In the below snapshot, Key2 works as expected, Foo1 and Foo2 are on different lines.
Key doesn't work, as there are two spaces in front of Foo1 and Foo2.
Both worked in version 0.8.0.
variable "test" {
  description = <<-EOT
    This is is a multline test which works

    Key
      Foo1: blah
      Foo2: blah

    Key2
    Foo1: bar1
    Foo2: bar2
  EOT
  type = string
  default = ""
}

That's interesting, I added a test and a fix for this particular case (i.e. consecutive lines with whitespace at the beginning of the lines, pkg/tmpl/sanitizer_test.go#L245-L256 and pkg/tmpl/testdata/multiline/indentations.golden).
Can you try to build the HEAD of master and try to see if you still have the problem?

Yes, it works now. Thank you!

Fixed as part of #208


content(cms): create Tag "skos"

Automatically generated by Netlify CMS

going to close this as it seems to have gone stale. feel free to reopen, or create a new one, if this is still needed.


cx_oracle module import fails due to restrictive umask

/roles/cxoracle/tasks/main.yml
We needed to add a more forgiving umask for the cx_oracle install task, otherwise only the root user can import the module.
pip: name=cx_oracle state=present extra_args="{{ extra_args }}" umask=0022

Fixed with 344325bc6b7a35376c0e32a3e79f29c5bf0c476d.
I've never seen issues with this, so I'm leaving the default as is, but I've made it configurable
e.g
cx_oracle_umask: 0022
The task does:
umask: {{ cx_oracle_umask | default(omit) }}

Confirmed that this works, however please note that when defining this parameter the value will need to be quoted:
cx_oracle_umask: "0022"


Provide CUDA context check function for multiprocessing

Prompted by #7309, it has been discussed how to give a comprehensible errors for mistakingly created CUDA context before fork (e.g. in MultiprocessParallelUpdater) without calling multiprocessing.set_start_method(). This is the summary of conclusion:

[ ] Create a function to ensure that no CUDA context has been created (using cuCtxGetCurrent).
[ ] Call the function at MultiprocessParallelUpdater.__init__().
[ ] As for #7309, it's ok to check the number of CUDA devices in chainer.get_device(). Those users who followed the introduction of chainer.get_device (v6) and are using MPU might be affected, but the error will be comprehensible enough.
[ ] Write in the upgrade guide about the check function and the change of behavior of MPU.
Those users who do their own multiprocessing might be given non-comprehensible errors. But this is highly advanced usage and we can consider they're responsible of doing a correct multiprocessing way.

Additionally,

[ ] Call multiprocessing.set_start_method() in examples using MultiprocessParallelUpdater which should have been done in its own.

We missed that MultiprocessParallelUpdater already does such check.
Only what should be done is to check the device count in  chainer.get_device (#7309).


Test code for flasher fractional error investigation

I've refactored my test code to make it more parameterisable with different NSB rates, illuminations and calibration regimes and to factor out the common code to determine errors / moving averages. This code requires my get_flasher_illumination changes in source.py to run.
There are various methods to show that we get flasher output and that uniform illumination works as this was changed to accept flasher errors as we discussed before :
multiple_flasher_call, single_flasher_call_no_err, single_flasher_call_with_err, plot_multiple_flasher, plot_uniform_illumination
The most interesting methods are PlotFractionalError which subtracts average of 20 NSB runs from  the measured and true charges as a pedestal and determines the fractional error (measured - true / true) from the moving average of measured and true . This recreates plots like those shown on the calls:

We can now also answer questions like "How many flashes reach a target fractional error ?" using GetNumberOfFlashesToReachTargetFractionalError. This is for a single pixel currently.
I can now specify a configurable observation regime that has the following steps (for a single camera pixel):
1)Specify number of NSB obs at a given rate  to take and compute an average, "a pedestal" (i.ve tried 20 and 50 NSB obs)
2) Specify flasher illumination, pulse width and error. Take a moving average over an incrementing number n of flasher events with NSB as step 1 with pedestal subtracted. Verify that target fractional error is achieved for a configurable number of independent but consecutive "runs" each with n flashes.
NB Fractional error (FE) is defined as Measured charge - True Charge / True Charge. True charge is flasher charge without NSB . Measured charge is flasher+NSB but with pedestal subtracted
With this scheme I can get fractional errors of 1.5% (pushing this lower only works for low NSB ) at the 100 and 200 p.e. flasher illumination level for approx 100 flasher events across NSB up to 300 MHz and approx 150 events for 1000 MHz (moonlight) as my test outputs below:
"/home/sheridan/miniconda3/bin/python /home/sheridan/sstcam-simulation/test_flasher/call_flashers.py
Begin GetNumberOfFlashesToReachTargetFractionalError - no of NSB obs 20
Illumination,100,NSB,50,Target Fractional error,0.015 achieved for 10 runs in 33 flashes
Illumination,100,NSB,100,Target Fractional error,0.015 achieved for 10 runs in 24 flashes
Illumination,100,NSB,200,Target Fractional error,0.015 achieved for 10 runs in 19 flashes
Illumination,100,NSB,300,Target Fractional error,0.015 achieved for 10 runs in 90 flashes
Illumination,100,NSB,1000,Target Fractional error,0.015 achieved for 10 runs in 177 flashes
Illumination,200,NSB,50,Target Fractional error,0.015 achieved for 10 runs in 10 flashes
Illumination,200,NSB,100,Target Fractional error,0.015 achieved for 10 runs in 8 flashes
Illumination,200,NSB,200,Target Fractional error,0.015 achieved for 10 runs in 17 flashes
Illumination,200,NSB,300,Target Fractional error,0.015 achieved for 10 runs in 55 flashes
Illumination,200,NSB,1000,Target Fractional error,0.015 achieved for 10 runs in 156 flashes"
And for 2% FE target
Illumination,100,NSB,50,Target Fractional error,0.02 achieved for 10 runs in 18 flashes
Illumination,100,NSB,100,Target Fractional error,0.02 achieved for 10 runs in 18 flashes
Illumination,100,NSB,200,Target Fractional error,0.02 achieved for 10 runs in 25 flashes
Illumination,100,NSB,300,Target Fractional error,0.02 achieved for 10 runs in 29 flashes
Illumination,100,NSB,1000,Target Fractional error,0.02 achieved for 10 runs in 89 flashes
Illumination,200,NSB,50,Target Fractional error,0.02 achieved for 10 runs in 12 flashes
Illumination,200,NSB,100,Target Fractional error,0.02 achieved for 10 runs in 18 flashes
Illumination,200,NSB,200,Target Fractional error,0.02 achieved for 10 runs in 15 flashes
Illumination,200,NSB,300,Target Fractional error,0.02 achieved for 10 runs in 6 flashes
Illumination,200,NSB,1000,Target Fractional error,0.02 achieved for 10 runs in 73 flashes
There is variability in all this on successive runs - I am not implying these are the optimal number of flashes, just ballpark
We can discuss how this all makes it into a notebook - might need some guidance there

Following good discussion with Jason on the next investigation, I've modeled the effect of decreasing over-voltage with increasing NSB, which Jon L. has shown decreases SiPM response to a laser pulse by 0.016% per MHz increase in NSB.
The method I've coded up is :

Initial pedestal subtraction at 50 MHz NSB
Increase NSB in a linear fashion 50 - 1000 MHz
Intersperce this NSB increase with flasher calibration runs where I measure the combined pedestal corrected flasher illumination + NSB ( "measured illumination" in attached plots), with each run having some number of flashes and flash rate
I simulate the decreased silicon response by dropping the flasher illumination, (so called "true illumination") depending on the NSB at the time of the flash
I determine the root mean square error (RMSE) to quantify the mis-calibration between "true" and "measured"

My code also now uses  the common NSB pedestal subtraction and charge extraction routines in the simulator which should supresses NSB in relation to flash signal as it considers a narrow 20 ns window rather than whole waveform integration.
I attach some plots which show how "measured" and "true" vary with different flash rates, NSB rate of change, and number of flashes in each calibration run (which are ten minutes apart):
VaryFlasherRate_5_20_Hz_Initial_NSB_50MHz_100Flashes.log
VaryNo_Of_Flashes_Initial_NSB_50MHz_1.log
VaryNSB_ROC_Initial_NSB_50MHz_100_Flashes.log

The plots show a largely linear relationship between measured and true (as expected) whose linearity improves with number of flashes. The  NSB rate of change plot is less useful as with increasing rate of change, we have fewer calibration runs and hence points on the plot, before getting to 1 GHz NSB.
I also generated some output log files (attached above) with my stats for these runs and the variable parameters for each run.  In the files "MPRC" is "measured" and "TPRC" is "true". The take home message is that the mis-calibration RMSE varies from approximately 4 to 14 p.e. which for an initial flash of 100 p.e. can be taken as 4 to 14 % miscalibration which is within the stated requirement of 20 % . This is pretty much as expected from Jon L. figures (1000 MHz * 0.016 = 16 %). Increasing the number of flashes also takes the standard deviation on the measured calibration runs down from 7 p.e to <2 p.e. for 10 to 100 flashes in each calibration run respectively.  Going to 200 flashes (files not attached) only takes standard dev down to around 1 p.e. so diminishing return.
I've also coded up a sin wave NSB generator so we can sample at any time within a max min NSB range with a given rise time and that is the next thing I will try to make sure the stats are not affected much by the level of the start NSB pedestal. This code will also improve on the linear NSB generator which stops at around 1 GHz and only allows me so much time to place the calibration runs in.


plumber --version is UNSET

Describe the bug
./plumber --version
UNSET

go.mod sets module github.com/batchcorp/plumber but the Makefile uses -X 'github.com/streamdal/plumber/options.VERSION=${VERSION}'

We renamed our company from Batch.sh to Streamdal but haven't gotten around to updating all of our source packages yet. Will get working on that shortly. The version should also include the version number like v2.4.0-854d258, but that is currently only set via CI, so we'll work on a solution for that also

@a-n-n-a-l-e-e this is fixed now. The CI process will inject the version into version.txt and then running the make build commands will pull this version in


Added Sum of pairwise bit differences

Have you read the Contributing Guidelines on Pull Requests?
Yes
Description
Added Sum of pairwise bit differences, Its a very important google interview question, with good use of binary number concept.
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
#5538

@HarshCasper Sir plz do review this, and let me know if it requires any changes.


add panos_ssl_decrypt_exclude_certificate_entry resource

Description
add resource panos_ssl_decrypt_exclude_certificate_entry which manages SSL Decrypt Exclusion setting without toching other SSL decrypt settings
Motivation and Context
#342
this is an update of https://github.com/PaloAltoNetworks/terraform-provider-panos/pull/345
How Has This Been Tested?
Tested using the resource to modify ssl decryption exclusion with exsiting forward_trust_certificate_rsa and other ssl drcrypt settings, nothing gets changed except ssl decryption exclusion setting.
Tested create, update, delete. All working correctly (only changing ssl decrption exclusion)
Tested with resource ssl_decrypt, working as expected.
Types of changes

New feature (non-breaking change which adds functionality)

Checklist

[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes if appropriate.
[x] All new and existing tests passed.

added this, so closing out this PR


[UI] LB state pill lacks left padding

Rancher Version: 1.3.0
Chrome 55.0.2883.95 macOS

Version - 1.3.0
For some reason this changed between 1.2.2 and 1.3.0 (and yes I checked 1.2.2 to make sure).  But it is indeed now cut off, all states are moved to the left, but you can really see it during activation and upgrading cause those have a blue background.

https://github.com/rancher/ui/pull/965

Version - master 1/17
Verified fixed


[SPLIT #221] Redux axios and loading bar

Hi @choonkending , as an author of original redux-axios-middleware I just want to warn you, that @18601673727 is attempting to merge into your project his fork of middleware called redux-axios. I don't want to sound bad, but I just don't want your work to get destroyed and to repeat this process of middleware selection few weeks later when you found out that forked middleware isn't developed and maintained anymore.

@svrcekmichal Thanks for letting me know buddy! I was not aware of this until you pointed it out to me. No you don't sound bad at all. I'll keep this in mind when I get the chance to review the PR.

Narrow minded people can always predict the future, can't they?
Just wanna to clear the air a little bit, redux-axios now is a fork of redux-axios-middleware, but redux-axios is designed to be a redux abstraction of axios(http client), not like redux-axios-middleware the bridge way, you don't have to new serval axios instances in configureStore.js or have a "axios": "x.x.x" in your package.json, to achieve the same thing.
redux-axios want to reduce users' code as much as possible. But it looks like someone pissed off, so maybe I'll rewrite it someday, not just a fork plus modification. 😄

Whoa that is a bit of a heated discussion going on here!
Here are my comments purely based on the code and changes in this PR.
Here's what I like:

Moving configureStore.js out and removing the store/ directory
Creating a client index and backend module that makes it easy to change configurations
res.sendStatus(500) change in server.js

I disagree with the addition of react-redux-loading-bar and redux-axios (still) unfortunately. We've had the redux-axios middleware discussion prior to this, and while it definitely is beneficial for many projects, I think that cost of adding this abstraction to this specific boilerplate outweighs the benefit.
I'm looking for ways to reduce the power of our code in a principled way https://github.com/choonkending/react-webpack-node/issues/364
I'm going to close this PR off, but will be happy to get the above changes as PR(s) or I can apply them when I am reducing this boilerplate. Sorry we did not get to merge this, and even though we did not apply this change, I think it still spurred the discussion regarding abstraction and side effects in the right direction as this boilerplate matures.


gateway-api: add appProtocol to gateway svc ports

Please provide a description of this PR:

actually should this be lower case? oddly I cannot find guidance in the API spec but everything seems to be lower case

actually should this be lower case?

I was wondering that myself. It works for Istio either way, but if you think it would be better, I'll ToLower the values.


TypeError: Cannot read property 'text' of undefined

Not sure where to start with this one.  Output works beautifully but the last lines are an error:
2 06 2017 11:52:50.122:ERROR [coverage]: TypeError: Cannot read property 'text' of undefined
    at C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:210:45
    at Array.forEach (native)
    at annotateStatements (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:193:33)
    at HtmlReport.writeDetailPage (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:428:9)
    at C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:489:26
    at SyncFileWriter.writeFile (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\util\file-writer.js:57:9)
    at FileWriter.writeFile (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\util\file-writer.js:147:23)
    at C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:488:24
    at Array.forEach (native)
    at HtmlReport.writeFiles (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:482:23)
    at C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:484:22
    at Array.forEach (native)
    at HtmlReport.writeFiles (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:482:23)
    at HtmlReport.writeReport (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\html.js:566:14)
    at LcovReport.writeReport (C:\Users\brian.cullinan\Documents\portal\node_modules\istanbul\lib\report\lcov.js:55:19)
    at writeReport (C:\Users\brian.cullinan\Documents\portal\node_modules\karma-coverage\lib\reporter.js:68:16)

I have the same problem, my repo is:
https://github.com/FezVrasta/popper.js/pull/314
To reproduce just run yarn test

I have the same issue. I did a bit of investigation and it seems to be at this point:
https://github.com/gotwarlost/istanbul/blob/master/lib/report/html.js#L288
At the point it errors on my project, structuredText is an array of 60 objects but startLine is 64, so it's returning undefined.
However the coverage report is generating correctly!

it is solved with: https://github.com/gotwarlost/istanbul/pull/819

This not "solved" until a fix is merged in and people stop getting poor
error messaging for framework bugs.
On Jun 22, 2017 3:55 AM, "Cenk Ebret"<EMAIL_ADDRESS>wrote:
it is solved with: #819 https://github.com/gotwarlost/istanbul/pull/819
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gotwarlost/istanbul/issues/817#issuecomment-310346550,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AX5XbpdFGqJUCtlFDwS5vjVrzHdo3WyRks5sGkgDgaJpZM4N3iuA
.

Hello. When it's going to be fixed? I cannot get proper html report

Congratulations!!  You win<PHONE_NUMBER> points for arriving here at the same
conclusion as we did!  Here is a joke while you wait for a response:
Q: Why was everyone so tired on April 1st? A: They had just finished a
March of 31 days.
On Wed, Nov 15, 2017 at 9:26 AM, Arsen Slyusarchuk<EMAIL_ADDRESS>
wrote:

Hello. When it's going to be fixed? I cannot get proper html report
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gotwarlost/istanbul/issues/817#issuecomment-344647091,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AX5XbhyFdjDF1pBvphV_3bFvPB1jgNWjks5s2xCfgaJpZM4N3iuA
.

--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure. It is then shared with tech companies, bots, hackers,
government agencies, and marketers.  The security of this message is none,
and it may be shared on Instagram at anytime.  If you are OK with this,
please respond.  There isn't really any security or privacy anywhere.  If
you disagree you may want to go camping and talk to people face-to-face
like in old times.

No news ?

Dev's on holiday? How long is this going to take?

I am using gulp-istanbul 1.1.3 which depends on istanbul 0.4.x and I encountered this issue (when it was using istanbul 0.4.5). Manually forcing gulp-istanbul to use istanbul 0.4.4 resulted in the same error.
@heronblade Is it possible another change helped you fix this or was it just the downgrade?
Also, this issue seems to be stuck without any updates or progress. Is there anything users can provide to help the project maintainers root cause this? It seems related to source maps but my mocha test setup doesn't really do anything with source maps so I'm not sure where to try to fiddle with things to get around this error.

Reverting to istanbul 0.4.4 did not solve the issue for me, either.

Bump. So this issue is an unsolved blocker? Any other workaround suggestions. Istanbul seems like the only game in time for modern js coverage so I'm a bit lost where to go from here.

FWIW, I gave up using gulp to drive my mocha test run. When I instead invoked nyc via an npm script, it worked fine. I think that means using this package by itself is no longer supported.

I am using simple istanbul and still recieveing this error. Does anyone know how to solve it? It is generating lcov.info correctly but not index.html

FWIW I switched to using nyc via its CLI and it's working. I think this means this package is unmaintained/dead for other use cases.

In my case, i am using protractor + istanbul to get coverage for integration testing.
So need to pre-instrument source file before test and get this error in istanbule report.
I find if i use origin source file to replace the instrumented source file, then it works.
Checking code, structuredText[startLine].text is undefined due to the instrumented source file text structure is changed and the [startLine] line is invalid.

I've started seeing this issue a couple days ago. A week ago everything worked smoothly. Looks like js source code was changed and somehow uncovered this bug.
    "istanbul": "^0.4.5",
    "istanbul-combine": "^0.3.0"

We are combining coverage from a set of files. Looks like calculation is fine, but the report generation is failed
=============================== Coverage summary ===============================
Statements   : 29.37% ( 1614/5496 )
Branches     : 9.96% ( 442/4436 )
Functions    : 28.57% ( 932/3262 )
Lines        : 30.26% ( 1276/4217 )
================================================================================
node_modules/istanbul-combine/node_modules/istanbul/lib/report/html.js:283
                    text = structuredText[startLine].text;

Fixed: I remove karma-coverage and used karma-coverage-istanbul-reporter

For rollup-plugin-istanbul users, the babel plugin has to come after istanbul plugin.
https://github.com/artberri/rollup-plugin-istanbul#other-usage-options

Hi ,
I am also facing this issue and this https://github.com/gotwarlost/istanbul/pull/819
Solved it for me . Please can this issue be resolved so that we don't have to patch Istanbul ourselves.

The real problem seems not to be in "html" reporter module.
I ran into the same problem and after some investigations, I make a patch myself on the "html" reporter module to skip the error and I realize that the highlighted uncovered lines seems to be shifted with the real code.
The reason was that in my bundler (webpack) I set another loader ("imports-loader") before the "babel" one which brings instrumentation in my configuration. So "babel" took an already transformed code before to send it to "istanbul", and all offsets were shifted and wrong.
I ordered my bundler loaders to pass by "babel" loader first and all is good now 🙂
Hope this trick can help another desperate code coverage lover, and sorry for my bad english

The real problem seems not to be in "html" reporter module.
I ran into the same problem and after some investigations, I make a patch myself on the "html" reporter module to skip the error and I realize that the highlighted uncovered lines seems to be shifted with the real code.
The reason was that in my bundler (webpack) I set another loader ("imports-loader") before the "babel" one which brings instrumentation in my configuration. So "babel" took an already transformed code before to send it to "istanbul", and all offsets were shifted and wrong.
I ordered my bundler loaders to pass by "babel" loader first and all is good now 🙂
Hope this trick can help another desperate code coverage lover, and sorry for my bad english


Display Original JSON Text

In addition to the current state source JSON, we should also preserve and display the original JSON text

Resolved by #16


[mysql] Optimize pure binlog phase check logic to improve performance

Pure binlog phase performace opt
Our online business found that the performance of the binlog phase of Flink jobs could not meet the demand. Through performance analysis, we found that the performance bottleneck in the phase was the comparison of binlog offset. For each binlog data, mysql-cdc needs to judge whether the offset of the binlog data is after the end of the full snapshot phase (max split high watermark). If so, it is in pure binlog phase, and can be directly output to the downstream. The comparison between binlog data offset and max split high watermark consumes CPU very much and has become a performance bottleneck (see the following figure).
Further analysis of the internal logic of mysql-cdc shows that for each mysql-cdc table, the state of incremental synchronization will remain unchanged after entering the pure binlog phase. Therefore, it is sufficient to keep a flag for each table to judge whether the table has entered pure binlog phase, so as to avoid binlog offset comparison of each data in the pure binlog phase, Improve the performance of incremental data synchronization.

Performance improvement
In the actual online scenario test (more than 180 tables are synchronized with many table fields in a flink job), the performance is improved by 3 times (from 5k/s to 2w/s), which meets the real-time synchronization needs of customers.

@leonardBang Would you have a look at if there is any problem with this optimization?


Electronic labbook integration

This is in reference to #867.
I have added a middle-ware in base.py. And there are some changes in the way the server is configured. I am adding comments to the changes on github to better explain why I made those changes.
I have only tested this behind an https nginx server with everything running in docker containers. And I have modified vital parts of the code. I tried to keep changes to the minimum. But there are a few things that might need more discussion in how to go about them.
The middle-ware adds a random session ID if none are given. This should mean every browser connected to LiberTEM should have a separate shared state.
Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases
[ ] I have included the rebuilt production build of the client (only if
changes were made to the GUI)

Thanks for your work @shabihsherjeel! I've done a first quick read through the code and I'll try to finish my review beginning of next week. Thank you also for the comments in the code, this makes the review much easier :+1:
One question from my first read through the code: how does the labbook server inject the data set parameters? Does it just use the DataSetDetailHandler? If so, what stops a user from opening an arbitrary data set? Could you maybe add a small script (prototypes folder?) to demonstrate the labbook <> libertem interaction?
I see that you have methods {get,add}_allowed_directories (unused for now?) - we need to be a bit careful with this. This assumes directories are "access boundaries" - so if a user puts a bunch of data into a directory, sharing this via a labbook becomes an all-or-nothing decision.
Also, imagine a user having access to a directory they can write to (via NFS or similar), they can then create symlinks to data of other users which they shouldn't be able to access. This can be nuanced in practice - for example, the opening path specified for some formats is just a header file, whereas the data set consists of multiple data files. So if we only check that the real opening path (with symlinks resolved) is in one of the allowed directories, the data files can still be symlinks which resolve to other user's data.
Maybe we should formalize a minimal threat model, to define the parameters on how LiberTEM can be run securely, and what the security boundaries are?

Maybe we should formalize a minimal threat model, to define the parameters on how LiberTEM can be run securely, and what the security boundaries are?

Sounds good! I've never done a threat model, so I'll just try to follow https://en.wikipedia.org/wiki/Threat_model
Model of the system:
The connection between browser and ElabFTW as well as LiberTEM GUI is SSL encrypted, excluding tampering and eavesdropping on the underlying
transport layer, right?

Clicking the "Load in LiberTEM" button in ElabFTW points the user to a LiberTEM instance. The button is not a hyperlink in the classical sense, but seems to trigger some kind of script?

Can a user spoof the action (request/JavaScirpt call/...) that triggers opening a LiberTEM window for a file/dataset in ElabFTW without having regular access to that dataset in ElabFTW?
Can a user create datasets in ELabFTW that point to arbitrary/unauthorized locations on the backend system?

A new tab with running LiberTEM opens, having the dataset loaded

Which steps are performed when opening a dataset between ElabFTW and LiberTEM? Can the request to open a dataset be spoofed on the client side?

The LiberTEM web GUI seems to work normally, just the buttons to open new datasets seem to be missing

What happens if the user spoofs the request(s) to the LiberTEM GUI to open a dataset, mimicking the action of the usual buttons?
Can the web GUI of LiberTEM be used to inject code that runs on the server?

The LiberTEM GUI seems to load, just without any open dataset, if one points a browser to the target URL.

Can the exposed web API be used to perform any actions or disclose information when opened from outside without redirection from ELabFTW?

Yes, it seems so. I just performed a DoS by pressing the "Shutdown" button from a second browser (no ElabFTW based cookies etc). Clicking on "Load in LiberTEM" in the primary browser doesn't work anymore.

Perhaps requests to the sandboxed LiberTEM should contain some form of session ID that is somehow generated from the ElabFTW session, and interaction should be completely disabled for unauthenticated users? That would already make exploits from the open internet a lot harder.
@sk1p, if I understand correctly the security model for JupyterHub  integration is based on LiberTEM running as the Unix user that is logged in to JupyterLab, right? That means until now we had the approach "if the OS lets me do it, it is probably OK". To be honest, I don't have a good feeling how to manage LiberTEM running as a kind of system user shared between different end users.

Model of the system:
The connection between browser and ElabFTW as well as LiberTEM GUI is SSL encrypted, excluding tampering and eavesdropping on the underlying
transport layer, right?

Agreed.
A threat model should also ask and answer questions like these:

What privilege boundaries exist or should exist?

Example: access restrictions in the labbook should imply access restrictions in LiberTEM
Example: in restricted mode, the user should not be able to shut down the LiberTEM server → there should be a boundary between privileged and user API accesses

Should a single LiberTEM process have read access to data across permission boundaries?
Can we trust the parameters for opening data sets, or do we need to take extra care that they are correct? Can a user elevate privileges by manipulating parameters?
What even is privilege elevation in this case? Remote code execution, but also information disclosure

Related: should access via LiberTEM imply that users should be able to access the raw data? (regarding the point analysis plus result download mentioned above, or equivalently just picking all frames individually...)

FYI @shabihsherjeel and I had a call last week about this, and discussed that it would make sense to have privilege separation between the API calls performed by elabFTW (creating a session, opening a data set, shutting down the LiberTEM server, ...) and by the user in the session (running analyses etc.). The elevated calls by elabFTW would be protected by an API token / shared secret that is put into both the LiberTEM server config and the elabFTW integration / plugin. Sorry for not updating here!

Clicking the "Load in LiberTEM" button in ElabFTW points the user to a LiberTEM instance. The button is not a hyperlink in the classical sense, but seems to trigger some kind of script?

Can a user spoof the action (request/JavaScript call/...) that triggers opening a LiberTEM window for a file/dataset in ElabFTW without having regular access to that dataset in ElabFTW?

This will probably change soon, to create the session from the server-side of elabftw, see note above.

Can a user create datasets in ELabFTW that point to arbitrary/unauthorized locations on the backend system?

That's a good point, but I think we should draw a boundary here: LiberTEM cannot know this, only the ElabFTW knows what an authorized location would look like. Of course we can try to create some defense in depth (like checking data locations), but this can often be circumvented (see: symlink attacks).

A new tab with running LiberTEM opens, having the dataset loaded

Which steps are performed when opening a dataset between ElabFTW and LiberTEM? Can the request to open a dataset be spoofed on the client side?

I think the current implementation relies on the semi-random dataset filenames, but that is subject to change (see above).

Can deeper information about the system be gained from an open dataset?

Possibly the "Dataset info" and the new "Download notebook" can disclose internal storage locations.

Yeah, the diagnostics API provides some information that will contain absolute paths - I would argue it's not a severe issue if LiberTEM is running in a docker container with the data bind-mounted to some generic path.

The LiberTEM web GUI seems to work normally, just the buttons to open new datasets seem to be missing

What happens if the user spoofs the request(s) to the LiberTEM GUI to open a dataset, mimicking the action of the usual buttons?
Can the web GUI of LiberTEM be used to inject code that runs on the server? Good candidates would be dataset parameters?

This is related to the question: how trustworthy are the data sets uploaded to the labbook? I can imagine that our readers contain bugs that, in case of numba-based decoding, could result in remote code execution. We should be relatively robust against rogue dataset parameters,

The "Download result" feature in combination with a collection of "point analyses" and crafted dataset parameters can be used to exfiltrate arbitrary data from the system, as soon as the LiberTEM GUI opens it as a dataset.

I think the key word here is "crafted dataset parameters" - untrusted API calls must not be allowed to open data sets.

Can a malicious website somehow inject requests into the LiberTEM GUI (XSS)?

React makes it hard to inject raw markup and thus scripts - you have to use the aptly named dangerouslySetInnerHTML function, which can be audited against, or you have to go out of your way and access raw DOM nodes.
We don't generate any dynamic HTML on the server side.
Together, these two things should make XSS in principle less likely. In general, it depends on React not being buggy, and changes to the GUI being properly reviewed.

What happens if a user sets parameters that overload the target system (memory, CPU)? In particular clustering analysis has the potential to set large number of reference points and clusters. Opening a large RAW file with parameters for very small signal or navigation dimension could do the same.

This is a good point - the first part relating to resource sharing and throttling by viewers, the second relating to abuse from the side of data uploaders.

The LiberTEM GUI seems to load, just without any open dataset, if one points a browser to the target URL.

This will change when the session is created by the elabFTW backend - then, in restricted mode, it should be an error if you visit without a valid session id set.

Can the exposed web API be used to perform any actions or disclose information when opened from outside without redirection from ELabFTW?

Yes, it seems so. I just performed a DoS by pressing the "Shutdown" button from a second browser (no ElabFTW based cookies etc). Clicking on "Load in LiberTEM" in the primary browser doesn't work anymore.

What happens if/when the LiberTEM GUI is updated? How do we make sure it doesn't introduce new interfaces that are harmless in the usual "per user" case, but can be exploited in the ElabFTW integration scenario?

LiberTEM uses Dask.Distributed

Is the Dask.Distributed scheduler exposed somehow?

When deploying LiberTEM, there should be a network barrier between the dask scheduler and the external network.

The "Download Notebook" feature gives information about any external Dask scheduler used.

We have to decide if we consider this privileged information and want to disable it in restricted mode

Perhaps requests to the sandboxed LiberTEM should contain some form of session ID that is somehow generated from the ElabFTW session, and interaction should be completely disabled for unauthenticated users? That would already make exploits from the open internet a lot harder.

Indeed, see above

if I understand correctly the security model for JupyterHub integration is based on LiberTEM running as the Unix user that is logged in to JupyterLab, right? That means until now we had the approach "if the OS lets me do it, it is probably OK".

Yeah, JupyterLab integration is a different beast - it means:

You can launch a libertem instance per user, because either there is such a thing as a unix user for the jupyter user, or the container/vm is "owned' by the jupyter user
Typically in a jupyterlab instance, there should be enough RAM available that we have enough resources for starting per-user LiberTEM instances
The user already has full access to the context where we are launching the libertem instance, so there are no privilege boundaries to cross
JupyterLab takes care of authorization/authentication, similar to the proposed session system

To be honest, I don't have a good feeling how to manage LiberTEM running as a kind of system user shared between different end users.

I think it can work - although it could be possible that the result of our discussion is that we only recommend deploying LiberTEM in a scenario where the data is open access, or "flat" access, such that anyone who can access any data, can access all data. This should match the use case of @shabihsherjeel (please correct me if I'm wrong)

Sorry for the delayed response. I thought of the points you put out. I decided to try another approach. I will have a middleware process launcher that will control Linux file access for different client instances of LiberTEM. I am planning to have a separate GID for every data file I wish to provide an instance of LiberTEM access to. And the file permissions as default for all the unnecessary files and other data files would be no read, write, or execute.
The same security issues will exist on the middleware as well. And I will setup privileged APIs for lab notebooks to spawn a LiberTEM process. But this means that LiberTEM will not have to handle creating a "jail" for different instances.
This is still work in progress. I have tested a very basic version of this. I will test out a few details and also design how I will manage concurrent users on the same file. Then I will update this thread with more info.
The only changes I am planning to do now to LiberTEM are to set default cluster configuration and dataset to load through the CLI. All those session details will be handled by a separate middleware process spawner. This way LiberTEM will not need to worry about being in a lab notebook mode and cause users confusion as this is more of a client side application.
I will love to discuss this more with you after I have a basic implementation up and running.

I thought of the points you put out. I decided to try another approach. I will have a middleware process launcher that will control Linux file access for different client instances of LiberTEM. I am planning to have a separate GID for every data file I wish to provide an instance of LiberTEM access to. And the file permissions as default for all the unnecessary files and other data files would be no read, write, or execute.
The same security issues will exist on the middleware as well. And I will setup privileged APIs for lab notebooks to spawn a LiberTEM process. But this means that LiberTEM will not have to handle creating a "jail" for different instances.
This is still work in progress. I have tested a very basic version of this. I will test out a few details and also design how I will manage concurrent users on the same file. Then I will update this thread with more info.
The only changes I am planning to do now to LiberTEM are to set default cluster configuration and dataset to load through the CLI. All those session details will be handled by a separate middleware process spawner. This way LiberTEM will not need to worry about being in a lab notebook mode and cause users confusion as this is more of a client side application.

Thanks for the update! This also sounds like a good approach, and a bit less complexity on the side of LiberTEM, delegating the security issues to the OS. Some questions about this:

Will data loading / browsing also be disabled if a dataset was loaded through the CLI?
I think the plan was to terminate LiberTEM after some inactivity. How does this work with the middleware? Does the libertem-server just quit on its own after some time of inactivity?
I'm not so clear about the GID scheme, I have to think a bit about this. But maybe as an idea of an alternative approach, what about launching the libertem-server instances in a container, and r/o bind-mount the file(s) that are needed in there? Then the files only need to be read-accessible by the user that LiberTEM is running as in the container, or LiberTEM could even be run as the UID/GID of the specific files.

I would be happy if the loading via CLI could use something in the direction of #768, if that works for you.

I will love to discuss this more with you after I have a basic implementation up and running.

Sounds good!

Thanks for the update! This also sounds like a good approach, and a bit less complexity on the side of LiberTEM, delegating the security issues to the OS. Some questions about this:

Will data loading / browsing also be disabled if a dataset was loaded through the CLI?

I am thinking of leaving that intact. I can see how it works out. But currently, I am managing file access on the OS level so even if you browse you can't open the other datasets.

I think the plan was to terminate LiberTEM after some inactivity. How does this work with the middleware? Does the libertem-server just quit on its own after some time of inactivity?

I am not at that stage of my implementation yet. But as the process will be managed by the middleware, having a time limit on the process is easy. I need to figure out how the middleware will know if the user still has the tab open. Currently, I am planning on using nginx as a proxy with directives to dynamically add routes for different instances. I guess I could monitor connections from the user's IP or somehow inject a simple script that tells the middleware when the browser tab has been closed.

I'm not so clear about the GID scheme, I have to think a bit about this. But maybe as an idea of an alternative approach, what about launching the libertem-server instances in a container, and r/o bind-mount the file(s) that are needed in there? Then the files only need to be read-accessible by the user that LiberTEM is running as in the container, or LiberTEM could even be run as the UID/GID of the specific files.

Currently, the way I have it setup is that whenever I want to give a new user/instance access to that file, I chgrp on the file and give that process the same GID. This gives me a bit of liberty in controlling access than the UID would.
As of now, my implementation runs in a docker instance with all the datasets mounted with read/write access to control their permissions through a tornado server that just launches processes within a single container. The reasons I am using docker containers and do not want to go with launching a separate container for every instance are the following:

Ease of setup for users already deploying ElabFTW as docker containers.
Isolation of the system in a container to add another security layer. (I don't think this is a great argument. But it helps with the infancy of the implementation.)
Less overhead every time a user wants to run a Libertem instance
If I want to spawn docker containers from my main docker container, I will have to give this main container access to the system docker daemon. And this doesn't feel secure to me.

Please let me know what you think of this.

I would be happy if the loading via CLI could use something in the direction of #768, if that works for you.

This is great for the actual loading of files. But I only implemented pre-loading the cluster info through the CLI. I do not mind having this too. It is a great idea.

I will love to discuss this more with you after I have a basic implementation up and running.

Sounds good!

I am thinking of leaving that intact. I can see how it works out. But currently, I am managing file access on the OS level so even if you browse you can't open the other datasets.

Okay, sounds good.

But as the process will be managed by the middleware, having a time limit on the process is easy. I need to figure out how the middleware will know if the user still has the tab open. Currently, I am planning on using nginx as a proxy with directives to dynamically add routes for different instances. I guess I could monitor connections from the user's IP or somehow inject a simple script that tells the middleware when the browser tab has been closed.

Sounds like a solution. Did you already find a way to implement the dynamic configuration? There seems to be a module for dynamic configuration, but that's only available in the commercial version of nginx.
If nginx doesn't work out, there are also some alternatives like traefik, caddy, haproxy etc. that should also have the possibility to create configuration via API.

Currently, the way I have it setup is that whenever I want to give a new user/instance access to that file, I chgrp on the file and give that process the same GID. This gives me a bit of liberty in controlling access than the UID would.

Okay - maybe we can later adapt this to different setups, as the data access will probably be different in each setup.

As of now, my implementation runs in a docker instance with all the datasets mounted with read/write access to control their permissions through a tornado server that just launches processes within a single container. The reasons I am using docker containers and do not want to go with launching a separate container for every instance are the following:

Ease of setup for users already deploying ElabFTW as docker containers.
Isolation of the system in a container to add another security layer. (I don't think this is a great argument. But it helps with the infancy of the implementation.)
Less overhead every time a user wants to run a Libertem instance
If I want to spawn docker containers from my main docker container, I will have to give this main container access to the system docker daemon. And this doesn't feel secure to me.

True, especially the last point: access to the system docker daemon is equivalent to root access on the system (example: an attacker can spawn a new container with a bind mount of /etc/shadow and /etc/passwd and add a new root user for themselves). So the middleware would have to be regarded the same as running as root on the host system itself.

@shabihsherjeel just re-assessing older PRs. Is this still relevant or is it superseded by token-based auth? Are there other ways to address the integration? :-)

This has indeed been superseded by a token-based auth. Now that Libertem supports auth with JupyterLab, I can work with that instead. I'm closing this pull request. I hope to have the new integration ready soon. :)


NSRegularExpression extensions

🚀
No more using NSString, NSRange and UnsafeMutablePointer<ObjCBool> with NSRegularExpression calls.
Checklist

[x] I checked the Contributing Guidelines before creating this request.
[x] New extensions are written in Swift 5.0.
[x] New extensions support iOS 8.0+ / tvOS 9.0+ / macOS 10.10+ / watchOS 2.0+, or use @available if not.
[x] I have added tests for new extensions, and they passed.
[x] All extensions have a clear comments explaining their functionality, all parameters and return type in English.
[x] All extensions are declared as public.
[x] I have added a changelog entry describing my changes.

          4 Messages

:book:
Thank you for submitting a pull request to SwifterSwift. The team will review your submission as soon as possible.

:book:
iOS: Executed 596 tests, with 0 failures (0 unexpected) in 13.162 (16.411) seconds

:book:
tvOS: Executed 571 tests, with 0 failures (0 unexpected) in 6.275 (8.685) seconds

:book:
macOS: Executed 429 tests, with 0 failures (0 unexpected) in 1.101 (1.439) seconds

  Generated by :no_entry_sign: Danger

@LucianoPAlmeida looks like the Linux implementation treats the block as @escaping. Does this look ok? https://github.com/SwifterSwift/SwifterSwift/pull/727/commits/6fe2f92bc4ed29fd4ca91a108f69a003773e4ae9


gltf: Fix mesh nodes which are also bones.

This resolves these errors related to skinned meshes with mesh nodes as bones:
ERROR: Condition "mi == nullptr" is true.
   at: GLTFDocument::_process_mesh_instances (modules\gltf\gltf_document.cpp:5969)
ERROR: Condition "sk == nullptr" is true.
   at: GLTFDocument::_import_animation (modules\gltf\gltf_document.cpp:5648)

I filed an issue specifically for what this PR solves at #49118
It affects both 3.x and master.
in the following glTF documents:
NestedSkeletonReproCaseV2Animated.glb
MeshSkinnedToItselfAndOthersV2.glb
NestedSkeletonReproV3.glb
However, it does not resolve the problems in the following glTF document. More extensive work will need to be done on models causing the ERROR: glTF: Generating scene detected direct parented Skeletons error:
DirectParentedSkeletons.glb
The following assume the more useful error messages implemented by #48912:
ERROR: glTF: Generating scene detected direct parented Skeletons
   at: (modules\gltf\gltf_document.cpp:5441)
ERROR: glTF: Generating scene detected direct parented Skeletons
   at: (modules\gltf\gltf_document.cpp:5441)
ERROR: glTF: Generating scene detected direct parented Skeletons
   at: (modules\gltf\gltf_document.cpp:5441)
ERROR: Unable to cast node 13 of type Skeleton3D to EditorSceneImporterMeshNode3D
   at: (modules\gltf\gltf_document.cpp:5987)
ERROR: Unable to find node 32
   at: (modules\gltf\gltf_document.cpp:5984)

Despite any reservations you may have about the usefulness of having mesh nodes as bones, The above glTF documents are conformant and can be generated from the UniGLTF unity plugin, and I have confirmed that this issue existed in the wild (in at least one model being sold for real money). Also, these models are confirmed to work in Blender, three.js (Don McCurty), Unity (UniGLTF) and Windows 3D Viewer.
I think we should test these fixes heavily. We don't want to break any working models. CC @fire

There is a bug with mesh + skeletons animations. Will be debugging with Lyuma on this.

Ok, I'm now pretty confident with this change. Here is my testing project with glTF files that trigger some of these bugs:
gltf_test.zip
The big change in this update is we now remove the fake_joints hack, and instead add a special case in _generate_godot_node where an extra bone attachment is created when a bone is also a mesh, camera or light.
There are still some cases where two skeletons are created which are directly parented to one another, but this change ensures that Godot behaves correctly when this happens. Whether or not two skeletons can be directly parented seems to depend on arbitrary conditions and not all test files reproduced the issue. Fixing this with the existing algorithm appears intractable to me...
The code in _determine_skeletons is supposed to detect this case in some big O(S * N^2) time complexity loop, but evidently it does not succeed in detecting them. It stands to reason that if an additional checking pass is added, perhaps that can create more direct-parent relationships.
Another approach to solve direct parented skeletons could be to recursively add all nodes to the Skeleton, which would solve the direct parented case, but this has other implications, including animation artifacts due to the inability to precisely represent bone pose animations against the rest pose.

Note about the blend shape issue, it's a couple lines and possibly could be in its own PR.
It seems related to issue #38751, fixed last year: the fix addressed most of the issues, but the animation code assumed that the blend_weights array has the correct length. The two line change included with this PR ensures this.
There is a separate bug caused by EditorSceneImporterMesh that causes weights to be ignored. I have filed it here: #49005

After most of the afternoon I and Lyuma reviewed this pr in depth.
We made changes to make the functions more direct and easier to understand.
I approve of this pr to be merged. The steps to add the models godot-test should be a later step.

Thanks!


Added a label MaximoID for Admin interface City field

Ready to merge to prod

Coverage decreased (-95.0%) to 3.65% when pulling 2599685ca0734e1682d69d31f85372f4c3e3bb17 on change-name-city-col-231 into 3db99fe9cdd53f25fa4dcb167c621ca2e5b38a29 on production.

Coverage remained the same at 98.618% when pulling 78099f06d1f528ae958593da3a2c6fe08ff8c40e on change-name-city-col-231 into 3db99fe9cdd53f25fa4dcb167c621ca2e5b38a29 on production.

:+1:


Meilisearch integration

Description
Add Meilisearch integration to the app host.
Fixes #5816
Checklist

Is this feature complete?

[x] Yes. Ready to ship.
[ ] No. Follow-up changes expected.

Are you including unit tests for the changes and scenario tests if relevant?

[x] Yes
[ ] No - not yet.

Did you add public API?

[X] Yes

If yes, did you have an API Review for it?

[x] Yes
[ ] No

Did you add <remarks /> and <code /> elements on your triple slash comments?

[x] Yes
[ ] No

[ ] No

Does the change make any security assumptions or guarantees?

[ ] Yes

If yes, have you done a threat model and had a security review?

[ ] Yes
[ ] No

[X] No

Does the change require an update in our Aspire docs?

[X] Yes

Link to aspire-docs issue:

[ ] No

Microsoft Reviewers: Open in CodeFlow

We need to import https://www.nuget.org/packages/Testcontainers/3.10.0, docker.io/getmeili/meilisearch:v1.10 and https://www.nuget.org/packages/MeiliSearch/0.15.3

Close PR moving to community toolkit.


deposit form: import files errors when being used on metadata-only records

Package version (if known):
Describe the bug
A clear and concise description of what the bug is.
Steps to Reproduce

Create and publish a metadata-only record.
Create a new version of the metadata-only record.
Click on 'Import files'

Expected behavior
I expect that either 1) the import files is not shown or 2) an error message is displayed that the previous version is a metadata-only record.

The solution already used for file upload in that same case is to precede the action with an API call to save the draft. This way the files.enabled is set correctly which allows subsequent action.


Unwanted error handling in command suggestions.

Description

The bot sometimes sends you error messages when you type invalid commands.
Steps to Reproduce
Type ...and or .cand or so on in a channel, and you'll be met with an embed by the bot saying "Command can only be used in October." This happens because the command handler tries suggesting the candy command but its checks raise a check failure, and the check failure is passed to the error handler again which is handled by it by posting that october embed.
Expected Behaviour

The bot shouldn't send any message at all.
Possible Solutions

157b19b5b99d5ab0ea5f62bb8231765916bf90d8 should fix it.
Would you like to implement a fix?
Note: For high-priority or critical bugs, fixes may be implemented by staff.

[X] I'd like to implement the bug fix
[ ] Anyone can implement the bug fix

@Qwerty-133  Actually this has already been fixed. Please have a look.

Yeah, looks like it was fixed by https://github.com/python-discord/sir-lancebot/pull/1096, thanks


Feature: respect indent in brace-based languages

Claude 3 Opus likes to append methods to a Java/Kotlin class by doing a search/replace on an empty block whilst providing indented text. Aider just adds this to the end of the file, but it would be smarter to spot when the file ends with a brace and position the newly inserted block within the right scope, as determined by computing the minimum indent of the provided block.
Example:
 <<<<<<< SEARCH
 =======
     private fun invokeTask(task: BuildTask, cacheKey: String?, retryCount: Int = 0): Any? {
         check(retryCount <= maxRetries)
         return try {
             executeTask(task, listOf(task), 0, cacheKey)
         } catch (e: RetryLater) {
             if (retryCount < maxRetries) {
                 Logger.info("Task ${task.taskDisplayName} requested retry (attempt ${retryCount + 1}): ${e.message}")
                 if (e.delay != null) {
                     Logger.info("Waiting ${e.delay} before retrying ${task.taskDisplayName}")
                     Thread.sleep(e.delay.toMillis())
                 }
                 invokeTask(task, cacheKey, retryCount + 1)
             } else {
                 Logger.warn("Task ${task.taskDisplayName} failed after $maxRetries retries")
                 throw e
             }
         }
     }

 >>>>>>> REPLACE

Here, the right place to add the new method is inside the class, so one line above where aider chooses to put it.

I think this is probably best treated as a generic "edit error". It would be hard for aider to enforce per-language grammar rules in edits. Sometimes the LLM will make edit errors unfortunately. Aider tries very hard to prevent this, but it can still happen.
I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.

Something similar happens to me as well, when using diff mode. I'm using gpt-4o. GPT correctly reasons where to make change, it was at the end of if branch in my example situation, but produces SEARCH block with indentation and closing curly brace. '       }'. This results in replacing wrong place in the file since there are 8 of such places in the file.
I spent half a day to try go around it mainly by modifying aider's system prompt in editblock_prompts.py, but it returned that in 9 out of 10 tries.
<<<<<<< SEARCH
                         }
=======
                             equationsToCalculate.push("factorial");
                         }
>>>>>>> REPLACE

So, I just reverted any changes made by aider (I don't use auto-commit feature) and followed with prompt the diff is not sufficient for <concerned_filename>, and it returned the correct SEARCH/REPLACE block.
<<<<<<< SEARCH
                             equationsToCalculate.push("exponent");
                         }
=======
                             equationsToCalculate.push("exponent");
                             equationsToCalculate.push("factorial");
                         }
>>>>>>> REPLACE


Support Windows Subsystem for Linux

So mvnd does not work in WSL right now? Or is this issue about checking first whether there are problems at all?
Is this about WSL1 or WSL2 or both?

@gnodet Seems to run ok so far in WSL2 (Ubuntu 20) on my Windows 10 Home box...

@gnodet Seems to run ok so far in WSL2 (Ubuntu 20) on my Windows 10 Home box...


Markdown relative links on edit mode are not working correctly

We have written the needed d
Issue Type: Bug
When I use the link syntax its not opening the notebook. If I use an  tag (i.e. FAB Parameters) it is getting the relative link locations as "Unable to open 'FABParameters.ipynb': Unable to read file 'c:\C:\Users\mml63\AppData\Local\Programs\Azure Data Studio\resources\app\out\vs\code\electron-browser\workbench\FABParameters.ipynb' (Error: Unable to resolve non-existing file 'c:\C:\Users\mml63\AppData\Local\Programs\Azure Data Studio\resources\app\out\vs\code\electron-browser\workbench\FABParameters.ipynb')."
This is not the relative link to the project files.
This started last update.
Azure Data Studio version: azuredatastudio 1.18.1 (5b3b97f4a34018f3b37b0fce30e32f2af7ad78fa, 2020-05-26T21:14:29.147Z)
OS version: Windows_NT x64 10.0.19041

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (4 x 2400)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_off_okwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
11.87GB (3.71GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (1)

Extension
Author (truncated)
Version

powershell
ms-
2020.4.0

ata into your clipboard because it was too large to send. Please paste.

@barbaravaldez can you work with @mikelawell to repro this issue?

Hi @mikelawell, are you trying to use markdown links?

Sorry, been awhile... this is using markdown links... I converted to an A tag to work around this. I need to try to repro to see if it got fixed in the past couple of releases.

There is still some weirdness. It looks like it is still adding a front slash to the link. I'm now using 1.20.1

Thank you for the screenshot 😄 Yeah, it seems that something weird is happening when it's trying to convert the relative link into an absolute path. I'll take a look at this. Thank you!! 😄

Hi @mikelawell, I tried to repro this issue using markdown relative links in Notebooks but it seems to be working fine. So, I just wanted to clarify does this issue happens on Notebooks or Books?

Closing this one as a duplicate of #11657

I was incorrect here; this affects text cells instead of markdown files, which use slightly different code paths. Reopening.

@mikelawell @barbaravaldez  - I'm using v1.25.1 right now and am unable to repro. Perhaps we can close this if it's no longer an issue?
I do notice that there is "/" in front of the beginning of the relative link. But this seems to be harmless on Windows. @barbaravaldez - what's the behavior on Mac or LInux?
A friendly reminder:  link sometimes does not work is that if a user forgets to mark their notebook as Trusted.

@mikelawell @barbaravaldez - I'm using v1.25.1 right now and am unable to repro. Perhaps we can close this if it's no longer an issue?
I do notice that there is "/" in front of the beginning of the relative link. But this seems to be harmless on Windows. @barbaravaldez - what's the behavior on Mac or LInux?
A friendly reminder: link sometimes does not work is that if a user forgets to mark their notebook as Trusted.

@MsSQLGirl - I was not able to repro on Mac, as well.

Given that we have some folks unable to repro, closing this issue. Feel free to reactivate if we have a consistent repro.


Add a hook for a future cast optimization

NSString->AnyHashable casts are remarkably common. We should make them faster someday.
In particular, NSDictionary bridges by default to [AnyHashable:Any], so when such a dictionary has NSString keys, every one gets bridged to AnyHashable.  Besides being very common, this is also one of the slower bridging paths, as it recurses through various parts of Foundation and the Swift standard library before it's done.

CC: @Catfish-Man

@swift-ci Please smoke test and merge


Unsupported OP_QUERY command

I have mongoDB 6.0. When I am trying to connect it from redash 8.0, I am getting error:
Unsupported OP_QUERY command: connectionStatus. The client driver may require an upgrade. For more details, see https://dochub.mongodb.org/core/legacy-opcode-removal.
After going through redash's code, I realized they are using pymongo 3.9.0 which only supports till mongo 4.2. Please upgrade the pymongo version to 4.3.3

Interesting.  Yeah, that pymongo version of 3.9.0 is even in the latest Redash source code:
https://github.com/RedashCommunity/redash/blob/d63ef905ba62e39307156f8e3fe9e5484951b488/requirements_all_ds.txt#L9
We're updating other dependencies over the last days and weeks anyway, so we might as well try a newer pymongo and see if it builds without needing other changes.

The newer PyMongo passed the current Redash test suite, though I'm not sure how much that actually tests the Mongo part of things.  That being said, I've merged it the PR into our master branch, so it's currently re-running the test suite on there + it'll automatically build a new preview docker image:
https://hub.docker.com/r/redashcommunity/redash/tags
You might need to give it an hour or so to finish the tests and building, then you could try that docker build out with your Mongo version 6 data source and see if it works ok. :smile:

Hey @justinclift. Thanks for the quick response. I will test it and let you know

Thanks, please do.  If the updated PyMongo library doesn't work properly, we'll probably need to change the version back to the older one until we can look at it properly.

I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.
But, when I am running the command docker-compose run --rm server create_db, I am getting the following error:
Starting redash_redis_1    ... done Starting redash_postgres_1 ... done Traceback (most recent call last):   File "/app/manage.py", line 6, in <module>     from redash.cli import manager   File "/app/redash/__init__.py", line 56, in <module>     import_query_runners(settings.QUERY_RUNNERS)   File "/app/redash/query_runner/__init__.py", line 445, in import_query_runners     __import__(runner_import)   File "/app/redash/query_runner/dynamodb_sql.py", line 10, in <module>     from dql import Engine, FragmentEngine   File "/usr/local/lib/python3.7/site-packages/dql/__init__.py", line 6, in <module>     from .cli import DQLClient   File "/usr/local/lib/python3.7/site-packages/dql/cli.py", line 21, in <module>     from .engine import FragmentEngine   File "/usr/local/lib/python3.7/site-packages/dql/engine.py", line 47, in <module>     from .grammar import line_parser, parser   File "/usr/local/lib/python3.7/site-packages/dql/grammar/__init__.py", line 37, in <module>     from .query import (   File "/usr/local/lib/python3.7/site-packages/dql/grammar/query.py", line 45, in <module>     from .parsed_primitives import primitive as parsed_primitive   File "/usr/local/lib/python3.7/site-packages/dql/grammar/parsed_primitives.py", line 30, in <module>     integer = pyparsing_common.signedInteger AttributeError: type object 'pyparsing_common' has no attribute 'signedInteger'

Interesting.  It looks like an error happening in the DynamoDB connection library.  Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?

Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?

Tonight, i will look at this. I'm busy right now. Sorry.

Tonight is completely fine, thank you. :smile:

I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.
But, when I am running the command docker-compose run --rm server create_db, I am getting the following error:
Starting redash_redis_1 ... done Starting redash_postgres_1 ... done Traceback (most recent call last): File "/app/manage.py", line 6, in from redash.cli import manager File "/app/redash/init.py", line 56, in import_query_runners(settings.QUERY_RUNNERS) File "/app/redash/query_runner/init.py", line 445, in import_query_runners import(runner_import) File "/app/redash/query_runner/dynamodb_sql.py", line 10, in from dql import Engine, FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/init.py", line 6, in from .cli import DQLClient File "/usr/local/lib/python3.7/site-packages/dql/cli.py", line 21, in from .engine import FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/engine.py", line 47, in from .grammar import line_parser, parser File "/usr/local/lib/python3.7/site-packages/dql/grammar/init.py", line 37, in from .query import ( File "/usr/local/lib/python3.7/site-packages/dql/grammar/query.py", line 45, in from .parsed_primitives import primitive as parsed_primitive File "/usr/local/lib/python3.7/site-packages/dql/grammar/parsed_primitives.py", line 30, in integer = pyparsing_common.signedInteger AttributeError: type object 'pyparsing_common' has no attribute 'signedInteger'

Please show me how do you connect it.

Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?

I think our docker image not build dql==0.6.2 in it. Thus causing the situation.

Oh.  I wonder if our docker image build is not using the requirements_all_ds.txt during build then...

Oh. I wonder if our docker image build is not using the requirements_all_ds.txt during build then...

Yes, you understand me.

Cool.  We need to fix that. :smile:
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt.  Any idea how to get them included?

Hmmm, it looks like GitHub now enforces a very narrow "maximum width" for wiki pages, making them useless for tables.  That's not good. :frowning:

I think there's a cut-n-paste mistake here too:
https://github.com/RedashCommunity/redash/blob/d2ac24585142a19f82794564abb8539786b2d1ac/redash/settings/__init__.py#L167-L174
Both REFERRER_POLICY and FEATURE_POLICY are being loaded from the environment variable REDASH_REFERRER_POLICY, even though they have different uses and take different values.  It probably hasn't been spotted before due to the default value for each of them being ok.
That FEATURE_POLICY one should probably be using a REDASH_FEATURE_POLICY environment variable instead.

Also, while I have been trying to run docker image for redashcommunity/redash:preview, I am getting error:
2023-06-20 10:43:44.157 UTC [1] FATAL:  database files are incompatible with server
2023-06-20 10:43:44.157 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 15.3.
I think postgresql version needs to be cross checked again.

Yes. It will be fixed.

On Jun 20, 2023, at 18:47, Aditya Khandelwal @.***> wrote:
Also, while I have been trying to run docker image for redashcommunity/redash:preview, I am getting error:
2023-06-20 10:43:44.157 UTC [1] FATAL: database files are incompatible with server
2023-06-20 10:43:44.157 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 15.3.
I think postgresql version needs to be cross checked again.
—
Reply to this email directly, view it on GitHub https://github.com/RedashCommunity/redash/issues/59#issuecomment-1598540673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBBQNGDJG4OJRDOM3IOOBTXMF5UBANCNFSM6AAAAAAZLXQQD4.
You are receiving this because you were mentioned.

Awesome, thanks heaps @jezdez. :smile:

Cool.  We need to fix that. :smile:
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt.  Any idea how to get them included?

Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base.

DS means "Data Source". :smile:
I can probably take a look at it tomorrow.  Badly need sleep at the moment after being up all night again. :man_facepalming:

DS means "Data Source". 😄
I can probably take a look at it tomorrow. Badly need sleep at the moment after being up all night again. 🤦♂

We can move ds lib to requirement.txt. Just a idea.

Cool.  We need to fix that. 😄
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt.  Any idea how to get them included?

Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base?

When you build images with docker, i want to know what args are you used?

In theory, it should be done by this:
https://github.com/RedashCommunity/redash/blob/master/.github/workflows/docker-preview.yml
That file seems to be mostly pre-defined GitHub Action er... "actions".  I'm not sure if the docker image build is being done by docker/setup-buildx-action@v2 or docker/build-push-action@v4 though.
Those actions have docs:

https://github.com/docker/setup-buildx-action
https://github.com/docker/build-push-action/

But after skimming those, it's still unclear if the build action is just directly running through the Dockerfile to create the image, or perhaps it's using the Makefile.
If it's building the image directly using the Dockerfile, then I thought it would be automatically using requirements_all_ds.txt already:
https://github.com/RedashCommunity/redash/blob/d844f2086663d75c53a204e7311417f2988f7e79/Dockerfile#L93-L94
That's supposed to install all of the data source dependencies unless the variable "skip_ds_deps" is set to any value and passed to it.

We can move ds lib to requirement.txt.

Yeah, it's probably the fastest way to get to a working result, while we're still getting a better idea of how all the pieces fit together.
a) Make it work  <-- we're still here
b) Optimise it
:wink:

Initial draft PR created.  No idea if it'll pass our CI though.  We may need to drop some of the data sources which pip (in my local install) complained about.
Lets see how it goes... :smile:

@Adityak4201 In the main getredash/redash repo we have a potential update for this almost ready.  But I'd like someone that uses MongoDB 6 to try it and make sure the data source is indeed working ok (once the update is merged).
Is that something you'd be ok doing?


Create several background processes from one project

The current solution that there's a one-to-one relationship between possible background processes and projects seems somewhat arbitrary (it covers the most common case, though).
If you would specify an id as well under which the task is managed you could create an arbitrary number of background processes.

+1


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:21: host_section_needs_exact_pinnings: Packages in host must have exact version pinnings, except python build tools.
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:22: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:41: missing_documentation: The recipe is missing a doc_url or doc_source_url
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:41: missing_license_family: The recipe is missing the about/license_family key.
WARNING: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:41: missing_description: The recipe is missing a description
ERROR: /tmp/abs_8a8j8ydouq/clone/recipe/meta.yaml:41: missing_dev_url: The recipe is missing a dev_url
Errors were found