Fix uncaught exception causing pipeline shutdown

Description
Fixes #3184 by:

Catch ClassCastException in JacksonOtelLog.toJsonString()
Add overwrite_if_exists option to parse-json processor. It defaults to true to avoid breaking change. Set it to false will prevent fields from being overwritten by the processor.

Issues Resolved
Resolves #3184
Check List

[x] New functionality includes testing.
[ ] New functionality has a documentation issue. Please link to it in this PR.

[ ] New functionality has javadoc added

[x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@oeyh , The unit tests are failing:
ParseJsonProcessorTest > test_when_dataFieldEqualToRootField_then_notOverwritesOriginalFields() FAILED
    java.lang.AssertionError at ParseJsonProcessorTest.java:430

Noticed this exception in the java 17 build action
ParseJsonProcessorTest > test_when_dataFieldEqualToRootField_then_notOverwritesOriginalFields() FAILED
    java.lang.AssertionError at ParseJsonProcessorTest.java:430


Use teeny-request instead of request

The teeny-request module can be used instead of request to improve require speed of the debug agent.

I'm pretty sure this library already uses teeny-request, all you need to do is remove two references to request types.

My local code was on the branch chore/update-teeny-request that was out of date while the chore/update-teeny-request-5 was the correct branch that was up-to-date with master.  I have PR #750 to resolve this issue.


Program:exit called before Vue nodes

It looks like a common pattern to collect info from multiple nodes and do final check in "Program:exit".
However, this pattern breaks for Vue nodes (e.g. VElement), because Vue nodes are called after "Program:exit".
I think this is not right because Program is the topmost node:

Using "VElement[name=template]:exit" does not solve the problem because
Is it possible to fix it? I.e. to visit template section before Program:exit?

Is it possible to fix it?

No. it cannot be changed. This is because we are traversing the template nodes in the Program:exit listener.


enumArrayForKey should not be exclusive to the enum type

Enum arrays (Decoder.decode(enumArrayForKey:)) forces you to have all values in the array the same type of the enum, e.g
enum Color {
  case blue
  case red
}

enum Element {
  case fire
  case wind
}

//JSON
"color_elements": {
  "blue",
  "fire",
  "red",
  'wind'
}

If I apply:
self.colors =  Decoder.decode(enumArrayForKey: "color_elements")(json) ?? []
I would expect to have an array back with only the colors, but I'm getting back an empty array because there is a different type within the color_elements object

that was intentional. arrays in Swift are homogenous, i would not expect to have an array of various enum types. if that is something your application does have, i'd contend you should create a custom transformation]

In my case color_elements is homogenous, but contains a value not present in the enum... I think that should not avoid the generation of the array... just ignore it... all values are the same type of the enums "Strings"

Let me know what do you think about this, otherwise I will follow your suggestion about creating a custom transformation for this case and we can close this issue.

it seems to me this issue is similar in nature to this one - https://github.com/hkellaway/Gloss/issues/189
i.e. what is the expected behavior for unexpected input. in that issue, we resolved that for unexpected input, decoding functions that returned arrays would not return partial or empty arrays but would return nil.
this is a bit different in that what programmatically unifies enums is that they are RawRepresentable - but from a model point of view, what unifies them is that they are of the same "type", i.e. belong to the same enum.
i'm still of the mind that having an enum array with the same types is what is expected.

Fair enough, I will go with the custom transformation.
Thanks 👍

Sorry for the re-opening...
I think actually that way of working is introducing limitations to the library.
For example I'm actually working on a news feed api, it could return different types of objects under the same root container "items" this way:
{
  "data": {
    "pagination": {
        "total": 11124,
        "per_page": 30,
        "offset": 0
    },
    "items": [
      {
        "picture": "Some picture"
      },
     {
       "story": "Some Story"
     },
    {
      "video": "Some video"
    }
    ]
 }
}

The library now requires all elements within items must be the same model, which is a very rude... so now I can't do something like this
pictureList = "items" <~~ data ?? []
storyList = "items" <~~ data ?? []
videoList = "items" <~~ data ?? []
because the library will stop parsing all element in the array....  I think the idea should be to facilitate functionality not to impose a particular way of thinking?
I think it should not be considered as a parsing error to find some different element type within the parsing array... the lib is not being open to "internet" in which many requirements and many situations might me present.
What do you think with this scenario?

no problem reopening, of course.
i don't see this as rude, but opinionated - the library expects homogenous arrays and will carry on effortlessly in that case. i find the potential for partial arrays adds an element of surprise to the output the library has, that i'd rather the specific application explicitly handle. that's how i personally  expect JSON libraries to behave; i took that original cue from other libraries like Mantle.

Yep, opinionated...
It is a restriction based on an opinion of how should behave, of course there is no right and wrong with this case.
Would be nice at least, to provide the option and let the users decide how they whant to use the library instead of forcing a particular point of view of how they should do it, well that's just my opinion.
So yes... I will have to follow the custom transformation way for everything that implies this scenario, even for normal arrays 😩

@hkellaway  Any chance to make the behaviour optional to let users decide how they need the lib to behave in these particular scenarios?
I'm ok dealing with a custom transformation for enums, but doing the same for general arrays doesn't feels right.

alright. something like including an 'allowPartialArrays' flag argument that is defaulted to 'false' in the operator definitions? or did you have another implementation in mind? and would you be willing to make a pull request to that effect?

Sure, I would make a pull request tomorrow I already did some testing using Protocol conformance instead of a flag.
My idea, in order to do not mess with the actual implementation is to create additional protocols
public protocol PartiallyDecodable: Decodable { }
public protocol PartiallyEncodable: Encodable { }

This way if you need to make your model "partially" decodable or encodable you just conform to this protocol, no need for a flag messing around 👍 and it could apply to any array type (enums, structs, class)

Not sure about naming though!..
Because the idea of this is that it will apply when parsing arrays of the models implementing those protocols, if I see any model implementing that I could think it means that the actual model could be somehow partially parsed which is not correct.
thinking about it...


Update readme to note that sage-modeler-site and building-models need to be deployed at the same time [#181597207]

Pure documentation update, no review required.


Add support for postcss-responsive-properties

Hey, there! I have a suggestion for the way how to improve writing of responsive styled-components. It seems, that you, guys using some lightweight version of PostCSS? So, I suggest to try figure out, could we add support for this postcss-plugin: https://github.com/alexandr-solovyov/postcss-responsive-properties ? So, we can write responsive values for our rules, thanks!
Even though it's a pretty simple example, I think it might help to make writing more complex styles a bit easier.
Instead of this:
const Example = styled.div`
  height: 3em;
  width: 3em;
  background: papayawhip;

  @media (min-width: 700px) {
    background: palevioletred;
  }

  @media (min-width: 1024px) {
    width: 5em;
    height: 5em;
  }
`;

We could write this:
const Example = styled.div`
  height: {
    0: 3em;
    1024: 5em;
  }
  width: {
    0: 3em;
    1024: 5em;
  }
  background: {
    0: papayawhip;
    700: palevioletred;
  }
`;

@alexandr-solovyov We unfortunately switched away from PostCSS and are now relying on stylis. We might in the future switch again to an ISTF/Sweetsour based pipeline, but that's far away.
So the thing with these plugins though is that we'd like to keep our supported CSS as closely to vanilla CSS as possible; so this probably won't be something we'll support, and only if our pipeline ever becomes customisable — no promises here :wink: — then this'd be in your hand.
Hope this clarifies it 😅 Sorry, if SC's internals are not as transparent as they maybe should be


Make annotation deletion more explicit

Currently, you delete an annotation by pressing the small cross handle on the top right corner.
To me, this button looked more like a "close" button than a "delete" button. We should probably have both a close button and a delete button, and make it clear what delete does.

We should probably consider having a similar UI to Google docs for removing annotations:


Fixing test_db errors

Work in progress, need comments. @tom-de-smedt  @guydepauw
Fixed test_db errors involving cmp and zip object.
Please have a look when you have some time and let me know if I am going in the right direction. I'll continue to fix more errors till then.

https://travis-ci.org/shubhangiKishore/pattern/builds/216032856
All tests for Python 3.4 are passing. (Excluding the skipped tests)


[Docs] Add note on serving documentation locally

This adds brief instructions to CONTRIBUTING.md on how to serve up the documentation website locally.
This is half of a proposed fix to the issue mentioned by @adityarb88 here: https://github.com/vmware/clarity/issues/1820#issuecomment-354438343
The other half is a pull request against the new-website branch here: https://github.com/vmware/clarity/pull/1823

Thanks for this, though its about to change significantly in a few days. Nothing wrong with this PR other than there will be a lot more detail added since we'll be able to use server-side rendering for our site.


Unoptimized memory usage during gameplay

Describe your bug here.
Hi fellas!
After updating a project of mine to the latest source version of Psych I happened to notice a memory usage issue: After the player hits their first note, memory usage exponentially grows. As a test, I played Blammed in both 0.5.2h and the latest source update, which helped me notice a few things:

Both songs ended off using 40MB more memory than they started (source went from 575 to 615, 0.5.2h went from 680 to 720). However, 0.5.2h's end result was due to a gradual climb throughout the song, while the latest source constantly had to reset itself down to 575 to avoid going into the 620s.

During the memory cleaning cycles in source's code, a small frame drop occurs. At some points it isn't noticeable but occasionally I've dipped below 60 fps. That ain't good chief.

The whole cycling of memory begins when the player presses their first note. This doesn't happen on botplay for whatever reason.

I don't know if this is a bug or not, but I wanna (hopefully) see the older, more optimized memory return. Good luck guys!
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Source
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No, this is a comparison between the latest source code push and release 0.5.2h

UPDATE: found the cause! Reverting how the score text is displayed in game seems to get rid of the erratic memory and frame drops.


Problem on WinogradConv Op when training from scratch

Hi, @xingyul :
Thanks for sharing your research, and its implementation.
When training ResNet-18 from scratch, I encountered the following problem to the WinogradConv operator:
'TypeError: 'NoneType' object has no attribute '__getitem__'
The problem pointer implies the bug occurred on this line.
Any suggestions would be appreciated!
Thanks in advance.
Cheers!
P.S.:
Ubuntu 16.06
gcc: 5.4.0
cuda: 8.0.61
tensorflow: 1.4
tensorpack: commit 0086e156c

This is a bug. "use_mask" should be used instead of "test". Thanks for catching this!

Thanks for fixing the issue!


FUEL

FUEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEL not feul

Yes. I will be going over the code with a fine toothed comb soon.
I'm no good with spelling.


[stable/anchore-engine] Certificates in secret don't get applied

Describe the bug
Using an existing configuration for values.yaml I used before (succesfully on the same cluster) I installed the 1.6.0 version of the chart. This installs 0.7.1 of anchore-engine. Upon starting the engine, the catalog component didn't want to start because of certification verification errors for the S3 archive storage. This storage is a private S3 instance with company signed certificates.
I solved this in the previous version of the chart (1.3.7, anchore-engine 0.5.2) by including a secret with the root- and intermediate certificate of the CA with anchoreGlobal:certStoreSecretName. This worked great. However, this doesn't work anymore. The secret is there and gets mounted in the pod to /home/anchore/certs/ just fine. It turns out that there's logic in the container (/docker-entrypoint.sh) which handles these custom certificates. But when I look in the deployment which this chart generates I see that both "command" and "args" are defined, which according to https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#notes will cause ENTRYPOINT to be ignored. This means /docker-entrypoint,sh never gets called. Compiling all this leaves me with a couple of questionmarks. First, how has this ever been able to work at all as I don't see any recent changes to either the chart or the container which touch this mechanism. Second: how do I inject my intermediate certificates into anchore-engine as using anchoreGlobal:certStoreSecretName cannot work.
Version of Helm and Kubernetes:
Kubernetes : 1.15.3
Helm : 2.16.5
Which chart:
stable/anchore-engine
What happened:
SSL validation errors for external services due to private CA SSL certificates missing despite being added with anchoreGlobal:certStoreSecretName
What you expected to happen:
anchore-engine to work with custom private CA certificates
How to reproduce it (as minimally and precisely as possible):
Try to use anchoreGlobal:certStoreSecretName to inject custom CA certificates, see if they get added properly (see: https://github.com/anchore/anchore-engine/blob/master/docker-entrypoint.sh). If in the container /docker-entrypoint gets executed by hand, everything works as expected. Ergo, in containers deployed with the chart the ENTRYPOINT gets ignored.
Anything else we need to know:
Probably.

/assign btodhunter

@FOTempel - thanks for pointing this out. I'm working on a fix right now to prevent the chart from overriding the entrypoint. I will try to get it pushed ASAP.

@FOTempel - thanks for pointing this out. I'm working on a fix right now to prevent the chart from overriding the entrypoint. I will try to get it pushed ASAP.

Thanks for looking into this, did you make any progress yet? Is there anything I can do to help?

I made some changes to the chart in our fork that I think will address this issue, I just need to find some time to test that it works as expected. Hopefully I'll be able to get that done early this week. If you want to check out my attempt at a fix, you can see it here

Tested the updated chart with the above use-case.
Steps to validate success:

Created a secret pointing to the cert key in the namespace: kubectl create secret <cert_name> --from-file=<cert_file>
Added secret name to anchoreGlobal.dbConfig.sslRootCertName and anchoreGlobal.certStoreSecretName and enabled ssl ssl=true
Deployed the helm chart and verified all pods enter running state
Confirmed in the logs that the services are able to communicate with external postgres using the certificate

Works great, thank you!


fix: prevent infinite GetTwinOperation loop

Added connection logic to on_get_twin_complete to ensure GetTwinOperations aren't tried indefinitely when not connected
Added appropriate unit tests
Fixed minor typos

/azp run Azure.azure-iot-sdk-python-horton-e2e


Feature request: Bind publisher to private app

No scope in the provider to bind a publisher to a private app, currently we have to fall back to the API which is suboptimal
eg
resource Private_app_add_publisher {
app_id = string
publisher_id = string
}
the inverse would be good as well for when a publisher is torn down
resource Private_app_remove_publisher {
app_id = string
publisher_id = string
}

I will review this but likely we will need to rely on Publisher tagging which you have also requested.
https://github.com/netskopeoss/terraform-provider-netskope/issues/5

From our conversations it does appear it will

Closing this as it will be addressed with https://github.com/netskopeoss/terraform-provider-netskope/issues/5


[Utils] Update the branch names in update-checkout

@swift-ci Please test

Build failed
Jenkins build - Swift Test Linux Platform
Git Commit - 7e70752706074a94638bf18573076553b35cb156
Test requested by - @shahmishal

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - 7e70752706074a94638bf18573076553b35cb156
Test requested by - @shahmishal


feat: Fabricpool panels - parity with 1.6

updated all panels with description and units.
Only pending panel is Top Volumes with Object Storage Requests graph, because the variable of type volName-Req type.

Added 5 panels as shown below.

Me and Rahul discussed regarding the 6th panel where panel would need filter based on volume-metric, but due to instances/metrics independent data structure, it's not feasible in backend. Also, I have tried to manage in prometheus query but label_join/label_replace function don't support filter on newly generated label volume-metric, and without this it's difficult to implement in panel with topk.
So, we will submit 5 panels for now and this 6th panel will be targeted later.


Fix byte compile warnings

This patch fixes following warnings.
steam.el:29:1:Warning: cl package required at runtime

In steam-get-xml:
steam.el:40:16:Warning: reference to free variable `url-http-end-of-headers'

In steam-download-logo:
steam.el:132:56:Warning: reference to free variable `url-http-codes'

In end of data:
steam.el:149:1:Warning: the function `org-current-level' is not known to be
    defined.

Thanks!


Automatic detection does not work after manual deletion

Hi,
It's a fantastic add-on.
I installed it two days ago.
I have about 50 pcs Ikea light.
At first, I tried 2 of them to add  manually (Settings->Devices&Services->Integration->Add Integration->PowerCalc->Virtual Power->.....)
I did not select the right options because I selected a fixed type. So I delete integration (Settings->Devices&Services->Integration->PowerCalc->Delete) Then I reinstalled these 2 Ikea lamps with LUE setting and they worked fine. At this point, I read through your documentation on GitHub. I read that the add-on automatically detects my Ikea lamps, so I delete the integrations of  2 lights again, and I configure the add-on in configuration.yaml:
powercalc:
  enable_autodiscovery: true
  scan_interval: 00:01:00 #Each minute
  power_sensor_naming: "{} Power"
  create_energy_sensors: true
  create_utility_meters: true

Unfortunatelly those 2 lights I set first manually don't appear in the entity list.
I stop the auto-discovery (enable_autodiscovery: false -> restart HA), delete powercalc entities (Settings->Devices&Services->Entities)
, restart auto-discovery (enable_autodiscovery: true-> restart HA), and Add-on recognized all my lamps (added power and energy sensors to the lamps) except the two originals.
What could be the problem?

You can enable debug logging, see last section of the readme. That should probably give some clues why no entities are created for this light or why the light is not discovered.
Also I would first suggest to install the latest v1.0.0 beta as this has a new feature where all the discovered lights appear in the integrations overview (as discovered) and allow you to include them in your HA. Also you can change some options for the power sensors later on.

Thank you for your advice.  I can see that the auto-discover can't discover the light:
'2022-10-25 23:19:30.912 DEBUG (MainThread) [custom_components.powercalc] light.stairs_led_up Model not found in library, skipping auto configuration'
But it's an IKEA LED1903C5/LED1835C6. As I have seen, the library includes this bulb

This alias has actually been only recently added two days ago. https://github.com/bramstroker/homeassistant-powercalc/pull/1185
This was not released yet. I just released v1.0.0-beta.14 which contains this addition.
When you don't want to install a beta version you can wait a bit. I expect to release stable version upcoming weekend or the week after.

I see, thanks!


Changed behavior of RouteAttribute template

Hi,
in 6.0.0-beta7 something like this works:
[Route("[controller]s")]

public OrganizationController(LegalContext context)
{
    ...
}

url was: /organizations
In 6.0.0-beta8 it stopped working - Controller isn't created and web server returns:
HTTP/1.1 500 Internal Server Error
Date: Fri, 16 Oct 2015 09:25:15 GMT
Content-Length: 0
Server: Kestrel

Is it by design or somethings didn't work as expected?

Ok, strange, but it now starts working - maybe this was problem related to something else.
Sorry for the confusion.

If you start seeing 500s, I strongly recommend hooking up a debugger or the error page middleware https://github.com/aspnet/Diagnostics/blob/dev/samples/DeveloperExceptionPageSample/Startup.cs#L10 so you can see what's failing.

Closing for now. Please re-open or log a new bug if you see anything odd again.


Avoid republishing for confluence pages not changed

It's possible to publish only the confluence pages changed? I don't know if you have a sort of metadata to know the pages really changed..

Hi @amanganiello90  I've deployed the version 6.0-SNAPSHOT with requested feature.
Could you test it and let me know ?
Thanks in advance

Thanks, I will check it.

Hi @bsorrentino,
I built the source code from develop to use the 6.0-SNAPSHOT version (it isn't been deployed on maven central repository http://search.maven.org/#search|gav|1|g%3A"org.bsc.maven" AND a%3A"confluence-reporting-maven-plugin").
Hovewer some pages (few pages) not changed are republished from the plugin.

Hi @amanganiello90
The plugin generate a file named .state.properties could you share it
Meanwhile i'll put more information on the log

Hi @amanganiello90
if you run maven with -X option you could see which page will be skipped
have you a chance to share the .state.properties generated file

Ok, as soon as I can, I will check

Where is stored the .state.properties file? However, both in confluence and with maven -X parameter, I see that the not changed pages are reloaded

The file is generated in the current directory (ie where maven start)
Please consider to use the deployed SNAPSHOT to be aligned

I have problems to resolve with maven this snapshot version also after configuring in maven settings the sonatype repo. Besides your code is not on develop, because after a local build and running I don't see any changing of your feature for publishing wiki pages. So, could you create a branch with this feature in order to build in local and run the confluence-reporting deploy goal? Thanks

Hi @amanganiello90
I'm surprised that you have problem to use SNAPSHOT, could you try the  configuration below?
<pluginRepositories>
    <pluginRepository>
        <id>sonatype-repo</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </pluginRepository>
</pluginRepositories>

However such implementation is on the branch 'feature/delta'

ok thanks, it works...but I don't find the .state.properties after publishing success :(

ahhhh ok, I found it into src/site/confluence directory !
I suppose that in this file there are the id for every page, and the pages already published in the same deploy have the same id :)

have you chance to join gitter ?

I'm connected on gitter

I don't see u ... please, when u ready,  join the maven-confluence-plugin room

Implementation released in 6.0-beta1

I'm worried that some uses cases will not be covered by this approach (based on timestamps of files specified in site.xml).
For example, if change homepage title in site definition site.xml - content of reflinks on other pages will be changed too because rendered reflinks includes homepage title.
I want to propose a more robust approach:
store in ./target/confluence-plugin/converted_pages internal representation of all pages  (in confluence wiki format) and attachments according site definition.
Then for every page and attachment compare it by content.
If content changes - publish, otherwise - not publish and print log message.

Hi @qwazer
This issue, raised by @amanganiello90,  was mostly to avoid re-publication during CI process.
Your suggestion is very interesting ... and I'd prefer that you'll open a new issue concerning comparing by content

Great @qwazer !


ci: standardize workflows using SciML's reusable workflows

Update the workflows in this repository to use SciML's reusable workflows.
This is part of a larger effort to standardize the SciML's CI workflows for more generic and common requirements, to keep the workflows uniform and easier to maintain.

This library needs a bit more care because its CI needs to install the right stuff in Python.


Discovery

GET /dns/{domain}/subdomains Host: dns.projectdiscovery.io Authorization: API_KEY Connection: close HTTP/1.1 200 date: Sat, 09 May 2020 09:18:12 GMT content-type: application/json content-length: 241 access-control-allow-origin: * cf-cache-status: DYNAMIC server: cloudflare cf-ray: 590a54eccbc1dd0e-SIN cf-request-id: 029a5367fc0000dd0e87bb1200000001 {"domain":"hackerone.com","subdomains":["a.ns","b.ns","email","go","","info","links","ns","o1.email","o2.email","o3.email","docs","api","www","","mta-sts","mta-sts.managed","mta-sts.forwarding","support"],"@timestamp":"0001-01-01T00:00:00Z"}


ask about Saga

Hi, i am reading your project and I love it. But i have a question, what queue do you use for the message queue, how do you manage the state of each service and how do you tackle and implement "Transaction outbox ". Thank u so much

The message-queue will be a bounded in-memory work queue in front of the thread-pool needed by the saga-execution-coordinator. The saga-log acts as persistent state cluster-wide and will handle forward (or backward) recovery so that we get eventual consistency of all saga executions.


Fix dblquad and tplquad not accepting float boundaries

This is a simple fix to make dblquad() and tplquad() functions accept numbers (instead of just functions) as boundaries for the integration. I'm not sure if the previous behaviour was a bug or done like that on purpose, but since nquad() (which dblquad and tplquad use) accepts numbers as boundaries, it seemed inconsistent to me.
I also changed the documentation and added a simple unit test to reflect this change.

Makes sense. Thanks @vkk800


Rank Polymorphism

Why not?
Currently these would be errors, but they could easily be rank polymorphic:
» [ 1 2 3 ] 1 + pl
[ 2 3 4 ]

» [ "it's" "too" "quiet" ] upcase pl
[ "IT'S" "TOO" "QUIET" ]

Technically this isn't rank polymorphism since dt arrays don't have rank i.e They cannot be multi-dimensional. However they do have depth(i.e nesting). Although that could be useful for dealing with tabular data
Also this means polymorphism means you will need some kind of rank/depth modifiers to control it e.g
example ← 10‿10⥊↕100 # Assume this is something useful
┌─
╵  0  1  2  3  4  5  6  7  8  9
  10 11 12 13 14 15 16 17 18 19
  20 21 22 23 24 25 26 27 28 29
  30 31 32 33 34 35 36 37 38 39
  40 41 42 43 44 45 46 47 48 49
  50 51 52 53 54 55 56 57 58 59
  60 61 62 63 64 65 66 67 68 69
  70 71 72 73 74 75 76 77 78 79
  80 81 82 83 84 85 86 87 88 89
  90 91 92 93 94 95 96 97 98 99
                                ┘
# table-reduce plus
Sum ← +˝

Sum example # Columnwise
⟨ 450 460 470 480 490 500 510 520 530 540 ⟩

Sum⎉¯1 example # Rowwise
⟨ 45 145 245 345 445 545 645 745 845 945 ⟩

I would recommend reading the J documentation on Rank

IMO rank is perfectly suited for the language. It would finally allow shell invocations to mirror scripts. Shell invocations would just be Array -> Array

Thanks for chiming in, the rank vs depth distinction is good to call out. I guess I was thinking more of depth as a proxy for rank
I think a proper rank really only makes sense when getting into math operations, so I might save it for later. I don't know what rank would mean for text processing, for example, and that's more of dt's bread and butter


How to tell commits ahead/behind remote?

Is there a way to get the number of commits a repository is ahead/behind a remote?

Sorry but this is not the right channel to ask questions please post your questions at:
https://stackoverflow.com/questions/tagged/go-git


Wrong return count in executeBatch()

The current implementation of executeBatch() returns a wrong number of effected rows given the underlying driver does not provide such information. In the line below, rowCounts[] is updated with 1 for each statement in batch:
https://github.com/ing-bank/cassandra-jdbc-wrapper/blob/3ab76c07bf043cf93670b636452e0b4de53af29c/src/main/java/com/ing/data/cassandra/jdbc/CassandraPreparedStatement.java#L282
Accordingly to executeBatch() javadoc it should be updated with SUCCESS_NO_INFO.

Good catch @stefanofornari, I'll fix this asap.

Will be fixed in the next release (see commit https://github.com/ing-bank/cassandra-jdbc-wrapper/commit/40f81093875331c967026c1de4fd2dbf5bc31219).


Safety improvements

Please accept my humble safety improvements to your Makefile and small polishes to README.md.

Merged, thanks.

Can you please make a new release with these changes? 2.0.1 supposedly.

Done. There should also now be an automatic release triggered on every push to the master branch tagged "latest".


Add proposalId to Sample entry

Most of the samples we receive are related to a particular proposal (except for water, simple dispersants and sample containers).
We would like to link sample entries to a particular proposal, in the same way that you can see which datasets are related to a particular proposal. This would require a sample entry in the database to carry information on which proposal it belongs to. The addition of a proposalId to the Sample class should be sufficient for this.
(cc. @ibressler)

Currently the datasets each store a sampleId and a proposalId, so to collect the samples in one proposal, once would have to first fetch the datasets and then each corresponding sample. Adding the proposalId to sample would shorten this process. Is there only one proposalId per sample, and do water, simple dispersants etc behave differently?

As there is a workflow to get samples for a given proposal, and a sample can have more than one proposal, we will not add the proposalId to the Sample model

Sorry for not responding sooner. Can you let us know (or link to) what the workflow is for getting the samples belonging to a proposal?


[Feature Request] Select2 focus to open the list

Feature Request
What's the feature you think Backpack should have?
When I'm on a create/update form, with some text fields and a Select2 field in betweens, I can navigate trhu the text fields with the TAB key. This behavior is very natural to users who are fluent with IT.
BUT, when the Select2 field gets the focus, the "typing box" field doesn't show, and the list doesn't pop out. The user need to click on the "fake text" field to make the Select2 field show.
Have you already implemented a prototype solution, for your own project?
Not yet. I'm pretty sure that some JS would do the trick.
Do you see this as a core feature or an add-on?
This should be a core feature. Using a form with TAB is something that should work by design in any application.

Edit : It seems that the Select2 is in fact based on the Select2 Jquery library ?
If yes, this could help : https://stackoverflow.com/questions/20989458/select2-open-dropdown-on-focus

Totally agree - it's something that keeps annoying me sometimes too. It's one of the often-used fields, so improving this bit would do a lot to improve the UX for keyboard-form-fillers like us.
Thanks for opening an issue for this @LemarinelNet - I've prioritised it, it's now on our to-do list. If you have any ideas about how to do this, it'll definitely move the issue forward.
Cheers!

Haven't look for any solution, but while searching for an "auto-complete-able" select component for some UI, I've came accross this :  https://selectize.github.io/selectize.js/
It's not a way to add focus to the current Select2 component, but a good candidate for replacement : It does supports everything (as far as I know) that Select2 does, support an "open on focus" option, add elements on the fly, populate with ajax, easy to code, and so on ...
However, I would understand that replacing the Select2 component with another one might not be a choice....If not, perhaps creating a new component would be a choice, but IMHO, it would make select2 redundant compared to Selectize, which does the exact same job !?
I think that if the choice would be mine, I would drop the Select2 component and replace it, even while keeping the same name ;)

I had a further look into Selectize, and did not managed to make it work !
So I stayed with Select2.
Now I have to look if I can make work the focus event ;)

Ugh - it looks like it's not as simple as just flicking a Select2 switch - https://stackoverflow.com/questions/20989458/select2-open-dropdown-on-focus So now I'm wondering if the functionality itself is worth the added complexity to the select2 fields. If it's worth the trouble.
To be fair, it does work to move through select2 fields with tab (at least in my experience), what's incovenient is that the dropdown doesn't show on focus, so can only select an item from the dropdown if you know what item you want beforehand. Which yeah now that I mention it it's a little inconvenient...

I had a further look into Selectize, and did not managed to make it work !
So I stayed with Select2.
Now I have to look if I can make work the focus event ;)

Can you explain what problems you have faced, i was thinking to integrate selectize.js

I had a further look into Selectize, and did not managed to make it work !
So I stayed with Select2.
Now I have to look if I can make work the focus event ;)

Can you explain what problems you have faced, i was thinking to integrate selectize.js

Totally agree - it's something that keeps annoying me sometimes too. It's one of the often-used fields, so improving this bit would do a lot to improve the UX for keyboard-form-fillers like us.
Thanks for opening an issue for this @LemarinelNet - I've prioritised it, it's now on our to-do list. If you have any ideas about how to do this, it'll definitely move the issue forward.
Cheers!

@tabacitu I want you to check this out and tell me whether this looks promising or not?

Thanks,
Anand

Totally agree - it's something that keeps annoying me sometimes too. It's one of the often-used fields, so improving this bit would do a lot to improve the UX for keyboard-form-fillers like us.
Thanks for opening an issue for this @LemarinelNet - I've prioritised it, it's now on our to-do list. If you have any ideas about how to do this, it'll definitely move the issue forward.
Cheers!

@tabacitu I want you to check this out and tell me whether this looks promising or not?

Thanks,
Anand

It looks like it's exactly what's needed from a UX perspective yes!
￼￼Take note though:

we do have to think long and hard before we replace ￼Select2 because we have 6-8 fields using it, some of them very complicated￼￼￼, and very heavily used (like 99% perhaps);
for the next major Backpack version (fall-winter 2021)￼￼, we're probably going to upgrade the Bootstrap version too; and Bootstrap 5 removed the jQuery dependency￼￼￼... Which means we could do that too - we can rewrite our JS code to not use jQuery and then the developer can choose whether to use it or not - just like Vue.JS, just like React;￼￼￼￼￼￼ just uncomment one line in the config; The biggest roadblock to removing the strict jQuery dependency in Backpack though... would be how much people use Select2; because even if we remove jQuery everywhere else￼, most forms have Select2 fields, so then most forms would load jQuery... which you know... defeats the purpose; ideally we would find a dropdown input that is vanilla JS (with no jQuery dependency), but that would also be able to pre-populate the list like selectize does; unfortunately, it looks like there are very few good options... I'll keep looking, but please let me know if you know/find anything that fits this description; if we are to rewrite some huge fields we'd better rewrite them using something that we can keep for a lot of years...￼

hello everyone @Adesin-fr @anandphulwani @tabacitu  i will be glad to take this task please let me know if it just require open select2 when focus is make, i make a small experiment with this behaviour in Backpack crud, let me know if this is how you expect this work and i will PR this.
https://recordit.co/uVSY6nXCxd
Cheers.

Hey @jorgetwgroup
I think @maurohmartinez tackled this issue in a previous PR where we added support for focus in repeatable too. #4491

Hi @pxpm ups, i try in main and didn't find this behavior on focus select2 in normal forms, this mean focus select2 in any position, not just in first place, maybe i miss something and i try to cover this on this branch https://github.com/Laravel-Backpack/PRO/tree/select2-open-on-focus i only think need to handle new parameter as default true.
To use it, add to any select2
'openOnFocus' => true,

Let me know.
Cheers.

@jorgetwgroup , if you want Pedro to take a look at your solution, please

create a PR from your branch to main; that way he can take a look at the code and evaluate the impact;
assign him to that PR; that way he knows the ball is in his court;
move that PR to "Needs Review";

Thanks!

Done.


Merge kirkstone 0.18.25-0.18.27 into kirkstone-next

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 4 committers have signed the CLA.:white_check_mark: JoergZeidler:white_check_mark: mlilien:x: JanZachmann:x: HarryWaschkeitYou have signed the CLA already but the status is still pending? Let us recheck it.


extensions: Separate generation of extension db from docs in CI

Signed-off-by: Ryan Northey<EMAIL_ADDRESS>Commit Message:  extensions: Separate generation of extension db from docs in CI
Additional Description:
As we will require the extension DB to be generated for docs and schema, it makes sense to separate this task so that those jobs can run in parallel
Im wondering if this job should be moved to a bazel build somewhere - but initially im just gonna  get it working by splitting the script/job
UPDATE
rather than separating the ci job, i have moved the db generation into a bazel job - which will allow other tasks that also require this data to make use of it more easily/bazely
this also adds buildozer into the bazel env, so we might want to make use of that elsewhere and potentially remove buildozer from the build-image env
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

im  not 100% how best to separate this, or exactly how it will be required by #15100 - so this PR is currently experimental to figure these things out

@htuch would you mind having a look at this one
im not sure if its done the best way - using /source/... feels a bit hacky
also, i had to change the visibility of the actual //source and include in the bazel data - not sure if i understand why

@phlax generally this is a really nice step. I don't mind hardcoding /source as the default for ENVOY_SOURCE, but maybe you want to use ENVOY_SRCDIR as done elsewhere in ci/. The visibility stuff would benefit from better understanding, since I rather keep the limitation in at least some CI checked build to avoid accidental dependencies across core/extensions happening.

@htuch this should be good for another review

/retesst

/retest

/retest

im pretty sure the coverage test fail is unrelated - retesting doesnt seem to fix

/retest

/lgtm deps

@phlax I'll be having a stern word with @htuch on his deps LGTM :smile: The version is hardcoded in strip_prefix and urls. Updated for 4.0.1 we should have
    com_github_bazelbuild_buildtools = dict(
        project_name = "Bazel build tools",
        project_desc = "Developer tools for working with Google's bazel buildtool.",
        project_url = "https://github.com/bazelbuild/buildtools",
        version = "4.0.1",
        sha256 = "0d3ca4ed434958dda241fb129f77bd5ef0ce246250feed2d5a5470c6f29a77fa",
        release_date = "2021-03-01",
        strip_prefix = "buildtools-{version}",
        urls = ["https://github.com/bazelbuild/buildtools/archive/{version}.tar.gz"],
        use_category = ["api"],
    ),

i can add a cleanup if you want

@moderation good point; can we add a check on this kind of hardcoding to the deps validation Python?

A check in deps validation definitely makes sense. Will make an attempt.


Removed redundant Source check

What does the pull request do?
It removes a redundant null check for the Source property which is set in the caller.

You can test this PR using the following package version. 11.0.999-cibuild0037374-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]


Add support for function_to_apply in classification

For classification models, users should have some way of specifying the function to be applied to the logits rather than softmax always being used since the logits do not always correspond to mutually exclusive classes.
Take, for example, a model's logits representing whether a piece of text conveys negative, neutral, or positive sentiment in a text classification task. In this case, the softmax activation function is apt as it ensures that the probabilities in the final output vector sum to 1. However, if the logits denote whether the text is toxic, obscene, insulting, etc. (as seen in Detoxify models), the sigmoid activation function would be more appropriate since the values in the final output vector do not necessarily have to sum to 1.
HF pipelines used softmax as the mandatory activation function over multi-class model logits until https://github.com/huggingface/transformers/pull/8328, when they added support for the function_to_apply parameter which can be specified by the user.
I am proposing support for a function_to_apply parameter here as well. The options are the same as implemented in HF: softmax, sigmoid, and none (none applies no activation function and just returns the raw logits). softmax is the default since it preserves backward compatibility and seems to be the most common case.

Thank you for your feedback, @jonatanklosko! I have implemented your suggested changes.


A11y: Tab order lands on element "Pane" with no focus indicator

Describe the bug
When opening a folder and tabbing through the UI, I noticed that focus can land on an element that is being announced "Pane" without a more descriptive name that also has no focus visual:
To Reproduce
Steps to reproduce the behavior:

Open folder in Files (in my case desktop)
Tab through entire app
Start narrator
Notice elements visited and announced
Notice pane without name or focus indicator

Expected behavior
Either give it a proper name and add focus indicator or skip element in tab order
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS Version: Win10 10.0.19042
App version: [<IP_ADDRESS>]

Additional context
Add any other context about the problem here. Does this problem occur again after restarting the app?
Log file
Please upload the log file here so that we can understand your issue better. You can access it from Settings->About->Open log location. You can also find the log file manually by following this guide.
debug_fulltrust.txt
debug.txt

@chingucoding This issue should be resolved with #3312.

@chingucoding This issue should be resolved with #3312.


Correct queue child command handling ECFLOW-1937

Codecov Report
Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (afd8444) 83.34% compared to head (f566d18) 83.32%.

Files
Patch %
Lines

Http/src/ecflow/http/ApiV1Impl.cpp
12.50%
14 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##           develop      #79      +/-   ##
===========================================
- Coverage    83.34%   83.32%   -0.02%
===========================================
  Files          590      590
  Lines        52614    52636      +22
===========================================
+ Hits         43849    43861      +12
- Misses        8765     8775      +10

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


got error when try to build, no buildable Go source files in /XXX/netlink/nl

XXX-M-Q0DJ:netplugin XXX$ make build
./scripts/checks
got below error when try to build it
i am new for go, please see if there are any inconsistency for code checkin.
Checking Go version...
Checking gofmt...
All checks passed!!
go get -d ./ ./netdcli/ ./mgmtfn/k8contivnet/
go install -v ./ ./netdcli/ ./mgmtfn/k8contivnet/
../go/src/github.com/vishvananda/netlink/netlink.go:14:2: no buildable Go source files in /Users/XXX/Desktop/Private/go/src/github.com/vishvananda/netlink/nl
make: *** [build] Error 1
Environment:
OSX 10.8
go 1.4.2 (with go-installer for mac)
GOPATH be well configured

@jsoyee
Looks like this error is specific to the mac environment, as the build seems to be going through fine in our ubuntu based environment for sanities.
I don't have a mac to try it out but the error seems to indicate that no go source was found at '/Users/XXX/Desktop/Private/go/src/github.com/vishvananda/netlink/nl'. This directory should have been created as part of downloading go dependencies. Can you please verify if it has the .go files downloaded? Also does the build fail always or does subsequent build goes through?
I will see if I can find a mac to repro the error locally.

@mapuri
thanks for your feedback. yes, i guess all .go files has been downloaded,
YIZH2-M-Q0DJ:nl yizh2$ pwd
/Users/yizh2/Desktop/Private/go/src/github.com/vishvananda/netlink/nl
XXX-M-Q0DJ:nl XXX$ ls
addr_linux.go			xfrm_linux.go
addr_linux_test.go		xfrm_linux_test.go
link_linux.go			xfrm_policy_linux.go
nl_linux.go			xfrm_policy_linux_test.go
nl_linux_test.go		xfrm_state_linux.go
route_linux.go			xfrm_state_linux_test.go
route_linux_test.go
XXX-M-Q0DJ:nl XXX$
Here is my go environment:
XXX-M-Q0DJ:netplugin XXX$ go env
GOARCH="386"
GOBIN=""
GOCHAR="8"
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/XXX/Desktop/Private/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_386"
CC="clang"
GOGCCFLAGS="-fPIC -m32 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"
XXX-M-Q0DJ:netplugin XXX$
i use go installer on my mac, so the environment should all come from what it go on my system.
I guess it should be some cross platform compiling issues, not sure CGO_ENABLED='1' triggered this.
thanks, i will use ubuntu to try it out.

We now support building netplugin on Mac by building binaries inside the VM.
Some of us are actively using this mode.
Please reopen if you still see this issue.


Candles, smoke from smelters, enchanting altar fps

Describe the bug
Looking at candles, the smoke coming off forges and smelters, enchanting tables causes a lot of fps loss. From solid 60 to 30s on a 3060ti. I assume its something to do with particles?
To Reproduce
Stare at the pretty lights lol
Expected behavior
A small loss would be fine but not 30+ fps lol
Screenshots
If applicable, add screenshots to help explain your problem. I highly recommend opening the console and clicking on the object that has a bug. If your bug is location based, take a screenshot of You are Here MCM Option from DynDOLOD as well as a screenshot of the world map.
Modlist version
1.2.1
Did you add any additional mods?
https://www.nexusmods.com/skyrimspecialedition/mods/50682

Fixed in 1.2.2


Add config to disable kube-proxy

There isn't a way to disable kube-proxy, as doucmented in the Talos Cilium documentation.
https://www.talos.dev/v1.6/kubernetes-guides/network/deploying-cilium/#machine-config-preparation
cluster:
  network:
    cni:
      name: custom
      urls:
        - https://server.yourdomain.tld/some/path/cilium.yaml
  proxy:
    disabled: true

You can use patches like this:
patches:
  - |-
     cluster:
       proxy:
         disabled: true

@budimanjojo @szechyjs this seems to be resolved. I'll close it as fixed for now. Feel free to reopen it if there are any further issues on the topic


version increment for release

Issue #, if available:
Description of changes: prepare to publish a new version with autoscaler and nginx add-on fixes.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/do-e2e-tests


Only students can submit: Why?

I think it would be useful for staff to submit, in order to test in the live system whether the grading is all correctly set up.
Would it make the system more complex if we allow staff to submit?

I think it would be really messy if staff could submit assessments. These submissions by staffs would clutter up and confuse the grading work space.
Could you specify exactly which part of grading staff might need to check?
I personally like this rule i think it keeps the data clean. but if really need be we could give allow admins to submit. Or a better way would just have test student accounts for this purpose.

Ideally, testing of missions should not be done on prod but on staging instead. But yes if you really think this would be helpful it definitely can be done! We gotta watch out on the backend, cause currently it allows us to mutate assessments before they are open on the premise that there are no submissions tied to them. This will change when and if we allow graders to submit before the open time.

Ideally, testing of missions should not be done on prod but on staging instead. But yes if you really think this would be helpful it definitely can be done! We gotta watch out on the backend, cause currently it allows us to mutate assessments before they are open on the premise that there are no submissions tied to them. This will change when and if we allow graders to submit before the open time.

Two things:
(1) We don't have a staging environment, at the moment.
(2) Even with a staging environment, the same issue arises: How do the authors test the whole workflow? Without staff submission, we would need to have dedicated accounts for mock students in the staging environment, the authors would need to log in using those accounts. All of that seems rather complex.
Overall, allowing staff to submit might be the most pragmatic option for now.

Okay I see where you're coming from prof! This feature is definitely possible just have to be a little careful on the backend! Just note that its really not best practice to test on prod.

We can plan to create one and do it write! Shouldn't be too much work once we get our deployment right and automated.
We will create student and staff accounts for this! I would argue that this would actually be closer to prod than staff submitting on prod cause you really get to test as a student.

I will explore how to send all testcases if the requester is staff, this way, we can run all testcases on the frontend to test the missions.

Okay I see where you're coming from prof! This feature is definitely possible just have to be a little careful on the backend! Just note that its really not best practice to test on prod.

We can plan to create one and do it write! Shouldn't be too much work once we get our deployment right and automated.
We will create student and staff accounts for this! I would argue that this would actually be closer to prod than staff submitting on prod cause you really get to test as a student.

I agree that we should do this properly. If we have a staging system, with proper test accounts, we will catch the use case that I have in mind.
There is another use case, however: The staff member who wants to upgrade their skills. For example, an Avenger may want to work on a Quest. Not being able to save is unnecessarily restricting the Avenger and in practice might deter them from upgrading their skills and interact with the system. In the light of this, I maintain my view that staff not able to save and submit assessments is an unneccesary restriction.

Conceptually there's nothing wrong with testing on production, provided there are no side effects. A large number of systems are actually designed to that effect.
The reason for this is that testing should be done in layers. The back end, front end and their integration should be tested during development/staging.
Content can and should be tested in production, especially since we are doing content creation on front end. This is a major student gripe actually. Why are the missions not tested? The answer is probably - it takes too long for the missions to be tested.

Conceptually there's nothing wrong with testing on production, provided there are no side effects. A large number of systems are actually designed to that effect.
The reason for this is that testing should be done in layers. The back end, front end and their integration should be tested during development/staging.
Content can and should be tested in production, especially since we are doing content creation on front end (similar to how blogs have post previews). This is a major student gripe actually. Why are the missions not tested? The answer is probably - it takes too long for the missions to be tested.

I agree with @blackening on this. We need to distinguish between the system as a tool for content providers and the system as a product. A good example is the "Mission Control" feature. It is currently a tool for content providers and deployed as such in Production.
If staff could submit, this feature would serve a similar purpose: It would allow mission authors to test missions before they open. I'm ok with using the production system for that.
I also agree that we should make the authoring and testing of missions as easy as possible. This was a severe pain point last year that affected to quality of the module.

This is possible now.


Request for sudo in the postBuild

Hi guys!
There are things I need to do post-Build, such as installing new apt repos, R packages and Ruby gems.  My post-build contains, for example:
add-apt-repository ppa:chris-lea/zeromq
apt-get update
apt-get install -y libzmq3-dev libzmq3
echo 'install.packages(c("devtools", "dplyr","ggplot","gapminder"),repos="http://cran.r-project.org" )' | R --no-save
echo 'devtools::install_github('IRkernel/IRkernel')' | R --no-save
echo 'IRkernel::installspec()' | R --no-save
gem update --no-document --system && gem install --no-document iruby rbczmq pry bio xml-simple gene_ontology
gem install --no-document rdf -v 2.0.2
gem install --no-document rdf-raptor -v 2.0.0
The errors are permissions errors, such as :
Error: must run as root
Installing packages into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning in install.packages(c("devtools", "dplyr", "ggplot", "gapminder"),  :
'lib = "/usr/local/lib/R/site-library"' is not writable
Error in install.packages(c("devtools", "dplyr", "ggplot", "gapminder"),  :
unable to install packages
Execution halted
ERROR:  While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /var/lib/gems/2.3.0 directory.
If I could sudo these commands, I think I'd be racing!

+1 for this, as it seems natural to run postBuild as root. A --debug Dockerfile for r2d when using postBuild shows the last lines as:
...
# We always want containers to run as non-root
USER ${NB_USER}

RUN ./binder/postBuild

@yuvipanda Is there a reason these lines couldn't be swapped so postBuild runs before the user change?

I'm mostly resistant to doing it for the following reason:
It means users would have to use sudo to a non-root user for 99% of tasks (like downloading files, enabling extensions, moving things around, etc). Otherwise you're going to end up with things accidentally owned by root & not usable by the non-root user we run as. It'll also make all the current postBuild files 'wrong'.
IMO, for the use case of installing software that isn't made available with the current apt.txt, users should use a Dockerfile directly. We should support the 80% use case, since the other 20% is going to be very different for everyone.
For R, there's install.R which triggers setting up a specific, non-root writeable library path users can write to. This is new, and is an example of how we should tackle figuring out which is the 80% case!
For installing language packages, finding a non-root location to set them up is almost always the right thing to do, rather than use root. For PPAs, I think we should recommend users use a Dockerfile for now.
I hope this makes sense?

@yuvipanda maybe worth adding this to the list of "when to use a Dockerfile" points somewhere?

If a Dockerfile is the answer then it may be worth reconsidering the "no you can't" guidance on using r2d to bootstrap a Docker file. The 80/20 rule may happen within a single project, in which case it may make sense to use r2d locally to build a base image within the r2d scope, then use a Dockerfile to build resources that require root on top of that layer. Images built locally with r2d don't always work on mybinder.org though.

How about adding/collecting the uses where we have said "for now use a Dockerfile" in the binder-examples repo? Or for higher efficiency: encourage people who have been using their own Dockerfile to add an example.
The problem with Dockerfiles is that they let you do too many things in too many ways for us to efficiently support them. You could have a base image you want to add to to make it work on Binder, you could want to start from something that works on Binder and layer on top, or you could want to do your own thing all the way. These are already three legit use-cases that are all completely different :-/
One way to tame this potential chaos is to have some good examples that are so nice to use that people use those instead of doing other things.
Not sure I understand the 80/20 scope thing. There is nothing stopping you from using your own FROM in a Dockerfile on Binder. So you could create one with repo2docker and then inherit from it. I'd be -1 for supporting multiple different FROMs in repo2docker, too much maintenance  effort.

I meant the 80 is everything that can be done as a non sudo user from a postBuild script, versus the 20 that must be done with root. The decision so far is that custom installation or configuration requiring root must be done with a Dockerfile.
My use case for needing root was having to run the normal install instruction inside an x11 virtual frame buffer to install the R package cairoDevice, which otherwise fails when it tries to open a graphic device to test the package install. Fortunately I got around the issue by installing r-cran-cairodevice using apt.txt. So don't have a use case for it anymore.

@brooksambrose great you could find a solution that works! IMO what you did is the 'right' way to do things, and providing root (with sudo or otherwise) in postBuild would lead to a similar sprawl problem as Dockerfiles - except we'll actually have to support it! I'm keenly aware that the team maintaining this is small, so we should keep the amount of advanced use cases we support small too. I'll posit that doing anything with X in a container counts as advanced...
@choldgraf That's a good idea!

@yuvipanda So the story so far seems to be that postBuild should give you all the customization options you need without root. Dockerfiles are all the customization you would need, period. So using Dockerfiles is a strong signal that this user is in the 20% that is beyond the support agreement. So this issue could be closed if that's the decision.
Here's a thought: imagine a composable solution from a single repo, appendDocker, that would marry the convenience of r2d with the flexibility of Dockerfiles. It would be a composable file that simply lets you keep writing in the Dockerfile after the final line of the r2d build packs are finished. Yeah YMMV, but you'd actually probably be in a stronger position than if you tried to write the whole Dockerfile from scratch.
FWIW my experience has been that 99% of my project is achievable through the available build packs, but there's just one last little piece that needs root. Then there's that realization that I might need to start from scratch with a Dockerfile after having made an investment in the build packs which are still super useful even if you do need to go rogue.
The current guidance on Dockerfiles with binder or r2d basically asks people to recreate some of the patterns already established in r2d. Building on top of r2d is the right approach to Dockerfiles imho, rather than writing them from scratch. My current complicated strategy, which isn't very obvious, is to to use r2d locally to build and push a base image composed from one repo, then have an additional repo or branch with a Dockerfile using the first image in FROM.
Dockerfiles are already supported but sort of in a noncommittal way. An "appendix to the composed Dockerfile" option might actually make the process of using them less error prone. It would work best if people knew to inspect the base Dockerfile using the r2d debug flag. I've come to regard r2d is a stand alone product anyway, with binder an added benefit, rather than r2d as simply binder under the hood.
Just a thought! Can't thank you enough for all the work the team is doing; it's already changed the way I do both research and teaching.

You are suggesting that appendDocker contains lines to append to the end of the Dockerfile that r2d generates?
I think we would like to maintain the freedom to change the contents and structure of the docker instructions r2d uses to build images "at any moment". I am not sure we can maintain that and have happy users who are appending instructions to them. It just feels like we'd either have to limit the amount of changing we can do or would end up breaking people's appended lines.
In the cases where I need to use r2d's escape hatch most of the work is in understanding the generated instructions from r2d. Given I have to do all that thinking it doesn't feel like a big burden to take the output of --debug and transfer it to a Dockerfile. This also means that if you don't understand the instructions r2d generates you will have a bad time appending lines to them.
For me the spirit of the "no you can't" advice is that if you look at the output of --debug and copy it into a file. If you do that though you are breaking the warranty seal on a radio or CD player. It might be a good idea, but you can't complain if all your favourite tunes sound funny afterwards.
Overall I think the right way forward is to collect uses cases that currently aren't supported "natively" in r2d and once there are a few people who want one particular thing discuss adding it.

In that spirit, the OP wanted:

ruby gems
R packages
custom apt repos

Ruby gems aren't something we have in r2d yet, but you can install them as user.
R packages are supported via install.R
Custom apt repos are the only thing we can't do right now. So far there have not been that many people who needed it (less than five).

I think we have resolved this issue since we don't need executable anymore right?

Closing this. If we want to discuss a appendToDockerfile build pack lets create a new issue.


MNT: Re-render the feedstock [ci skip]

Hi! This is the friendly conda-forge-admin automated user.
I've re-rendered this feedstock with the latest version of conda-smithy (2.1.0) and noticed some changes.
If the changes look good, then please go ahead and merge this PR.
If you have any questions about the changes though, please feel free to ping the 'conda-forge/core' team (using the @ notation in a comment).
Remember, for any changes to the recipe you would normally need to increment the version or the build number of the package.
Since this is an infrastructural change, we don't actually need/want a new version to be uploaded to anaconda.org/conda-forge, so the version and build/number are left unchanged and the CI has been skipped.
Thanks!

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

This suffers from the same OSX issue as in #9 and #19.

@danielballan @jakirkham @183amir
The failing OSX builds are now obstructing the Py3.6 distibution of ffmpeg. I cannot debug anything on OSX, so I hope someone has time at some point to repair the OSX build.
In the meantime, we might skip the OSX build. If I am correct, the already existing packages for OSX will then not be removed, while Py36 packages will be added for linux and windows systems. Is that an idea?

TBH this is a really old version of FFmpeg. I added this version as defaults had something similar and this is what I could build on my machine at the time. Really we should try updating to a newer version and see if it builds ok. That should be pretty easy to test if someone wants to try it.


Unable to open overview dashboard

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
The Dasboard remains on "Fetchin",
All the others work correctly, (even OverView - LFP)
Expected Behavior
Show the Overview
Steps To Reproduce

Tableau de bord
Overview

Relevant log output
2024-03-26 14:06:04.565417+01:00logger=context userId=2 orgId=1 uname=boced66 t=2024-03-26T13:06:04.564236177Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=<IP_ADDRESS> time_ms=2 duration=2.204443ms size=0 referer= handler=/api/live/ws status_source=server

Screenshots

Additional data
No response
Type of installation
Docker
Version
1.28.5

the log is from grafana,
I have no other log at the time of clicking
Click at 9:05:46 :
Teslamate log :
2024-03-27 09:03:21.383333+01:002024-03-27 09:03:21.381 [info] GET /
2024-03-27 09:03:21.385004+01:002024-03-27 09:03:21.384 [info] Sent 200 in 2ms

Grafana log :
2024-03-27 09:05:30.725465+01:00logger=provisioning.dashboard t=2024-03-27T08:05:30.725348651Z level=info msg="finished to provision dashboards"
2024-03-27 09:05:30.867124+01:00logger=plugins.update.checker t=2024-03-27T08:05:30.867002718Z level=info msg="Update check succeeded" duration=462.777187ms
2024-03-27 09:05:30.921424+01:00logger=grafana.update.checker t=2024-03-27T08:05:30.92129967Z level=info msg="Update check succeeded" duration=518.911182ms
2024-03-27 09:05:31.334015+01:00logger=grafana-apiserver t=2024-03-27T08:05:31.333893296Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
2024-03-27 09:05:31.344427+01:00logger=grafana-apiserver t=2024-03-27T08:05:31.344313587Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
2024-03-27 09:05:46.069386+01:00logger=context userId=2 orgId=1 uname=boced66 t=2024-03-27T08:05:46.069270373Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=<IP_ADDRESS> time_ms=1 duration=1.833038ms size=0 referer= handler=/api/live/ws status_source=server
2024-03-27 09:06:49.419032+01:00logger=infra.usagestats t=2024-03-27T08:06:49.418525181Z level=info msg="Usage stats are ready to report"

I restarted grafana just before the test

Well, it seems to be a rights issue.
I have two users, an admin and a viewer.
I've tested with the admin and it works.
If I change my "viewer" user to "editor", it works.

The "viewer" would have to have access to the view too, since all the other views work.
Thanks for your time,


ci: run all golangci-lint linters because pre-commit only runs fast linters

on modified files.
See https://pre-commit.com/hooks.html
cc @greg-szabo

If we expect the full linting to pass, shouldn't we request that in pre-commit too?

Short answer: I don't know. As of now, pre-commit only runs fast linters (slow linters are skipped)
golangci-lint - Fast linters runner for Go. Note that only modified files are linted, so linters like 'unused' that need to scan all files won't work as expected.
golangci-lint-full - Fast linters runner for Go. Runs on all files in the repo. Use this hook if you use pre-commit in CI.

I.e. it's not enough to run golangci-lint for all files, we also want to run both fast & slow linters

Do we expect the full linting to uncover issues that the user caused?

Yes

Is the user required to fix linting errors uncovered by the CI, even if it wasn't detected locally?

Yes

Shall we request a full linting in pre-commit? (The pre-commit command has an --all-files option and we can also set the full golangci-lint parameters in the .pre-commit-config.yml.)

I would say no because it would be slow (this is my hunch, but let's verify) and break the developer process. I.e. if he/she commits often, it will be annoying to wait for full lint to finish every time.

I.e. if he/she commits often, it will be annoying to wait for full lint to finish every time.

1

Running linting locally shouldn't take too much time, at least in my experience.
I'm also a bit perplexed with the type of linters and their configuration:
I need to find where slow/fast linters are configured. The golangci-lint config only makes a slight difference between the two ways of running: using --new-from-rev HEAD  or not. The configuration should come from our repository, which is preconfigured for a specific set of linters.
I would really like it if the developer had a way to run these checks locally without parsing what's in the GitHub Actions, but maybe that's only my preference. The only case where this becomes problematic is security vulnerability fixes that are created on a private branch. Those branches can't run CI for obvious reasons and the only linting (if it ever happens) is on the developer machine.

Running linting locally shouldn't take too much time, at least in my experience.

I just checked it and it doesn't add too much difference. Changed pre-commit job to golangci-lint-full.


Bump to Snowplow.Tracker.PlatformExtensions.Dotnet to 4.8

The last supported NET Framework release with MSMQ is 4.8. We can bump to this and then leave this as is for the forseeable.

Closing as we will leave the extension at 461 for compatibility.


Consuming cpprestsdk

Created a static library project for cpprestsdk source code and included necessary header and cpp files, preprocessor defs, etc.
Added required references in the other projects that will look at cpprestsdk project.
Manually tested consuming cpprestsdk httpclient, json and others in the project and the requests work as expected using the library.

Microsoft Reviewers: Open in CodeFlow

Let's modify the AppInstallerCLI.vcxproj, AppInstallerCLITests.vcxproj and other vcxprojs when you start to actually using the cpprestsdk. Since I think most of the changes are unnecessary. For example, I don't think AppInstallerCLi.vcxproj needs to have those additional includes.

wingetutil.vcxproj probably does not need to be changed, I don't see a usage for cpprestsdk in wingetutil.

Refers to: src/WinGetUtil/WinGetUtil.vcxproj:1 in 9eb7179. [](commit_id = 9eb71799c4eef9ec48982ed221f01300729c9826, deletion_comment = False)


Unable to Add Username

Support guidelines

[X] I've read the support guidelines

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description
When I try to add a new username, I get the error message "An error occurred. Please try again later" in the bottom right corner.
So I cannot add a new username.
Expected behaviour
When I click on "Add Username" and then type a new username and again Add Username, a new username is added to the list of usernames.
This used to work in the past.
Actual behaviour
The following error appears when trying to add a new Username:

Looking at the container logs, the following line appears:
<IP_ADDRESS> - - [27/Jun/2024:18:14:53 +0200] "POST /api/v1/usernames HTTP/1.1" 401 41 "https://mydomain.com/usernames" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"

Steps to reproduce

Login to the web app
Click Usernames on the left bar
Click on Add Username button on the top right
Type a new Username
Click on the Add Username button
The error message appears in a red box on the bottom right

Docker info
Client: Docker Engine - Community
 Version:    27.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 8
  Running: 6
  Paused: 0
  Stopped: 2
 Images: 6
 Server Version: 27.0.1
 Storage Driver: fuse-overlayfs
 Logging Driver: journald
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 6.8.0-31-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.863GiB
 Name: e4wardtest01
 ID: a9b388d8-6451-4b1f-873e-714d96ed3a4f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: true

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Docker Compose config
No response
Logs
<IP_ADDRESS> - - [27/Jun/2024:18:14:53 +0200] "POST /api/v1/usernames HTTP/1.1" 401 41 "https://mydomain.com/usernames" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"

Additional info
Not sure if this is related to this container image or to the anonaddy app itself. Difficult for me to tell from the logs...

401 response means unauthorized.
See https://github.com/anonaddy/docker/issues/163 and update your SANCTUM_STATEFUL_DOMAINS=

Looking at your reply, this reminded me of #191
So I realised there was a typo in the value of my APP_URL= variable. Setting it to the correct webapp URL fixed the issue.
Many thanks for the prompt reply!!!! :-D


Weird letter spacing for 0.43.0 vs 0.42.0 | Answer: Use monospace font families

Is it just my installation or is anyone else seeing this? My font family and font size are the same between both versions.
0.43.0

0.42.2

Same issue here. Every time I reopen Extraterm, the font looks like the first screenshot. Also, on mine, there's a weird, thick border around the frame, which didn't use to be there before. Using it on Win10 x64. I'll grab a screenshot later at home (at work, I'm still on 0.42, exactly for those 2 reasons).

@do-kevin  The problem is that in a terminal you are meant to use a monospace font because all of the rows and columns of characters are supposed to line up neatly. You're using a proportional font where the width of the character varies. The bug is in the 0.42.2 screenshot. Extraterm used to assume that all characters had the same width. Since the change in how the text is drawn, that assumption is fixed.
So basically, no bug, but you should choose a monospace font, like Consolas or Dejavu sans mono.

@cobrabr  do post a screenshot here so that we check whether it is the same issue.

Same issue here. Every time I reopen Extraterm, the font looks like the first screenshot. Also, on mine, there's a weird, thick border around the frame, which didn't use to be there before. Using it on Win10 x64. I'll grab a screenshot later at home (at work, I'm still on 0.42, exactly for those 2 reasons).

Could you provide a screenshot so we could see that thick border? I think it could actually be the margins.

@do-kevin The problem is that in a terminal you are meant to use a monospace font because all of the rows and columns of characters are supposed to line up neatly. You're using a proportional font where the width of the character varies. The bug is in the 0.42.2 screenshot. Extraterm used to assume that all characters had the same width. Since the change in how the text is drawn, that assumption is fixed.
So basically, no bug, but you should choose a monospace font, like Consolas or Dejavu sans mono.

Okay, I'll test it out. Thanks seward.

I'll post the screenshot later, but in my case, I was already using a monospace font (tested with Oxygen Mono and Fira Code), and I had the same issue.

Here is the 0.42.2 screenshot (which is working fine):

Here is the screenshot for 0.43:

Notice the font and the grey border around the frame. The configuration is the same as the one from the screenshot above.

@cobrabr Thanks for that. I've seen this before on macOS but no where else until now. The background colour of the terminal itself seems to change for no reason. I'll try poking at it on Windows to see if I can find a way of reproducing this problem on demand. Then I have a chance of figuring it out.

@cobrabr  Looking at your screenshots, it appears that there is some tweaking to do with respect to the character spacing. It looks a bit too wide and big on the 0.43.

Thanks for the quick reply, @sedwards2009, I appreciate it. :)
When I change the font back to Oxygen Mono (which only lasts until I close the application; it reverts back to the font on my screenshot every time I open it), the spacing looks about the same as on 0.42.2 (i.e. it wasn't noticeable/didn't bother me). With the "weird" font, yeah, the spacing is all over the place, but I'm thinking it's the font itself more than Extraterm.

And now that you mention it (and looking at both screenshots), you're right, it's not the border that's the problem, it's the terminal background colour that changes incorrectly.

@sedwards2009, FYI 0.44 still has the same problem. Should I open a new issue for it?

@cobrabr  This issue is fine. I still have to look into that Oxygen Mono problem.

@cobrabr  I was just able to reproduce the totally-wrong-font problem here.
Oxygen Mono looks good when it works.

Thanks again, @sedwards2009. Did you manage to find the reason behind the background colour changing as well?

Fixed in v0.44.1.

@cobrabr  That background color thing is on my TODO list. I've got a bunch of other little things which need attention.

@sedwards2009 Thanks for the heads-up. Do you need me to open a new issue for it?
Also, thanks for fixing this one! 😄

@cobrabr  please do


what are the event handlers for? how should we use them

I have a working notebook based on your sample, so thanks for getting me started.
https://github.com/Mike-Honey/powerbi-export-all-visuals
But I don't understand what the "event handlers" are for? How should we use them?  Is there any documentation or examples of this?
All I've figured out so far is that they seem essential - when I removed them from my code, the report object stopped responding.

Event handlers in Power BI are essential for enabling communication between an embedded component (such as a report, dashboard) and your web application.
When something happens inside the embedded component—like a user interaction or an automated event, such as a visual being rendered—an event is triggered. Event handlers allow you to capture these events and define how your application should respond.
You can set up event handlers to listen for specific actions, like when a report button is clicked (buttonClicked),  a data point is selected (dataSelected), a page is changed (pageChanged), or the report is fully rendered (rendered).
Refer to this documentation for more details and let us know if you have any further queries.
How to handle events in a Power BI embedded analytics application | Microsoft Learn

We are waiting for your response. If no response is received within the next couple of weeks, the issue will be closed.


Remove static image name and put a dinamic

Let's remove static image name and put a unique dinamic font name, created by I + id.

Closed by 7d91f79.


Proposal/Request: Decouple coat roughness from transmission/refraction

Is there a means to decouple specular roughness from affecting transmission/refraction? In the Standard Surface, this could be accomplished with coat roughness which did not affect the transmission roughness. In Open PBR the coat roughness appears to affect the transmission, as does the specular roughness. I would wish that the coat did not affect the transmission so for example one could create scratches on a transparent surface without making the underlying substrate into (expensive to render) frosted glass.

Do you mean that for example you want a thin pane of glass, which appears rough/frosted viewed from one side, but clear/smooth on the other side?
That's a bit problematic for OpenPBR since it would be explicitly unphysical (at least, I can't think of any physical way of achieving that). But even with OpenPBR it could be done I assume with some shader hacks, i.e. making the roughness depend on the view direction, which I think might be a better approach than baking such a non-physical effect into the model.

If you model the grime as the fuzz layer, then it would behave something like what you describe, as currently we assume the fuzz does not roughen the transmission. I think that could make sense physically, since e.g. if the fuzz is just absorbing the incident light without scattering, it would not affect the transmission roughness.
For the coat, we do assume it explicitly roughens the transmission (if the coat is rough), since physically it must as the rough microfacet surface will roughen both the reflection and transmission.  In the case of a real pane of glass, I think it must be the case also that if it is scratched up to the point that it looks frosted from the outside, the transmission (seen from the inside) must appear frosted too. So if one wants to avoid that, some hack is needed like just turning off the roughness if the camera is inside.

Ah yes, that makes sense. Thanks very much for explaining that workflow. Always good for us artists to learn how to do things in a physically correct way, especially with a shader with PBR in its name!


renderToolbar not loading any views

Hi, I have tried to use the renderToolbar method to design my collapsible but only the image in imageSource is rendered, regardless of what I do. If I remove the imageSource prop then my app breaks, telling me "Either an imageSource or custom component is required".
How can I avoid the imageSource and design my toolbar using the renderToolbar method itself?

renderToolbar should return a function. If both renderToolbar & imageSource is given as props, renderToolbar takes priority. So it should work without any problems. Just use the renderToolbar instead of imageSource.


Travis tweaks v2

Description:
use glide install
update vendored go-zfs
drop go 1.5 from travis and add tip.
fix a go vet error when running under tip
Issues affected:
tests for #233

This change is

Reviewed 4 of 4 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks broke.

.travis.yml, line 31 [r1] (raw file):

  - mkdir -p $ZFS_CACHE
  - .travis/deps.sh fetch
  - glide install

Does travis come with glide preinstalled or does this need to be added to deps.sh?

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks broke.

.travis.yml, line 31 [r1] (raw file):
Previously, TauZero (Brian Bickerton) wrote…
> Does travis come with glide preinstalled or does this need to be added to deps.sh?

get out of here with you rational comments.

Comments from Reviewable

Review status: 2 of 5 files reviewed at latest revision, 2 unresolved discussions.

.travis.yml, line 31 [r1] (raw file):
Previously, mmlb (Manuel Mendez) wrote…
> get out of here with you rational comments.

Done.

glide.yaml, line 6 [r2] (raw file):

  version: ^0.10.0
- package: github.com/coreos/etcd
  version: ~2.1.1

fyi this just means take any patch level updates (which we should see none of). I changed it so that it matches what we had back in mistify.

Comments from Reviewable

Reviewed 3 of 3 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks broke.

Comments from Reviewable

Reviewed 1 of 1 files at r3.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Problem with embedded IFrames with different origin

While highlighting I'm getting error:
Uncaught SecurityError: Failed to read the 'contentDocument' property from 'HTMLIFrameElement': Blocked a frame with origin "http://www.polskieradio.pl" from accessing a frame with origin "http://platform.twitter.com". Protocols, domains, and ports must match.
It's being thrown at https://github.com/julmot/jmHighlight/blob/master/src/jquery.jmHighlight.js#L295
IFrames at some point should be excluded from highlighting in them (at least the ones the are not in the same domain CORS-like).
Steps to reproduce

Open http://www.polskieradio.pl/7/1691/Artykul/1392115
Open dev console
Load jmHighlight version mentioned below
Run in console $('body').jmHighlight('Debata polityczna Jedynki');

Environment

Google Chrome 48.0.2564.116
jmHighlight 4.0.1
jQuery v1.12.0

Hi @KrzysztofMadejski,
Thanks for submitting. Unfortunately I am unable to reproduce this issue in a small kept environment. However, I would like to debug this issue. Would you mind to readjust the attachment to show this issue?
highlight-test.zip
Thank you in advance.

Will do on Saturday!

Thank you. I will close this issue in the meantime.

@KrzysztofMadejski Did you forget it?

Ok, I've nailed it down:
jQuery's ready function is getting called before contents of iframe are loaded.
I've changed your test example to jQuery 1.12.1 (or 1.12.0, both doesn't work) and:
<script>
		$(document).ready(function(){
			setTimeout(function() {
			$('body').jmHighlight('Lubię', {
				"debug": true
			});
			}, 2000);
		});
	</script>

and that causes the exception ->
highlight-test.1.12.1.zip
Though on jQuery 2.2.1 (the one you've attached to the test example) it works ok.
Do you think it should be reported to jQuery as a bug?

@KrzysztofMadejski Well, this isn't a bug of jQuery. Referring to this article it is a wanted behavior:

Code included inside $( document ).ready() will only run once the page Document Object Model (DOM) is ready for JavaScript code to execute. Code included inside $( window ).load(function() { ... }) will run once the entire page (images or iframes), not just the DOM, is ready.

The question I ask myself is, do users (like you) really want to highlight things inside an iframe?! For example in a Twitter iframe like in your example, you wouldn't expect anything to get highlighted, right? Also, if the plugin would set the highlight class to an element inside an iframe, then this iframe would also need a CSS that contains this class, not just the parent container.
If you really have a use case for this I think I will implement iframe support as enhancement rather than as bug fix. The plugin will wait for the .load() event to get fired before highlighting things in it.
In the meantime you can use a filter array filter: ["iframe"] or call jmHighlight within a $(window).load().

Speaking about bug to report to jQuery 1.12 I've meant getting exception when calling $('iframe').contents() when entire page is loaded.
As to use case: I do not want to highlight anything in an iframe. The thing is because of jmHighlight code throwing an exception (with jQuery 1.12), nothing gets highlighted, even in the main page. So if you support jQuery 1.12 then this is a bug imho.
BTW: I think due to the browser security you can't get into iframe, so such enhancement won't work. I believe that jQuery 2.2.1 just skips over iframes when contents() get called, that's why it doesn't throw an error.

BTW: How can I write tests targeting jQuery 1.12? Some extending karma.config? Bower would need to be updated as well I believe.

@KrzysztofMadejski

Speaking about bug to report to jQuery 1.12 I've meant getting exception when calling $('iframe').contents() when entire page is loaded.

This is because the iframe is external (CORs like). When having an iframe pointing to an internal file no error will be thrown. Example:
highlight-test.zip

I do not want to highlight anything in an iframe.

Ok, so you also can't imagine a use case for also highlighting inside iframes?

I think due to the browser security you can't get into iframe, so such enhancement won't work.

You can also access iframes from different origins. Having for e.g. a browser extension like this one or when developing a web extension (browser plugin) where you can define permission to a specific host or when configuring your browser to disable the SOP. So this is not generally an exclusion criterion. But of course, there would need three option values for an option iframes:

false (Default)
true
true but without CORs like iframes

How can I write tests targeting jQuery 1.12? Some extending karma.config? Bower would need to be updated as well I believe.

This is an interesting point. With bower you can load multiple versions of jQuery, so that shouldn't be the problem. But the test would need to run against two versions of jQuery. So I think it would be awkward and maybe impossible to replace a jQuery version in runtime and  jQuery.noConflict() seems to be overhead. Calling the test runner (karma) with two versions of jQuery seems to make more sense. So it would be necessary to make karma.conf.js the common config submission (with empty files array). Then defining new karma tasks in the Gruntfile.js for multiple jQuery versions containing the relevant jQuery file reference inside files array.

Ok, so you also can't imagine a use case for also highlighting inside iframes?

Well, actually I see an use case (if it's possible). I'm developing factchecker browser plugin which highlights politicians statements on web pages. Some of the statements are taken from social media, like twitter: ie. text "Taki skrajny populizm na Węgrzech sprawił" on http://demagog.org.pl/politycy/ryszard-petru/. While there's no problem to highlight this phrase on twitter itself, such enhancement would allow to highlight embedded tweets. It's nice-to-have for me.

Ok, so as I mentioned parsing iframes from a browser extension can be permitted. So theoretically it is possible. But, you would have to either

use iframe's seamless attribute (just a joke :bowtie: ~ compatibility)
insert the style of your highlight class into the iframe to actually highlight a matched term
or access the iframe after highlighting yourself, find all highlighted matches by class ($("span.highlight")) and highlight them with JS

or do you have any other idea?

@KrzysztofMadejski Regardless the way a match is acutally highlighted with CSS inside an iframe, I will implement an option to allow highlighting inside iframes and to prevent it. As the .load() event is asynchronous it will be API breaking (currently everything is synchronous). So I have added this to the next major release milestone, due at the end of the month.

@KrzysztofMadejski I'm happy to announce that the option iframes has been implemented. It was a hard job to get this to work, as supporting iframes meant to refactore the plugin to work asynchronous. The challenge here was to observe when an iframe is ready for DOM modifications, as stated on SO.
You now have the opportunity to enable iframes or disable them. I've waived the option "CORS" as it seemed to be unnecessary after I've been thinking hard about this. Usually you know when you are developing a browser extension that you have to set permissions to access different origins (e.g. <all_urls> inside permissions of a web extension). And if you are implementing a webpage it's known that you can't access different origins. So from a developer point of view you aren't expecting anything to happen when not having the necessary permissions. Therefore it seems to be ok to silently skip iframes where you don't have access to it. And in case you really want to mark things in internal iframes but not in external where you also have permissions, you can ignore those iframes by passing a filter array.
I have added several test cases to make sure this functionality works as expected. However, please give it a try and let me know if it worked in your real application.
:wave: :blush:

Wohoa! Amazing job, sure I'll try it! :-)
On 26 March 2016 at 12:55, Julian Motz<EMAIL_ADDRESS>wrote:

@KrzysztofMadejski https://github.com/KrzysztofMadejski I'm happy to
announce that the option iframes has been implemented. It was a hard job
to get this to work, as supporting iframes meant to refactore the plugin to
work asynchronous. The challenge here was to observe when an iframe is
ready for DOM modifications, as stated on SO
http://stackoverflow.com/a/36155560/3894981.
You now have the opportunity to enable iframes or disable them. I've
waived the option "CORS" as it seemed to be unnecessary after I've been
thinking hard about this. Usually you know when you are developing a
browser extension that you have to set permissions to access different
origins (e.g. <all_urls> inside permissions of a web extension). And if
you are implementing a webpage it's known that you can't access different
origins. So from a developer point of view you aren't expecting anything to
happen when not having the necessary permissions. Therefore it seems to be
ok to silently skip iframes where you don't have access to it. And in case
you really want to mark things in internal iframes but not in external
where you also have permissions, you can ignore those iframes by passing a
filter array.
I have added several test cases to make sure this functionality works as
expected. However, please give it a try and let me know if it worked in
your real application.
[image: :wave:] [image: :blush:]
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/julmot/jquery.mark/issues/16#issuecomment-201785001

--
Krzysztof Madejski
<EMAIL_ADDRESS>for Poland Program Manager / Koordynator
Koduj Dla Polski
<EMAIL_ADDRESS>/ +48 664 082 823

http://epf.org.pl/*

@KrzysztofMadejski How was the result?

I had to freeze the development for some time, but probably will get back to it soon.


View all tunes

Since I've done the Repertoire tab, which should cover all tunes. I think I'm doing to call this done.


Run CQs in parallel

Feature Request
System info: Influx 1.2.4, Linux 4.1.19_1, 4x E5-2660 v4 @ 2.00GHz (56 cores in total), 256GB RAM, 2TB SSD RAID 10 (8 Disks)
Current behavior: We run a 1.6M cardinality InfluxDB  with ~500 CQs. We cannot add any further CQs as they're processed sequentially and overall runtime must be below the lowest CQ interval (of 5m).
We already tried to use kapacitor batch tasks to offload expensive CQs. But enabling kapacitor with two subscriptions introduces an 3x increase in memory consumption and 2x increase in CPU without having any enabled kapacitor task.
Desired behavior: Introduce parallelization. I've seen the pro and cons regarding parallelism https://github.com/influxdata/influxdb/issues/8005#issuecomment-298076222. I think on the long term there is no way around it as this will become a limiting factor for larger setups.
To avoid the introduction of a dependency tree I could imagine to introduce parallelization only for CQs operating on a single DB as a first implementation.

Have you tried using a cron-triggered script that performs the query?
i.e something like:
influx -execute 'SELECT * INTO dest FROM src WHERE time > ... AND time < ...'

Not sure if this more performant than CQ. I had to use this solution to handle some cases in a previous setup.

Thanks @bbczeuz for the hint! Wrote a script triggered by cron that runs CQ-like queries as proposed by you. The load is much lower compared to using kapacitor.
Unfortunately absolute time must be used. Using relative time (e.g. WHERE time > now() AND time < now - 10m) would be much easier to implement, but leads to wrong results (as the cronjobs are always slightly delayed and aren't able to run exactly at the desired time).

I'd like to bump this feature request a bit.
We're running a system with many CQs. Ideally we would like to have all of our data aggregated every 5 minutes, but since the sum of all CQs run in series takes over 10 minutes... we're backing up, and it's leading to a poor customer experience.
The queries are largely looking at completely separate measurements, so we're not facing a condition where locking should be a big issue.