[Proposal] Add ability to issue GraphQL queries that will transform to Rapid

I went through the documentation and got some ideas.
The main chance to get more coverage and love of the developers is to simplify some of the client-side work.
My thoughts

Selection and expansions are very hard to write and unnatural.

Batching in 2020 should be first-class citizen of every query transport.
OData has very robust specification for batching however it is not as easy to use from frontend

Developers can do $query with POST which is nice, but when target is to keep things simple it is challenging to get served multiple ways to work with the api.

This cannot be done in spec. There should be some implementation like Apache Olingo that will implement or address those concerns (unless spec will address this by using
Idea
As we have GraphQL based DSL maybe we could also explore adding a layer of graphql queries. Wrapping OData (and OpenAPI in general) is a very common use case with dozens of libraries for the client.
https://github.com/microsoftgraph/graphql-demo/
Even Microsoft tried something like this at some point.
With additional translation layer to URLs we will get a number of benefits:

Capabilities will be easy to write thanks to various GraphQL tools
Batching, Expansion, arguments, filtering available in format that is easy to read and work with.
Multiple features will be hidden in the translation layer so people will not need to explore them - they will just follow the GraphQL Document spec
Full compatibility with OData always guaranteed thanks to transforming documents to rest queries.
It can be delivered as command line tooling (Node.js) that will perform number of transformations for people.

Maybe not now, but next logical step to specifying a model in RSDL is to making queries in RSDL and expressing payloads in RSDL.  This feels like just another negotiated representation choice.

This has implications. What we have now is Dev time model that is not being used at runtime. If we decide that this would be wort considering it will be easier to keep RSDL at runtime - at least for reference implementation

I think we can:

Use GraphQL-Mesh - Developers can get their ODataAPI
I have been working closely with Urigo so if there will be need for any changes we can commit them directly to the repository. Developers can use this as proxy if they want.

Work on the abstraction for filtering (the biggest change is to define what filters are available.
For that we need to start with RDSL ability to define querability - similar to the ones we used in GraphQL based solution

If developers do not want to use this as proxy we will provide GraphQLQuery to URL translation client that can be used as simple client here (however we going to loose cache and graphql compatible responses. What I can think of is if we actually do it for some specific graphql client we can format Odata Responses to look like they came from the GraphQL server (that will be matter of removing some of the metadata and adding operation header. This library will be best to be done as separate repository.

CC @mikepizzo @ralfhandl

Note that, for (1), we have added a tutorial (#301) for using GraphQL with RAPID via GraphQL Mesh.  The current GraphQL Mesh ODataHandler passes through things like $filter and $top, but that may not be the way that we want to pass through such query refinements, in which case we would want to propose modifying this to address (2).
That leaves (3), defining a tool for converting GraphQL requests to OData URLs (and responses to GraphQL)


Prometheus fetching data from exporter even http_sd end point is DOWN

What did you do?
Run a prometheus server with http_sd targets.
Prometheus fetching the targets from http API and collecting the metrics using exporter for those metrics.
Now make down the http_sd end point API.
But, still prometheus fetches the metrics from the old targets.
What did you expect to see?
Prometheus should stop the metric collection since the http_sd endpoint is DOWN
What did you see instead? Under which circumstances?
Prometheus still collecting the metric values for the old targets
System information
Linux
Prometheus version
2.40.3

Prometheus configuration file
# my global config
global:
  scrape_interval: 900s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'snmp_exporter'
    #scheme: https
    http_sd_configs:
    - url: https://localhost/webnms/api/v1/me/targets
      tls_config:
        insecure_skip_verify: true
    metrics_path: /snmp
    params:
      module: [pm_mib]
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: _meta_snmp_exporter_port
      replacement: "9116"
    - source_labels: [_meta_ems_name,_meta_snmp_exporter_port]
      separator: ":"
      target_label: host
    - target_label: __address__
      source_labels: [host]
    - action: labeldrop
      regex: "_meta_.+"
    - action: labeldrop
      regex: "host"

Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response

Hello BrivinTech,
Thank you for reaching out to us on this issue. In such cases, Prometheus by design, tries to retain the old targets as it would be difficult to handle if all the targets disappear at once when SD fails. So Prometheus keeps continuing to fetch metrics from the last known targets when the http_sd endpoint is down.
One possible solution is to update your http_sd to respond with an empty target list to Prometheus and delay the closing of the http_sd endpoint as long as prometheus has not fetched that empty list. This way, you can ensure that Prometheus does not continue to fetch metrics from an outdated source.


Why the epoch is 25?

Is it a bigger epoch would be better ?

Might be slightly better. But in most of the cases, it has converged after 25 epoch.


server.bat - java not found (although it exists)

When starting the orientdb server via console, in the script "server.bat" there is a line:
if exist "%JAVA_HOME%\bin\java.exe" goto setJavaHome
the issue is that when the variable %JAVA_HOME% contains itself double-quotes, the above line doesn't function as expected, because it is interpreted like:
if exist ""C:\Program Files\Java\jre7"\bin\java.exe" goto setJavaHome
notice the double double-quotes inside the already double-quoted string.
this makes the full path incorrect and java is not found, although it exists right there.
so, this line in the script "server.bat" should actually be:
if exist "%JAVA_HOME:"=%\bin\java.exe" goto setJavaHome
notice the :"= added inside the variable.
this removes any double quotes from the variable value and the script begins to work correctly.

same issue for another line below:
set JAVA="%JAVA_HOME%\bin\java"
in this case, the possible double-quotes have to be removed again, like this:
set JAVA="%JAVA_HOME:"=%\bin\java"
again, notice the :"= added inside the variable.

Please update the "server.bat" script ASAP.
Also, please search for other occurrences of this bug in other .bat scripts in the installation.
Thank you.

@asherchayyim WDYT about providing a Pull Request with such changes? This is the best way to become a contributor. Please send it against "develop" or "2.0.4-hotfix" branch.

@G4br13l3 can you please test this?

With this alternative syntax the server.bat script works even with paths already containing double quotes.
@asherchayyim thanks for your contribution.


[Suggestion] Emojicord

The mod Emojicord adds well emojis to minecraft chat. However the discord chat seems them as their base commands, ie a 😄 in minecraft will appear as  :smile : in discord. And the emoji in discord shows up as a square [] in minecraft.
Unless I completely missed something in the config (100% possible) would something like this be possible? Would be really neat.
Mod: https://www.curseforge.com/minecraft/mc-mods/emojicord

This feature should already be implemented for 1.14 and 1.15. Won't backport it to 1.12 and, as there is no emojicord for 1.16, i can't port it to 1.16 right now. If you are on 1.15 and it does not work, please comment and I re-open this

I am so sorry, I linked the wrong mod all together. The mod is Emojiful NOT emojicord. My bad. I am on 1.16.3 currently. They are so close name wise..
https://www.curseforge.com/minecraft/mc-mods/emojiful

Oh, great to know there is an up-to-date alternative. Adding to to-do list

I see what is happening here. This also counts as bug, as I was already converting emotes into thier text form (  :smile:  -> :smile: ). But it seems that JDA does not count most default emoji as an Emoji object inside of the message. It rather uses unicode symbols which get displayed as [] ingame.
This is what I could produce with an library which tries to convert the unicode emotes back into text:

Going to close this as it now is halfway implemented in the latest 1.16 beta release and I can't really go any step further unless I manually replace every single emote with the text version of it


Updated the view of Directory Structure.

Improved the directory structure so that the user will understand the directories and files inside them.

Thanks for the pull request! However, this repository is outdated and no longer the canonical source. I've updated the README.md to explain where the main projects now live.


Testing that drafts do not appears in discord

Task linked: CU-86bx9x8bb Batching (OpenGL Optimizations)


Consider google/shaderc for shader compilation.

Compiling libshaderc google/shaderc might work for better integration.

Oh, and about dynamic size stack-allocated arrays: can't you just have a max-sized [i32; 10] that you can allocate on the stack and used that? It doesn't matter how big it is since it's allocated on the stack.

Resolved by https://github.com/vulkano-rs/vulkano/pull/947


feat: Update Quorum struct, handle EventElected Quorum, and trigger DKGInitiate,HandleAcks,GenerateKeySet event

This pull request includes the following updates:

Update Quorum struct: The Quorum struct is modified to include IP address and Raptor ports. This enhancement allows for improved network communication and connectivity.

Handle EventElected Quorum at Broadcast Engine: The Broadcast Engine is now equipped to handle the EventElected Quorum. This ensures that when a quorum is elected, the peer list is promptly updated with the latest information.

Trigger DKGInitiate event: Once the peer list is updated, the DKGInitiate event is triggered. This event initiates the Distributed Key Generation process, enabling secure and efficient cryptographic operations.

Trigger HandleAcks events once sufficient acks for commitments are received from peers
( No of acks=QuorumSize^2),then triggers Generate KeySet

Marked this as draft for the time being until we resolve other networking PRs.

Marking this as draft to prevent accidental merges as I want to review it in detail early next week.

Closing this PR as a great deal of the changes within the node crate are invalid as per https://github.com/vrrb-io/vrrb/pull/402.
@vsawant10949 please create another PR containing only the changes to the Quorum and the Event struct so the aforementioned PR can use those.


Typo in Package Promotion Command Feedback:  "version-umber" instead of "version-number"

Note
Before you submit your issue, make sure that:

You're using the latest version of Salesforce CLI.
You've searched both open and closed issues for related posts.
You've used the doctor command to diagnose common issues.
You understand that GitHub Issues don't adhere to any agreement or SLA.

If you require immediate assistance, use official channels such as Salesforce Customer Support.

Summary

Seems to be typo in promotion command feedback when attempting to promote version that already has a promoted package
Steps To Reproduce

IMPORTANT
Provide a repository that's configured to reproduce the issue. If you are unable to provide a repo, please explain why not. The more info we have from the start, the faster we can resolve your issue.
We may close your issue if you don't include proper instructions.

Generate a project with sf project generate or fork dreamhouse-lwc.
Provide detailed step-by-step instructions on how to reproduce the issue.

create a package version 1.1.0, promote it.
create a new package version off of that same convention, promote it.
Expected result

Here's the results of the SFDX CLI Promote Command: {   "code": 1,   "actions": [     "Create a new package version with a different --version-number, then promote the package version.\nsf package:version:create -p  -n  -k \nsf package:version:promote -p 05ixxx"   ],   "context": "PackageVersionPromoteCommand",   "commandName": "PackageVersionPromoteCommand",   "message": "You already promoted a package version with this major.minor.patch version number. For a given major.minor.patch number, you can promote only one version.",   "name": "DUPLICATE_VALUE",   "status": 1,   "stack": "DUPLICATE_VALUE: You already promoted a package version with this major.minor.patch version number. For a given major.minor.patch number, you can promote only one version.\n
Actual result

Here's the results of the SFDX CLI Promote Command: {   "code": 1,   "actions": [     "Create a new package version with a different --version-umber, then promote the package version.\nsf package:version:create -p  -n  -k \nsf package:version:promote -p 05ixxx"   ],   "context": "PackageVersionPromoteCommand",   "commandName": "PackageVersionPromoteCommand",   "message": "You already promoted a package version with this major.minor.patch version number. For a given major.minor.patch number, you can promote only one version.",   "name": "DUPLICATE_VALUE",   "status": 1,   "stack": "DUPLICATE_VALUE: You already promoted a package version with this major.minor.patch version number. For a given major.minor.patch number, you can promote only one version.\n
System Information

@salesforce/cli/2.1.7 linux-x64 node-v18.17.0
{
  "cliVersion"<EMAIL_ADDRESS>  "architecture": "linux-x64",
  "nodeVersion": "node-v18.17.0",
  "osVersion": "Linux 5.15.0-1041-azure",
  "shell": "bash",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.2 (core)",
    "@oclif/plugin-commands 2.2.18 (core)",
    "@oclif/plugin-help 5.2.13 (core)",
    "@oclif/plugin-not-found 2.3.[31](https://github.com/department-of-veterans-affairs/dtc-release-hub/actions/runs/5679964641/job/15393204221#step:4:31) (core)",
    "@oclif/plugin-plugins 3.1.6 (core)",
    "@oclif/plugin-search 0.0.20 (core)",
    "@oclif/plugin-update 3.1.26 (core)",
    "@oclif/plugin-version 1.3.7 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.43 (core)",
    "@oclif/plugin-which 2.2.25 (core)",
    "@salesforce/cli 2.1.7 (core)",
    "apex 2.3.6 (core)",
    "auth 2.8.9 (core)",
    "data 2.5.1 (core)",
    "deploy-retrieve 1.16.0 (core)",
    "info 2.6.31 (core)",
    "limits 2.3.26 (core)",
    "login 1.2.20 (core)",
    "org 2.9.25 (core)",
    "schema 2.3.20 (core)",
    "settings 1.4.20 (core)",
    "sobject 0.1.37 (core)",
    "source 2.10.25 (core)",
    "telemetry 2.2.3 (core)",
    "templates 55.5.5 (core)",
    "trust 2.4.[32](https://github.com/department-of-veterans-affairs/dtc-release-hub/actions/runs/5679964641/job/15393204221#step:4:32) (core)",
    "user 2.3.24 (core)"
  ]
}

Additional information

Thanks for reporting this @jdschleicher! I've got a fix in and it will be release in latest in the coming weeks.


Break down RiskProfile calculation

Create a feature branch for work to help with high prevalence situations + add to CI
Breaks down RiskProfile calculation in individual's actions, # of housemate's, # of other traceable contacts; standardizes how RiskProfile is calculated
Adds hooks for adding "symptom free" features

That this change makes no user-facing changes (except adding a new parameter in the url) - verified by calculate.test.ts

Also looks like there are some merge conflicts to resolve.


cardano-cli: drop --tracing-off & improve error printing

cardano-cli:

Drop --tracing-off -- it was implemented with a hard-to-fix bug.
Fix error printing -- it was over-quoting.

bors r+


An error occurred (RequestLimitExceeded) when calling the CreateTags operation (reached max retries: 4): Request limit exceeded.

running:

name: ebs-copy-instance-tags
resource: ebs
filters:

type: value
key: "Attachments[0].Device"
value: not-null
actions:
type: copy-instance-tags
tags:

Name

configuration:
ran via c7n-org across 50 accounts
handled exception:
2017-11-20 17:56:40,770: custodian.actions:ERROR Exception renaming tag set
An error occurred (RequestLimitExceeded) when calling the CreateTags operation (reached max retries: 4): Request limit exceeded.

Looks like your account is hitting an aws limit.
I would recommend extending the filter to only apply when the ebs name is null.

looks like some backoff retry was added to this action (and its intelligent about not copying if the values are current), but in general if account is under stress, we won't continue indefinitely in the face of continuous throttle errors.


Include restriction in name query results in CLI

Summary
The name service includes the attribute restricted on each NameRecord.  When querying for a name from the name services using the CLI, this restricted attribute is not returned.
Command:
provenanced query name lookup <address name(s) bound to>
Current Result:
name:
- pb
- io
- pio
- provenance
pagination:
next_key: null
total: "0"
Problem Definition
The NameRecord.restricted boolean controls if this name can be extended and used as a parent name by any account or if it can just be extended using the name owner's account.  As a Provenance SDK client, when creating a name it is important that I choose a parent/root name that that I am allowed to extend.  Because the CLI does not provide the restricted attribute in the query results, it is only by trial and error creating and executing a transaction with a MsgBindNameRequest with desired root names as parent names until I did not receive a message similar to this:
UNKNOWN: failed to execute message; message index: 0: parent name is restricted and does not resolve to the provided parent address: invalid request: invalid request
Proposal
When using the CLI to query a name from the name service ("query name lookup"), include "restricted: " on each name returned.

For Admin Use

[x] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

might be good if the provenanced query name resolve <name> would also return this... or if that might cause problems for something downstream for some reason, maybe a separate query endpoint for just getting the details of a name (address/restricted status/whatever else is available)

Would also be helpful if the events emitted during creation would include this flag as well


GPS transactional location fetch

Resolves #18

@raddishiow can you please reviews this so I can merge and release?
Sorry for nagging...

Yeah of course, sorry - I'll take a look ASAP.

@HarelM When you're ready, I think we're all good to merge and make a 1.1.0 release?


feat: Scanner improvements

This PR contains several improvements for the scanner:

Move the decoding part in an Isolate
Improve the MLKit algorithm by limiting the types of supported barcodes
"Adaptive algorithm" to start the decoding (cf below for more details)
Ask for a camera lower resolution (high -> medium)

"Adaptive algorithm" :

The duration of each decoding is now stored.
An average value of the last 10 decodings is computed.
The average time (what I call a window) is then multiplied by 10 (this value is arbitrary, we can change it)
This sum is then used a the time between two decodings

What's still missing: pause the barcode when scrolling between the carousel.

Codecov Report

Merging #1781 (3974c1a) into develop (2ea0da3) will decrease coverage by 0.89%.
The diff coverage is 0.56%.

@@            Coverage Diff             @@
##           develop   #1781      +/-   ##
==========================================
- Coverage     8.86%   7.96%   -0.90%
==========================================
  Files          161     171      +10
  Lines         6623    7495     +872
==========================================
+ Hits           587     597      +10
- Misses        6036    6898     +862

Impacted Files
Coverage Δ

...h_app/lib/cards/category_cards/abstract_cache.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/asset_cache_helper.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/raster_async_asset.dart
0.00% <0.00%> (ø)

...oth_app/lib/cards/category_cards/raster_cache.dart
0.00% <0.00%> (ø)

..._app/lib/cards/category_cards/svg_async_asset.dart
0.00% <0.00%> (ø)

...smooth_app/lib/cards/category_cards/svg_cache.dart
0.00% <0.00%> (ø)

...t_cards/knowledge_panels/knowledge_panel_card.dart
0.00% <0.00%> (ø)

...t_cards/knowledge_panels/knowledge_panel_page.dart
0.00% <0.00%> (ø)

...knowledge_panels/knowledge_panel_summary_card.dart
0.00% <ø> (ø)

...s/knowledge_panels/knowledge_panel_table_card.dart
0.00% <0.00%> (ø)

... and 65 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b0410c...3974c1a. Read the comment docs.

Hi @M123-dev & @monsieurtanuki, I have implemented a big part of the improvements for the scanner/camera.
Everything is not yet finished, but I would appreciate if you could send me your feedback on the current implementation.
Thanks ;)

After discussing with @teolemon, we will measure if these improvements are enough.
And if necessary the start/resume detection will be implemented.

Some merge conflicts @g123k

It should now be OK :)

@g123k the analyzer goes wild, I guess a file has change or something like that


Perfomance issues with big list ( array with ~250 items )

Modal selector is opened after big delay, on iPhone5s delay is longer then on iPhoneX for e.g. But anyway, even on IphoneX it's taking too long.

This is also reported in issue #45 , and as I mentioned back then, the ScrollView may be replaced by FlatList. We are actively merging PR's, do you have the time to test if FlatList would fix the performance problem, and eventually make PR? It is of course fine if you got other suggestions than FlatList :-)

Any finding on this? I facing the same where the modal is too slow to pop up, don't have time to do testing with code changes though. If anyone experienced with this and came up with solution do share. Thanks a lot, I still prefer this modal selector.


testing new pr

testing things

merge this please

merge this please

retest this please

merge this please

merge this please


Resolve called twice when state url needs to be encoded

I've seen issue #37 which was resolved, but the problem still occurs if the state url needs to be encoded. Here's how it works:
If we have a state like this:
.state('state', {
        url: '/state/?path',
        templateUrl: 'template.html',
        resolve: {
          test: function ($stateParams) {
            console.log("Resolve called. Path = " + $stateParams.path);
          }
        }
      })

and have two links:
Test 1
Test 2

Then for the first link resolve will be called once with path equal testPath, but for the second link resolve function will be called once for /testPath/ path and once for %2FtestPath%2F.

Im going to call this a duplicate of the #1973 (might not be the right one). Once the double encoding problem is fixed then if this isn't it can be reopened

@eddiemonge
I still experience this issue with ui-router#0.2.18 if I use slug in urls:
.config(['$stateProvider', '$urlRouterProvider', '$urlMatcherFactoryProvider',
        function($stateProvider, $urlRouterProvider, $urlMatcherFactoryProvider) {
                $urlRouterProvider.otherwise('/');
                $urlMatcherFactoryProvider.type('slug', {
                        encode: function(str) { return str && str.replace(/ /g, '-'); },
                        decode: function(str) { return str && str.replace(/-/g, ''); },
                        is: angular.isString,
                        pattern: /[^/]+/
                });
        }
])

Is here any workaround?


remove coverage, docs, test directories in npm package

Good catch. We're working on top of https://github.com/gaearon/redux/pull/195 right now so I excluded coverage there. I want to keep the docs though.


Added the ability to copy selected text to HTML.

I bound a new function copy_html_action() to the Ctrl+Super+C hotkey. This triggers the new get_selected_text() method inside the DocumentView class, which returns selection and runs it through the new  html process_selection function.
This implementation returns the minimum HTML result, meaning no enclosing html tags or embedded styles. Eventually I may write a clause that checks if the selection == the entire buffer, and if so, switch to the original process function that returns the enclosing html tags and embedded styles.

Nice work!
A few suggestions

[ ] since the first 8 lines of process and process_selection are the same, can you re-factor that to a new function? or make one use the other (renaming the process_selection)
[ ] can you remove stdout.printf from get_selected_text
[ ] is there a reason to keep the reference to Gdk.Display? it could just be called in the call to Clipboard.get_for_display

Thanks for the suggestions! This is the very first thing I've done in Vala, so bear with me here. I'll make the changes in the morning :)

@voldyman do you mind taking a look at the code now? I made the suggested changes, if you notice anything else please let me know :blush:

Nice!
just one small thing, the process function, instead of adding a branch(if) i meant that you could create a function A that is called by both process and process_for_selection.
for a first time vala developer, you are doing good work!

I've been giving this some thought, and what about changing the process function to return the minimum html result with the given string, and then adding a process_page function, that return the fully process HTML content in page format (with embedded styles, enclosing tags, etc). That way the process_page function could call the process function to process the markdown content.
In my mind this seems like the most logical approach to this.

this latest commit implements the function structure that I mentioned above.

Great work! i was going to merge this but the feature didn't work for me. It works when i apply this diff though, can you confirm?
diff --git a/src/Widgets/Window.vala b/src/Widgets/Window.vala
index cbff796..17b7b1e 100644
--- a/src/Widgets/Window.vala
+++ b/src/Widgets/Window.vala
@@ -315,8 +315,8 @@ public class Window : Gtk.ApplicationWindow {
                                               Gdk.ModifierType.SUPER_MASK);

         switch (ev.keyval) {
-        case Gdk.Key.c:
-            if (ctrl_pressed && shift_pressed) {
+        case Gdk.Key.C:
+            if (ctrl_pressed) {
                 handled_event = true;
                 copy_html_action ();
             }

Yes, that was because I ended up switching the modifier to SUPER instead of shift and didn't change the variable name. But I actually prefer the shift modifier anyhow, so that diff will work

Great work! :+1:


Algorithm

In which file i can check the algorithm of Machine learning?

which algo is used to train the model


[RFR] Strip restangular methods from entries

Angular adds its own method to every entity fetched. As they are not namespaced, this has side effects.
I removed them, but I'm not sure we don't use them. @manuquentin, any insight?
Fixes #273

Restangular methods aren't used on the object himself, we can remove them

As a matter of fact, my patch doesn't solve #273, because properties named like Restangular methods (like "head") are added then removed, so there is no way to read them. We need a way to prevent Restangular from decorating them in the first place.

We can't prevent Restangular from decorating, but we can get the original entity by adding a listener (see https://github.com/mgonto/restangular/issues/100#issuecomment-19174352). Back to WIP.

I've fixed the bug, but there are two drawbacks:

This breaks all tests using Restangular (long and cumbersome to fix)
This slows down the app, as we have to clone all entries in order

So this needs more work.


Fix make starter.

I don't know exactly why this worked but I just copied the line from starter_dev in between the steps and it installs so :shrug:


https://raw.githubusercontent.com not working

It looks like https://raw.githubusercontent.com is not available anymore or temporaly unvailable.
It gives
curl -sL https://raw.githubusercontent.com/home-assistant/hassio-build/master/install/hassio_install.sh | bash -s -- -m raspberrypi3    bash: line 1: 404:: command not found

mabye an outage on github


Revert recent schema removals

This reverts the schema removals we did to get back to the state where all schema definition is in codepropertygraph. We will bring this change back in once it no longer causes trouble down the line.

revert of the revert: https://github.com/ShiftLeftSecurity/codepropertygraph/pull/1125


Review IR controls with 3PAO

Review this control set with Veris, using an appointment booked via the slots on this calendar.
Acceptance Criteria

[x] Any notes on changes needed in the text is recorded as "docs" in our followups doc.
[x] Any notes on practices we're missing are recorded as "practices" in our followups doc.

@LinuxBozo Reviewed with Veris yesterday, comments in docs.


Discussion: reopen log files without hot restart envoy

Title: Reopen log files without hot restart envoy
Description:
I'm trying to use logrotate + hot restarting envoy to dealing with increasing envoy access logs.
And I have read https://github.com/envoyproxy/envoy/issues/1109 .
My concern is that frequent hot restarting affect performance (we use envoy as API gateway), and wonder if envoy can implement a more lightweight approach (like what nginx do, receive a signal and just reopen log files).

This is already implemented. Send SIGUSR1 IIRC.

Thanks! My missing when reading docs.


/content/* url undercases id's

Issue Summary
A content ID like 1e8bCHOJlGrr53HPLli4ApF-KMqdBuJtu (especially if it looks like a slug) is converted to all lowercase once the request is processed. This is problematic because image plugins that are case-sensitive read the request / path data and are unable to remove the content from external providers. See #36
before:

after:

Steps to Reproduce

Install my ghost-google-drive adapter (my repo does this easily)
Upload an image after configuration
Observe that it is immediately unable to resolve the image in preview (id sent from ghost is lower-cased)

Ghost Version
4.41.3
Node.js Version
16.x
How did you install Ghost?
from dist
Database type
MySQL 5.7
Browser & OS version
edge, chrome latest
Relevant log / error output
[2022-04-11 15:37:58] INFO "GET /content/images/size/w2000/1awWwRzaPOmoAweUE1e_T-rKeNc66fJpV.jpg" 302 109ms
1E6VSyQh-U1hJFY38wMcuPY0-7dYYSD4M
[2022-04-11 15:37:58] WARN {{#get}} helper took 1107ms to complete

{{#get}} helper took 1107ms to complete

Error ID:
    5d57aaf0-b9ad-11ec-b5b8-35a24307f111

Error Code:
    SLOW_GET_HELPER

Details:
    api:          canary.postsPublic.browse
    apiOptions:
      limit:   3
      include: authors
      context:
        member: null
    returnedRows: 3

[2022-04-11 15:37:58] INFO "GET /content/images/1lxvhkbytjygdymfuavqepltuj-jzuaz6.jpeg/" 404 2853ms
1lxvhkbytjygdymfuavqepltuj-jzuaz6
[2022-04-11 15:37:58] WARN {{#get}} helper took 842ms to complete

{{#get}} helper took 842ms to complete

Error ID:
    5d6825b0-b9ad-11ec-b5b8-35a24307f111

Error Code:
    SLOW_GET_HELPER

Details:
    api:          canary.postsPublic.browse
    apiOptions:
      limit:   3
      include: authors
      context:
        member: null
    returnedRows: 3

[2022-04-11 15:37:58] WARN {{#get}} helper took 656ms to complete

{{#get}} helper took 656ms to complete

Error ID:
    5d6e4030-b9ad-11ec-b5b8-35a24307f111

Error Code:
    SLOW_GET_HELPER

Details:
    api:          canary.postsPublic.browse
    apiOptions:
      limit:   3
      include: authors
      context:
        member: null
    returnedRows: 3

[2022-04-11 15:37:58] INFO "GET /members/api/site/" 200 373ms
[2022-04-11 15:37:59] WARN {{#get}} helper took 680ms to complete

{{#get}} helper took 680ms to complete

Error ID:
    5d7481c0-b9ad-11ec-b5b8-35a24307f111

Error Code:
    SLOW_GET_HELPER

Details:
    api:          canary.postsPublic.browse
    apiOptions:
      limit:   3
      include: authors
      context:
        member: null
    returnedRows: 3

[2022-04-11 15:37:59] INFO "GET /content/images/1s_yizxraixwwhe5ltydmmmqokhx3pggn.png/" 404 3075ms
[2022-04-11 15:37:59] INFO "GET /content/images/1awwwrzapomoaweue1e_t-rkenc66fjpv.jpg/" 404 3315ms
[2022-04-11 15:37:59] INFO "GET /content/images/1e8bchojlgrr53hplli4apf-kmqdbujtu.jpg/" 404 3046ms
1awWwRzaPOmoAweUE1e_T-rKeNc66fJpV
[2022-04-11 15:37:59] INFO "GET /content/images/size/w2000/1zD16OIPC_l22qsMSt5DouGudHwICmK0N.jpg" 302 7ms
[2022-04-11 15:37:59] INFO "GET /content/images/size/w2000/1e8bCHOJlGrr53HPLli4ApF-KMqdBuJtu.jpg" 302 11ms
1zD16OIPC_l22qsMSt5DouGudHwICmK0N
1e8bCHOJlGrr53HPLli4ApF-KMqdBuJtu
1lxvhkbytjygdymfuavqepltuj-jzuaz6  not found
[2022-04-11 15:37:59] INFO "GET /content/images/1LxvHkByTjygDYMFuAvQEPltuJ-jzuaZ6.jpeg/" 301 1153ms
1lxvhkbytjygdymfuavqepltuj-jzuaz6
[2022-04-11 15:37:59] INFO "GET /content/images/1E6VSyQh-U1hJFY38wMcuPY0-7dYYSD4M.jpg/" 301 1017ms
[2022-04-11 15:37:59] INFO "GET /content/images/1S_YizXrAiXWWHe5LtyDmmMQokhx3PGGN.png/" 301 1223ms
1e6vsyqh-u1hjfy38wmcupy0-7dyysd4m
1s_yizxraixwwhe5ltydmmmqokhx3pggn
[2022-04-11 15:37:59] INFO "GET /content/images/1awWwRzaPOmoAweUE1e_T-rKeNc66fJpV.jpg/" 301 971ms
1awwwrzapomoaweue1e_t-rkenc66fjpv
1lxvhkbytjygdymfuavqepltuj-jzuaz6  not found
[2022-04-11 15:38:00] INFO "GET /content/images/1zD16OIPC_l22qsMSt5DouGudHwICmK0N.jpg/" 301 1011ms
[2022-04-11 15:38:00] INFO "GET /content/images/1e8bCHOJlGrr53HPLli4ApF-KMqdBuJtu.jpg/" 301 1029ms
1e6vsyqh-u1hjfy38wmcupy0-7dyysd4m  not found
1s_yizxraixwwhe5ltydmmmqokhx3pggn  not found
1zd16oipc_l22qsmst5dougudhwicmk0n
[2022-04-11 15:38:00] WARN {{#get}} helper took 470ms to complete

{{#get}} helper took 470ms to complete

Error ID:
    5e2e82f0-b9ad-11ec-b5b8-35a24307f111

Error Code:
    SLOW_GET_HELPER

Details:
    api:          canary.postsPublic.browse
    apiOptions:
      limit:   3
      include: authors
      context:
        member: null
    returnedRows: 3

[2022-04-11 15:38:00] INFO "GET /content/images/1lxvhkbytjygdymfuavqepltuj-jzuaz6.jpeg/" 404 1409ms
1awwwrzapomoaweue1e_t-rkenc66fjpv  not found
1zd16oipc_l22qsmst5dougudhwicmk0n  not found

Code of Conduct

[X] I agree to be friendly and polite to people in this repository

Apologies, looks like the problem was on my end.


Delete the dependency swift-tools-support-core

I'm deleting the dependency in favor of using Foundation.Process. I've taken the opportunity to add support for Windows and extend the CI pipeline to ensure the package is compatible with Windows, Ubuntu, and macOS.

I've decided to skip Windows from CI. All the attempts yielded non-debuggable errors like this where the swift test command exited with an error 1 with no information about what caused the abortion:


Import NPC AI and Whisper packages

Relates: #636

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Imports the VR4VET NPC AI package and the Whisper Package it depends on.

What is the current behavior? (You can also link to an open issue here)
One existing NPC prefab with no ability for players to talk using microphone inputs.

What is the new behavior? (if this is a feature change)
NPC and NPC AI prefabs have had their dialogue boxes updated. Adds NPC AI prefab which allows player conversations. Both prefabs should work alongside one another.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
I am not sure which scripts to overwrite and which ones to keep when importing. I suggest someone with experience working on Blue Sector follows the steps show in the wiki below to replicate this import. I believe the issues stem from parallell work on scripts the NPC prefab depends on in both the VR4VET and Blue Sector repositories.

Other information:
I created this PR mostly to show which files have conflicting changes --- you may see there are deletions in for instance the Tablet UI scripts. Again this is a consequence of differences between the two repositories. A lot of the scripts the prefab depends on can be toggled when importing. The NPC AI package contains all the assets it depends on, but most of these are not necessary to import as they already exist in Blue Sector.  For more information, visit the wiki.

NB: Ny versjon av packagene er tilgjengelige under releases i vr4vet.

Closing pull request because the import of NPC AI and Whisper packages is being dealt with in the branch IntegrateNPCAI.


Content downloader tests

:warning: Another giant PR!
This is big, but the vast majority is test data and scaffolding for running the tasks worker. If you're interested in getting into the minutiae of navigating Kolibri, Django and pytest-django, that's great. However, the interesting part is the last commit. The first commit has an internal interface change, and the second commit changes an internal interface return value. The rest is all test stuff including a vast pile of test data.
I'd really like to land this so I can tackle #890 with at least some confidence.
Fixes: #778

The downloader test was a bit flaky locally. After running it repeatedly with extensive logging, it seems that Kolibri's task worker gets hung occasionally. I couldn't figure out why that would happen, so I added pytest-retry to retry the test when it times out in either the foreground or background downloads. The 3rd commit attempts to handle one potential issue I thought of while debugging.

This is looking good to me, and it works on my system, albeit with some sporadic failures which I'm gathering are more related to problems we need to fix in the code being tested. I do have one suggestion with regards to making this a little less huge, but all seems good otherwise :)

I'm still getting the sporadic failures, too. I thought pytest-retry would fix that, but it actually doesn't do anything useful because it causes an error in the teardown of the failed test. I really wanted to get to the bottom of why Kolibri's worker stalls.

This is impressive! Should we revert the retry as you guys have found that is not preventing the sporadic failures? From my side this is good to go and very welcome, even with those failures.

I think so, but let me play with it for a little bit longer. I was using pytest-flaky, but I think I should use pytest-rerunfailures, which has almost the same interface but is maintained by the pytest developers. I think that's more likely to work correctly.

This is impressive! Should we revert the retry as you guys have found that is not preventing the sporadic failures? From my side this is good to go and very welcome, even with those failures.

I think so, but let me play with it for a little bit longer. I was using pytest-flaky, but I think I should use pytest-rerunfailures, which has almost the same interface but is maintained by the pytest developers. I think that's more likely to work correctly.

Yeah, pytest-rerunfailures seems to work correctly. Let me update the PR.

This is impressive! Should we revert the retry as you guys have found that is not preventing the sporadic failures? From my side this is good to go and very welcome, even with those failures.

I think so, but let me play with it for a little bit longer. I was using pytest-flaky, but I think I should use pytest-rerunfailures, which has almost the same interface but is maintained by the pytest developers. I think that's more likely to work correctly.

Yeah, pytest-rerunfailures seems to work correctly. Let me update the PR.

If you're running this locally (@dylanmccall), you should pip uninstall pytest-retry since it provides the same flaky mark and I don't know how pytest will decide which one to use.

Great, the rerun worked this time:
kolibri_explore_plugin/test/test_collectionviews.py::test_download_manager_clean[athlete-0001] RERUN [ 63%]
kolibri_explore_plugin/test/test_collectionviews.py::test_download_manager_clean[athlete-0001] PASSED [ 63%]

I'm going to merge this now.


Add RBI for Gem::Security::Policy

Motivation
As defined by https://docs.ruby-lang.org/en/master/Gem/Security/Policy.html.
Test plan

See included automated tests.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_LdQM6h2UV5GdBv
→ https://go/builds/bui_LdQMN0rBjGQRLk


Code explanation request

This may be a big ask (you could consider me to be --less than-- a junior developer), but can anyone help me understand by explaining the code for the InitReceives() method in the DomainEventSubscriber base class?
It looks like maybe the method is converting the derived class Handle methods into Receive methods of the DomainEventSubscriber's base ReceiveActor class.  I'm guessing its just using a lot of reflection to extract the method info from the Handle method and magic that method info into a Receive method.... but i'm not totally sure.  The last foreach loops is confusing to me (because i don't really yet know anything about reflection), but i'm guessing that is the final conversation and creation step to actually make the final Receive methods ?

Hi @woodpk,
It's my pleasure.
You are correct: InitReceives does use reflection to programmatically call the DomainEventSubscriber's Receive functions. The Receive method is, of course, a function in Akka's ReceiveActor base class. Here, Akkatecture is using the power of convention to make development simpler at design time.

In the InitReceives method, a collection of System.Type instances is gathered. This collection is all of the DomainEvent types to which the actor has a subscription. It gets this list using the GetDomainEventSubscriberSubscriptionTypes extension method (described below).
Next, an instance of MethodInfo is created that represents the generic Receive<T>(Func<T, bool> handler) function of the base class and is stored in method.
Each time through the foreach loop of subscription types, a MethodInfo instance representing an instance of Receive is created, where that method's generic parameters are closed over. In other words, instead of an open generic method such as...

Receive<T>(Func<T, bool> handler);

... the instance would be closed like this:
Receive<MyEvent>(Func<MyEvent, bool> handler);

Finally, the closed method is invoked passing in the subscriber's corresponding methodInfo gathered in step 1.

With all of this code in the library, a developer only needs to extend the DomainEventSubscriber class and add however many ISubscribeTo<,,> interfaces they want. For each subscription interface/type, they would create a Handle method with the related type as the only parameter.
Example here

GetDomainEventSubscriberSubscriptionTypes
A parameterless extension method of the Type class, it...

Filters the interfaces extended by the type in question to those that are implementations of  ISubscribeTo<,,>
Selects the third of the three generic types specified in that ISubscribeTo  implementation: the IDomainEvent<,,>
Returns the complete list of domain event types.


FirebasePerformance start trace dishonest API

[REQUIRED] Step 1: Describe your environment

Xcode version: 12.5 (12E262)
Firebase SDK version: 8.6.0
Installation method: Swift Package Manager
Firebase Component: Performance

[REQUIRED] Step 2: Describe the problem
Steps to reproduce:
I was trying to log some traces for my network code and I found a crash when tracing certain endpoints. The crash report says that logged named cannot be nil. I struggled looking at my own code until I noticed that the problem was of at the FirebasePerformance side. Taking a look at the code I notice that if you call this method with a String/NSString * larger than 100 character it will crash. The problem is that the static method signature for this method is not being honest and will assert here.
Why it's not truncating the String/NSString * or saying that the return type for the FPRReservableName is nonnull NSString * instead of NSString *?

The reason that we did not make the method nonnull is because this return value is nullable and we want to check the return value in other functions.
We use assert to inform that the trace name is invalid for debug build. If you enabled the log message, you should be able to see the reason why the name is invalid, including the trace name length being too long. Note that the assert is disabled in release build so your app will not crash if the trace name is too long.
Closing the issue for now. Please re-open if you have any other questions.


401 on SkypeImageMsg.fileContent

When I try to get the fileContent of a SkypeImageMsg or SkypeFileMsg, I get this response
 [traceback details in my own app]
 File "/usr/local/lib/python3.5/dist-packages/skpy/msg.py", line 498, in fileContent
    auth=SkypeConnection.Auth.Authorize).content
  File "/usr/local/lib/python3.5/dist-packages/skpy/conn.py", line 215, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
 skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/0-eus-d2-9fc51dfcc7c37f83eac51aa51031ceb3/views/imgpsh_fullsize', <Response [401]>)

I'm not sure what to do about this. Is it the wrong authentication method?

Can't seem to reproduce here.  Please set SKPY_DEBUG_HTTP=1 in your environment, try retrieving the file again, and include the response from Skype below.
$ cat a.txt
A

>>> m
SkypeFileMsg(...)
>>> m.file
File(name='a.txt', size='2', ...)
>>> m.fileContent
<= [28/02 10:29:24] GET https://api.asm.skype.com/v1/objects/0-weu-d2-.../views/original
{}
=> [28/02 10:29:25] 200
A
b'A\n'

@dstelzer: Any updates here?

Tried it again but can't see any way to reproduce this...  😞

Here's what I saw with HTTP debugging.
`[snip]
{'clientmessageid': '8611425457298071227',
'composetime': '2017-03-30T22:21:53.953Z',
'content': '<URIObject type="Picture.1" '
'uri="https://api.asm.skype.com/v1/objects/0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4" '
'url_thumbnail="https://api.asm.skype.com/v1/objects/0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4/views/imgt1" '
'ams_id="0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4">You've '
'received a new picture. View it at: <a '
'href="https://login.skype.com/login/sso?go=xmmfallback?pic=0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4">https://login.skype.com/login/sso?go=xmmfallback?pic=0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4<FileSize '
'v="146231" /><meta '
'type="photo" originalName="" />',
'conversationLink'<EMAIL_ADDRESS>'from': 'https://bn2-client-s.gateway.messenger.live.com/v1/users/ME/contacts/8:live:[username_redacted]',
'id': '1490912513963',
'messagetype': 'RichText/UriObject',
'originalarrivaltime': '2017-03-30T22:21:53.953Z',
'type': 'Message',
'version': '1490912513963'}]}
<= [06/04 20:39:00] GET https://api.asm.skype.com/v1/objects/0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4/views/imgpsh_fullsize
{}
=> [06/04 20:39:01] 401`
Strangely enough, waiting a couple minutes and trying again can sometimes avoid this problem.
(Also, successfully downloading an image with HTTP debugging on often does strange things to my terminal. But I don't know if there's a way to avoid that.)

Presumably the fallback link in there works for you?  (I get an expired error but that's probably just masking the account mismatch -- if you get one now, try a fresh file upload first and test the corresponding link.)  If that fails too, it's probably Skype serving you a dud file for some reason.
If you're signed in to Skype for Web, the direct file link should also just work in your browser (it only appears to require cookies for auth).
<= [06/04 20:39:00] GET https://api.asm.skype.com/v1/objects/0-cus-d4-f4633e3c29a1bbd194f4a3a46d800dd4/views/imgpsh_fullsize
{}
=> [06/04 20:39:01] 401

Have you missed a line under that?  Presumably you didn't get the file, but did you get any form of response?
Curiously, following your link, I get redirected to nus1-api.asm.skype.com and the same path, and then get a 403, as opposed to a file of mine that has no such redirect.

(Also, successfully downloading an image with HTTP debugging on often does strange things to my terminal. But I don't know if there's a way to avoid that.)

Indeed, that would be it trying to print binary files to your terminal.  If you just send text files around whilst testing, you should be ok.

hi, I have also same problem.
however this happens on Centos 7 Ubuntu seems to work ok.
Traceback (most recent call last):
File "./andy.py", line 36, in
sk.loop()
File "/usr/lib64/python2.7/site-packages/skpy/main.py", line 218, in loop
self.cycle()
File "/usr/lib64/python2.7/site-packages/skpy/main.py", line 207, in cycle
self.onEvent(event)
File "./andy.py", line 21, in onEvent
print event.msg.fileContent
File "/usr/lib64/python2.7/site-packages/skpy/util.py", line 227, in wrapper
cache[key] = fn(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/skpy/msg.py", line 593, in fileContent
auth=SkypeConnection.Auth.Authorize).content
File "/usr/lib64/python2.7/site-packages/skpy/conn.py", line 217, in call
raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/0-cus-d2-1f8840328850101b4d1631f4552d5653/views/imgpsh_fullsize', <Response [401]>)

There's no differences with SkPy between the two OSes, nor should Skype be able to detect any due to the consistent user agent.  Please follow the previous requests to reproduce the issue with debugging, and to confirm if the fallback link still works.

I'm having the exact same problem.
With SKPY_DEBUG_HTTP set to 1:

Getting the message (its a text file with "test" inside):

>>> m = sk.chats['8:username'].getMsgs()
<= [15/10 21:29:59] GET https://db5-client-s.gateway.messenger.live.com/v1/users/ME/conversations/8:username/messages?syncState=250c000000383a732e6d696b756c69637a01831072875e010000a43ef1215f01000000&startTime=0&view=msnp24Equivalent
{'params': {}}
=> [15/10 21:29:59] 200
{'Cache-Control': 'no-store, must-revalidate, no-cache',
 'Content-Encoding': 'gzip',
 'Content-Length': '634',
 'Content-Type': 'application/json; charset=utf-8',
 'ContextId': 'tcid=8250543446289258611,server=DB5SCH101101028',
 'Date': 'Sun, 15 Oct 2017 21:29:58 GMT',
 'Pragma': 'no-cache',
 'X-Content-Type-Options': 'nosniff'}
{'_metadata': {'lastCompleteSegmentEndTime':<PHONE_NUMBER>127,
               'lastCompleteSegmentStartTime':<PHONE_NUMBER>091,
               'syncState': 'https://db5-client-s.gateway.messenger.live.com/v1/users/ME/conversations/8:username/messages?syncState=250c000000383a732e6d696b756c69637a01831072875e010000578cf1215f01000000&startTime=0&view=msnp24Equivalent'},
 'messages': [{'clientmessageid': '1508102989845',
               'composetime': '2017-10-15T21:29:52.373Z',
               'content': '<URIObject type="File.1" '
                          'uri="https://api.asm.skype.com/v1/objects/0-weu-d3-4bd84805043a69cec674293762cced9d" '
                          'url_thumbnail="https://api.asm.skype.com/v1/objects/0-weu-d3-4bd84805043a69cec674293762cced9d/views/thumbnail"><Title>Title: '
                          'test.txt</Title><Description> Description: '
                          'test.txt</Description><a '
                          'href="https://login.skype.com/login/sso?go=webclient.xmm&docid=0-weu-d3-4bd84805043a69cec674293762cced9d"> '
                          'https://login.skype.com/login/sso?go=webclient.xmm&docid=0-weu-d3-4bd84805043a69cec674293762cced9d</a><OriginalName '
                          'v="test.txt"/><FileSize v="5"/></URIObject>',
               'conversationLink': 'https://db5-client-s.gateway.messenger.live.com/v1/users/ME/conversations/8:username',
               'conversationid': '8:username',
               'from': 'https://db5-client-s.gateway.messenger.live.com/v1/users/ME/contacts/8:username',
               'id': '1508102992377',
               'messagetype': 'RichText/Media_GenericFile',
               'originalarrivaltime': '2017-10-15T21:29:52.373Z',
               'type': 'Message',
               'version': '1508102992377'}]}

Downloading the file:

>>> m[0].fileContent
<= [15/10 21:42:01] GET https://api.asm.skype.com/v1/objects/0-weu-d3-4bd84805043a69cec674293762cced9d/views/original
{}
=> [15/10 21:42:02] 401
{'Content-Length': '0',
 'Date': 'Sun, 15 Oct 2017 21:42:01 GMT',
 'Server': 'Microsoft-IIS/8.5',
 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains',
 'X-Content-Type-Options': 'nosniff',
 'X-Powered-By': 'ARR/2.5, ASP.NET'}

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shyman/.pyenv/versions/skycord/lib/python3.6/site-packages/skpy/util.py", line 227, in wrapper
    cache[key] = fn(*args, **kwargs)
  File "/home/shyman/.pyenv/versions/skycord/lib/python3.6/site-packages/skpy/msg.py", line 568, in fileContent
    auth=SkypeConnection.Auth.Authorize).content
  File "/home/shyman/.pyenv/versions/skycord/lib/python3.6/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/0-weu-d3-4bd84805043a69cec674293762cced9d/views/original', <Response [401]>)

If I open the api.asm.skype.com link in a browser I can download the file no problem. Opening the fallback link in browser shows me "This is your message" and a play button, pressing it does nothing.

Let's reopen this as it's clearly affecting multiple users, despite not being reproducible over here.  🙁

Opening the fallback link in browser shows me "This is your message" and a play button

Assuming that's still test.txt, I have no idea why a play button appears...
@Ashymad, @wilder11: Can you try sending a file through Skype for Web, and then capturing and downloading it from SkPy?  You can run a SkypeEventLoop and react to a SkypeNewMessageEvent where the event's msg property is a SkypeFileMsg, then try retrieving the content like before.

Here is the code I used:
from skpy import *

class MySkype(SkypeEventLoop):
    def onEvent(self, event):
        if event.type == "NewMessage":
            if event.msg.type == "RichText/Media_GenericFile":
                file = event.msg.fileContent

username = "BotAccount"
password = ""

Skype(username,password,".tokens-test")

MySkypeLoop = MySkype(tokenFile=".tokens-test", autoAck=True)
MySkypeLoop.loop()

Then:
export SKPY_DEBUG_HTTP=1
python ./test.py > test.log 2>&1

Then I've sent a text file from my account (through Skype for Web) to the bot.
Here is test.log
Maybe the problem is that I am running the script on a remote server?

Ok, so it looks like it's at least parsing the message correctly, but we're sending the wrong kind of credentials.  It's possible something has changed within Skype for Web, in which case it may be using different authentication now.
If you've still got that message hanging around, can you try using https://web.skype.com to fetch the attachment?  Open developer tools, try to download the file, find the request in the Network tab (on Chrome at least), and get a copy of the request headers and body.

Request headers:
GET /v1/objects/0-weu-d3-50424a9ccc973ea131da4209652afa83/views/original HTTP/1.1
Host: api.asm.skype.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://web.skype.com/en/
Cookie: SC=CC=:CCY=:LC=en:LIM=:RS=d:TM=1508355247:TS=1506340084:TZ=:VAT=:VER=; SWADCC=TS=1506340091:TM=1506340091; SWACC=TS=1506626683:TM=1506626683; SSP=linux; MSFPC=ID=63ce316bcc101742b49adbda8785d49d&CS=1&LV=201709&V=1; skype-cookie-policy=1; MSCC=1506756172; s_pers=%20s_fid%3D5A081F7341145EFA-11333A4437F17D54%7C1570361758933%3B%20gpv_p23%3Dno%2520value%7C1507291558935%3B%20s_nr%3D1507289758936-Repeat%7C1570361758936%3B; skypetoken_asm=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjEzIn0.eyJpYXQiOjE1MDgzNTUyNDUsImV4cCI6MTUwODQ0MTY0Mywic2t5cGVpZCI6ImxpdmU6bjk5ODUxMCIsInNjcCI6OTU4LCJjc2kiOiIxIiwiY2lkIjoiZTNhZDQ0OWExMzhiODViMyIsImFhdCI6MTUwODM1NTA5NX0.cuTcXXQkY2eNi-_SUOynXZGoTD5Rx0Gm_GXGYwpecx4uf5iUJPeLEdKJWMu5ae8wQZz8jVP2b7EZWEcoIhwYigrL54t27RBe6KA6HMyzvpxSKRqDl8aDcZGhhM2kp5mqR61xVf0xXJgqG5fLmsWUu-dN_39QBhG45Vkv1OxhRc_NEOedroMoHZvEsKUZ0aedVFFiWnYR0ITjFgqm; platformid_asm=908; skplet=1508441645
Connection: keep-alive
Upgrade-Insecure-Requests: 1
DNT: 1

Response headers:
HTTP/1.1 200 OK
Cache-Control: no-cache, max-age=0, s-maxage=0, private
Content-Range: bytes 0-4/5
Server: Microsoft-IIS/8.5
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Content-Disposition: attachment; filename=test.txt
X-Powered-By: ARR/2.5, ASP.NET
Date: Wed, 18 Oct 2017 19:34:24 GMT
Content-Length: 5

Response:
test

I've also tried
curl 'https://api.asm.skype.com/v1/objects/0-weu-d3-50424a9ccc973ea131da4209652afa83/views/original' -H 'Host: api.asm.skype.com' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://web.skype.com/en/' -H 'Cookie: SC=CC=:CCY=:LC=en:LIM=:RS=d:TM=1508355247:TS=1506340084:TZ=:VAT=:VER=; SWADCC=TS=1506340091:TM=1506340091; SWACC=TS=1506626683:TM=1506626683; SSP=linux; MSFPC=ID=63ce316bcc101742b49adbda8785d49d&CS=1&LV=201709&V=1; skype-cookie-policy=1; MSCC=1506756172; s_pers=%20s_fid%3D5A081F7341145EFA-11333A4437F17D54%7C1570361758933%3B%20gpv_p23%3Dno%2520value%7C1507291558935%3B%20s_nr%3D1507289758936-Repeat%7C1570361758936%3B; skypetoken_asm=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjEzIn0.eyJpYXQiOjE1MDgzNTUyNDUsImV4cCI6MTUwODQ0MTY0Mywic2t5cGVpZCI6ImxpdmU6bjk5ODUxMCIsInNjcCI6OTU4LCJjc2kiOiIxIiwiY2lkIjoiZTNhZDQ0OWExMzhiODViMyIsImFhdCI6MTUwODM1NTA5NX0.cuTcXXQkY2eNi-_SUOynXZGoTD5Rx0Gm_GXGYwpecx4uf5iUJPeLEdKJWMu5ae8wQZz8jVP2b7EZWEcoIhwYigrL54t27RBe6KA6HMyzvpxSKRqDl8aDcZGhhM2kp5mqR61xVf0xXJgqG5fLmsWUu-dN_39QBhG45Vkv1OxhRc_NEOedroMoHZvEsKUZ0aedVFFiWnYR0ITjFgqm; platformid_asm=908; skplet=1508441645' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'DNT: 1'

with mixed success. At first I didn't get anything then an unknown SSL protocol error, finally it started working.
After I did that, SkPy suddenly was able to download files and images as well...

Ugh.  I wonder if there's another API call that needs to be made first, before the link in the message becomes valid (after which both Skype for Web and SkPy can use it because the account / session is "authorised").

Today SkPy is able to download again. For no reason at all.I was not at home even, but my bot on the server suddenly became able to download them.

I ran into the same problem. I have done a little troubleshooting and it appears there is a similar issue as with login.skype.com failing to redirect to api.asm.skype.com; in this case api.asm.skype.com fails to redirect to the proper "cloud server" resulting in the file not being found and returning a 401.
I did a silly little mod to get around this problem by first trying urlFull as returned from the API, and if that doesn't work I step through a ladder of exceptions for each API-server that I have identified so far, until I find the correct one for my file. If all fail I just leave it. Ugly I know, but it works for now. ;)
From what I have seen so far these are the servers api.asm.skype.com redirects to:
nus1-api.asm.skype.com
weu1-api.asm.skype.com
neu1-api.asm.skype.com
as-api.asm.skype.com
sa1-api.asm.skype.com
eus1-api.asm.skype.com
ea1-api.asm.skype.com

there is a similar issue as with login.skype.com failing to redirect to api.asm.skype.com; in this case api.asm.skype.com

Is that from opening the fallback link in the message in a browser, or from opening a file directly from Skype for Web, or both?

I step through a ladder of exceptions for each API-server that I have identified so far

There's bound to be another way to figure out where a file is.  Not sure how they're allocated -- if you send multiple files, are they consistently on the same file server?  IIRC I've personally only ever seen weu1 and neu1 from my testing.

I'm having same issue. It happens sometimes and I absolutely have no idea why. It doesn't happen in web skype.

Some strange logs:
>>> sk.conn
SkypeConnection(userId='live:blablabla', tokenFile='token', connected=True, guest=False)
>>> url = "https://weu1-api.asm.skype.com/v1/objects/bla/views/original"
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
  File "/media/work/envs/mtbz/lib/python3.5/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('400 response from GET https://weu1-api.asm.skype.com/v1/objects/bla/
views/original', <Response [400]>)
>>>
>>> url = "https://weu1-api.asm.skype.com/v1/objects/bla/views/imgpsh_fullsize"
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
<Response [200]>
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
<Response [200]>
>>>
>>> url = "https://weu1-api.asm.skype.com/v1/objects/bla/views/original"
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
  File "/media/work/envs/mtbz/lib/python3.5/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('400 response from GET https://weu1-api.asm.skype.com/v1/objects/bla/
views/original', <Response [400]>)
>>>

Looks like we should use /views/imgpsh_fullsize instead of /views/original! But...
>>> url
'https://api.asm.skype.com/v1/objects/bla/views/imgpsh_fullsize'
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
  File "/media/work/envs/mtbz/lib/python3.5/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/bla/views
/imgpsh_fullsize', <Response [401]>)
>>>
>>> url = "https://api.asm.skype.com/v1/objects/bla/views/original"
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
  File "/media/work/envs/mtbz/lib/python3.5/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/0-weu-d10-f82deea033cef2fc326c77e42300cc92/views
/original', <Response [401]>)

I've noticed that problematic urls contains name of server. Like this:
urlFull: https://api.asm.skype.com/v1/objects/0-weu-d3-759ce324c4d88734b603e03cbde1333d
urlView: https://login.skype.com/login/sso?go=webclient.xmm&pic=0-weu-d3-759ce324c4d88734b603e03cbde1333d
It contains "-weu-".  I open first one in browser it returns 401 error. When I open second one it returns 200 and redirects to weu1-api.asm.skype.com.
So if I replace api.asm.skype.com with weu1-api.asm.skype.com it loads fine.
Same in python shell:
>>> url = "https://{0}api.asm.skype.com/v1/objects/{1}/views/imgpsh_fullsize".format('', '0-weu-d3-759ce324c4d88734b603e03cbde1333d')
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
  File "/media/work/envs/mtbz/lib/python3.5/site-packages/skpy/conn.py", line 217, in __call__
    raise SkypeApiException("{0} response from {1} {2}".format(resp.status_code, method, url), resp)
skpy.core.SkypeApiException: ('401 response from GET https://api.asm.skype.com/v1/objects/0-weu-d3-759ce324c4d88734b603e03cbde1333d/views/imgpsh_fullsiz
e', <Response [401]>)
>>>
>>>
>>> url = "https://{0}api.asm.skype.com/v1/objects/{1}/views/imgpsh_fullsize".format('weu1-', '0-weu-d3-759ce324c4d88734b603e03cbde1333d')
>>> sk.conn("GET", url, auth=sk.conn.Auth.Authorize)
<Response [200]>
>>>

According to @dforsell there is a lot of servers. Also seems like all image urls contains /0-something-d.

I'm testing your hack, @SergeiMinaev , and it seems to work.

Interesting, I wonder which clients are generating the dud links then.  I suppose Skype for Web rewrites them somewhere, similar to the workaround above.
I'm going to be off the grid for the rest of this week, but I'll take a proper look after that.  Hopefully, if it proves successful, I'll get such a workaround in place.  🙂

Based on @SergeiMinaev's sample above, I've pushed a potential fix to the file-url-fix branch.
For those who have seen issues reading file content from messages, please try this branch and report back on whether it works for you or not.
I'm not particularly happy on using {server-id}1-asm.skype.com as the hostname (with server-id extracted from the object URL) as this feels rather brittle, so I'm keen to hear if there are any server names that don't satisfy this pattern (@dforsell mentioned as-api.asm.skype.com so I'm unsure if/when that will come up).  So far I've only come across weu1 and neu1 during my testing.

I've been testing the fix for the last week or so and everything is working quite nicely.

I've merged it as-is for lack of a better solution.  Hopefully this puts us in a better position than without.

I still see this error in 0.10.6
An error occurred: 401 Client Error: Unauthorized for url: https://api.asm.skype.com/v1/objects/0-sa-d1-86d6de52532838282bd170491a147ffc/views/original

Can you please help if there is a workaround to get rid of this error 401 ?

If you try to download the same file in Skype for Web, what does the URL look like?

This is as seen from the browser
Request URL:
https://gateway.bingviz.microsoftapp.net/receive?app=skype
Request Method:
POST
Status Code:
200 OK

Prior to the above request there was a
Request URL:
https://gateway.bingviz.microsoftapp.net/receive?app=skype
Request Method:
OPTIONS
Status Code:
204 No Content

gateway.bingviz.microsoftapp.net appears to be client analytics/tracking and is unrelated to downloading images.  I'd expect the request you're looking for to look like the one SkPy is trying to make but on a different domain.

I tried another instance of download from the code and the browser(skype web). The URL that is being used seems to be the same in both the case.
code:
An error occurred: 401 Client Error: Unauthorized for url: https://api.asm.skype.com/v1/objects/0-sa-d6-d1754454c51db5164aa91d2078ae8983/views/original
skype web:
Refer attachment.

The relevant request:
curl "https://api.asm.skype.com/v1/objects/0-sa-d6-.../views/original"
  ...
  -H "cookie: ..., skypetoken_asm=eyJhb..."
  ...

Looks like they may have switched to cookie authorisation instead of headers.  The existing header authorisation continues to work for me, but I've experimentally pushed branch asm-cookie which does the cookie authorisation instead (which also works for me so I'm hoping is valid for existing files).
Can you check if that branch works for you?

$ pip install git+https://github.com/Terrance/SkPy.git@asm-cookie
DEPRECATION: Loading egg at c:\users\91821\appdata\local\programs\python\python311\lib\site-packages\alright-2.5-py3.11.egg is deprecated. pip 23.3 will enforce this behaviour change. A possible replacement is to use pip for package installation..
DEPRECATION: Loading egg at c:\users\91821\appdata\local\programs\python\python311\lib\site-packages\platformdirs-3.10.0-py3.11.egg is deprecated. pip 23.3 will enforce this behaviour change. A possible replacement is to use pip for package installation..
Collecting git+https://github.com/Terrance/SkPy.git@asm-cookie
Cloning https://github.com/Terrance/SkPy.git (to revision asm-cookie) to c:\cygwin64\tmp\pip-req-build-l2n06uo2
Running command git clone --filter=blob:none --quiet https://github.com/Terrance/SkPy.git 'C:\cygwin64\tmp\pip-req-build-l2n06uo2'
Running command git checkout -b asm-cookie --track origin/asm-cookie
Branch 'asm-cookie' set up to track remote branch 'asm-cookie' from 'origin'.
Switched to a new branch 'asm-cookie'
Resolved https://github.com/Terrance/SkPy.git to commit 22443921bd1365d8032aba07c8a4c47a25b2708d
Preparing metadata (setup.py) ... done
Requirement already satisfied: beautifulsoup4 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from SkPy==0.0.0) (4.12.2)
Requirement already satisfied: requests in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from SkPy==0.0.0) (2.31.0)
Requirement already satisfied: soupsieve>1.2 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from beautifulsoup4->SkPy==0.0.0) (2.4.1)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from requests->SkPy==0.0.0) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from requests->SkPy==0.0.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from requests->SkPy==0.0.0) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\91821\appdata\local\programs\python\python311\lib\site-packages (from requests->SkPy==0.0.0) (2023.7.22)
I did run pip install command as above and re ran my listener, but no change in behavior. Am I missing anything here ?

I can't tell from that output, as it doesn't shows SkPy itself actually being updated -- I'd expect after the various "Requirement already satisfied" messages something like...
Installing collected packages: SkPy
  Attempting uninstall: SkPy
    Found existing installation: SkPy 0.10.6
    Uninstalling SkPy-0.10.6:
      Successfully uninstalled SkPy-0.10.6
  Running setup.py develop for SkPy
Successfully installed SkPy-0.0.0

I also haven't seen the request made to download your file again this time -- as with the previous reports above, set SKPY_DEBUG_HTTP=1 in your environment to actually log the request/response, and if you're running the updated branch then you should see the request as...
<= [11/10 23:52:52] GET https://xyz1-api.asm.skype.com/v1/objects/0-xyz-000-0000000000000000646c76a75b0e4cc1/views/original
{'cookies': {'skypetoken_asm': '***'}, 'headers': {}}
=> [11/10 23:52:52] 200


connman uses only 3 candidates for nameservers which are chosen non-deterministic

If DHCP gives 2 IPv4, say <IP_ADDRESS> and <IP_ADDRESS> and 2 IPv6 ones then we have  total of 6 DNS nameservers. Only 3 of them will get serialised to /etc/resolv.conf and in no particular order. That means that a device that got the network configuration once and connected successfully isn't guaranteed to do the same in the event of a reboot or a carrier loss.
Also, we specify <IP_ADDRESS> and <IP_ADDRESS> both in /etc/connman/main.conf as FallbackNameservers and in /var/lib/connman/network.config in Nameservers but that doesn't seem to imply anything in terms of priority.

@floion this was implemented by dnsmasq. Please confirm and close this issue. It would be good to mention the issues in the future PRs so we can trace and link them.

This can be closed


Support configuring msg deduplication strategy

From https://github.com/nats-io/jetstream/blob/master/README.md#message-deduplication , it seems to me that jetstream currently uses first_arrived_wins strategy for deduplication. I would like to request an option to configure that:

last_arrived_wins
Msg-Version: Int64 where user can pass an integer that represents version number and the higher version wins.

My use-case is that I am routing Kubernetes resource updates via jetstream and in my case I only need to process the latest version of an object (msg-id).

Personally with a stream I dont think last wins is a good option to have, you have a few options:

Make it a work queue and acknowledged messages will be removed, so you'd only be processing the last
Use a CQRS like model to apply changes to your representations

wdyt @derekcollison

Msg-Version will be helpful since k8s objects have generation field.

If the message is the same and susceptible to deduplication why does it matter, both should be the same no?

In my case, the messages are Kubernetes update events. The message body is about the same object (say, pod or service) but the actual body is the updated version (not same). In Kubernetes world, we only need to reconcile the updated version only.

For them would we just have a jetstream stream of limit of messages of 1, meaning last one in is always the one available?

I am thinking more like a subject k8s.c1.pods where all update events related to pods from a single k8s cluster will be sent to a jetstream . So, I don't think message limit 1 will work.

Depending on how many distinct events you could use a template and have each one be its own stream?

In theory yes. But this feels a bit extreme :) . How many streams can be 2 CPU 4-8GB ram jetstream deployment handle?

Streams should be considered as fairly light weight from a construct point of view. I would not hesitate to have a template create hundreds or thousands of streams.
Easy to test out and play with..

"last arrived wins" makes sense. For example, you have an entity, e.g. order. Instead of emitting an event with order delta, you emit an event with full order snapshot on any change in that order. Thus, you have a stream of order snapshots. If a stream has multiple snapshot events for a certain order, only the last snapshot for that order makes sense.
To get advantage of deduplication with "last arrived wins", you should assign order ID (primary key) to Nats-Msg-Id. Thus, you cover all these requirements for cross-service data synchronization:
1. No data loss on service-consumer downtime. - this is clear - durable consumer
2. Feeding new services with fresh data. - a new empty service only needs the latest snapshot event for each order.
3. Limited stream size due to deduplication instead of retention. - if you enable retention in case of event sourcing, a new empty service can't restore full state due to events loss. If you disable retention - you have eternal event stream and a new empty service has to waste time on replaying old snapshots. "last arrived wins" strategy allows having predictable size of event stream due to deduplication and having durable and optimized data synchronization between services. This is some kind of "hybrid event sourcing" when you emit snapshots instead of deltas.
By the way, Liftbridge implements "last arrived wins" strategy, they call it "log compaction":

Additionally, Liftbridge supports log compaction. Publishers can, optionally, set a key on a message envelope. A stream can be configured to compact by key. In this case, it retains only the last message for each unique key. Messages that do not have a key are always retained.
https://liftbridge.io/docs/concepts.html#stream-retention-and-compaction

I previously wrote up a internal feature request, currently not being implemented though:
**Message Rollups**

It would be great to support rollups of messages, meaning if 100 previous messages is combined
into the last 1 message then the previous 100 can be deleted.

One possible way to do this is using headers, if messages are published to subject `in.x` when a
message with header `JS-Rollup: in.x` is received on subject `in.x` then all previous messages
received on that subject can be removed.  No doubt other ways but this is one possibility

This could be done with templates if you could purge up to a specific message ID but today purge
is all messages.  Also templates mght not scale to a large case for this feature
via discussion with Igor Anic on slack

I vacillate back and forth between making some of these small changes and swinging back to this should be a different stream.
That being said, we have conditional publish now with headers to match last msgId or seq, we could add one that says "iff" you accept this message purge til this point. Not sure that gets us everything we need, but I do support a single stream that can be rolled up and snapshotted effectively.

rollups into a seperate stream would be very hard to use, ideally rolling up would not affect any of the current consumers - they'd just get the latest rollup data.  Would need to support wildcards, and will finally get me my requested purge up to seq feature.
Agree though, we should do this after GA after a period of focussing on higher order use cases which is why I never really passed on this feature request properly

I have done these types of systems in the past, they are useful but also tricky to get really right. We need to think more about consumers who are behind the stream itself and what happens to them when you rollup and purge.

We have dedup now, I still want rollups. But lets close this one for the moment.


Keep getting 'API Rate Limit Exceeded'

Everything appears to have installed correctly, unfortunately I haven't been able to speak to ChatGPT as I keep receiving this error.

Thanks for the report @Yegiyan , I'll be looking into it!

Same issue

still an issue

i gass  your free api token is  expired，have to pay for api token

I also keep having after a fresh installation. Waiting for a minute doesn't help, and I couldn't send a single message with an API token generated minutes ago.

邮件发送失败

This problem is equals to the following json response
{
    "error": {
        "message": "You exceeded your current quota, please check your plan and billing details.",
        "type": "insufficient_quota",
        "param": null,
        "code": "insufficient_quota"
    }
}

a fix might be use custom API endpoint, this might be a new feature

邮件发送失败

In my case the solution was to add some credit balance in billing page.

Hello @everyone:
This issue is related to Account <> OpenAI. TerminalGPT uses OpenAI API and the user key(encrypted) to send request and  return an answer. The error message is been returned from OpenAI, so I consider this a non issue of TerminalGPT.
Since this issue where created from old versions, I suggest everyone to update terminalGPT and make new tests to verify if the error still exists.
Reminder
Before reopen this issue, always try to verify if your openAI account is not blocked or have insufficient funds.
More to come
This is a good issue to write about troubleshooting, if someone wants to contribute, feel free to do so. ATM I'm working on a new version of terminalGPT, in a new repo, I'll be announcing soon!


Slideshow mode for viewing gallery?

Hi! Really nice looking gallery and project :+1: The demo album and everything looks really smooth. I like your focus on self hosting and privacy, that's why I came here.
I stumbled on this project when I was trying to find software for turning my tablet into a digital picture frame while it's not in active use. Having it showing random images from my file server and slowly changing between them. So I can passively enjoy and remember past events in my life. I suppose nothing like this is built in? Would you be willing to merge such functionality if someone contributed it?
Any recommendation for other software that would solve this use case?
Absolutely simplest solution, that is not really useful, but still: Add a timer that just clicks the "next" button in the full screen image view. Trivial, but does not add much flexibility.
The best for me personally is if it could be made both to go through a specific event/tag/geographic location slowly, and if it can be run on the full image archive as well. For the latter it would be great if it could be limited to include/exclude certain tags. And it would be great if the selection of the next image would be semi-random but with some decently smart logic. If it just picks any arbitrary picture from the album there will be no consistency at all, one picture from a hike up a mountain and the next a blurry picture at 2am from a party four years earlier in a different country. That would not be nice. But also going through every picture in chronological order would be boring. Then you would be stuck viewing the same hike forever if you were too trigger happy on the camera during that trip. The middle ground is of course the hardest to implement! Staying at each event* for a decent amount of pictures, then transitioning to another event*
* Might be hard for the software to know what one "event" is? It can of course use both timestamp and geotag to correlate pictures. Personally I also sort everything into very hierarchical folders, so having support for #129 would also help the software with knowing what belongs together :)
Thank you for working on this software :pray:

Hi @faern
thank you for your interest and your detailed description.

I stumbled on this project when I was trying to find software for turning my tablet into a digital picture frame while it's not in active use.

Currently HomeGallery does not have a slideshow feature in the view mode of the web app.

Would you be willing to merge such functionality if someone contributed it?

Sure, absolutely. If your need some help to get into it, let me know.
I know you would like your tablet as play back device. But have you tried the cast feature of HomeGallery for your TV? It can cast media in ordered or random order any search to a Google ChromeCast. See cast docs for it.

it would be great if the selection of the next image would be semi-random but with some decently smart logic

Well, the "smart logic" needs to be implemented quite statically at the end. For me pictures transports some moods or emotions and my emotions are changing over time. So the "smart logic" becomes very tricky at the end because the computer can not guess my current (and maybe changed) mood. I would suggest to keep it simple. You define your subset via a search and let the slideshow start chronological or in random order.
... or I did not understand your description, how the "smart logic" can be implemented in simple rules. Maybe you can give another shot how your algorithm would work to be smart?

I know you would like your tablet as play back device. But have you tried the cast feature of HomeGallery for your TV? It can cast media in ordered or random order any search to a Google ChromeCast. See cast docs for it.

Interesting. I did not know it had a cast mode. A slideshow mode should probably share a lot of code with that cast mode, since they basically do the same thing, but to different targets. I would personally not be satisfied with fully linear or fully random. But any smarter shuffle could of course be shared with the cast mode also. I don't have a TV and even if I did, I would probably not want to leave it on 24/7, as I intend to do with this "picture frame" I'm trying to create. So it does not cover my usage even if the randomness was satisfactory.

You define your subset via a search and let the slideshow start chronological or in random order.

This would not work for my use case. I don't want to make searches. I want the picture frame to slowly visit my entire picture catalogue (20000+ pictures spread out over 20+ years) but not be completely random. As I described in my initial post, that would not be nice.
Let's say we define a couple of axis to put each picture on. These axis can be: Time taken, geographical position, people in the picture (face detection?), tags applied to the picture etc. Then the next-picture-to-show algorithm can look something like this:

Pick a picture completely randomly
Show 10-20 pictures that are taken close in time to the previous picture (This is what I call showing one "event"). These should preferably be shown in chronological order so you only travel forward in time as you view this one event.
Compute the union of all axis defined above for the 10-20 pictures you just showed: Time, place, people in the picture, tags.
Find a new picture from the entire catalogue where at least one of the axis from the previous event is very similar, the rest can differ completely. If the previous event was a trip to Tokyo in 2010 with Alice and Bob, and the pictures had the tags adventure, bestfriends, sushi. Then find a new picture where at least one of the following is true:

Taken around 2010
Alice and/or Bob is in the picture
The picture is taken close to Tokyo
The picture has as many tags as possible overlapping with adventure, bestfriends, sushi.

Repeat from step 2 above, but with this new picture as the base picture.

You talk about folders a lot. But my understanding from #129 is that HomeGallery does not support folders?
Very vaguely I imagine the algorithm to be something like:
event_base_image = random_image_from_catalogue()

do forever:
  event_images = select_ten_random_images_in_timespan(
    start: event_base_image.time,
    end: event_base_image.time + 24h
  ).sort_chronologically()

  event_image_attributes = AttributeSet()
  for image in event_images:
    fade_to_image(image)
    event_image_attributes.append(image.attributes)
    sleep(8 seconds)

  event_base_image = pick_random_image_sharing_some_similarity_with_attribute_set(event_image_attributes)

The attributes are very fuzzily expressed here. But I imagine the attribute set would have one geographical coordinate, maybe simply from the first image, a timestamp that is also just the first image timestamp, a tag set that is the union of all the images and a face set being the union of all faces etc.

Closing this issue due inactivity.
Please reopen it, if it becomes active again.

I took the liberty of implementing something for this since I also want to have a slideshow.
The PR #167 is currently marked as WIP since I wanted to start getting feedback.
Right now it's a "dumb" and basic slideshow, it just picks randomly a single image from the entries' store (which is taken from the list shown on the screen, so it can already be filtered or based on similarity, tags, dates, and any other filter home gallery already provides) and displays it similar on how the details view does it.
This is a list of improvements I have in mind, which might be included in this initial PR or in independent ones because they might require more work.

[ ] Different overlays for the slide (show 2 or 3 images at the same time to cover more screen).
[ ] Different transitions between slides.
[ ] Screen to configure slideshow (timeout, transition, overlays active, etc).
[ ] Configure things to always exclude (tags, folders, sources, etc).
[ ] Smart selection of entries (as best as we can define "smart").
[ ] Store history and add controls to return to previous slide.


Wrap location field option

Every day I have meetings across many different conference rooms and the only way I can see the entire location right now is too use very small text. Would love the option to wrap location to multiple lines just like you have with title now. Would also me to see where I'm going at a glance, with a little larger text. For context I'm using a 2 column wide by 4 row tall size.

+1 on this. Making subject/summary and location fit single line is just wrong: they have different type/importance. Also the stock widget (emulator image) uses slightly different fonts for summary and where/when (typeset difference emphasising difference in type?). Not sure if stock widget is the right reference, but it somehow looks good with 2 lines for summary, 1 line for location and 1 line for time (putting time on the right iOS style is actually a nice touch, albeit looking very cramped. Line spacing option might save the day...)
this one rather unusable:

stock handling long summary gracefully:

even when made tiny:

also, color bar being a space waste for single-calendar/type-of-events widget?

@mattpanther The widget has "Multiline title" option for years! It allows to see the whole content, including the whole location string.
In v.3.0 (currently in Beta, see #320) text scaling in greatly improved: size of all text fields changes, not of the event Title only.
@sgserg Color bar can also be hidden in the v.3.0


es6 problem

I'm making a Javascript class,it is  vaild。
class MyClass {
  static myStaticProp = 42;

  constructor() {
    console.log(42); // 42
  }
}

but jshint show:
in lin2,: static myStaticProp = 42;
 Class properties must be methods. Expected '(' but instead saw ')'.

jshint
jshint v2.9.4
.jshintrc
{
"asi": true,
"laxcomma": true,
"es5": true,
"node": true,
"strict": false,
"esnext": true
}

The code you have shared is not valid according to the more recent language standard. Specifically, it makes use of the proposed language extension, public class fields. That proposal is currently at stage 2 in the standardization process, however, so in accordance with our policy on new language features, we would accept a patch to support it on an experimental basis. I've created a new issue to track the status of the feature request: https://github.com/jshint/jshint/issues/3139

I believe this is part of the language now, yet...


fix: ensure api code preview indentation is correct

Tweaks the code preview spacing and indentation in the API view:

Vertically aligns the chevron and dashed line
Reduces the indentation for nested data

Before

After

Might need some more thought, might be in the rendering logic of the view, but I assume @jmikrut would like it to look similar to this:

Might need some more thought, might be in the rendering logic of the view, but I assume @jmikrut would like it to look similar to this:

Ah okay I've updated it to match


Allow subseconds precision for the watch latency measurements for write and update operations

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR inserts a client-side timestamp annotation for every write and patch operations and it allows subsecond precision measurements for the "watch latency" out to watch or informer clients.
a) CREATE
Name:         big-deployment-94
Namespace:    test-0wzi9x-1
Labels:       <none>
Annotations:  client-side/createTimestamp: 2021-11-12 15:31:10.482 +0000

b) UPDATE
Name:         big-deployment-0
Namespace:    test-0wzi9x-1
Labels:       <none>
Annotations:  client-side/createTimestamp: 2021-11-12 15:31:10.128 +0000
              client-side/updateTimestamp: 2021-11-12 15:31:10.874 +0000

Which issue(s) this PR fixes:
This PR allows finer precision for the "watch latency", when we need subsecond precision timestamps - it addresses the no milliseconds time precision in kubernetes.

Fixes #
Special notes for your reviewer:
NONE
Does this PR introduce a user-facing change?:
"NONE"

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@dumb0002 - please sign the CLA
This isn't universal solution, because many objects (e.g. pods) are not written from CL2 itself. But I'm personally not completely against that change.
/assign @mborsz

@wojtek-t : you raise a very good point, which I think begs a much bigger observability question --- being able to observe relationships between different writes and string them together by causality.  If our control plane were imperative this would be solved by distributed tracing.  For a kube-style control plane we need something different but analogous, with a focus on object writes rather than remote invocations.
The point of this here PR is to facilitate study of the primitive step, the latency from one client submitting a write to another client getting notified of the updated state.  It is not intended to solve the higher level problem, but rather support study of the primitive step.

signed CLA

@wojtek-t: signed the CLA

Could you describe more how are you going to use this? Are you going to implement some clusterloader measurement that will use that? Or some internal tool you own requires this?
Overall, I'm not fan of this solution: it adds a piece of code to the shared code that will be useful only for your specific use case.
Instead of this pattern, I would suggest adding a template function that returns current time and using it as part of your own test configs. This will be simpler, more reusable and will not change objects in all our tests.

@mborsz , yes, I am going to implement a clusterloader measurement to measure the "watch latency" for CL2 object writes. This new measurement will use the changes from this PR. Right now CL2 only measure the watch latency for pods. Measuring the watch latency for other objects (e.g., configmaps, secrets, etc) which have subsecond precision is also useful for the k8s scalability analysis.

/easycla

/remove-lifecycle stale

/remove-lifecycle stale


Move example.json and schema.json out of the doc folder

Hi @jmcanterafonseca , I moved the example.json and schema.json to dispose the files in a similar way to other data models. Please, check if this is the desired change by #127.
Let me know if more changes are required.
BTW, in a short future I will propose a new data model, the idea is provide information about detected near wifi devices.

LGTM
thanks


Remove numGPUs/ithGPUs argument from CARLsim constructor

I am wondering about this. For CARLsim4 do we still need to specify which GPU want want to run on (e.g. 0, 1, 2, etc..) or is this being taken care of by CARLsim. In the printed output from sim.runNetwork() I no longer see anything specifying which GPU it was run on.

In CARLsim4, you do not need to specify the CPU/GPU ID at the CARLsim constructor. This is because you can use a separate CPU core or GPU card for each neuron group. Therefore, you specify the desired CPU/GPU at group level, through the createGroup() function. This section of the user guide discusses this:
http://uci-carl.github.io/CARLsim4/ch12_advanced_topics.html#ch12s3_multi_gpu

Great. Should the random seed then be the 4th input to the constructor (as opposed to the 5th)?


How can i use Drawer inside PlatformScaffold?

There doesn't appear to be a way to programatically open the drawer with this method? Can't access Scaffold.of(context).openDrawer();
Any thoughts?

As long as the Widget is part of the BuildContext and you are on Android then you should be able to do it. You can use Builder which should give you access to the BuildContext with the scaffold on it
     body: Builder(
           builder: (BuildContext context) => ...

OK, Been playing with it and with the example I have all you need to do is wrap a Builder instance around the button you want to open
This is on the trailingActions on the PlatformAppBar
trailingActions: <Widget>[
        Builder(
          builder: (BuildContext context) => PlatformIconButton(
                onPressed: () {
                  Scaffold.of(context).openDrawer();
                },
                iosIcon: Icon(
                  CupertinoIcons.share,
                  size: 28.0,
                ),
                androidIcon: Icon(
                     Icons.share,
                ),
              ),
        ),

OK, Been playing with it and with the example I have all you need to do is wrap a Builder instance around the button you want to open
This is on the trailingActions on the PlatformAppBar
trailingActions: <Widget>[
        Builder(
          builder: (BuildContext context) => PlatformIconButton(
                onPressed: () {
                  Scaffold.of(context).openDrawer();
                },
                iosIcon: Icon(
                  CupertinoIcons.share,
                  size: 28.0,
                ),
                androidIcon: Icon(
                     Icons.share,
                ),
              ),
        ),

I'll have a play with this tonight, I hadn't thought to use a builder inside the appBar
I'll let you know if it works in my situation


Correct required Windows account permissions

Remote Management Users instead of Remote Desktop Users local group.

@ouissameb : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi, this change has already been made. Thx!
#please-close


dashboard

Summary by CodeRabbit

New Features

Introduced German translations for new storage methods, dashboard elements, and material types.
Added English translations for new error messages, dashboard elements, and material types.
Implemented a new dashboard page for managing user uploads, displaying them in a table, and providing a form for new uploads.
Added a new history page to display user history with data fetched from Firebase Firestore.

@coderabbitai full review


2D Delaunay Filter is misspelled

The 2D Delaunay filter on the vtki.PolyData class is misspelt as .delauney_2d which is inconsist with the spelling in VTK and the 3D Delaunay filter under vtki.DataSetFilters.
This is leading to confusion among new users.
I’m afraid this might not be a simple fix by refactoring since .delauney_2D filter has been around for a long time and other dependencies might call it.

Better to fix it now than to let it stick around.  If you think it would be helpful, you can keep the old function, wrap the new correctly spelled one, and add a warning for those still using the old function.
Since DeprecationWarning is ignored by default, I'd add a general user warning.  Or you can just raise an exception with a helpful error message.

Good plan. Will implement that soon